UFS 1/2 and Reiser ...
 
Notifications
Clear all

UFS 1/2 and Reiser test Image files

16 Posts
4 Users
0 Likes
1,633 Views
CyberGonzo
(@cybergonzo)
Posts: 100
Estimable Member
Topic starter
 

Hi,

I'm currently looking into easily detecting EXT 1/2/3/4, UFS 1/2 and Reiser on media so the best way to implement this is to work on image files.

I don't run Linux myself, one of the reasons why for the moment I simply and only want to detect the presence of these file systems.

I found EXT image files fairly easily and the implementation is done.
But I can't seem to find UFS image files and I haven't even tried finding Reiser image files.

If you can point me to a few downloads that would be great !

 
Posted : 27/05/2014 8:36 pm
(@athulin)
Posts: 1156
Noble Member
 

I'm currently looking into easily detecting EXT 1/2/3/4, UFS 1/2 and Reiser on media so the best way to implement this is to work on image files.

Take a look at the Sourceforge Disktype project. It is very closely related to what you're doing. It comes with lots of test images, though not comprehensive in any sense of the word.

You may also have some luck with the magic(5) pattern database, which does include a number of file system patterns. (Added Though if you're not into Linux or Unix, that may be somewhat foreign ground.)

I found EXT image files fairly easily and the implementation is done.

Don't be too sure. Remember, all file systems come in versions and variants. An Ext2 dating from 1993 is not necessarily the same as an Ext2 created yesterday. And there may even be implementation differences I think I've seen reports that EnCase 6 failed to recognize Ext4 images from some platforms, while working with Ext4 images from others.

And most tools I've looked at – including Disktype – don't show any signs of recognition if you feed them a filesystem from a big-endian or other-endian system.

But I can't seem to find UFS image files and I haven't even tried finding Reiser image files.

The best thing is to create your own. VMware Player or Oracle VirtualBox are very useful if you're running Windows. I'm sure there are similar environments on Mac OS X. Or, even VMWare vSphere hypervisor which is free for some types of use.

For UFS, you probably need to go back to the early BSD releases. (I fish out FreeBSD 2.1 from … ahem, call it my 'archive' …, and after some brief digging around it looks as if UFS/FFS is the only thing it does support. For Reiser, I suspect you need to look for the Fedora Core releases).

I find Reiser 3.5, 3.6 and 4 in the Disktype (mentioned above) test image set, as well as a number of UFS images from different sources (Darwin and FreeBSD – see freebsd-disk.image).

 
Posted : 27/05/2014 9:47 pm
jaclaz
(@jaclaz)
Posts: 5133
Illustrious Member
 

Hi,

I'm currently looking into easily detecting EXT 1/2/3/4, UFS 1/2 and Reiser on media so the best way to implement this is to work on image files.

I don't run Linux myself, one of the reasons why for the moment I simply and only want to detect the presence of these file systems.

I found EXT image files fairly easily and the implementation is done.
But I can't seem to find UFS image files and I haven't even tried finding Reiser image files.

If you can point me to a few downloads that would be great !

NOT what you asked, mind you, but it would be not that difficult IMHO to download a distro and quickly install it in a VM.
Example for UFS
http//www.syslinux.org/wiki/index.php/FreeBSD_disk_image_creation

And you can have a look at UFS2TOOOLS
http//ufs2tools.sourceforge.net/

ReiserFS is well documented in this project
http//p-nand-q.com/download/rfstool.html
http//p-nand-q.com/download/rfstool/reiserfs_docs.html

But it has become, for the known (unrelated to computing) issues of it's Author sort of "abandoned", you will need the Wayback Machine
https://web.archive.org/web/20060203041344/http//www.namesys.com/

So you will probably need (still in the suggested VM approach) and older Suse release, as I believe that support in newer Linux distro's is reduced (and limited anyway to version 3)
http//en.wikipedia.org/wiki/Reiser4
to create new images.

As Athulin pointed out, relying on existing images without verifying results on "expressly made" ones may lead to "partial" support/detection.

jaclaz

 
Posted : 27/05/2014 9:54 pm
CyberGonzo
(@cybergonzo)
Posts: 100
Estimable Member
Topic starter
 

Take a look at the Sourceforge Disktype project. It is very closely related to what you're doing. It comes with lots of test images, though not comprehensive in any sense of the word.

I'm not good with other people's code but I'll have a look, nice lead anyways.
I can't seem to find the mentioned images but I'm a novice navigating sourceforge.

I found EXT image files fairly easily and the implementation is done.

Don't be too sure.

True but I'm not really digging deep (yet). Just recognition. I take it the superblock and signatures are not messed with ?

I did only find (what I think are) EXT 2 and EXT 3 versions.
Frankly I find it bloody confusing to determine the exact version of EXT.
None of the documentation I found really gave me a conclusive way to determine the version.
At the moment I assume version 2 when the major version is 'dynamic' and version 3 if dynamic + a journal.

And most tools I've looked at – including Disktype – don't show any signs of recognition if you feed them a filesystem from a big-endian or other-endian system.

I see, can even the signatures can be swapped for these file systems ?
Weird.

The best thing is to create your own.

Right, but I have never even seen Linux from close, so the threshold is high.
I was hoping for a quick win on the short term and later invest time to actually implement the file systems properly

I find Reiser 3.5, 3.6 and 4 in the Disktype (mentioned above) test image set, as well as a number of UFS images from different sources (Darwin and FreeBSD – see freebsd-disk.image).

nice … so where is that … sorry I can't seem to find it (yet).

 
Posted : 27/05/2014 11:57 pm
CyberGonzo
(@cybergonzo)
Posts: 100
Estimable Member
Topic starter
 

NOT what you asked, mind you, but it would be not that difficult IMHO to download a distro and quickly install it in a VM.

Right, but the threshold is quite high at the moment.

Thanks for the other tips, I will certainly check them out once I'm ready to actually implement these file systems.

 
Posted : 28/05/2014 12:09 am
CyberGonzo
(@cybergonzo)
Posts: 100
Estimable Member
Topic starter
 

It comes with lots of test images, though not comprehensive in any sense of the word

Found it

http//disktype.cvs.sourceforge.net/disktype/file-system-sampler/

 
Posted : 28/05/2014 3:13 am
(@athulin)
Posts: 1156
Noble Member
 

I see, can even the signatures can be swapped for these file systems ? Weird.

In some – I would not say definitely that that will happen for Ext2/3, without verifying it. But the really old file systems were not designed for portability between architectures, and usually did not have a defined On-Disk File Format specification. Generally, the disk label and superblocks were read and written straight out to/from memory, as a sector, all at once. Thus, any byte-ordering effects present would be present also in the disk sectors.

I've seen at least one file system (UFS/FFS) from an embedded NetBSD running on a printer. The CPU was a MIPS, which is very flexible when it comes to endianness. In that case, it was all 'wrong' for any standard CF tool to identify it, so it had to be disentangled by other means.

 
Posted : 28/05/2014 8:43 pm
CyberGonzo
(@cybergonzo)
Posts: 100
Estimable Member
Topic starter
 

In some –

From what I read (and have seen) EXT should always be little endian and UFS big endian.

I can however imagine that for embedded systems, essentially black box devices, the developers basically do what works best, and fastest, for them.

PS. my code seems to work nicely on the image files you pointed me to. Thanks for that.

A few things that still 'bother' me are

- The blocksize (bytes / block) for both EXT and UFS image files is often > 512 bytes, for instance 1024 or 4096 etc. Personally I find that weird, since all HDs and devices I have come across so far (except for optical) always seem to have a blocksize of 512.
I would have guessed that the file system stored block size would be the same as the device blocksize. Or are these blocksizes more like the cluster sizes as used in FAT and NTFS ?

- One of the EXT4 image files I can only detect as EXT3. I don't know what else to check than the (in)compatibility flags.

- The two UFS image files seem to be UFS1, no UFS2 image.

Tomorrow I will look into Reiser.
Cheers.

 
Posted : 28/05/2014 11:42 pm
(@athulin)
Posts: 1156
Noble Member
 

- The blocksize (bytes / block) for both EXT and UFS image files is often > 512 bytes, for instance 1024 or 4096 etc. Personally I find that weird, since all HDs and devices I have come across so far (except for optical) always seem to have a blocksize of 512.

Not sure I understand that. If you had said that HDDs usually have a sector size (sector = the smallest directly addressable unit) of 512 bytes, sure. 'Block' – at least in my world – tends to be used for other 'units' than sectors. (The 'logical block' of an ISO-9660 file system being a fairly good example, as it must be kept notionally separate from the 'logical sector' despite that it usually is the same size.)

I would have guessed that the file system stored block size would be the same as the device blocksize. Or are these blocksizes more like the cluster sizes as used in FAT and NTFS?

The purpose is the same there's an upper limit to how much you can address with 32-bit addresses. If the units are 512 bytes, the total data limit is around 2 Tbyte. And that would have been recognized as a bit small for scaling well, except perhaps in the very first version of the file system. Adding a 'logical block' layer on top of the physical block/sectors allows the file system to scale – at least until the file system needs a more fundamental overhaul, such as going 64-bit.

You also write

From what I read (and have seen) EXT should always be little endian and UFS big endian.

ExtX was developed for Linux, which is predominantly used on the Intel platform, hence little-endian ExtX file systems would be most common.

But Linux (and ExtX) runs on many hardware acitectures. A check at the Linux kernel archives, I find kernel ports to IA64, PA RISC and PPC, which are bi-endian, so I would not be surprised to find big-endian Ext, even if it probably would be quite unusual to encounter. (Added I have not examined the actual file system code – if it rearranges things into a single on-disk file format, my expectation would obviously be wrong. That would be putting portability before speed, though, and that I don't really think I'll find …)

UFS is rooted in Berkely Unix development, and BSD was then developed on and for VAX, which is a little-endian architecture. Sun Microsystems used BSD Unix for their initial range of workstations, which were based on 68k platform, hence big-endian … so I would not be surprised to find UFS of both kinds either. And there were companies, like Microport, who ported both BSD and AT&T UNIX to custom hardware, some of which became brand names.

NetBSD runs on everything and their toaster, so I would expect NetBSD would be a good environment for research in this area, as it supports both Ext2 and FFS (64- as well as 32-bit) – as long as the required hardware or a good emulator could be found.

 
Posted : 29/05/2014 12:24 pm
CyberGonzo
(@cybergonzo)
Posts: 100
Estimable Member
Topic starter
 

'Block' – at least in my world – tends to be used for other 'units' than sectors.

They are the same to me, I use them both, an LBA (logical block address) may be different from a physical sector address, but both sizes are the same.

(The 'logical block' of an ISO-9660 file system being a fairly good example, as it must be kept notionally separate from the 'logical sector' despite that it usually is the same size.)

Please explain. ISO9669 only exists on optical and optical is always 2048 bytes per block.

The purpose is the same there's an upper limit to how much you can address with 32-bit addresses. If the units are 512 bytes, the total data limit is around 2 Tbyte

Painfully aware of that yes.
I however use the physical block size only, as I show multiple file systems on a disc (e.g. per partition) and they may all have different internal 'block' sizes.

I'm also 'used to' file systems using the physical block size (UDF, FAT. NTFS) with maybe a cluster size on top of that, so this is new to me.

That would be putting portability before speed, though, and that I don't really think I'll find …)

That surprises me. You'd want to format a USB stick in a format that you can move to the next (similar) OS, whether it runs on intel or motorola (for instance). So I would expect a fixed agreed on (per the specs) endianess. Again, not what I'm used to (joliet, UDF, etc.)

UFS … so I would not be surprised to find UFS of both kinds either.

Could be, I may need to resort to some beta testing for all this. Problem is, I write Windows software and most people able to test this will likely only run Linux. Ah well, it is a first stab at it, it doesn't need to be perfect (yet).

 
Posted : 29/05/2014 2:47 pm
Page 1 / 2
Share: