Fastest forensics d...
 
Notifications
Clear all

Fastest forensics disk image

Passmark
(@passmark)
Active Member

After playing around with our disk imaging code I got it to image a 512GB SSD to an external USB drive in 4min 35sec. This works out at 1.7GB/sec average.

This was for OSForensics. Before I mistakenly claim some type of world record, is anyone aware of any better results from any other tool?

Comparison Criteria:

Raw image format (immediately compatible with every other tool without post processing)

Full physical drive (allocated and unallocated space)

Hashing (SHA1 in this case)

No compression.

 

I would accept E01 output, with compression, as a valid comparison, as E01 is almost as useful as a raw image. MD5 or SHA2 as an alternative hash would also be fair to compare against.

Interesting discovery along the way: Default E01 output is crippling slow. This all comes down to the use of the super old zlib compression as part of the format. It is so slow that I'm calling the whole E01 format dead & buried on modern hardware. Even modern CPUs can't come close to keeping up with modern SSDs, as the data compression step is that slow. Raw images, without compression, can be up to 7x faster to create, with the right code.

Quote
Topic starter Posted : 05/11/2021 6:21 am
JimC
 JimC
(@jimc)
Member

That is indeed very fast but, except for the hashing, really just a statement of how fast raw data can be copied from one drive to another. All software being equal, this means it is really a test of hardware performance. To make the comparison easier, it would be helpful to know:

1. What hardware was the USB drive? (e.g. SSD or "spinning rust", USB3?)

2. What CPU was used? e.g. How many threads were available for hashing

Raw imaging will sometimes be faster than compressed formats (like E01) simply because no compression is required but the downside of the this is that the user must then store (and backup) the huge raw image. Conversely, depending on the data, compressed imaging may actually be quicker if the source material contains a lot of empty (or easily compressed) data. This is because, in the raw scenario, every byte read must then be written but in the compressed scenario only a fraction of the data read will be written. This is why the hardware is so relevant to compression. I have certainly seen some "real world" images which compressed in much less time than it would have taken to do a raw image.

I agree the E01 format is lame but there are some tricks that be used to overcome this - for instance, using a bigger chunk size or more threads for compression. E01 is considerably faster if you use a bigger chunk size.

 

Jim

www.binarymarkup.com

 

ReplyQuote
Posted : 05/11/2021 10:08 am
Passmark
(@passmark)
Active Member

...just a statement of how fast raw data can be copied from one drive to another. All software being equal, this means it is really a test of hardware performance

A few weeks ago I would have agreed, but as it turns out it is pretty easy to write slow code and rather hard to write code that fully uses the latest SSDs & CPUs.

What hardware...

Modest, but new, desktop machine, Ryzen 5 5600X CPU, 16GB RAM. SDDs for both the internal and external drive. Single USB3.2 interface for the external. Single nVME for the internal.
(No RAID or striping)

How many threads were available for hashing

Lots, but they weren't required. Hashing and Zlib compression are both pretty resistant to being threaded. A 4 core machine would do as well as a 32 Core machine, all else being equal.
(Of course multiple threads are possible with multiple disk images, but in this case only 1 drive was being imaged).

Conversely, depending on the data, compressed imaging may actually be quicker if the source material contains a lot of empty (or easily compressed) data.

A few weeks ago I would have also agreed with this. But after testing, zlib (as used in E01 images), it turns out to be really really slow, especially at the high compression setting. And remember that even if you write nearly nothing to the output drive (because the input is all zeros) you still need to take the time to read the source drive. So the time saving isn't as much as it might at first appear.

zlib made sense when external devices were USB2.0 (~30MB/sec). But it makes no sense at all for modern hardware. Depending on how good your code is and how good the CPU is, zlib tops out at around 100MB/sec for in memory compression.

In some cases we measured E01 compression to be x20 slower than no compression (worst case). But of course for some people disk space is critical and time isn't. Nevertheless I am predicting that imaging time will become more important for people than disk space savings, and then E01 will fade away as a format. It's no longer the right tool for the job.

 

ReplyQuote
Topic starter Posted : 06/11/2021 11:44 pm
mokosiy
(@mokosiy)
Junior Member
Posted by: @passmark

Conversely, depending on the data, compressed imaging may actually be quicker if the source material contains a lot of empty (or easily compressed) data.

A few weeks ago I would have also agreed with this. But after testing, zlib (as used in E01 images), it turns out to be really really slow, especially at the high compression setting. And remember that even if you write nearly nothing to the output drive (because the input is all zeros) you still need to take the time to read the source drive. So the time saving isn't as much as it might at first appear.

 

I would like to support you this. According to our regular performance tests, E01 compression degrades performance when it comes to imaging speeds above 700 MB/s (approximately). Even when an evidence drive contains 5% of data and 95% of zero sectors. However, the compression is valuable for saving target drive space if you are not using file system with sparce files.

If one wants a target image forensically stored and compressed in a faster manner, I would recommend using AFF4 with either Snappy or LZ4 compression selected. In practice, they are 3-5 times faster than zlib.

With all that said, though, who cares about time taken by imaging fast SSDs? It's a matter of dozens of minutes every time 🙂 Multi-terabyte HDDs are the toughies eating out days.

 

 

ReplyQuote
Posted : 08/11/2021 6:32 am
v.katalov
(@v-katalov)
Member
Posted by: @passmark

After playing around with our disk imaging code I got it to image a 512GB SSD to an external USB drive in 4min 35sec. This works out at 1.7GB/sec average.

This was for OSForensics. Before I mistakenly claim some type of world record, is anyone aware of any better results from any other tool?

Comparison Criteria:

Raw image format (immediately compatible with every other tool without post processing)

Full physical drive (allocated and unallocated space)

Hashing (SHA1 in this case)

No compression.

 

I would accept E01 output, with compression, as a valid comparison, as E01 is almost as useful as a raw image. MD5 or SHA2 as an alternative hash would also be fair to compare against.

Interesting discovery along the way: Default E01 output is crippling slow. This all comes down to the use of the super old zlib compression as part of the format. It is so slow that I'm calling the whole E01 format dead & buried on modern hardware. Even modern CPUs can't come close to keeping up with modern SSDs, as the data compression step is that slow. Raw images, without compression, can be up to 7x faster to create, with the right code.

GREAT speed! Could you please share more details on SSD models (both internal and external), as well as motherboard model (looks like you almost reached USB Gen 2x2 limit), and what particular external case you are using?

ReplyQuote
Posted : 29/11/2021 4:15 pm
Passmark
(@passmark)
Active Member

Was ASRock B550M motherboard.

Internal Source SSD drive: WD Black SN850 WDS500G1X0E 500GB NVMe M.2 

External Destination SSD: External USB 3.2 Gen2x2 Type-C, Silverstone MS12 with Samsung 980 Pro NVMe M.2.

So a consumer motherboard and SSDs. High end, but not bleeding edge.

 

ReplyQuote
Topic starter Posted : 30/11/2021 11:45 pm
Share:
Share to...