±Forensic Focus Partners

Become an advertising partner

±Your Account


Username
Password

Forgotten password/username?

Site Members:

New Today: 0 Overall: 36296
New Yesterday: 0 Visitors: 163

±Follow Forensic Focus

Forensic Focus Facebook PageForensic Focus on TwitterForensic Focus LinkedIn GroupForensic Focus YouTube Channel

RSS feeds: News Forums Articles

±Latest Articles

±Latest Videos

±Latest Jobs

Hashing and wear-levelling

Computer forensics discussion. Please ensure that your post is not better suited to one of the forums below (if it is, please post it there instead!)
Reply to topicReply to topic Printer Friendly Page
Forum FAQSearchView unanswered posts
Page 1, 2, 3, 4, 5, 6, 7  Next 
  

thefuf
Senior Member
 

Hashing and wear-levelling

Post Posted: Feb 23, 09 23:59

Hi all!

I have 2 USB flash drives:

Transcend JF V30 / 1 GB
Transcend JF V33 / 2 GB

I'm trying to hash both drives using md5sum / md5deep on Helix and Slackware (without mounting). When I'm hashing 1 GB drive everything is OK, but I'm getting different hash values for 2 GB drive every run:

$ md5deep -e /dev/sdb1 /dev/sdc1
07db72d1b01b7afa55ce1b17fe06d2eb /dev/sdb1
9efc72df43454390b314d68aa6ebd3bc /dev/sdc1
$ md5deep -e /dev/sdb1 /dev/sdc1
8becb409bac99171fa5f03fbaa327e2d /dev/sdb1
9efc72df43454390b314d68aa6ebd3bc /dev/sdc1

(similar results with /dev/sdb and /dev/sdc; similar results when I'm formatting 2 GB drive to JFS)

From www.sandisk.com/Assets...lv1.0.pdf:

each time a host writes data
to the same logical address (CHS or LBA), data is written into a newly assigned, empty physical
block from the “Erase Pool”.


As I can understand, wear-levelling works on different level than block device I'm trying to hash and is not filesystem-aware. But why I'm getting different hash values in this case?

From digfor.blogspot.com/20...tion.html:

Some USB devices (approximately one in every ten from my experience) will produce different cryptographic hash every time you calculate it, despite the fact that no write is allowed. So, by simply reading such devices, we are changing something inside these drives.


So the questions are:

- Is wear-levelling responsible for different hash values?
- If yes, how does it actually work?
- If no, then why I'm getting different hash values?

Thanks.  
 
  

mscotgrove
Senior Member
 

Re: Hashing and wear-levelling

Post Posted: Feb 24, 09 05:50

I had a memory chip recently that was failing in such a way that sectors would read differently each time. This could be your problem.

My first approach would be to take an image (DD type image) of the chip twice, and see if they match. Any differences would explain the different hash value and let you track the problem down. If they are both identical, then I will be as confused as you are.  
 
  

ecophobia
Senior Member
 

Re: Hashing and wear-levelling

Post Posted: Feb 24, 09 08:14

There could be several reasons for getting different hash values. It could be faulty controller or chip but as I mentioned on my blog, it is probably wear-levelling. The best way to deal with this issue is to get winhex and create a hash set of all files. Open the second image (with the different hash value) and use the hash set you just created to filter out the differences. Another approach is to use md5deep or similar tool and create md5 hash for every sector. You may then compare both images and identify sectors that changed and go from there. Most likely you will find that your existing files will produce the same hash value and free space will give you different hash every time you try to image it. Good Luck and don't forget to post your findings on this forum.

(Prover' sam esli somnevaishsia Smile  
 
  

PaulSanderson
Senior Member
 

Re: Hashing and wear-levelling

Post Posted: Feb 24, 09 14:23

- ecophobia
but as I mentioned on my blog, it is probably wear-levelling.


Why - logically this sounds ridiculous.

1. wearlevelling is used to even out the wear of individual sectors which are written to more than once. In the example given the drive is not being written to so why would wear leveling come in to play.

2. When I do write data to a given sector, irrespective of what goes on behind the scene, I expect to read back the same data from the same LSN. If, for example, I wrote a new boot sector and for some reason decided to write the same data to the boot sector again - even if it was swapped by the wear leveling algorithm I would expect to read the same boot sector back when I came to do my hash.

It is more likely to be a faulty memory location/faulty controller chip as suggested.
_________________
Paul Sanderson
SQLite Forensics Book
www.amazon.com/SQLite-...entries*=0

Forensic Toolkit for SQLite
sandersonforensics.com...for-SQLite 
 
  

ecophobia
Senior Member
 

Re: Hashing and wear-levelling

Post Posted: Feb 24, 09 16:49

Why - logically this sounds ridiculous.


'Truth is stranger than fiction'

Paul,

The concept may be a bit confusing and requires some attention to details.

What I said is that "existing files will produce the same hash value and free space will give you different hash". So, you will read back the same data from the same LSN.

All Fun starts when you delete the file. You may see the deleted file being mixed up with a piece of another previously deleted file(s). Essentially, what you see is a logical representation of physical sectors (not a new concept). The controller inside the USB device performs dynamic mapping of logical to physical sectors and (this is the new concept) shuffles everything else marked as a free space (incl. deleted files) according to the pre-programmed algorithm. To make things even more complicated, there are Static Wear Leveling and Dynamic Wear Leveling and they operate quite differently compared to each other. Wear-levelling is not a problem for ordinary users, but forensic people for some strange reason always trying to recover the deleted data and even calculate hash values Smile

Instead of Faulty device OR Wear-Levelling I prefer
IF only free space is changing every time md5 hash is calculated THEN it is probably wear-levelling ELSE it could be faulty device.

BTW. Different sector count may indicate if the device is faulty or not. Wear-levelling will produce the same sector count and the spare ones will be hidden/reserved.

Hope this helps.  
 
  

mscotgrove
Senior Member
 

Re: Hashing and wear-levelling

Post Posted: Feb 24, 09 18:16

Surely a memory chip device knows nothing about unallocated space. A sector is a sector is a sector. Anything that changes a sector apart from the operating system is a failure.  
 
  

thefuf
Senior Member
 

Re: Hashing and wear-levelling

Post Posted: Feb 24, 09 22:24

Thanks for replies.

It seems that the problem is with USB flash drive only, because when I'm trying to hash files on a mounted filesystem I receive tons of I/O errors. Very strange, because I was using this new flash drive for some time without errors Wink

All Fun starts when you delete the file. You may see the deleted file being mixed up with a piece of another previously deleted file(s). Essentially, what you see is a logical representation of physical sectors (not a new concept). The controller inside the USB device performs dynamic mapping of logical to physical sectors and (this is the new concept) shuffles everything else marked as a free space (incl. deleted files) according to the pre-programmed algorithm


This "pre-programmed algorithm" knows about the filesystem used on a drive? Smile  
 

Page 1 of 7
Page 1, 2, 3, 4, 5, 6, 7  Next