ExFAT vs FAT32 (deeper mechanics)

Canot say if specifically it can help you, but maybe something in the source here
https://code.google.com/p/exfat/
is of use.
jaclaz

I can't say that I'm good with reading third party code but I had a look and it looks like they check the FAT for the root. Thanks for the tip

While tweaking my implementation I noticed that the checksum for the dir records doesn't seem to match. I use the little checksum calculation from the sans institute document.
So I figured I'll check mentioned code as well, turns out they do it the exact same way I implemented it. Or at least as far as I can see. Is anybody aware of any pitfall here that I'm not aware of yet ? I do not check it on deleted files, I know that won't work, but the checksum doesn't seem to match regular files and folders (small W7 formatted USB stick).

Hi,

I'm looking into ExFAT and as a first introduction I read through this pdf
reverse-engineering-microsoft-exfat-file-system_33274 (actually half way through)
It's getting a bit confusing and because it's been some time since I implemented FAT I'm not sure how different ExFAT is from FAT32. I feel I need to write some code first to see how things work rather than just reading text and not seeing the light.

A decision I need to make now is whether to extend my existing FAT implementation or start fresh for ExFAT (if too different). Right now I'm enclined to start again for ExFAT because the differences seem too great and it would clutter my FAT32 code too much.

To help me see the light I was wondering if anybody knows of a document / text that explains the *differences* between FAT32 and ExFAT on file-system level ? Or a document that explains what to add to an existing FAT32 implementation to support ExFAT (if that is at all possible).

As I said, from what I read, it seems quite different, but I haven't compared directory table structures yet etc. and maybe they're closer related than I feel right now.

Anyway, some input appreciated. I'm signing off. Reading that technical document while being tired is probably also not a good idea 😉

Hello, I am new to the forum, I found this post via a Google Search on SANS + exFAT, and I thought I'd chime in.

First of all, let me identify myself - I am the author of the SANS paper mentioned in this thread and post. The e-mail address (earthlink) in the paper is current, so you can reach me directly, but posting here would be fine, I can't say I will be able to monitor this thread much.

Some of what I will mention regards the timezone questions that also appear elsewhere in this thread.

I am a graduating Masters student at John Jay College, CUNY in the Forensics Computing curriculum and in Fall 2009 I was taking a digital forensics course and my term project was on the exFAT file system. SO the SANS paper was first a class paper. In October 2009, as part of my going to conferences as I did for my job, I attended a session at the Techno Forensics conference held at NIST that was given by Jeff Hamm that was an intro to exFAT. Jeff's talk provided a lot of insight for me to start my paper, but I dug in deeper and did research and relied heavily on the Microsoft Specification for exFAT. By the time I finished the paper, I had a lot more detail, and even found errors or inconsistencies in some of Jeff's presentation, including his interpretation of the time zone values. I also discovered that time zone values varied with Windows 7 and Windows XP, I'll leave that variation for later.

Now Jeff was practical, in that he taught and performed forensics investigations, and at the time he worked for paradigm solutions and had a blog (still there) at http//paradigmsolutions.wordpress.com/2009/12/10/extended-fat-exfat/. Jeff also had limited information, but he was the leader in exFAT and getting things done. From my perspective, I was being more theoretical, so I was heavier on the research end. When I came up with the inconsistencies in the time zone calculation, I had e-mailed Jeff, and we collaborated on a final interpretation. You see, 99% of the information for exFAT is in the specification released by Microsoft as part of a patent for a different patent (the exfat patent itself did not have the spec), and the time zone calculation, as well as the location was not provided. As a matter of fact the current time zone layout is different than what was released in the specification, so this led to trial and error.

So I brute forced the time zone, I changed the time zone, created a file, and dumped the exFAT filesystem and collected the values, and came up with a table will all the time zones. I presented it to Jeff, and he came back with the 7-bit number calculation. As it says on page 40 of the paper "A formula
was developed (by Jeff Hamm) that shows the time zone offset to be a 7 bit signed
integer. The purpose of the high order bit has not been determined." And Jeff was given credit in the paper for his contribution. Actually the meeting with Jeff at Techno is on page 4 of the paper as well.

In Dec 2009 I submitted my proposal for a SANS GCFA gold paper, and published by Mid Feb 2010. During 2010/2011 I presented at several conferences of exFAT, including Computer Forensics Show, Techno Security, HTCIA, and at a SANS What's New in Incident Response and Forensics in July 2010. You might find slide presentations online, but I also maintain a blog at rshullic.wordpress.com for exfat which has the slide decks and some additional stuff discovered after the release of the paper. The paper is also mentioned now as a footnote in two Sybex 2012 versions of forensics books, as a source for additional information.

Although most of my presentations were solo, the SANS What's Works conference was a co-presented presentation given by Jeff and myself. That PDF is at http//computer-forensics.sans.org/summit-archives/2010/10-exfat-ham.pdf and you will see both Jeff & Myself listed in the copyright.

Now I mentioned that I gave a presentation at the computer forensics show (CFS), well I gave it at two different CFS, the 2nd was April 2011 where the session was video taped for the AT&T Tech Channel. You can Google for the AT&T Tech Channel + exFAT and should find the link, which when I do it it is at http//techchannel.att.com/play-video.cfm/2011/8/16/Conference-TV-Computer-Forensics-Show-Introduction-to-exFAT but sometimes the link doesn't work unless you click it from the search return. Also keep in mind that this presentation, which is NOT like the Sans What Works presentation, if more than a year after the paper was published.

So, that was a little about history of the paper, and provides some additional resources for exfat. This post might not answer the questions you need answered, but hopefully may provide some insight. I will try to answer questions where I can, I have selected the "Notify me when a reply is posted" so I should at least get an alert.

Thanks for listening.

I wasn't able to put the time in today, but tomorrow I intend to start coding.

Question. I scanned the documents and (at least one) question is bothering me. Maybe I read over it ?

The root cluster is recorded in the VBR.
I read that the FAT is only consulted when a flag is set in the directory entry for a file.
I take it this is also true for folders ?

Which leads me to the root folder. Since there is no record describing it, I don't know if this folder can be fragmented, needing the FAT to determine its extents ?

Should I assume it's unfragmented, or should I consult the FAT for the root, flag or not ?

There are three (3) special files that exist in the Cluster Heap (Data Area), and they are

Root Directory
UP Case Table
Allocation Bit Map
(In the case of TexFAT) there will actually be TWO Allocation BitMaps, bringing the special files to 4.

Each of these files do not have a standard directory, and in the case of the Root Directory does not even have a directory entry at all, as stated - it is pointed by a value in the VBR.

I was always curious that when I formatted a exFAT file system, and then dumped the filesystem using a DD command, that the FAT entries for those three (3) clusters were always 0xFFFFFFFF. (In computer programming speak - we call that "High-values" where 0x0 is called "low-values". In FAT, including legacy FAT (FAT 12/16/32) and exFAT, we call high-values the end of file (EOF) marker. Then, it came to me that since these special files did not have normal directory entries, they would always have a FAT cluster chain, even if the cluster is built and maintained in a contiguous organization. The BitMap and UPCase table may take multiple clusters but they will never fragment because they are allocated once (at format time). But they will have a FAT chain. The Root directory, will fragment, and keep in mind that the root does not have a limit in size, where a subdirectory is limited to 256MIB in size.

Any processing of these 3 special files should consider the file as fragmented and assume that the FAT invalid bit (which is not available for these files) as if the bit was set to Zero, meaning the FAT is VALID and you have to refer to it.

Yes, I know there is more in the answer than you asked, but I wanted to be more general in the answer.

Thanks for listening.

You are very welcome ) , very interesting/clear post.

jaclaz

Thank you Robert, for your detailed input !

It's been a while now since I implemented ExFAT, I had to go look in my code, but yes, I do use the FAT to explore the root, I answered my own question through testing, so that is OK !

IB with ExFAT support has not been released yet, I only do two updates per year, but if you like a copy to try, let me know.

Best Regards,
Peter

IB with ExFAT support has not been released yet, I only do two updates per year, but if you like a copy to try, let me know.

Just for the record, and as another example of my (mostly vain) perpetual fight against acronyms
http//www.forensicfocus.com/Forums/viewtopic/p=6561838/highlight=acronyms/#6561838
http//www.forensicfocus.com/Forums/viewtopic/p=6561872/#6561872

the missing piece of info is that Cybergonzo is the developer of IsoBuster
http//www.isobuster.com/

jaclaz

jalaz Thanks, I didn't know and thought it was some unix variant.

But that is an interesting subject, i.e. data recovery. There are two things that I encountered in research AFTER the paper was written, it should be in my blog at rshullic.wordpress.com, but I will mention them here since they involve both exFAT and recovery.

The first is that if a directory entry is marked not in use, it does not mean the file is deleted. Due to the structure of the file set, if the file name is renamed and you exceed a fifteen character boundary, a new file set may need to be created. the old file set is marked Not In Use, and a new one is created. So, a rename can cause file directory entries to be marked not in use, but the file was not deleted.

The second is very interesting. It has to do with how legacy FAT (FAT 12/6/32) deletes a file, and also leads to an issue that I have had arguments with other people with. In Legacy FAT if the FAT cell is marked with a zero, the cluster is unallocated. In exFAT - zero is not significant. It has no meaning, it is UNDEFINED. The argument is that if the cell is zero, the file is not fragmented, which implies if the cell is non-zero the file is fragmented, and that is not totally true.

So, one of the target media for exFAT is flash memory, including camera media and USB drives. These devices are constantly overwritten and change a lot, but they use electronics and not magnetism underneath, so they wear out over time, usually sections may go bad as the gates wear out. The longevity of the memory can be maintained by minimizing the amount of changes.

This is accomplished by NOT writing anywhere if you can avoid it. If you can't avoid it, write is as few times as possible. What would change the most? The FAT, which by the way is not in the data heap (it is not in cluster space) so if the FAT became bad - large sections of the data heap would become unavailable.

How is this accomplished, besides using the Allocation BitMap, which BTW is in cluster space?

When a file is created, if it is contiguous, don't put the link chain in the FAT (like in legacy FAT), just set bit to ignore the FAT and read the clusters sequentially. All well and good, but suppose the file system is getting crowded and file fragmentation is now required, now you are forced to start writing into the FAT of an exFAT file system. Now, here is the interesting part for both file recovery AND forensics recovery when a fragmented file is deleted - the FAT is not cleared!

Remember, in legacy FAT, when we deleted a file, the file system had to zero out all the FAT cells to indicate that the cells were unallocated and available for use. But in exFAT, that function is now in the Allocation bitmap, so we don't need to clear out the FAT, all those links are still intact until clusters in that chain are overridden.

The significance of this change (the adding of the BitMap) in operation of the file system can be substantial for a file recovery. In legacy FAT, once a fragmented file is deleted, it becomes a puzzle to put the pieces back together because the pieces are not contiguous, might not actually be in the proper correct order, and in cases of pictures and videos, may be harder to recreate the original file with the pieces in the proper order. What you are trying to do is recreate the cluster link chain, i.e. the FAT entries that were erased. But in exFAT they are not erased, and as long as the clusters were not corrupted after the file delete, even a fragmented file should be easier to recover for an exFAT system.

But in exFAT they are not erased, and as long as the clusters were not corrupted after the file delete, even a fragmented file should be easier to recover for an exFAT system.

Interesting !
Thanks Robert !

The second is very interesting. It has to do with how legacy FAT (FAT 12/6/32) deletes a file, and also leads to an issue that I have had arguments with other people with. In Legacy FAT if the FAT cell is marked with a zero, the cluster is unallocated. In exFAT - zero is not significant. It has no meaning, it is UNDEFINED. The argument is that if the cell is zero, the file is not fragmented, which implies if the cell is non-zero the file is fragmented, and that is not totally true.

It's been a while since I looked at ExFAT, but I thought that a zero value in the FAT meant the cluster had never been allocated, and any other value meant it was either in use or had been at some point. Hence I wouldn't have classified Zero value as undefined.

What a zero in a FAT cell of exFAT would mean is that since the file system was created/last formatted, a fragmented file never used that cluster.

However, if during a forensics analysis you find a zero in the cell, it tells you nothing about the data in the associated cluster, i.e. whether the cluster is allocated, or if allocated how many different files were created and then deleted using that cluster. It is considered "undefined" because the value does not specifically tell you something.

FatEntry[2] … FatEntry[ClusterCount+ I] Fields
[0175] Each FatEntry field in this array represents a cluster in the Cluster Heap. FatEntry[2] represents the first cluster in the Cluster Heap and FatEntry[ClusterCount+l] represents the last cluster in the Cluster Heap.
[0176] The valid range of values for these fields is
[0177] Between 2 and ClusterCount+l, inclusively, which points to the next FatEntry in the given cluster chain; the given FatEntry shall not point to any FatEntry which precedes it in the given cluster chain
[0178] Exactly FFFFFFF7h, which marks the given FatEntry's corresponding cluster as "bad"
[0179] Exactly FFFFFFFFh, which marks the given FatEntry's corresponding cluster as the last cluster of a cluster chain; this is the only valid value for the last FatEntry of any given cluster chain.

The above is directly out of the exFAT specification, as provided in one of the Microsoft Patents. Valid values that can appear in a FAT Entry cell is 2-cluster count + 1, and FFFFFFF7 (Bad Cluster) and FFFFFFFF (Eof). This implies that values of 0 & 1 are not valid, and therefore undefined.