ExFAT vs FAT32 (dee...
 
Notifications
Clear all

ExFAT vs FAT32 (deeper mechanics)  

Page 1 / 2
  RSS
CyberGonzo
(@cybergonzo)
Active Member

Hi,

I'm looking into ExFAT and as a first introduction I read through this pdf
reverse-engineering-microsoft-exfat-file-system_33274 (actually half way through)
It's getting a bit confusing and because it's been some time since I implemented FAT I'm not sure how different ExFAT is from FAT32. I feel I need to write some code first to see how things work rather than just reading text and not seeing the light.

A decision I need to make now is whether to extend my existing FAT implementation or start fresh for ExFAT (if too different). Right now I'm enclined to start again for ExFAT because the differences seem too great and it would clutter my FAT32 code too much.

To help me see the light I was wondering if anybody knows of a document / text that explains the *differences* between FAT32 and ExFAT on file-system level ? Or a document that explains what to add to an existing FAT32 implementation to support ExFAT (if that is at all possible).

As I said, from what I read, it seems quite different, but I haven't compared directory table structures yet etc. and maybe they're closer related than I feel right now.

Anyway, some input appreciated. I'm signing off. Reading that technical document while being tired is probably also not a good idea 😉

Quote
Posted : 23/01/2013 12:57 am
twjolson
(@twjolson)
Active Member

I did a detailed study of exFAT last year for Lock and Code's Reference Guide. The paper you describe is the best source I found for exFAT. The other one is a snippet from Jeff Hamm titled "exfat-excerpt-1-4" (I forgot where I found it).

If you are writing a program for exFAT, you are going to have to make quite a few alterations. Honestly, I don't recall anything that remained exactly the same. The difference isn't as stark as FAT vs. NTFS, but still, you probably have to write new code (maybe use the FAT code as a starting point).

Sorry I don't have better information, but to go into every little thing that's different would take up a whole document, and you already have it.

ReplyQuote
Posted : 23/01/2013 1:54 am
mscotgrove
(@mscotgrove)
Senior Member

I would start from scratch. Most of exFAT does not rely on a FAT as most files are not fragmented. (I presume you are talking about Read only)

ReplyQuote
Posted : 23/01/2013 3:30 am
Patrick4n6
(@patrick4n6)
Senior Member

You can download a version of Jeff Hamm's ExFAT info at http//computer-forensics.sans.org/summit-archives/2010/10-exfat-ham.pdf

ReplyQuote
Posted : 23/01/2013 6:29 am
CyberGonzo
(@cybergonzo)
Active Member

I presume you are talking about Read only

That is correct.
Thanks for your input

You can download a version of Jeff Hamm's ExFAT …

I will check it out, thanks.
Thanks for your input

you are going to have to make quite a few alterations

Thanks all, I think I know enough, a fresh start it will be.

ReplyQuote
Posted : 23/01/2013 1:16 pm
CyberGonzo
(@cybergonzo)
Active Member

www.cnwrecovery.com

Michael,

Please check your website, and check if it hasn't been compromised !!
My virus scanner, BitDefender, did not allow me to go to your website, due to malware !!

I assume this provides for your bread on the table, the roof over your head, I hope you can fix this soon !!

Cheers,
Peter, a collegue

PS. I quickly did a scan with virus total as well
https://www.virustotal.com/url/8cac1a7c4b2663278e98d4ea7651d0a5ae80356bf19a1fb16503efcd2a4ec197/analysis/1358929538/
No issues found … so maybe this is a false positive by BitDefender ?
In any case, worth chasing still.

ReplyQuote
Posted : 23/01/2013 1:26 pm
mscotgrove
(@mscotgrove)
Senior Member

Peter,

Thank you for your concern over my website. It looks OK with Norton. I was not aware of any problems - but will keep checking.

Michael

ReplyQuote
Posted : 23/01/2013 5:28 pm
CyberGonzo
(@cybergonzo)
Active Member

I wasn't able to put the time in today, but tomorrow I intend to start coding.

Question. I scanned the documents and (at least one) question is bothering me. Maybe I read over it ?

The root cluster is recorded in the VBR.
I read that the FAT is only consulted when a flag is set in the directory entry for a file.
I take it this is also true for folders ?

Which leads me to the root folder. Since there is no record describing it, I don't know if this folder can be fragmented, needing the FAT to determine its extents ?

Should I assume it's unfragmented, or should I consult the FAT for the root, flag or not ?

ReplyQuote
Posted : 24/01/2013 12:07 am
CyberGonzo
(@cybergonzo)
Active Member

Implemented.

FYI, and for other people landing here, in search for the same feedback, I ended up simply adding code to my existing FAT code. Ok, there are differences, but there are also lots of things that simply could be re-used.

My FAT code already had if(type == 12) {} else if (type == 16) {} if (type == 32){} conditionals in critical code, so I only needed to add ExFAT to that (which I gave type=64 eventhough the FAT is still 32 bits). And because of the latter I could actually simply change if (type == 32) to if (type >= 32) in FAT handling code.

The only bigger difference (with lots of resemblances however) is the scanning of the directory entries. A unique subroutine for that fixed that too.

The status and open issues

- Still need to test on larger data sets. Now implemented on a formatted USB stick, and gradually added data to test.

- My previous question about the root directory (possibly fragmented or not) is still open

- The SANS Institute document is not correct or incomplete (unless I missed that) when it comes to time zone offsets. I ran into a file time/date bug that I could not figure out until I read this (very useful) article http//computer-forensics.sans.org/blog/2010/07/19/exfat-file-system-time-zone-concerns on the use of the time zone byte. Bottom line, the offset is a 7-bit value, and not an 8-bit value as I first thought.

And now some more testing … but that's for Monday. It's (nearly) weekend … whoohoo …

ReplyQuote
Posted : 25/01/2013 8:51 pm
jaclaz
(@jaclaz)
Community Legend

Canot say if specifically it can help you, but maybe something in the source here
https://code.google.com/p/exfat/
is of use.
Since it recently went "Version 1.0" it should have overcome the issues you mentioned.

jaclaz

ReplyQuote
Posted : 26/01/2013 12:25 am
CyberGonzo
(@cybergonzo)
Active Member

Canot say if specifically it can help you, but maybe something in the source here
https://code.google.com/p/exfat/
is of use.
jaclaz

I can't say that I'm good with reading third party code but I had a look and it looks like they check the FAT for the root. Thanks for the tip

While tweaking my implementation I noticed that the checksum for the dir records doesn't seem to match. I use the little checksum calculation from the sans institute document.
So I figured I'll check mentioned code as well, turns out they do it the exact same way I implemented it. Or at least as far as I can see. Is anybody aware of any pitfall here that I'm not aware of yet ? I do not check it on deleted files, I know that won't work, but the checksum doesn't seem to match regular files and folders (small W7 formatted USB stick).

ReplyQuote
Posted : 29/01/2013 12:41 am
Rshullic
(@rshullic)
New Member

Hi,

I'm looking into ExFAT and as a first introduction I read through this pdf
reverse-engineering-microsoft-exfat-file-system_33274 (actually half way through)
It's getting a bit confusing and because it's been some time since I implemented FAT I'm not sure how different ExFAT is from FAT32. I feel I need to write some code first to see how things work rather than just reading text and not seeing the light.

A decision I need to make now is whether to extend my existing FAT implementation or start fresh for ExFAT (if too different). Right now I'm enclined to start again for ExFAT because the differences seem too great and it would clutter my FAT32 code too much.

To help me see the light I was wondering if anybody knows of a document / text that explains the *differences* between FAT32 and ExFAT on file-system level ? Or a document that explains what to add to an existing FAT32 implementation to support ExFAT (if that is at all possible).

As I said, from what I read, it seems quite different, but I haven't compared directory table structures yet etc. and maybe they're closer related than I feel right now.

Anyway, some input appreciated. I'm signing off. Reading that technical document while being tired is probably also not a good idea 😉

Hello, I am new to the forum, I found this post via a Google Search on SANS + exFAT, and I thought I'd chime in.

First of all, let me identify myself - I am the author of the SANS paper mentioned in this thread and post. The e-mail address (earthlink) in the paper is current, so you can reach me directly, but posting here would be fine, I can't say I will be able to monitor this thread much.

Some of what I will mention regards the timezone questions that also appear elsewhere in this thread.

I am a graduating Masters student at John Jay College, CUNY in the Forensics Computing curriculum and in Fall 2009 I was taking a digital forensics course and my term project was on the exFAT file system. SO the SANS paper was first a class paper. In October 2009, as part of my going to conferences as I did for my job, I attended a session at the Techno Forensics conference held at NIST that was given by Jeff Hamm that was an intro to exFAT. Jeff's talk provided a lot of insight for me to start my paper, but I dug in deeper and did research and relied heavily on the Microsoft Specification for exFAT. By the time I finished the paper, I had a lot more detail, and even found errors or inconsistencies in some of Jeff's presentation, including his interpretation of the time zone values. I also discovered that time zone values varied with Windows 7 and Windows XP, I'll leave that variation for later.

Now Jeff was practical, in that he taught and performed forensics investigations, and at the time he worked for paradigm solutions and had a blog (still there) at http//paradigmsolutions.wordpress.com/2009/12/10/extended-fat-exfat/. Jeff also had limited information, but he was the leader in exFAT and getting things done. From my perspective, I was being more theoretical, so I was heavier on the research end. When I came up with the inconsistencies in the time zone calculation, I had e-mailed Jeff, and we collaborated on a final interpretation. You see, 99% of the information for exFAT is in the specification released by Microsoft as part of a patent for a different patent (the exfat patent itself did not have the spec), and the time zone calculation, as well as the location was not provided. As a matter of fact the current time zone layout is different than what was released in the specification, so this led to trial and error.

So I brute forced the time zone, I changed the time zone, created a file, and dumped the exFAT filesystem and collected the values, and came up with a table will all the time zones. I presented it to Jeff, and he came back with the 7-bit number calculation. As it says on page 40 of the paper "A formula
was developed (by Jeff Hamm) that shows the time zone offset to be a 7 bit signed
integer. The purpose of the high order bit has not been determined." And Jeff was given credit in the paper for his contribution. Actually the meeting with Jeff at Techno is on page 4 of the paper as well.

In Dec 2009 I submitted my proposal for a SANS GCFA gold paper, and published by Mid Feb 2010. During 2010/2011 I presented at several conferences of exFAT, including Computer Forensics Show, Techno Security, HTCIA, and at a SANS What's New in Incident Response and Forensics in July 2010. You might find slide presentations online, but I also maintain a blog at rshullic.wordpress.com for exfat which has the slide decks and some additional stuff discovered after the release of the paper. The paper is also mentioned now as a footnote in two Sybex 2012 versions of forensics books, as a source for additional information.

Although most of my presentations were solo, the SANS What's Works conference was a co-presented presentation given by Jeff and myself. That PDF is at http//computer-forensics.sans.org/summit-archives/2010/10-exfat-ham.pdf and you will see both Jeff & Myself listed in the copyright.

Now I mentioned that I gave a presentation at the computer forensics show (CFS), well I gave it at two different CFS, the 2nd was April 2011 where the session was video taped for the AT&T Tech Channel. You can Google for the AT&T Tech Channel + exFAT and should find the link, which when I do it it is at http//techchannel.att.com/play-video.cfm/2011/8/16/Conference-TV-Computer-Forensics-Show-Introduction-to-exFAT but sometimes the link doesn't work unless you click it from the search return. Also keep in mind that this presentation, which is NOT like the Sans What Works presentation, if more than a year after the paper was published.

So, that was a little about history of the paper, and provides some additional resources for exfat. This post might not answer the questions you need answered, but hopefully may provide some insight. I will try to answer questions where I can, I have selected the "Notify me when a reply is posted" so I should at least get an alert.

Thanks for listening.

ReplyQuote
Posted : 18/03/2013 12:09 am
Rshullic
(@rshullic)
New Member

I wasn't able to put the time in today, but tomorrow I intend to start coding.

Question. I scanned the documents and (at least one) question is bothering me. Maybe I read over it ?

The root cluster is recorded in the VBR.
I read that the FAT is only consulted when a flag is set in the directory entry for a file.
I take it this is also true for folders ?

Which leads me to the root folder. Since there is no record describing it, I don't know if this folder can be fragmented, needing the FAT to determine its extents ?

Should I assume it's unfragmented, or should I consult the FAT for the root, flag or not ?

There are three (3) special files that exist in the Cluster Heap (Data Area), and they are

Root Directory
UP Case Table
Allocation Bit Map
(In the case of TexFAT) there will actually be TWO Allocation BitMaps, bringing the special files to 4.

Each of these files do not have a standard directory, and in the case of the Root Directory does not even have a directory entry at all, as stated - it is pointed by a value in the VBR.

I was always curious that when I formatted a exFAT file system, and then dumped the filesystem using a DD command, that the FAT entries for those three (3) clusters were always 0xFFFFFFFF. (In computer programming speak - we call that "High-values" where 0x0 is called "low-values". In FAT, including legacy FAT (FAT 12/16/32) and exFAT, we call high-values the end of file (EOF) marker. Then, it came to me that since these special files did not have normal directory entries, they would always have a FAT cluster chain, even if the cluster is built and maintained in a contiguous organization. The BitMap and UPCase table may take multiple clusters but they will never fragment because they are allocated once (at format time). But they will have a FAT chain. The Root directory, will fragment, and keep in mind that the root does not have a limit in size, where a subdirectory is limited to 256MIB in size.

Any processing of these 3 special files should consider the file as fragmented and assume that the FAT invalid bit (which is not available for these files) as if the bit was set to Zero, meaning the FAT is VALID and you have to refer to it.

Yes, I know there is more in the answer than you asked, but I wanted to be more general in the answer.

ReplyQuote
Posted : 18/03/2013 12:27 am
jaclaz
(@jaclaz)
Community Legend

Thanks for listening.

You are very welcome ) , very interesting/clear post.

jaclaz

ReplyQuote
Posted : 18/03/2013 12:29 am
CyberGonzo
(@cybergonzo)
Active Member

Thank you Robert, for your detailed input !

It's been a while now since I implemented ExFAT, I had to go look in my code, but yes, I do use the FAT to explore the root, I answered my own question through testing, so that is OK !

IB with ExFAT support has not been released yet, I only do two updates per year, but if you like a copy to try, let me know.

Best Regards,
Peter

ReplyQuote
Posted : 18/03/2013 12:12 pm
Page 1 / 2
Share: