mft2csv - NTFS syst...
 
Notifications
Clear all

mft2csv - NTFS systemfile extracter and $MFT decoder

68 Posts
9 Users
0 Likes
8,710 Views
joakims
(@joakims)
Posts: 224
Estimable Member
Topic starter
 

I post it here as well since it is in fact open source and free..

So I finally finished first version of my mft2csv application. I registered it at the project hosting at Google code; http//code.google.com/p/mft2csv/

Edit New home is; https://github.com/jschicht

It is 2 applications
- One that will carve files directly from physical disk by reading sectors as specified in $MFT. Several modes are available.
- The main application is the mft2csv tool that will decode and log large amounts of data from $MFT and to csv format. The current base of 126 variables are a very good starting point for further improvement (and maybe some are unneeded).

Check out the site, there is source, compiled exe and descriptions. The source is free and preatty much without restrictions.

I really recommend doing something like this if you want to dig into NTFS and learn more about it the interesting and exciting way. It is very likely that I will expand it with more features soon. Since it is written in the scripting language AutoIt it is Windows only. However it is very easy to work with.

 
Posted : 11/08/2011 4:40 am
(@htcicolonial)
Posts: 1
New Member
 

Great tool with an easy-to-look-at output!
I was having some problems with it recognizing a volume mounted with FTK Imager, but it could see the same image mounted in EnCase with PDE, so must be an issue with Imager.
I also was able to carve MFT records and rename the output $MFT.bin and the MFT2CSV utility parsed it out well.
As a suggestion for leaning the $MFT layout, it might be good to put the attribute byte offset in each column header.
Thanks, Joakim!

PS - Darren Freestone's booklet at lockandcode.com is also a great resource for parsing the $MFT manually.

 
Posted : 11/08/2011 6:54 pm
joakims
(@joakims)
Posts: 224
Estimable Member
Topic starter
 

Thanks for the feedback. Suggestions noted.

New version of both apps are just uploaded.

In mft2csv I've fixed the issue with more than 3 ADS's. Have also added file size as a new csv field. It is for the first $DATA attribute I think it is easier to also have it available this way instead of as separated on resident/non-resident fields only.

In the extracter there was an error in the function _GetAllRuns() that incorrectly solved runs in certain cases. That is now fixed. Note however, that the experimental functionality to rip files by their MFTnumber, is not fully working. I did not account for the fragmentation of MFT itself. But since I have the runs solved I think I can handle it too.. But until it's fixed I've put a hardcoded exit after record 1000, since fragmentation is rare so early in the file. The functionality would work when MFT is not fragmented though..

 
Posted : 12/08/2011 2:13 am
joakims
(@joakims)
Posts: 224
Estimable Member
Topic starter
 

While fixing a tiny bug in the timestamps, I started wondering what kind of timestamp is the best to use. Is it as UTC or local time?

 
Posted : 13/08/2011 6:50 pm
(@kovar)
Posts: 805
Prominent Member
 

Greetings,

Well, the timestamp is what it is. What you're asking is what timezone you should present the information using. And my answer is - I don't adjust the timestamp at all. The user should record the timezone settings of the system they pulled the $MFT from.

You could add a switch to allow the user to report all times using a specific timezone.

-David

 
Posted : 14/08/2011 7:43 am
joakims
(@joakims)
Posts: 224
Estimable Member
Topic starter
 

@David
Sorry for late reply. I was trying to completely understand the timestamp and timezone handling, and I think it's clear now. As you suggested it only really matters what timezone configuration is on the system that analyzes the $MFT (and not the system it was taken from). For that reason I've implemented a choice when launching the app to choose if UTC/local time is wanted in the timestamps. Also added information in the csv header about what format (utc or local) the timestamps are shown in, as well as info about what timezone configuration is present (the file_time_delta/36000000000=hours). Btw, milliseconds are now as they should, after a bugfix.

Sample csv header

#Timestamps presented in Local time
#Current timezone configuration (bias) including adjustment for any daylight saving = -12 hours

Now also the $MFT runs are solved, so kind of any file on the volume with a valid record should be extractable from the fs (with certain limitations).

 
Posted : 17/08/2011 5:57 pm
 Ddan
(@ddan)
Posts: 42
Eminent Member
 

Hi,

I'm not an Autoit programmer, but I think you have a small logic error in your mft2csv program. The line

If NOT StringMid($MFTEntry,1,8) = '46494C45' Then

does not do what you think it does. I think it should be

If NOT (StringMid($MFTEntry,3,8) = '46494C45') Then

with the two values enclosed in brackets. Note also the offset should be 3 not 1.

Regards,

Ddan

 
Posted : 08/09/2011 9:44 am
joakims
(@joakims)
Posts: 224
Estimable Member
Topic starter
 

Thank you very much for taking your time and comment on the app. You are absolutely right about the error in detecting invalid records. It will be fixed.

 
Posted : 08/09/2011 12:58 pm
 Ddan
(@ddan)
Posts: 42
Eminent Member
 

You're welcome, it's a fascinating file system.

I've been following the references that you gave and just realised that the original Autoit script and the python one before that all seem to omit one important aspect of mft records. An mft record needs to be corrected before it can be decoded properly. In Linux I've seen the process referred to as 'fixup'. It uses the update sequence to provide a coarse integrity check on each sector of the record. Using the partial record shown below, it works as follows

At offset 0x04, the two bytes, 30 00, give the offset to the update sequence (0x30) and the next two bytes, 03 00, give the size of the update sequence array in words (0x03=6 bytes).

At offset 0x30 the 6 bytes are 09 00 31 0b 00 00. The first two bytes are used to replace the real bytes at the end of each sector in the record. So at offset 0x01fe and at 0x03fe the bytes 09 00 appear again as shown. This provides an integrity check for the sectors. If the bytes are not the same the sector, and hence the record, is probably bad.

Fixing up a good record is done by moving the second pair of bytes in the update sequence array to the end of the first sector and the third pair of bytes to the end of the second sector. In this case, 31 0b is moved to 0x01fe and 00 00 is moved to 0x03fe.

For this particular partial record, the change at 0x03fe makes no difference, but the one at 0x1fe is part of the data run for the file and would make a huge difference when reading the file. The bottom line is that if you don't do the fixup, it will bite you at some stage.

0000–46 49 4c 45 30 00 03 00-56 e6 03 1d 07 00 00 00
0010–14 00 02 00 38 00 01 00-10 02 00 00 00 04 00 00
0020–00 00 00 00 00 00 00 00-05 00 00 00 d6 a7 00 00
0030–09 00 31 0b 00 00 00 00-10 00 00 00 60 00 00 00
….
….
01f0–31 89 72 05 31 32 b8 98-fa 31 2d 19 91 02 09 00
0200–2c 72 fd 00 d0 30 94 e2-ff ff ff ff 82 79 47 11
….
….
03f0–00 00 00 00 00 00 00 00-00 00 00 00 00 00 09 00

Please let me know if you can't follow my explanation and I'll have another go.

Ddan

 
Posted : 09/09/2011 6:58 am
joakims
(@joakims)
Posts: 224
Estimable Member
Topic starter
 

@Ddan
Thank you for explaining that. I was partly aware of some of it, but not all. I will look into and send you a message if I get stuck or run into errors. Nice explanation!

 
Posted : 10/09/2011 6:50 pm
Page 1 / 7
Share: