±Forensic Focus Partners

Become an advertising partner

±Your Account


Username
Password

Forgotten password/username?

Site Members:

New Today: 0 Overall: 33166
New Yesterday: 0 Visitors: 73

±Follow Forensic Focus

Forensic Focus Facebook PageForensic Focus on TwitterForensic Focus LinkedIn GroupForensic Focus YouTube Channel

RSS feeds: News Forums Articles

±Latest Articles

RSS Feed Widget

±Latest Webinars

mft2csv - NTFS systemfile extracter and $MFT decoder

Forensic software discussion (commercial and open source/freeware). Strictly no advertising.
Reply to topicReply to topic Printer Friendly Page
Forum FAQSearchView unanswered posts
Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Thu Sep 08, 2011 6:58 am

Thank you very much for taking your time and comment on the app. You are absolutely right about the error in detecting invalid records. It will be fixed.
_________________
Joakim Schicht

github.com/jschicht 

joakims
Senior Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Fri Sep 09, 2011 12:58 am

You're welcome, it's a fascinating file system.

I've been following the references that you gave and just realised that the original Autoit script and the python one before that all seem to omit one important aspect of mft records. An mft record needs to be corrected before it can be decoded properly. In Linux I've seen the process referred to as 'fixup'. It uses the update sequence to provide a coarse integrity check on each sector of the record. Using the partial record shown below, it works as follows:

At offset 0x04, the two bytes, 30 00, give the offset to the update sequence (0x30) and the next two bytes, 03 00, give the size of the update sequence array in words (0x03=6 bytes).

At offset 0x30 the 6 bytes are 09 00 31 0b 00 00. The first two bytes are used to replace the real bytes at the end of each sector in the record. So at offset 0x01fe and at 0x03fe the bytes 09 00 appear again as shown. This provides an integrity check for the sectors. If the bytes are not the same the sector, and hence the record, is probably bad.

Fixing up a good record is done by moving the second pair of bytes in the update sequence array to the end of the first sector and the third pair of bytes to the end of the second sector. In this case, 31 0b is moved to 0x01fe and 00 00 is moved to 0x03fe.

For this particular partial record, the change at 0x03fe makes no difference, but the one at 0x1fe is part of the data run for the file and would make a huge difference when reading the file. The bottom line is that if you don't do the fixup, it will bite you at some stage.

0000--46 49 4c 45 30 00 03 00-56 e6 03 1d 07 00 00 00
0010--14 00 02 00 38 00 01 00-10 02 00 00 00 04 00 00
0020--00 00 00 00 00 00 00 00-05 00 00 00 d6 a7 00 00
0030--09 00 31 0b 00 00 00 00-10 00 00 00 60 00 00 00
....
....
01f0--31 89 72 05 31 32 b8 98-fa 31 2d 19 91 02 09 00
0200--2c 72 fd 00 d0 30 94 e2-ff ff ff ff 82 79 47 11
....
....
03f0--00 00 00 00 00 00 00 00-00 00 00 00 00 00 09 00

Please let me know if you can't follow my explanation and I'll have another go.

Ddan  

Ddan
Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Sat Sep 10, 2011 12:50 pm

@Ddan
Thank you for explaining that. I was partly aware of some of it, but not all. I will look into and send you a message if I get stuck or run into errors. Nice explanation!
_________________
Joakim Schicht

github.com/jschicht 

joakims
Senior Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Wed Dec 07, 2011 9:06 am

I have recently updated a few things in the package:


In the file extractor:
- Added a FileSelectFolder function to specify where to save the output.
- Removed the default ".bin" extension, so that the outputted extension is as given in $MFT.

In WinTimeFunctions:
- Created a new format like YYYY-MM-DD HH:MM:SS:MSMSMS, which is more filter friendly in Excel. But I'm open for suggestions on other datetime formats.

In mft2csv:
- Added nanoseconds to the datetime variables.
- Default outputted datetime format changed to "YYYY-MM-DD HH:MM:SS:MSMSMS:NSNSNSNS" for all datetime variables.
- MSecTest changed to evaluate the last 7 digits (msec + nsec)

Also included 64-bit compiled binaries for both applications.

@Ddan
I forgot about the issue you described, but will fix that next time.
_________________
Joakim Schicht

github.com/jschicht 

joakims
Senior Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Fri Dec 09, 2011 6:41 am

Nice to see you are still progressing the utility, but disappointed you have not corrected the programming errors.

Just a couple of quick questions and a piece of information.

In your code, when you use the Select statement, you end the cases with a long "AND" expression. Is there some reason you can't simply use "Case Else" which basically means any other value not already cased?

I'm sure this is a silly question and the answer is obvious but I just can't see it. In your csv output file, why are all the terms in inverted commas? Surely these are redundant in a csv file?

The information I have is that you only accept vales of 0-3 for the $HEADER_Flags variable but, if you look at the mft record for the $Secure metafile, you will see its flag value is 9. Also the values for $Quota, $ObjId and $Reparse are 13. Unfortunately in terms of files/folders etc, I don't know what these other bits mean. Maybe one has to do with file locking?

Ddan  

Ddan
Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Thu Dec 22, 2011 10:55 am

Updated to version 1.4 of mft2csv.

Fixed:
- A bug in the record signature evaluation.
- Added csv entry for record signature evaluation.
- Added record integrity check by comparing the Update Sequence number to the record page (sector) end markers (2 bytes for each sector).
- Some cosmetic changes.

@Ddan
Answers to your questions:

1. Quotation marks (inverted commas) in csv.
Honestly I don't remember. I agree with you and they are now removed.

2. Unknown $HEADER_Flags.
I have not found documentation for these, nor found any logic for it. If you (or anybody else) are certain about some new flags, I am happy to add it. And that's very fast and easy to change anyway.

3. The logic error when evaluating record signature.
You where absolutely right and it is now fixed. I added a new variable ($Signature) to the csv as the second field. Implementation is a little bit of a shortcut, because I just put "GOOD" to all 0x46494C45 and "BAD" to all others.

4. The fixups.
You are right about it (of course). For now I have implemented a basic integrity check by comparing those 3 values. If any of those differ, then the variable $IntegrityCheck in the csv will get the value "BAD", else it's "OK". Implementing a full fixup is thus on the todo list.

5. The Select statement.
That can probably be done in several different ways. If I get the time, I can test to see if any speed can be improved by changing it. However, the speed issue, is a drawback and will be looked into in the future.
_________________
Joakim Schicht

github.com/jschicht 

joakims
Senior Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Mon Dec 26, 2011 1:24 am

Re the unknown $HEADER_Flags, like you I have not found any documentation for any of the bits other than the first two. It is very definitely a four bit field though. If you examine the mft record for the metafile $Secure, its bit field is 1001 (ie 9). Also the three metafiles $ObjId, $Quota and $Reparse which are in the $Extend folder have bit fields 1101 (ie 13).

I can give you a rational explanation of what these extra bits are likely to mean but first you need to look at the first two bits a little differently.

Normally we have 00xy, where x indicates file/folder and y indicates in use/not in use. One of the basic features of the NTFS file system though is that everything is a file. So if instead we view the x bit as indicating a file containg data/file containing an index, this provides the key into the unknown bits.

If you look at the mft records for the above named metafiles, you will see that they all contain sets of indices not data. I think bit four, when set, refers to a file containing security indices. Similarly a set bit three refers to a file containing "other" indices.

As far as I am aware these are the only files that use these extra bits.  

Ddan
Member
 
 

Page 2 of 10
Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next