±Forensic Focus Partners

Become an advertising partner

±Your Account


Username
Password

Forgotten password/username?

Site Members:

New Today: 0 Overall: 33166
New Yesterday: 0 Visitors: 80

±Follow Forensic Focus

Forensic Focus Facebook PageForensic Focus on TwitterForensic Focus LinkedIn GroupForensic Focus YouTube Channel

RSS feeds: News Forums Articles

±Latest Articles

RSS Feed Widget

±Latest Webinars

mft2csv - NTFS systemfile extracter and $MFT decoder

Forensic software discussion (commercial and open source/freeware). Strictly no advertising.
Reply to topicReply to topic Printer Friendly Page
Forum FAQSearchView unanswered posts
Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Tue Dec 27, 2011 11:48 pm

Both apps with new update;

mft2csv v1.5
Changed record signature detection to:

46494C45 = GOOD (although FILE is strictly speaking the correct one)
44414142 = BAAD
00000000 = ZERO
all other = UNKNOWN

Added header flag interpretation:
0x9 = FILE+INDEX_SECURITY (ALLOCATED)
0xD = FILE+INDEX_OTHER (ALLOCATED)

Fixed crash on decoding of certain corrupt $MFT, by skipping decode of those records.

Extractor v1.4
Solved several bugs in the solution that calculated runs. Negative moves where reloved wrongly. And very fragmented files with runs located past record offset 0x1fd was also incorrectly solved. Actually that last fix also fixed decoding of attributes extending over that same offset.

Note however that compressed and/or sparse files are not yet supported for extraction. I therefore added a break when trying to extract such a file.

I'm now very sure that runs are correctly solved. My "worst" $MFT at 97 MB and with 121 runs (including lots of negative ones) are correctly extracted!

@Ddan
I just made a small discovery when redoing the runs. Byte 3-4 (in big endian) in the Update Sequence Array is the missing 2 bytes at offset 0x1fe-0x1ff when an attribute streches beyond that (for instance runs within $DATA). Ie, in such a case (for instance with resident data or maybe other attribute) you must not forget to copy the 2 bytes of missing data that should have been at 0x1fe-0x1ff, back in.

Edit: Just uploaded version 1.5 of the extracter with a fix on resident data extraction.
_________________
Joakim Schicht

github.com/jschicht 

joakims
Senior Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Wed Dec 28, 2011 11:51 pm

That 'small discovery' was what my earlier post about Fixup was all about. I did say that it would bite you at some stage if you didn't do it. It is not unusual for mft records to extend past the 0x1fe-0x1ff point, but I don't think I have seen any where the bytes at 0x3fe-0x3ff make any difference.

Having now discovered for yourself that you really do need to do the fixup before processing the mft record, you will undoubtably find that it will also resolve the problem with filenames that have bad unicode characters!

Those bytes at 0x1fe-0x1ff can have a dramatic effect on things like long filenames, times and dataruns when they are not corrected.

Ddan  

Ddan
Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Thu Dec 29, 2011 5:42 am

I'm sorry to keep harping on about this, but it really is important. I thought from my first post about Fixup that you had understood, but re-reading your last post makes it clear that you didn't. So let's be unequivocal.

When you read an mft record from disk, you do not get the real record. What you get is a modified version. It has been modified so that it has an inbuilt integrity check. You need to undo those modifications before you process the record. If you don't undo the modifications, you are not processing the real record. The undo process is called Fixup. It consists of putting two pairs of bytes back into their proper position. The two pairs of bytes are in the update sequence array and their proper positions are the last two bytes in each sector of the mft record.

You have already seen examples of what happens when you don't do Fixup. It causes dataruns to go haywire, filenames to have bad unicode, resident data to contain bad values, etc, etc, etc.

Ddan  

Ddan
Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Thu Dec 29, 2011 12:04 pm

He he, I now feel like a moron.. Anyways, have moved over to reassembling runs with compression (including decompressing it), and it's going really good. But this time I will keep my mouth shut a little longer. Smile
_________________
Joakim Schicht

github.com/jschicht 

joakims
Senior Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Thu Dec 29, 2011 7:37 pm

A question concerning ntfs compression.

I have resolved runs, meaning I can extract the raw compressed data of the file. I have also identified the winapi rtldecompressbuffer being able to handle at least parts of the compressed data. But I'm facing the issue that only the first 8192 bytes are decompressed, so I'm wondering if:

1. There is a better api or method to use?
2. More modification to the reassembled compressed parts are needed?

Anybody familiar with this?

Edit: Turns out decompression is working just fine if applied to each individual run, so reassembly must be done on decompressed chunks.
_________________
Joakim Schicht

github.com/jschicht 

joakims
Senior Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Thu Dec 29, 2011 11:19 pm

I'm not familiar with the api so cannot help there.

With the 8192 bytes though, was the cluster size 512 bytes? If it was, then it probably is ok.

The compression works on a block-by-block basis. A compression block is 16 clusters. The compression flag is always 4, and 2 to the power of 4 is 16. The fact that the flag is set doesn't actually mean that the file is compressed. It only means that the file is scheduled to be compressed.

If the file is resident, it is never compressed. A non-resident file is first split into 16-cluster compression blocks. Each block is then examined and if compression saves at least one cluster, then it is compressed. Otherwise it is not compressed. This means that the compressed file can have blocks which are compressed and some which are not compressed. The first two bytes in the compressed block tell you which is which.

To determine whether a file is compressed, you need to split the data run into 16-cluster subsets. A compressed subset would be of the form (x data clusters) plus (y sparse clusters) where x+y=16. Note that after decompression, the x data clusters will expand to 16 clusters so the sparse clusters are not actually used.

Hope this helps.

Ddan  

Ddan
Member
 
 
  

Re: mft2csv - NTFS systemfile extracter and $MFT decoder

Post Posted: Thu Dec 29, 2011 11:44 pm

Learning from past mistakes, I will not be too bombastik in my statements. Anyways, I just uploaded a new version of extracter with preliminary support for compressed files. It has been tested OK on a few files using the same api as mentioned before.

Thanks for pointing out that the flag only indicates that a file, at least, is to be compressed, and not already compressed with the magic 16 clusters as a big clue. I think the existing solution is very close to that, but probably will fail for files with an uncompressed run and the compressed flag set simultanously.

1 question though:
When you say the first two bytes in the compressed block tells you which is which, what is the actual meaning/interpretation of these 2 bytes. I noticed some patters but did not identify the difference. If I understand you right it is therefore 2 ways of identifying whether a block is compressed/uncompressed;
a. Looking at the runs and their clusters.
b. Looking at the first 2 bytes of a given data block (non-sparsed run).
_________________
Joakim Schicht

github.com/jschicht 

joakims
Senior Member
 
 

Page 3 of 10
Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next