Notifications
Clear all

Problem reading MFT  

  RSS
westor2010
(@westor2010)
New Member

Hi,

I need to make a little program that reads MFT from a NTFS drive and lists all files in it. So, reading here and studing about the MFT and NTFS, i figured out how to do it.
Basically, i first read the first clusters from the drive and obtains data like
The bytes per sector, sectors per cluster, and the cluster number of MFT.
Using this info I read data at the sector defined by the start cluster of mft and find the first
attribute that is $MFT.
It's ok, then i continue reading, for example in my disk each mtf record is 1024 bytes, so i continue reading and each portion of 1024 bytes is a file on disk, I take its name and display it.
Everything is ok, i read exactly 102928 sectors and i find the last FILE block in the MFT.
My problem started when i realized that there were some files that were not in that list.
So, i downloaded a program called NTFS Walker which lists all files in a drive read from MFT.
The files are listed in the same order I list them with my program until it reaches the last file
I list in mine. But the list of files from NTFS walker continues and mine, doesn't.
So, I used a HEX editor to read the disk directly and know why my code stopped at some place and did not find any files.
If I read my disk with a hex editor i go to the last sector my code finds a file.
I see that if I go two the next sectors, they have no data and then, other data that doesn't seem to be a MFT file record (i distinguish then because they start with FILE).
So, i made a raw search to find the next file that NTFS Walker showed after the last my code shows, and i found its entry (and all the other) at other sector near the end of the disk. So, i think my MFT is fragmented. What i need to know is, how I know where the next MFT blocks are (or all the others if my mft is fragmented in more that two parts). I thought i could do it by reading the $MFT attribute and analizing the Data runs in the $DATA attribute, but i cant get the value of the sectors where the next part of MFT starts, so, how can i do??

Please Help!!!
Thanks from Argentina

Antonio Briones
.NET Software Developer

[email protected]

Quote
Posted : 24/03/2010 8:46 pm
mscotgrove
(@mscotgrove)
Senior Member

The first MFT entry contains a datarun field to show where all the fragments are stored

ReplyQuote
Posted : 24/03/2010 10:05 pm
westor2010
(@westor2010)
New Member

The first MFT entry contains a datarun field to show where all the fragments are stored

Thank you very much! I've been asking this question in other forums and no one could reply!!.
I thougt it could be this way so started to read about the $DATA attribute and its content to apply this to first sector, the one you refer to, i think the $MFT should be.
So I search for the $DATA attribute, I localize it, 80 00 00 00 (in red) then its lenght 50 00 00 00 (in blue) but I dont know if what Im doing is ok.
I read out there that the $DATA attribute should finish with 00 00 and the, when attributes list finishes, FF FF FF FF.
In my case I find the FF FF FF FF (in green) but not the 00 00.
On the other hand, i read that the lenght (50, 80 in dec) is the byte count from the start of the attribute, so the attribute should finish at the number marked in yellow.

So, where does the data run start? At offset xxxx140?? (32), in that case, why does it start here??
If it starts there, 32 means the lenght is 5 (as i read out there), so the first data run is 42 32 00 00 0C???? the next data run should also start in the next byte, 32, its 5 bytes, so it should be FE 1C C0 D0 FB??? Then 00 00 means there are no more data runs?? and so 4C and 8C what mean??
Anyway, if all this is correct, I found the second part of mft starts at sector 262945984 (each sector 512 bytes), but i can't get this value anyway using the data above!!
Thanks for your help, it's the first place where someone replies and I cant find any help for this anywhere!

Thanks!!

ReplyQuote
Posted : 24/03/2010 11:42 pm
Wardy
(@wardy)
Active Member

Hi,

it sounds like you are assuming the MFT is one contiguous block. It can be fragmented, which would explain the issue you are getting.

As mentioned you need to process the $MFT entry, so that you get the cluster runs for the whole MFT.

ReplyQuote
Posted : 24/03/2010 11:54 pm
westor2010
(@westor2010)
New Member

Hi,

it sounds like you are assuming the MFT is one contiguous block. It can be fragmented, which would explain the issue you are getting.

As mentioned you need to process the $MFT entry, so that you get the cluster runs for the whole MFT.

Ok, the point is that I dont know a lot of all this, the $MFT entry youre refering to is the one Im showing above???

ReplyQuote
Posted : 25/03/2010 12:10 am
Ivalen
(@ivalen)
Junior Member

Well I cant confirm you found the very start of the $MFT at the sector you mentioned. But the second data run is to be interpreted as follows

Starting cluster 16,502,976 for a length of 7422 clusters.

ReplyQuote
Posted : 25/03/2010 12:43 am
westor2010
(@westor2010)
New Member

Well I cant confirm you found the very start of the $MFT at the sector you mentioned. But the second data run is to be interpreted as follows

Starting cluster 16,502,976 for a length of 7422 clusters.

Ok, I think this is the first MFT record, since the file attributes of the next blocks are the ones I read are for ntfs itself and then start the rest of the files blocks.
$MFT
$MFTMirr
$LogFile
$Volume
$AttrDef
.
$Bitmap
$Boot
$BadClus
$Secure
$UpCase
$Extend

$Quota
$ObjId
$Reparse
$RmMetadata
$Repair
$TxfLog
$Txf
$Tops
$TxfLog.blf
$TXFLO~1 $TxfLogContainer00000000000000000001
$TXFLO~2 $TxfLogContainer00000000000000000002
MAINQU~2.QUE MainQueueOnline1.que
CONTEN~2.DIR Contents1.dir
Setup.evtx SETUP~1.EVT
AITEVE~1.003 AitEventLog.etl.003
AGGLGL~1.DB AgGlGlobalHistory.db
PFSVPE~1.BIN PfSvPerfStats.bin
AgRobust.db
mxdwdui.BUD
AGGLFA~1.DB AgGlFaultHistory.db
Panther

So, as you say the second data run is cluster 16502976 with a lenght of 7422.
The first data run is cluster 786432 with a lenght of 12866.
Cluster 786432 * 8 (sectors per cluster) is sector 6291456, which is the sector where this $MFT starts as you see in the pick above.
But If I do the same with the second, 16502976 * 8 = sector 132023808, theres nothing there.
I think I have to add it the previous localization(786432 + 16502976) * 8, in this sector I have nothing.
Anyway, I cant get to the next sector where the next FILE block for the next file is, It should be 262945984

ReplyQuote
Posted : 25/03/2010 1:29 am
westor2010
(@westor2010)
New Member

Also, how I know that the firs pick is the firs $MFT block?
I read the first sector, and go to offset 0x30 and the 8 bytes in little endian are the first cluster of MFT as in this pick.

In dec it is cluster 786432, sector 6291456, which is the sector where i find the entry $MFT, in the first pick i enclosed above.
In this entry is where i read the $data attribute and find this data runs ive commented in the previous post.

ReplyQuote
Posted : 25/03/2010 1:58 am
Wardy
(@wardy)
Active Member

Ok…. I am doing this from memory and not referring to notes I made while parsing the MFT, I don't have the source code since parting ways with e-fense.

Ideally, you want to process the "attributes list" attribute. Although the MFT is unlikely to be so fragmented that it requires one, other files may… When parsing them, bare in mind it is possible for the attributes list to become "non resident". If it out grows the amount of space available within 1024 bytes, it will need to be stored somewhere else… The attribute list becomes a bit compressed cluster run and once parsed, the cluster run on disk contains the attribute list…

This will allow you to get all of the entries for a given file should it be very fragmented. I suggest you do this, as it's important regardless of file to be certain you always read everything.

My parser started by reading the first 5 * 1024 bytes of the MFT, giving me the entry containing the $MFT. From there, check for an attributes list.

The cluster runs are bit compressed within the Data attribute. The values are signed, so can be positive or negative. If for example you start off at sector 1000, it is possible for the next part of a segmented file to be located prior to sector 1000.

As the Data attribute is bit compressed, what you have said is actually correct. You need to add the next value within the data attribute to your current location.

Hope this helps.

ReplyQuote
Posted : 25/03/2010 2:22 am
mscotgrove
(@mscotgrove)
Senior Member

Data runs are offsets frim the previous data run. They can be positive or negative, hence the for a positive value, often a 0 has to be added to the offset if it is inthe range 0x8??? to 0xf???

Thus in your example, the second run will be

Cluster 0xc0000 + 0xfbd0c0 =

or 0xc0000 - 0x42f40
= 0x7d0c0 (approx, not entirely sure about arithmetic)

As the second value is 0xfbd0c0 this will be treated as a negative number. If the lengtn was 4, and not 3, then it would be a positive number.

NB, it is MUCH simpler to think in hex, rather than decimal when working with sectors, offsets etc

ReplyQuote
Posted : 25/03/2010 2:59 am
westor2010
(@westor2010)
New Member

Data runs are offsets frim the previous data run. They can be positive or negative, hence the for a positive value, often a 0 has to be added to the offset if it is inthe range 0x8??? to 0xf???

Thus in your example, the second run will be

Cluster 0xc0000 + 0xfbd0c0 =

or 0xc0000 - 0x42f40
= 0x7d0c0 (approx, not entirely sure about arithmetic)

As the second value is 0xfbd0c0 this will be treated as a negative number. If the lengtn was 4, and not 3, then it would be a positive number.

NB, it is MUCH simpler to think in hex, rather than decimal when working with sectors, offsets etc

Yes, youre right!!!!! I had read that the localizacion could be negative but i thought that it was when the first bytes where between 80 a FF, not the last ones, but in little endian, the last are the first.
So, I searched that cluster and there I found the next part of MFT (its not the one I was saying before, but it also contains the other files, maybe the one I said is a copy or something like this).
Now, I need to make a c# app to read all this files, summarizing the concepts i learned, the first Data run in MFT attribute is the same cluster I'm reading (in one of my posts I showed how the localization of the first data run was pointing to the mft cluster itself), then the next data runs point to the localization and clusters count of the next parts of MFT (in my case theres only one but there could be more). I know which is the last data run when i reach a 00 00 (as found after the second attribute), is this ok??

And the last question, as you can see in the first image I uploaded in this thread, I marked in red where the data attribute started 80 00 00 00 and then its lenght 50 00 00 00 in green, and when it fisished in yellow 8c (counting the byte lenght 80 dec). Now the first data run starts in 00C0000140 being 32 42 32 00 00 0c and then the second. But, from the start of the data attribute until I reach this offset where starts the first data run, there 64 bytes I don't know what they are, anyway, how do I know the first attribute is there and not before??

Thanks to all!!!

ReplyQuote
Posted : 25/03/2010 7:06 am
mscotgrove
(@mscotgrove)
Senior Member

http//www.reddragonfly.org/ntfs/concepts/data_runs.html

This site may give you more help

The 0x40 bytes you mention as defined by the value in offset 0x20 of the Data record.

The byte 0x8c is actually a random value at the end of the data run. The data run is actually terminated by a single byte of 0x00 where the next attribute byte should be. The attribute byte is actually 2 nibbles, one for length of length, and other length of offset .

I have sent you a PM that may assist

ReplyQuote
Posted : 25/03/2010 2:22 pm
westor2010
(@westor2010)
New Member

http//www.reddragonfly.org/ntfs/concepts/data_runs.html

This site may give you more help

The 0x40 bytes you mention as defined by the value in offset 0x20 of the Data record.

The byte 0x8c is actually a random value at the end of the data run. The data run is actually terminated by a single byte of 0x00 where the next attribute byte should be. The attribute byte is actually 2 nibbles, one for length of length, and other length of offset .

I have sent you a PM that may assist

Ok, thank you very much for your help, googling about all this I couldnt find any info, you all solved my problem, thanks!!

ReplyQuote
Posted : 25/03/2010 4:47 pm
Share: