Notifications
Clear all

$MFT Resident data  

Page 1 / 2
  RSS
jaclaz
(@jaclaz)
Community Legend

A few days ago, for completely unrelated from "forensics" reasons (grub4dos related matters), I happened to delve deeper in the way small files can be "embedded" in their correspondent $MFT entry.
The thread is here
http//reboot.pro/topic/18315-easy2boot-v1-beta-install-windows-from-isos-on-a-flash-drive-beta-testers-wanted/
but maybe it is useful to report the results here in a dedicated thread.

First thing the "mouth of the wolf" roll , I was able to find two references on Microsoft sites, evidently contrasting
http//technet.microsoft.com/en-us/library/cc781134(v=ws.10).aspx

Files less than 900 bytes or so are stored with the directory entry in the MFT.

(same info reported on Wikipedia, BTW
http//en.wikipedia.org/wiki/NTFS

Small files and folders (typically, 900 bytes or smaller) are entirely contained within the file’s MFT record.

the other MS resource is - to say the least - queer 😯
http//technet.microsoft.com/en-us/library/cc976808.aspx

Small files and directories (typically 1,500 bytes or smaller) are entirely contained within the file's MFT record.

as I presume that you can store at the most 1,024 bytes in a record that is 1,024 bytes in size wink .

So I decided to make a few tests (and Steve6375 did some others, having already found and discussed the grub4dos issue).

Steve was originally convinced that the limit was 680 bytes
https://code.google.com/p/grub4dos-chenall/issues/detail?id=110&can=1&q=ntfs
which would be "compatible" with this article
http//computer-forensics.sans.org/blog/2012/10/15/resident-data-residue-in-ntfs-mft-entries#
that cites Brian Carrier as source for

less than 700 bytes if you want the file to be stored as a resident attribute

The results of the tests, made with different programs/approaches (and on XP and 7) are variable

  • using fsz (part of the DSFOK toolkit) the limit results in 720 bytes
  • using ECHO (or ECHOO.COM) the limit is 736
  • using fsutil, the limit becomes 728

Not that a few bytes represent much of an issue, but it would be nice to understand the WHY and the HOW this limit is not "fixed" and "same" for whatever approach.

Ideas for new experiments (and/or results of different ones) are very welcome.

jaclaz

Quote
Posted : 18/03/2013 10:44 pm
Patrick4n6
(@patrick4n6)
Senior Member

The resident data attribute is just another attribute that needs to fit into a fixed record size. Therefore the number and size of other attributes can effect that. Consider for example that the $File_name attribute can vary in size depending on the number of characters in the file name. It's therefore not surprising to me that the maximum size floats.

ReplyQuote
Posted : 18/03/2013 11:08 pm
jaclaz
(@jaclaz)
Community Legend

The resident data attribute is just another attribute that needs to fit into a fixed record size. Therefore the number and size of other attributes can effect that. Consider for example that the $File_name attribute can vary in size depending on the number of characters in the file name. It's therefore not surprising to me that the maximum size floats.

Yes, of course, but the three tests have been made under the same conditions, i.e.

  • same XP machine
  • same user
  • files created in the same directory
  • using the same file name

And the other set of tests were made on a different 7 machine, evidently with a different logged in user, under a different directory, on a different volume, but under the same conditions among the three tests.

It seems like the way the file is created affects the result. ?

jaclaz

ReplyQuote
Posted : 18/03/2013 11:24 pm
keydet89
(@keydet89)
Community Legend

as I presume that you can store at the most 1,024 bytes in a record that is 1,024 bytes in size wink .

I wouldn't think that would be the case at all. An MFT record contains a header, as does the attribute. You've have to extract at least that much from the amount of data stored.

I updated my own MFT parser to display the contents of resident and non-resident $DATA attributes, in part to illustrate what Hal talked about here
http//computer-forensics.sans.org/blog/2012/10/15/resident-data-residue-in-ntfs-mft-entries

I've seen data 707 bytes in length (that's the data, not including the attribute header), and even some slightly more, stored in the attribute.

I have yet to see a hard drive with 4K MFT records, so I can't speak to those.

ReplyQuote
Posted : 19/03/2013 12:04 am
jaclaz
(@jaclaz)
Community Legend

as I presume that you can store at the most 1,024 bytes in a record that is 1,024 bytes in size wink .

I wouldn't think that would be the case at all. An MFT record contains a header, as does the attribute. You've have to extract at least that much from the amount of data stored.

Sure ) , it was ironical against the 1,500 bytes stated by the good MS guys on the referenced article.

I updated my own MFT parser to display the contents of resident and non-resident $DATA attributes, in part to illustrate what Hal talked about here
http//computer-forensics.sans.org/blog/2012/10/15/resident-data-residue-in-ntfs-mft-entries

Good, I already cited that article.

I've seen data 707 bytes in length (that's the data, not including the attribute header), and even some slightly more, stored in the attribute.

Yep, and I have seen (as said) as much as 720, 728 and 736 bytes (still "pure" data), that's why I am perplexed and bringing up the matter here.

I have yet to see a hard drive with 4K MFT records, so I can't speak to those.

Same here, all the tests were made on "normal" 1,024 bytes MFT records.

jaclaz

ReplyQuote
Posted : 19/03/2013 12:12 am
joakims
(@joakims)
Active Member

I got 744 bytes into it by setting file name to 1 character. Maybe you can get a few more on an outdated Win version.


0000 46 49 4c 45 30 00 03 00 8c 12 6e 41 07 00 00 00 FILE0.....nA....
0010 07 00 01 00 38 00 01 00 00 04 00 00 00 04 00 00 ....8...........
0020 00 00 00 00 00 00 00 00 03 00 00 00 d1 0a 06 00 ................
0030 02 00 00 00 47 11 00 00 10 00 00 00 60 00 00 00 ....G.......`...
0040 00 00 00 00 00 00 00 00 48 00 00 00 18 00 00 00 ........H.......
0050 2f 1a 8c 3f 34 24 ce 01 50 3e 93 3f 34 24 ce 01 /..?4$..P>.?4$..
0060 50 3e 93 3f 34 24 ce 01 50 3e 93 3f 34 24 ce 01 P>.?4$..P>.?4$..
0070 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...............
0080 00 00 00 00 de 03 00 00 00 00 00 00 00 00 00 00 ................
0090 c0 4c 98 8d 01 00 00 00 30 00 00 00 60 00 00 00 .L......0...`...
00a0 00 00 00 00 00 00 02 00 44 00 00 00 18 00 01 00 ........D.......
00b0 0c 02 04 00 00 00 1d 00 2f 1a 8c 3f 34 24 ce 01 ......../..?4$..
00c0 50 3e 93 3f 34 24 ce 01 50 3e 93 3f 34 24 ce 01 P>.?4$..P>.?4$..
00d0 50 3e 93 3f 34 24 ce 01 00 00 00 00 00 00 00 00 P>.?4$..........
00e0 00 00 00 00 00 00 00 00 20 00 00 00 00 00 00 00 ........ .......
00f0 01 03 77 00 00 00 00 00 80 00 00 00 00 03 00 00 ..w.............
0100 00 00 18 00 00 00 01 00 e8 02 00 00 18 00 00 00 ................
0110 00 c9 20 20 20 20 20 20 20 20 20 20 20 20 20 20 ..
0120 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
0130 80 00 80 00 5f 44 45 46 41 55 4c 54 2e 42 41 54 ...._DEFAULT.BAT
0140 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 .
0150 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
0160 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
0170 20 20 20 10 00 00 20 20 20 20 20 20 20 20 20 20 ...
0180 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
0190 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
01a0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
01b0 20 20 20 20 20 00 00 00 00 00 00 00 00 00 00 00 ...........
01c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01f0 00 00 00 00 00 00 01 00 ff 19 50 00 00 07 02 00 ..........P.....
0200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0220 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0 ................
0280 20 4d 49 43 52 4f 53 4f 46 54 20 50 49 46 45 58 MICROSOFT PIFEX
0290 00 87 01 00 00 71 01 57 49 4e 44 4f 57 53 20 33 .....q.WINDOWS 3
02a0 38 36 20 33 2e 30 00 05 02 9d 01 68 00 80 02 80 86 3.0.....h....
02b0 00 64 00 32 00 00 00 00 00 00 04 00 00 00 10 02 .d.2............
02c0 00 1f 00 00 00 00 00 00 00 00 00 00 00 64 00 32 .............d.2
02d0 00 00 00 00 00 00 00 20 20 20 20 20 20 20 20 20 .......
02e0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
02f0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
0300 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
0310 20 20 20 20 20 57 49 4e 44 4f 57 53 20 32 38 36 WINDOWS 286
0320 20 33 2e 30 00 21 02 1b 02 06 00 00 00 00 00 00 3.0.!..........
0330 00 57 49 4e 44 4f 57 53 20 4e 54 20 20 33 2e 31 .WINDOWS NT 3.1
0340 00 ff ff 37 02 8c 00 00 00 00 00 00 00 00 00 00 ...7............
0350 00 00 00 25 53 79 73 74 65 6d 52 6f 6f 74 25 5c ...%SystemRoot%\
0360 53 59 53 54 45 4d 33 32 5c 43 4f 4e 46 49 47 2e SYSTEM32\CONFIG.
0370 4e 54 00 00 00 00 00 00 00 00 00 00 00 00 00 00 NT..............
0380 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0390 00 00 00 25 53 79 73 74 65 6d 52 6f 6f 74 25 5c ...%SystemRoot%\
03a0 53 59 53 54 45 4d 33 32 5c 41 55 54 4f 45 58 45 SYSTEM32\AUTOEXE
03b0 43 2e 4e 54 00 00 00 00 00 00 00 00 00 00 00 00 C.NT............
03c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
03d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
03e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
03f0 00 00 00 00 00 00 00 00 ff ff ff ff 82 79 02 00 .............y..

Edit that was not easy to ready..
Edit 2 Here's the correct file; http//www.mediafire.com/view/?o8xuqaur8563cl4

ReplyQuote
Posted : 19/03/2013 4:12 am
randomaccess
(@randomaccess)
Active Member

the mft record size is set in the boot sector, but is typically 1024.

An MFT record will always have a header and standard information and filename attribute.
The other attributes are optional.

From parsing mft records manually recently (CFCE), it seems like there is empty space between the attributes. Sometimes its a few bytes, sometimes its more. The more wasted space between attributes the less resident data can be stored.

If you look at the files and compare them side by side, assuming everything has actually remained the same, my guess would be the amount of space between the end of one attribute and the start of the next.

But without examining the files I wouldnt be able to say.

ReplyQuote
Posted : 19/03/2013 6:06 am
Patrick4n6
(@patrick4n6)
Senior Member

From parsing mft records manually recently (CFCE), it seems like there is empty space between the attributes.

Attribute sizes are always a multiple of 8. This works great for the SI attribute, not so much for filename and resi data.

ReplyQuote
Posted : 19/03/2013 6:54 am
joakims
(@joakims)
Active Member

Here is the correct MFT record file; http//www.mediafire.com/view/?o8xuqaur8563cl4

Sorry for the previous one.

Happy to see someone beating that, given 1024 byte sized records )

ReplyQuote
Posted : 19/03/2013 9:30 am
jaclaz
(@jaclaz)
Community Legend

Happy to see someone beating that, given 1024 byte sized records )

Joakims, besides posting the file, can you explain how you are creating/writing it?
What do you mean 1 character name?
Without extension?
The tests I made were making a file named "1.txt".
What do you mean "outdated" Windows versions?

jaclaz

ReplyQuote
Posted : 19/03/2013 11:53 am
joakims
(@joakims)
Active Member

Creating the file was just a matter of making a copy of a file I knew was resident, and named it "w", with no extension. I knew the original file was 707 bytes, so it was already close to limit. Next I ran my own mftrcrd tool on the file to see visually what it looked like in the MFT record. I ran it like this


mftrcrd_x64 F\w -a attriblist=off indxdump=off

I had the file open in a hex editor and added bytes, then re-ran the tool, and repeated until it turned non-resident.

Point about the $FILE_NAME attribute is that the filename length will impact on the attributes size. The $STANDARD_INFORMATION attributes is as far as I know fixed in size. The record header is also fixed, however was different in earlier implementation (like in 2k I think), in that it did not have file references as is used now (8 bytes). I am not able to reproduce that here, but if you can, then you should be able to increase the max size of resident data by at least 8 bytes (maybe more, not sure).

ReplyQuote
Posted : 20/03/2013 1:15 am
jaclaz
(@jaclaz)
Community Legend

….
I had the file open in a hex editor and added bytes, then re-ran the tool, and repeated until it turned non-resident.

Point about the $FILE_NAME attribute is that the filename length will impact on the attributes size. The $STANDARD_INFORMATION attributes is as far as I know fixed in size. The record header is also fixed, however was different in earlier implementation (like in 2k I think), in that it did not have file references as is used now (8 bytes). I am not able to reproduce that here, but if you can, then you should be able to increase the max size of resident data by at least 8 bytes (maybe more, not sure).

Very good, thanks. D

But this does not answer the original question, which I will try to re-word.

I used three (actually 4) different methods to create a file with the SAME name (which is/was 1.txt) under the SAME account and in the SAME directory, with increasing sizes, until it "exited" the $MFT and it occupied a cluster.

In a "perfect world" the size that should trigger the "migration" of the file from $MFT to "cluster" would be the SAME, then if you change the name, create it under a different folder (permissions) or under a different user (ownership) or on a different system and what not, this "fixed no matter the tool used" limit may change (as an example resulting a bit higher for a shorter filename and a little bit lower for a longer filename).

I will do more tests, in a fully reproducible way, adding a few other "writing approaches" and will check if examining the results with your mftcrd tool I can pinpoint - if not the reason why - at least the HOW/WHERE the diffference comes out.

jaclaz

ReplyQuote
Posted : 20/03/2013 1:59 pm
randomaccess
(@randomaccess)
Active Member

can you post up the files when they are at the maximum data attribute length for each process?

(I dont know why that sentence was so difficult to phrase)

ReplyQuote
Posted : 20/03/2013 3:13 pm
jaclaz
(@jaclaz)
Community Legend

can you post up the files when they are at the maximum data attribute length for each process?

(I dont know why that sentence was so difficult to phrase)

OK, started again from scratch. )

Here
http//www2.zshares.net/vac5zybch4of

is a small (around 3 Mb) "floppy-like" image (suitable for being mounted in IMDISK or similar) with a set of pre-made files AND the tools and batches used to make them.
The size was chosen as it is (roughly) the smallest size you can make a NTFS filesystem in windows with a 4096 bytes cluster (i.e. same size as most "common" NTFS filesystems on disk) with the built-in in XP format command.

There is (as seen) a connection between filename length and "available space", but still fsz and fsutil *somehow* manage to make smaller sized files.
744 is confirmed to be the most you can put in a file which filename is within these lengths/form

1
12
1.1
123

The algorithm (valid for "normal" file generation through dsfo or ECHO/ECHOO and similar) is simple enough (by design this applies only up to 8.3 filenames), in batch

SET Max_size=744
SET /A Sum=%Name_length%+%Ext_length%-1
IF %Sum% geq 3 Set Max_size=736
IF %Sum% geq 7 Set Max_size=728

fsz and fsutil "steal" 😯 8 bytes from the above.

jaclaz

ReplyQuote
Posted : 22/03/2013 10:00 pm
jaclaz
(@jaclaz)
Community Legend

Re-uploaded file here
http//www.filedropper.com/ntfs3mb

(just in case someone wants to try and reproduce)

jaclaz

ReplyQuote
Posted : 17/06/2016 3:55 pm
Page 1 / 2
Share: