If you want to be technical, the formula for calculating RAM slack is
FileSize MOD 512
File slack is
FileSize MOD ClusterSize
For those who didn't study computer science, MOD is http//
[*]"file slack" <- unindexed space between file size and allocated space, two types
- "sector file slack" <- unindexed space between file size and next sector border (Maxsize=SectorSize-1, i.e. on a 512 byte/sector device at the most 511 bytes)
- "cluster file slack" <- unindexed space between file size and next cluster border
(Maxsize=ClusterSize-1, i.e. on a typical 4096 byte/cluster filesystem at the most 4095 bytes)
"cluster file slack" is comprehensive of "sector file slack" .[/listo]
It's not at clear what purpose the definition of 'sector file slack' serves – unused space is unused space, no matter where it appears. And 'sector file slack' is an 'interesting' definition only for file system implementations where that particular space may contain interesting data.
Some versions of NTFS (XP and up) has a special type of file slack inside files, always at the end, where file contents is specified to be zero, yet there are allocated clusters (which will not be read for data). The system call SetValidData() describes this particular functionality. It's a kind of strange 'sparse file'. It doesn't fit the usual definition of 'slack' (as data in 'gaps'), but as the effect of it is much the same as that of ordinary slack, it may deserve a nod of recognition.
If you want to be technical, the formula for calculating RAM slack is
FileSize MOD 512
NO.
It is
FileSize MOD SectorSize
We do have most devices having 512 bytes/sector, but this is not *always* the case and it will be increasingly less so in the future.
It's not at clear what purpose the definition of 'sector file slack' serves – unused space is unused space, no matter where it appears. And 'sector file slack' is an 'interesting' definition only for file system implementations where that particular space may contain interesting data.
The idea (at least mine) was to use the question by yunus as a start point to "consolidate" all types of "unindexed" areas (and possibly the reason why this happens), putting together all the experiences/suggestions as to provide a "common definition".
Some versions of NTFS (XP and up) has a special type of file slack inside files, always at the end, where file contents is specified to be zero, yet there are allocated clusters (which will not be read for data). The system call SetValidData() describes this particular functionality. It's a kind of strange 'sparse file'. It doesn't fit the usual definition of 'slack' (as data in 'gaps'), but as the effect of it is much the same as that of ordinary slack, it may deserve a nod of recognition.
Can you expand on this?
Is it not a "plain" sparse file?
From what you write it sounds more like some areas "inside" the files, if you prefer a "very soft" form of steganography…. ?
jaclaz
Can you expand on this? Is it not a "plain" sparse file?
I'm sure you know it already – it made a bit of a stir when it was introduced in XP.
Sparse files in Windows are created only by DeviceIoControl() calls. But this thing is done by SetValidData(), which would be called (roughly) as
SetValidData( ..., GetFileSize() + 65536 or whatever);
which adds 65536 (or whatever) bytes at the end of the file.
This allocates enough clusters to hold the extra space at the end of the file. To make this fast, NTFS never initializes those clusters, but remembers the previous file size (sometimes called 'logical file size', but that's a very misleading term). Anyway, any read from the area between that byte offset and the new end of file will always return zeroed bytes, regardless of what the allocated clusters actually contain. They don't get new data until something is actually written to this area.
It's a kind of special file slack, in that there are clusters at the end of the file, the actual contents has not been overwritten, but which cannot be read by the user process. And it's kind of like sparse files in that if the user process tries to read these clusters, it only gets zeroed bytes back.
It's not at clear what purpose the definition of 'sector file slack' serves – unused space is unused space, no matter where it appears. And 'sector file slack' is an 'interesting' definition only for file system implementations where that particular space may contain interesting data.
It's not "sector file slack", it's RAM slack, and the importance is knowing how the data past EOF got there. For most file slack, it's remnants from the last file in that cluster. For RAM slack, it's potentially data from RAM and has no relevance to the last file that resided on the disk. We are not just in the business of WHAT, but also of WHY and HOW (plus WHERE, WHEN and WHO). A large part of this industry is also about knowing the exceptions to the rules as they sometimes make or break.
A large part of this industry is also about knowing the exceptions to the rules as they sometimes make or break.
Unfortunately, that is true. The reason for that is, as far as I can find, poor rules, bad definitions, and retaining outdated concepts for too long – that's when knowing exceptions becomes a 'large part'. With more considered definitions, it can be made smaller. Which I believe would desirable, as otherwise knowing all these exceptions will become an area of specialization.
So it becomes important to get definitions and terminology as straight and unambiguous as possible, and ensure that any rules are created with a view of their intended purpose.
To my mind, RAM slack does not belong in this list. It's not defined by an on-disk file system structure, but by the source of the content of such a structure. If the contents comes from RAM, it's 'RAM slack', if it doesn't, the term is best not used, as it is liable to be misunderstood.
In that context, and given that RAM slack still is a concept that needs to be understood, I would accept 'sector file slack' as the on-disk structure on which the idea of 'RAM slack' can be built, provided that considerable emphasis was placed on the fact that it is not something that is particularly common today, and that you would have go back to … Win9x? early NT? … to find it, at least in the Windows world.
The presence of 'sector file slack' areas with a content that is not zeroed bytes would therefore be an anomaly, which just might be worth investigating. But that's the next step when the 'slack' structures are defined, their contents and the source of it can be discussed.
Sparse files in Windows are created only by DeviceIoControl() calls. But this thing is done by SetValidData(), which would be called (roughly) as
Call me tough as much as you want, but I completely fail to understand.
In which practical instance such "semi-sparse" file is created?
Through which command? (or procedure)
Or someone needs a "special program" to invoke the SetValidData() function?
@Patrick4n6
About "sector file slack", that is the name of an unindexed area, defined tentatively as
"sector file slack" <- unindexed space between file size and next sector border (Maxsize=SectorSize-1, i.e. on a 512 byte/sector device at the most 511 bytes)
wheteher this area contains (on "lazy" systems, whatever they are) part of the contents of the RAM is another thing.
RAM slack may be the name of the contents of the area, not of the area itself.
You normally say a beer bottle, but it is a glass bottle containing beer, as soon as you drink the contents wink and re-fill it with water, it will become a bottle of water, but the bottle itself has not changed name and it is still a glass bottle.
jaclaz
In which practical instance such "semi-sparse" file is created?
They're created when a program calls SetValidData() for an opened file. Exchange creates them for some files, and I'm fairly certain the old event log files (.evt) also could be of this type.
Exchange creates them for some files, and I'm fairly certain the old event log files (.evt) also could be of this type.
I have found this
Feature is seemingly called "File Initialization"
http//
fsutil can use setvaliddata allright !
Would this be "file initialization slack" or "file allocation slack" (to be coherent with "file sector slack" and "file cluster slack" ) ? ?
Uninitialized space is similar in concept to file slack except that it is contained within the logical file size. Unlike file slack which is no longer associated with a file, data in uninitialized space is in a kind of limbo, trapped at the end of an allocated file but not actually part of that file.
jaclaz
I have found this
Feature is seemingly called "File Initialization"
http//blog.cmdlabs.com/2010/03/17/the-pitfalls-of-file-initialization-for-forensic-analysts/
That may be what that writer calls it, but I find it misleading as the whole thing is about *avoiding* having to do any sector/cluster initialization until it is absolutely necessary.
Would this be "file initialization slack" or "file allocation slack" (to be coherent with "file sector slack" and "file cluster slack" ) ?
I would prefer to keep to Microsoft's terminology, and use something about 'valid data' – 'file valid data slack' seems better, though it's a bit of a mouthful. Anyway, using those words makes it possible to search in the MSDN or Microsoft TechNet documentation, and actually get relevant hits fairly high up in the search hit list. It's still a bit confusing so I would wish for a better term. Though with an appropriate definition of 'file slack' that confusion may be minimized. It should be strongly connected to NTFS as well at least until it has been observed in other file systems.
The thing to recognize however, is that it's not something new it's still 'file slack'. The only difference is that it isn't the file length (end of file offset) that decides where it starts, but the 'file valid data length' offset.
Uninitialized space is similar in concept to file slack except that it is contained within the logical file size. Unlike file slack which is no longer associated with a file, data in uninitialized space is in a kind of limbo, trapped at the end of an allocated file but not actually part of that file.
That seems confusing.
File slack, while not part of a file, is still associated with a file, as all file slack is part of the file extent the actual clusters used for the file data. (That 'no longer associated with a file' doesn't match anything very well that I can think of.)
This 'file valid data slack' is not different in any way it's not part of the file, and it's still part of the extent of the file.
(I'm trying to avoid the terms 'physical file size' and 'logical file size', as those are fairly specific to Windows. The abstract concept of a file has only one the file size.
And perhaps I need to say that I personally define slack as the difference between the extent of the file and the contents of the file. If it's part of the file extent, but not part of the file, it's slack, no matter where it appears.)