±Forensic Focus Partners

Become an advertising partner

±Your Account


Username
Password

Forgotten password/username?

Site Members:

New Today: 2 Overall: 34815
New Yesterday: 8 Visitors: 170

±Follow Forensic Focus

Forensic Focus Facebook PageForensic Focus on TwitterForensic Focus LinkedIn GroupForensic Focus YouTube Channel

RSS feeds: News Forums Articles

±Latest Articles

±Latest Webinars

Volume slack vs file system slack; confusing definition

Computer forensics discussion. Please ensure that your post is not better suited to one of the forums below (if it is, please post it there instead!)
Reply to topicReply to topic Printer Friendly Page
Forum FAQSearchView unanswered posts
Go to page Previous  1, 2, 3, 4, 5, 6, 7  Next 
  

Re: Volume slack vs file system slack; confusing definition

Post Posted: Mon Jul 16, 2012 9:25 pm

Adding "sparse file" concept just confuses the definitions.

In general sparse files are OS/application specific "compression", nothing more. From the perspective of a drive, a file, be it logical sparse file or physical sparse file, still remains a file.

I have seen two types of sparse files. One, where the OS/application interprets a "empty" section of a large data set and compressed it on the disk. Second, where a an application reserves a large amount of disk space by creating a large file with empty content, to be used in future.

In either case, from the drive perspective, file slack may exist. In the first scenario there is no virtual "slack", as the empty space only exists as it is referenced from the OS/application. In the second scenario, there is "internal slack space" that may or may not have been used, but currently not in use.  

jhup
Senior Member
 
 
  

Re: Volume slack vs file system slack; confusing definition

Post Posted: Tue Jul 17, 2012 8:06 am

- jhup
Adding "sparse file" concept just confuses the definitions.

I am not sure to understand you.
If I got this right, a "sparse" file is different from the above.
A "sparse" file has TWO different "sizes":
  1. the declared one (Bigger and fixed)
  2. the actually used on disk one (smaller and growing up to the above size) corresponding to increases of content
Apart from a minimal initial "overhead", the actual used disk space "grows" when the content is added or, if you prefer, no real disk space is "claimed" until actual contents are added/increase.

The "set valid data" is different, it still has TWO different sizes, but they are actually THREE of which two are the same:
  1. the declared one (fixed)
  2. the actually used on disk one (same as above and fixed)
  3. the actual size of content (either growing or shrinking)

I.e., when you create an empty "sparse" file only the very minimal disk space is "claimed/used" and this amount changes as soon as you add content, with an empty "set valid data" file, you directly claim a given space on disk (even if you write to it NO content) and this space may contain residual "deleted data" or "freed clusters previously occupied by *something* of use".

jaclaz
_________________
- In theory there is no difference between theory and practice, but in practice there is. - 

jaclaz
Senior Member
 
 
  

Re: Volume slack vs file system slack; confusing definition

Post Posted: Tue Jul 17, 2012 2:12 pm

- jaclaz

"valid data file slack" <- unindexed space between actual file size and declared file size (only on NTFS filesystems where the SetValidData() function or "fsutil file setvaliddata" has been used to allocate to the file a size exceeding the size of it's contents)


I'm afraid I don't find the definition enlightening -- it doesn't explain anything, particularly not to someone who doesn't already know what the definition is about. It also relies on definitions of 'actual file size' and 'declared file size' which needs to be provided somewhere.

That said, I do appreciate the difficulty in creating a definition. Myself, I typicall explain it from the other end: I walk the person (who has to be a programmer!) through the variuous steps of extending a file by adding zeroed bytes at the end, through the recommended coding pattern that was found to have unforeseen consequences for NTFS, and end up with the solution that Microsoft came up with to solve that particular problem. And with a reasonably experienced Windows programmer, I may discuss alternative ways of solving it -- like using sparse files.

My own attempt at a definition:

valid data file slack -- file slack created by NTFS when a file is extended by a number of zero bytes through the use of the system call SetValidData(). While a suitable number of clusters are allocated and added to the file to support the extra bytes, the previous cluster contents is not wiped. Instead NTFS uses special code to ensures that any attempt to read from the additional bytes result in a suitable number of programmatically zeroed bytes, instead of actual cluster content. Not until the program writes to one (or more) of these clusters will they be initialized and from there on be treated as 'normal' clusters.

But I expect that's just as difficult to understand. Trying to get it into a single sentence is probably doomed to failure.

By the way, what is 'unindexed space'?  

athulin
Senior Member
 
 
  

Re: Volume slack vs file system slack; confusing definition

Post Posted: Tue Jul 17, 2012 2:25 pm

We are in agreement in both scenarios, jaclaz.

Two types.

Let's presume OS/Application uses memory for declared data.
Let's also presume D represents data, 0 represents zeroed out data, and X just general disk space.
Also, let's agree that { and } represents beginning and ending of files, and | represents "clusters" (or whatever the chunking is for the storage media).
Finally, the underlined characters are slack space as I understand it in each scenario.

Type 1:
Memory content:
DDDDD00000DDDDDDDDDDD
Disk content allocated to the "file" (declared size to OS):
|{DDDDD|DDDDD|DDDDD|D}XXXX|XXXXX

Type 2:
Memory content:
DDDDDDDDDDDDDDDD
Disk content allocated to the "file" (declared size to OS):
|{DDDDD|00000|DDDDD|DDDDD|DXXXXX|}XXXXXX

In type 1, it is simply a "compression". This is what most people understand under sparse files.

In type 2, the OS/application reserves disk space in essence for its own use. We used this in large scale distributed databases to prevent running out of storage before the systems can fully synchronize. In some implementation the allocate space is wiped, and in some other, the OS is told "this is part of the file", yet nothing is really written to it - until needed.

In either case, the OS or file system does not know that actual data content size.

In type 1, the slack is outside of the file.
In type 2, the slack is inside of the file. Note that type 2 may still have an outside slack (but in my experience this is not the case, as apps tend to take advantage of the storage segmentation).  

jhup
Senior Member
 
 
  

Re: Volume slack vs file system slack; confusing definition

Post Posted: Tue Jul 17, 2012 2:43 pm

- athulin

valid data file slack -- file slack created by NTFS when a file is extended by a number of zero bytes through the use of the system call SetValidData(). While a suitable number of clusters are allocated and added to the file to support the extra bytes, the previous cluster contents is not wiped. Instead NTFS uses special code to ensures that any attempt to read from the additional bytes result in a suitable number of programmatically zeroed bytes, instead of actual cluster content. Not until the program writes to one (or more) of these clusters will they be initialized and from there on be treated as 'normal' clusters.


File slack is created by many other file systems, not just NTFS.
What is "SetValidData()"? In what programming language? The only system calls I know of for disk are INT 13h calls . . .

Mr. Green

My point - i think you overcomplicated it.  

jhup
Senior Member
 
 
  

Re: Volume slack vs file system slack; confusing definition

Post Posted: Wed Jul 18, 2012 4:52 am

- jhup
File slack is created by many other file systems, not just NTFS.


At present, the only known source of this particular type of slack is NTFS. Strictly speaking, it has only been verified for Windows; status on non-Microsoft implementations of NTFS is still unknown.

What is "SetValidData()"?


A Microsoft Windows system call -- though 'system call' should not necessary be interpreted to mean 'kernel call' or 'BIOS call'. It's implemented by 'kernel32.dll' -- that's why I call it a system call. You find the details in the usual MS places -- try googling for 'MSDN SetValidData' for one possibility.

I know that EnCase 6 supports this kind of file system structure, though it's not classed as file slack. I'm less sure what EnCase 7 does, though it probably does the same, and I have no idea how other forensic suites or toolkits deal with it.

My point - i think you overcomplicated it.


It may not be a type of file slack that is the most important thing to get right in a list of definitions of slack in general, true. But it does belong on a list of types of file slack.  

athulin
Senior Member
 
 
  

Re: Volume slack vs file system slack; confusing definition

Post Posted: Wed Jul 18, 2012 5:13 am

- jhup

Type 1: [...]
Type 2: [...]


Just to make sure, SetValidData() creates a structure that could be called Type 3:

Memory content: (added spaces for legibility)
DDDD DDDD DDDD DDD0 000
Disk content allocated to the "file" (declared size to OS):
|{DDDD|DDDD|DDDD|DDDX|XXX}X|  

athulin
Senior Member
 
 

Page 4 of 7
Go to page Previous  1, 2, 3, 4, 5, 6, 7  Next