I considered an application that allows the user to defined the header and footer of a test 'target file'. Then allow them to define the size of the file. Here I am thinking that rather than having to recreate say a picture. Just recreate the header and footer, choose the size and randomise its content with any binary data. Then hash and mark the physical location of the 'test picture'. Repeat the process for as many pictures are the user wants and with a defined structure - the user can choose back to back files or a defined gap or randomised. Then, once generated, automated map the position of each fake picture, size and physical location.
Then run the carver, which will essentially extract based on header and footer, hash the files and acquire the physical location and match the result.
This would essential create fake test content, but with the key elemnts of header and footers and hash to establish that it has been extracted properly. It also takes the overheads of having to source and develop test data which consists of real files. Essentially it would allow a user to test carve for anything they like, creating test data in second.
….I hope ive explained that clearly enough, cant tell if im doubting my own approach now.
I considered an application that allows the user to defined the header and footer of a test 'target file'.
That sounds as if it would be useful for a particular branch of carvers that pay attention only to that kind of files. It may be the most interesting subcase for all I know – I don't rely much on carved data myself as I don't trust carving tools very far – but it might cause carvers that take deep content into consideration (if there are any) to get confused.
Notionally, it seems a carver should do something like 'here's a list of possible carves for the file that starts in sector X. Let's measure how well they follow the basic file structure', and chose the file with the least number of non-legitimate format errors. Or at least the files that have less than Y such errors. And perhaps do that for a major set of carves, minimizing problems not over a single file, but over the entire carve job. Well, something on those lines, assuming a fairly structured file format. (Yes, I'm aware I'm not being very practical here.)
Then run the carver, which will essentially extract based on header and footer, hash the files and acquire the physical location and match the result.
OK, you have a different view of how the carver works. No problem.
It also takes the overheads of having to source and develop test data which consists of real files. Essentially it would allow a user to test carve for anything they like, creating test data in second.
So the next question is fairly obviously what does the user want to do? A test should preferably convince someone of something. If you can state what that would be, I think the most important problem would be be solved. That the carver works as specified? As expected? Or perhaps, carves at least 90% of the files (conforming to the model you have) from a disk image? (That last requires some kind of automation 'add 500 files of types A, B, C' to image, like.)
Me, I'd probably like to see what happens when I
have two or more files b**t-to-head to test if the carver is over-greedy or not.
distribute all sectors of a real file (or some subset) all over an image to simulate and extremely fragmented file. Other cases are, of course, a file split in the middle, with some degree of separation between the two parts for a less fragmented file
Those could be done with real file data, or with synthesized data. Various variations are fairly simple to imagine. One interesting would be to allow the footer to be split over a cluster border in different byte positions, to see if that is a fail-case for the carver.
But I'd also like to
add fragments of NTFS MFT with wholly (or partially?) resident files to see that they would be carved, even if they don't start on a cluster boundary.
have real files with 'ambiguous' contents say contents that looks like a footer, but can't be as the file header clearly says that more data must be present.
but those probably can't be tested with synthesized content.
But it's pretty clear now that I don't want to validate that a tool works correctly – I want to test what works and what doesn't work. So I may not be the intended end-user of this tool.
I agree in terms of testing, I think all these things need testing - but this level of depth is something I doubt can be automated.
In addition, do the mainstream carvers - EnCase, FTK, Xways claim to be able to do this, or do they offer just basic functionality? I suppose the key is to make sure that you evaluate a carver against what it claims to do.
I do agree with you in terms of what needs to be achieved - i think its a case of finding out what it doesn't do, not just verifying that it can do something.
From what I have seen so far and the validation, which passed UKAS from both LE and private sector, isn't worth the paper it's written on.
For those of us who are geographically and mobilely disadvantaged, could you clarify what document you're referring to?
Unfortunately organisations are not required to provide their method validations to outside agencies as standard. I've had copies of a few for various legitimate reasons (review of tender contracts, sharing by agencies) however cannot re-post these or provide links to them.
I agree in terms of testing, I think all these things need testing - but this level of depth is something I doubt can be automated.
I honestly believe the answer is not in tool testing by user knowledge. If the user's are knowledgeable enough in their field and understand the requirements to check results etc it would ensure that erroneous/incorrect data is prevented from going into reports far more than a limited, worthless test.
I agree in part, but is the greater risk not erroneous interpretation (where you would hope knowledge and experience might pick something out as being a bit odd), but the potential to actually never come across it in the first instance. If something is regularly missing large volumes of data in a 1TB drive, I doubt anyone is ever going to pick that up manually. So I guess in that sense, testing may have more of a focus on identification, and a secondary emphasis on interpretation…
I agree in part, but is the greater risk not erroneous interpretation (where you would hope knowledge and experience might pick something out as being a bit odd), but the potential to actually never come across it in the first instance. If something is regularly missing large volumes of data in a 1TB drive, I doubt anyone is ever going to pick that up manually. So I guess in that sense, testing may have more of a focus on identification, and a secondary emphasis on interpretation…
The conditions under which this would probably occur would never be picked up by testing either. Tools which completely fail to carve images of documents are likely to be identified very quickly by simple dual tooling/tool comparisons.
Its more likely that items missed would be a specific set of circumstances which are slightly unusual, a picture file which has a slightly different or compression/encryption at file system level prevented carving, and to test for this would be at best massively resource intensive and at worst a fools errand.
Furthermore your suggestion of automating test data would never likely be feasible to test these eventualities.
I have a feeling judging from the way this has gone that the following may well be accurate=
1. We cannot effectively test tools in DF
2. We are reliant on vendors to produce accurate tools - however even they may not be able to fully test them.
3. We have to accept this.
Interesting. But massively controversial. Surely we are the only branch of forensics where it is acceptable to use tools which have not been effectively calibrated. 😯 😯
Evening all.
I agree that it is definitely a troublesome area at the moment, especially with the deadline looming! There seems to be a common misconception as to what areas need to be validated. From my understanding of the standards and documentation which I have read, it will not be the tool itself which you will have to get validation for, it is the process and procedure which you follow whilst using the tool. This is rather than having to validate whether the tool works or not, as this is generally out of our hands, although we are required to be aware of any changes which may occur from using that tool….
3. We have to accept this.
Interesting. But massively controversial. Surely we are the only branch of forensics where it is acceptable to use tools which have not been effectively calibrated. 😯 😯
Almost, we know that we cannot validate the tools for every type of eventuality and so do not trust the results absolutely and use our expert knowledge and experience to determine if they are correct. As we use digital image files for most of our work, what we have done (outside of imaging) can always be replicated by someone else for verification purposes. In other fields the sample is destroyed when examined and therefore cannot be verified in the same way.
We are also the only branch of "forensics" that deals with investigations. I put "Forensics" as I do not consider the field to be "Digital Forensics" but rather Digital Investigations.
Forensic accountants are more in line with what we do than fingerprints or DNA labs. Also they don't have any calibrated tools P
Also a lot of the time it can be argued that not using an older 'validated' version of a tool is a greater risk of missing evidence than using a newer 'non-validated' version. Given the rate of change in the Digital Investigations field and the time it takes to validate a tool, we would always be behind and potentially missing vital evidence waiting for tools to be validated.