I mentioned my concern earlier about operators having blind faith in a tool (it was nothing personal, but there are posts most weeks that concern me).
If tools are verified (something I think we have seen is largely impossible) there may be more blind faith with the results.
If everyone knows that results are made with best endeavour, rather than perfect, the culture of double checking may become more common.
I think the way forward should be more based on procedures, rather tool validation. ie can we see the same result from difference approaches.
Doing (domestic) data recovery I will often run a straight forward recovery process, but then also an analytical data carve which will display the areas of disk with data, type of data and a simple report of number of files and file types found. The results should be similar, and significant differences investigated.
Concentrate on the operator - but also make sure the tools do are they say.
Mscotgrove - Understood.
jaclaz
If both tools validate across the same data sets, what you choose at that moment would then be a matter of preference. Of the two tools you described I would want the first. I would assume most would. However, we must assume that no one knows of the possible issue you describe (as you noted) until it is in fact discovered.
Until then, the validation for either tool described would stand from both a practical and technical perspective. The value of both validations would be equally beneficial until such time as the unknown error causing issue becomes known. The ramifications of the discovery should then prompt remediation and call for better programming practices. So then focus would/should be on the first tool because as you previously described, it was designed to fail/alert noticeably, which is a better design.
Bottom line, without additional information (which is usually absent), all one can do is work/test with either/both tools until the discovery is made. To prevent the scenario from even occurring is unlikely to impossible considering there is no way to be aware of, or account for, every scenario or every action that can be taken against the data by something else.
It could then be said then why validate at all? I’m sure a counter would be that you can’t do anything about what you don’t know, but you can do something about what you do and shouldn’t one be obligated to do so? (From a criminal justice, or science-based perspective)
Perhaps validation isn’t necessary and the focus should be on verification/confirmation of results primarily?
Perhaps validation isn’t necessary and the focus should be on verification/confirmation of results primarily?
Exactly. )
It is not like validation is not necessary, the point is more around the relevance or "weight" that should be given to validation.
It seems like a lot of people think that once a tool is validated (for a single, specific task/operation BTW) it becomes infallible in all operations and on all data sets.
Surely a firm or programmer or Author with a record of reliability can be trusted more than a newcomer, but very little more than that.
And of course one should be aware of the risk of falling in traps like rating "accuracy" in percentage.
When you have a tool that has a rated accuracy of 98% you should be very careful, the (re-known) probability paradox is the following.
There is a "simplified" test that is performed on all the population to early diagnose cancer.
Tests are performed on groups of 10,000 people at the time.
The known statistical average is that cancer affects 0.5% (one person every 200) of population.
You go through the test and a few days later your doctor tells you in a sad voice that your test resulted positive.
Should you begin to really worry or you have good reasons to be only mildly preoccupied until you can do the "full test"?
Think a bit about it, you have this tool that is 98 percent of the times accurate, how high is the probability that in your case it is a false positive?
Click below for the whole story
http//
jaclaz
It is not like validation is not necessary, the point is more around the relevance or "weight" that should be given to validation.
It seems like a lot of people think that once a tool is validated (for a single, specific task/operation BTW) it becomes infallible in all operations and on all data sets.
This is exactly my position. The key question for pro-validation becomes how much is enough? And in terms of data sets, you can use known sets to test but in the wild all your examinations are on different data sets so there is no operational reality of standard input (theoretical maybe).
That is why I wanted to evaluate and understand the notion described by my OP question. I try not to make a habit of jumping to conclusions unless I actually grasp a concept so I can agree/disagree on its merits.
So the category portion of my OP becomes most pertinent. What separates "forensic" tools from "other" tools (regardless of what you call them)? I know of know definitive definition so I am looking for perspectives to evaluate (my OP).
What separates "forensic" tools from "other" tools (regardless of what you call them)?
Nothing, I don't think. The insistence on calibration most probably come from the forensic *sciences* you can't do science unless you have errors and error propagation under control. (Some observational sciences even extended this to the observers involved – in astronomy you had to understand your 'personal equation' the systematic errors produced by personal differences.)
If there is a difference, it has to do with computer forensics. A pair of scales might last 20 years, but need constant zero-point calibration depending on temperature and air moisture and barometric pressure to keep errors to a minimum. But it's not quite as obvious where sources of errors are in a tool for examining NTFS file systems. Until those sources are known, however, calibration seems a moot point you can't really reduce errors unless you know what the errors are and their source. So … there has to be some kind of study of error sources in software – as well as in the operators of such software. One source is new releases of software both forensic software as well as the artifact-producing software. However, there's little if any calibration here – it's mainly validation. Even so, a tool may produce consistent errors. They should be known.
Other considerations are damage and cost. If a validation is too costly to make each time an observation is performed, some other approach is required and if errors don't change, a 'validate once and use many times' may work until such a time when a new release may introduce new errors.
Time stamp translation. for example, is probably a fairly typical validation object. If there's a problem it's agreeing just what should tested (dates from the early 1600s?, late 30212?, sub-microsecond resolution? GMT time only, or include local time adjustments? That can be quite hairy.)
And the cost of trusting a mistranslated timestamp can be quite high. (This extends also to other sources of error – trusting a time stamp from a bad time source can have devastating consequences.)
While testing can't show the absence of bugs, testing at least provides a basis for statistical statements about tested samples, and thus about confidence And that's where we rejoin the world of science again. Hopefully.
athulin
Yeah, I've been apart of discussions regarding "calibration" in my time at an ASCLD lab whose director also expressed the same sentiment. I also agree. For all of the criticisms some want to attribute to Digital Forensics not being "scientific," they do all they can to put DF into a context that isn't appropriate and then complain about what it isn't. I believe DF is science-based even if it didn't "develop" that way. It has grown as a discipline of practice as opposed to theory. Many other STEM breakthroughs weren't "science" at first either. They just progressed and took form quickly and thus had basis for future breakthroughs.
I think validation is the most appropriate mitigator for the moment. The key is to identify how much is too much and how can it be assessed in an appropriate fashion? There has been some work in appropriate error recognition such as
While testing can't show the absence of bugs, testing at least provides a basis for statistical statements about tested samples, and thus about confidence And that's where we rejoin the world of science again. Hopefully.
No doubt, but DF will need more than that overall. As was touched on earlier in the thread, it sounds good to make statistical statements about "tested samples" but they only exist in those controlled environments where the testing was done and once you get into the wild its all uncontrollable and the statistical statements would actually have little to no weight "scientifically" or even be misleading. The other sciences benefit from natural laws that in theory do not change. There is no such thing for DF. The underlying basis for DF is the very thing researchers like Fred Cohen discuss
I agree that there is no general separation. Perhaps focusing on the functionality would at least lend itself to an objective separation that wouldn't be arbitrary. I'll need to research further.
once you get into the wild its all uncontrollable and the statistical statements would actually have little to no weight "scientifically" or even be misleading.
You want validation, you validate. But you don't validate on 'wild' data. You validate using carefully created data sets, following a designed test method, or the validation has no value.
However, there's an almost total lack of useable data sets in DF. So the inability to validate is understandable you have to create your own validation data. And that's not something most people associate with DF work. It's closer to security testing or software testing. Lack of validation data or effort comes from an immature field of work one that leaves things to trust.
The other sciences benefit from natural laws that in theory do not change. There is no such thing for DF.
Not sure there such thing as a natural law either. All there seems to be is statistical maxima, or insufficiently analyzed data. And you have those in DF, just as everywhere else. There's just more of them.
File system artifacts are very close to 'natural laws'. They may need a verification now and then, especially if they are undocumented, but well-documented artifacts, like NTFS file stamps, have been the same since the first release of NT, for example.
I've dabbled a little in validating translation of NTFS file time stamps. It seems like one of the most fundamental validations that are needed. And that is a field where cross-testing doesn't always work different tools tend to make the same fundamental errors. The best tools – of those I tested – were actually non-forensic. I hope things have changed since.
I don't know of anything more extensive. Elizabeth Zwicky created a Unix file system test, intended to validate Unix backup tools and archivers. The latest iteration I know was published at LISA '03 as "Further Torture More Testing of Backup and Archive Programs", and the source code seems to be on the net still (Perl). Handing a TAR or ZIP archive of that file system over to a forensic toolkit for analysis is usually very enjoyable. That could perhaps be the basis of additional validation work. Anyway, the article illustrates some of the processes behind the design.
I would also like to see a 'standard' data set (or two) for file carving. Properly designed and implemented that could help get an estimate of how good file carvers really are, on different data formats.
I would add to athulin's comments ) another aspect.
Fabricated data are - guess what - fabricated wink .
The whoever is going to prepare the dataset is likely to either (if he/she is a practitioner) affect the contents with some involuntarily bias (just an example I expect someone dealing every day with - say - corporate emails and security videos to be somewhat less proficient with - still say - WhatsApp deleted messages and the dataset is likely to be more representative of real cases of the first kind than of the second) or (if he/she is a university researcher or however more theoretical in mindset) to represent not "real-life" cases.
On the other hand practitioners often work with "sensitive" data in "real" cases and they cannot distribute actual images/datasets for obvious legal/privacy/secrecy issues.
We touched somehow the topic here
http//www.forensicfocus.com/Forums/viewtopic/t=11023/
But it's not like an actual related idea/prototype
http//articles.forensicfocus.com/2013/10/18/forge-computer-forensic-test-image-generator/
was universally acclaimed or appreciated.
The "solution" would be in theory to make a committee of experts, some would be practitioners, some would be programmers, some coming from academics to study, evaluate and create a "standard validation dataset" and then carry actual tools validation.
This more or less would equate to a sort of "certification" of the tool (at least against a given set of well thought and cleverly crafted datasets) BUT of course by the time members of the committee will manage to agree on a dataset and/or "validate" a tool there will be
- three or four new OS releases
- success of tens of new softwares/protocols/cloud services/file formats/etc. that will become in common use
- failure and consequent obsolescence of common (at the time the committee is nominated) softwares/protocols/cloud services/file formats/etc.
- n updates of the forensic tool under test
- something else …
[/listo]
So, in any given moment *whatever* this hypothetical committee produces will be partially or largely outdated.
If you think a bit about it, this is what happened (and is happening) with NIST. (
JFYI (and only if you have some time to go through it) it does exist a software that is reportedly "certified" by NIST (and that actually exceeds their criteria)
http//www.forensicfocus.com/Forums/viewtopic/p=6562161/#6562161
http//www.forensicfocus.com/Forums/viewtopic/t=8679/
jaclaz
athulin
You want validation, you validate. But you don't validate on 'wild' data. You validate using carefully created data sets, following a designed test method, or the validation has no value.
Which confirms the point. That said validation is then used as a basis for expected and consistent operation against “wild data,” not the carefully created data set. You don’t validate a tool to then use it on other carefully created data…."fabricated data" as jaclaz put it. You validate it to support its use in the uncontrolled environment or controlled environment with uncontrolled data.
As you note, there are not a ton of sources, but wouldn’t
jaclaz
Not a “committee” but rather an actual organization/entity. If it was the express purpose and existence of such an entity, there would be no competing agenda, just a single goal. Everyone would have a job to work for the organization and do these tasks. I don’t know if NIST had ever endeavored to keep up, but as you hint they cannot. This
There are organizations that demonstrate it can be done more timely. The Defense Cyber Crime Center has a research arm, the Defense Cyber Crime Institute. DCCI does, among other research things, tool validations. Their validations are more extensive and much faster than NIST (though not perfect of course). Unfortunately for everyone else, they are only available to DoD and Federal LE and intelligence.
I don’t think any good solution can ever be bleeding edge, but keeping up can be done if it is uniquely purposed and open for all.
Geesh. I need some verbiage reducer or some validating tool for the content here, to see if I grasped all the to and fro … mrgreen
presuming tool = software tool
In my experience tool validation validates very specific requirements; rarely the whole package. In general, most of the digital forensics tools are (or should be) validated through similar to ISO/IEC 250102011 framework.
This is the "Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models". Yes; the one thing you should learn from it is never leave it to committees to come up with titles.
Most lawyers, in my experience are looking for "error rate", when talking about tool validation.
When we, forensicators talk about tool validation, we are discussing external validation (black-box measurements, or quality in use), not necessarily internal, specifically correctness, reliability, and integrity (as McCall & Matsumoto describes it).
Of course, there are various methodologies and taxonomies, including the above mentioned ISO/IEC.
[…]why validate at all?
Because forensicators deal in facts, to the best of our abilities.
[…]can’t do anything about what you don’t know,
But, we can. Mitigating solutions do not always have to know everything about possible outcomes to be able to mitigate the unknown.
[…]you can do something about what you do and shouldn’t one be obligated to do so?
In my opinion, either do something about it, or note it explicitly that it is known, and the risk is accepted, without action. That is a proper response to certain things.
[…]Perhaps validation isn’t necessary and the focus should be on verification/confirmation of results primarily?
Verification of results is a type of validation. Some call it back-box methodology or testing. It is a perfectly acceptable method and is used in various industries.
You want to be really, really scared, but is a perfect example of black-box method? Look into how pharmaceuticals are tested for psychological disorders.
Disclaimer I work for an organization that also does software validation. I also validate software myself and publish the outcomes.