±Forensic Focus Partners

Become an advertising partner

±Your Account


Username
Password

Forgotten password/username?

Site Members:

New Today: 0 Overall: 32221
New Yesterday: 1 Visitors: 115

±Follow Forensic Focus

Forensic Focus Facebook PageForensic Focus on TwitterForensic Focus LinkedIn GroupForensic Focus YouTube Channel

RSS feeds: News Forums Articles

±Latest Articles

RSS Feed Widget

±Latest Webinars

Thoughts on testing tools

Computer forensics discussion. Please ensure that your post is not better suited to one of the forums below (if it is, please post it there instead!)
Reply to topicReply to topic Printer Friendly Page
Forum FAQSearchView unanswered posts
Go to page 1, 2, 3, 4, 5  Next 
  

Thoughts on testing tools

Post Posted: Wed May 17, 2017 3:00 am

Hi all,

What are everyone's thoughts on testing - in terms of the tools that we use. I have been thinking about strategies etc given the importance (albeit it has always been important) with ISO standards etc.

Does everyone have their own test data, strategy that they roll out on new software/releases? I have been thinking about he potential value of an automated test data generator for know good content from which to evaluate parsing/carving algorithms against. Just wanted to gather thoughts on such a thing, what it would need to do and how valuable. I was probably thinking about carving validation so test data would be geared towards such algorithms.  

tootypeg
Member
 
 
  

Re: Thoughts on testing tools

Post Posted: Wed May 17, 2017 5:24 am

From what I have seen so far and the validation, which passed UKAS from both LE and private sector, isn't worth the paper it's written on. Not the fault of the producer, its that testing a tool which examines 2000 mobile phones over 4 different OS's with 20 OS versions and 1000's of app's is a job that would never be complete before it needed redoing even if a national unit was doing it.

My thinking would be that to achieve a "tick" re-use the same data again and again for carving/imaging type validation and simply say the last version got the same results as the previous version therefore it is equally good.

Its not actually guaranteeing very much, but then ISO 17025 guarantees f*** all anyway.  

minime2k9
Senior Member
 
 
  

Re: Thoughts on testing tools

Post Posted: Wed May 17, 2017 6:11 am

What about the generation of test data which is different (content-wise) but verified in terms of structure, so that more exhaustive testing could be provided.

....or are you saying that we just cant/dont sufficiently test tools.  

tootypeg
Member
 
 
  

Re: Thoughts on testing tools

Post Posted: Wed May 17, 2017 6:27 am

More the latter.
In terms of imaging storage devices (hdd's, ssd's , pen drives etc) validation is possible to cover the majority of situations if you assume a toshiba spinning hdd is not different from a hitachi hdd in terms of imaging.
"Imaging" or extraction of mobile devices - any testing is the smallest percentage of use cases and the devices/os/apps have changed while you were writing up your validation.
Carving of images - its never possible to prove that a method brings back everything when you start talking about deleted, fragmented files. Only that what you get is real. As long as you get data to manually verify your results then that's basically enough. I can't imagine a scenario where a software bug causes Indecent Images to be carved from nothing!  

minime2k9
Senior Member
 
 
  

Re: Thoughts on testing tools

Post Posted: Wed May 17, 2017 7:01 am

I see your points,

I guess in terms of carving validation, it would be a case of identifying and acknowledging the weaknesses of a certain carving algorithm (if any) and that it returns results consistently when the environment variables are X.  

tootypeg
Member
 
 
  

Re: Thoughts on testing tools

Post Posted: Wed May 17, 2017 7:49 am

- minime2k9
More the latter.
In terms of imaging storage devices (hdd's, ssd's , pen drives etc) validation is possible to cover the majority of situations if you assume a toshiba spinning hdd is not different from a hitachi hdd in terms of imaging.
"Imaging" or extraction of mobile devices - any testing is the smallest percentage of use cases and the devices/os/apps have changed while you were writing up your validation.
Carving of images - its never possible to prove that a method brings back everything when you start talking about deleted, fragmented files. Only that what you get is real. As long as you get data to manually verify your results then that's basically enough. I can't imagine a scenario where a software bug causes Indecent Images to be carved from nothing!


Been of the same opinion for a long time with regards this long running saga of testing/validation with respect ISO stuff.
What are you going to test for? What are you going to test against? As you say there's endless tools, endless functionality within said tools, endless potential data sets.
Even just picking one function, within one tool, running it against one set of data, and seeing expected results, is no guarantee it will on the next set of data. How many times, with how many different sets of data, would you want to run it, to have confidence it works as intended? Even if you ran the test 100 times over with 100 different sets of data would you be confident it was reliable on the 101th or 1001th etc?
Then multiply that for every tool, every function, and so forth?

I can never get past the opinion that it is fundamentally a giant waste of time and that you're always better off "trusting" the tool to some extent. Trying to dual-tool, or manually verify the results, every time, or at least periodically / where more appropriate, seems endlessly more practical/sensible.

In reality, the testing and validation will simply leave you with an unjustified sense of overconfidence in the tools and methods being used, or simply be a giant waste of time (or both).  

Rich2005
Senior Member
 
 
  

Re: Thoughts on testing tools

Post Posted: Wed May 17, 2017 8:18 am

Really interesting point - so is the argument then that we cant and never will be able to effectively test tools - sorry reading that back, I couldnt help but feel like I sounded like some sort of interviewer!

I hear the points, just wondered whether that is the whole point of continuing to push for testing and that every little helps. Or without any form of testing, is that not going to completely undermine our field?  

tootypeg
Member
 
 

Reply to topicReply to topic

Share and Like this forum topic to get more replies




Page 1 of 5
Go to page 1, 2, 3, 4, 5  Next