Greg,
Your response is interesting? I really do not wish to be ambiguous – I was actually trying to be direct by indicating that it may not be as straight forward as any of us would like it to be.
Far from having any admissions that I wish to make - the truth is I would welcome this if we could achieve it. What I was trying to explain (and clearly not doing a very good job at it, if I have to make two posts) is that the reality of the situation is far harder.
As a vendor I would love to be able to stamp a 'Certified' badge on my products, in fact I have been trying to find a way to achieve that unsuccessfully so far.
There are five releases of .XRY per annum (other providers are available). Some are for new features but the majority of these software releases are for increased phone support, we have to do this in order to simply keep pace with the market place.
Despite numerous meetings with different parties - there is no certification body able to check, validate and keep up to date with this release rate either in the UK or anywhere else for that matter. NIST for example in the USA - tested v3.6 of XRY and we had released v3.8 by the time the report was actually published. This invalidates any Certification before it is even issued.
I am sure that most other vendors like Micro Systemation are all for Validation. The issue is one of practicality as to how it is done – so if someone here can offer a solution and explain to me how it would work then I will jump at the chance. The Forensic Regulator is actively seeking input on how to better improve these processes for validation and certification and I am sure they would welcome your input if you had a suggestion.
In the absence of that information, my conclusion at present is that it is not practical at the moment for vendors to achieve this validation – hence the request for an additional poll option?
I do not believe that is ambiguous or avoiding the issue in any way – it is simply my view.
Mike
Mike
No, actually, you didn't follow what you now state above. You used the suggestion I had been trivial and unprofessional in the Poll questions. You cannot escape that (not to put too fine a point on it).
When challenged on whether you made admissions about the product you sell you side step the issue.
The question remains should all tools be Certified/Validated? The question isn't how can I steer round it.
It is not in issue whether tools are needed, we all agree they are needed. The mere fact that the very simple, basic question of this thread creates such apparently overwhelming perceived challenges to that it invokes responses that something isn't possible without even having tried, is the bit that speaks volumes.
You say Updates happen five times a year? And…? So what? How many examiners agree to having five Updates a year? How many of those Updates do you say are suggested by users of XRY? Does that mean you are saying if users suggest updates that are not 'evidentially sound' they get put in anyway? So is it that why you say your users are to blame for not having a Certified/Validated tool? Why would a manufacturer ask its users to go to Court on its behalf to defend its product when a manufacturer would be unwilling to defend its product at Court?
We could go on like this for a long time Mike, or you could explain technically why it is not possible to have a Certyified/Validated XRY product? What is wrong with the product that you say why it cannot achieved?
Greg,
I have to go with Mike and Sean on this one. The poll questions are written such that you agree with choice 1 or 2 or your professionalism is brought into question. As it stands, I do not believe that certification/validation can be done by the vendors because there will always be questions about their motivation( as evidenced by your response to Mike). Until there can be a govorning body that has the resources to validate all of these tools at the speed in which they are brought out for use, I do not belive that there can be such a thing.
Eric Wahlberg
Eric I see you are a first time poster and I hope you will forgive me if I get straight to the point.
I am going to disagree with you and that also means from your comments disagreeing with seanmcl and Mike too.
No one has ever produced a Poll that provides the questions that everyone wants. It is impossible. As sure as night follows day, there will be someone who says the questions do not fit with their style of thinking.
Ultimately does the forum understand the questions? Yes they do.
Would examiners like Certified/Validated tools? Yes, so far apparently they would.
End of story.
The Questions were set from best case scenario to worse case scenario. I had no intention of undermining anyone. Perhaps sensitivity should be left at home and to remember that data is dispassionate, so should the expert be.
Where there is the will, there is a way. Where there are excuses, confusion surely follows in its wake.
seanmcl are you saying you need to be convinced about the need to have a Certified/Validated tool which you use as forensic practitioner?
Not all all. What I am saying is that until someone proposes a process by which a tool is validated or certified and shows that that process, when applied to other tools, can be demonstrated to be accurate and neutral, I am not in a position to say whether I favor it or not. Even the NIST process is very focused on the objectives and outlines specific tasks to be performed and expected outcomes. Saying, however, that EnCase is NIST validated speaks only to acquisition/restore and not to any other part of the tools.
In my experience, which I do not mean to compare to yours, there are different tools out there with different capabilities even with respect to the same handsets. This suggests that a certification or validation process would need to be either very focused (in which case, it would be like NIST), or very broad (in order to not favor a particular vendor).
Or might you be deviating from the Poll and actually saying forensic tool cannot come about until you are consulted on what makes up the processes involved Certified/Validated?
I am not asking to be consulted. I am saying that A) no such certification/validation currently exists and, therefore, B) I reserve my opinions until I have some facts upon which to base them. Would I like to see some formal mechanism for testing of tools be established which can verify the tool's accuracy. Sure (assuming that it meets other standards which would be the same in any industry). But since it doesn't, yet, exist, I cannot take a position for or against it.
Do remember the Poll merely seeks a show of hands that people believe they should be using Certified/Validated tools.
That is not how I read the questions. If, for example, I choose the first answer regarding ALL tools. How do I answer the question, if answered in Court, as to how much confidence I have in the tools that I am using, none of which is certified or validated?
And if I answer with the second option, it is more problematic. What basis is there for me accepting the results of older, established products, which have no compliance with certification/validation when newer products are forced to conform?
The middle option is meaningless, unless you are going to argue that the market will fix this. If I don't demand certification/validation, why should any vendor pay to have it done?
The last two options are, I think, a bit insulting (at first I thought that they were tongue in cheek). I do care and I don't think of what I am doing, today, as "getting away" with anything. I am doing the best that I can given the state of the art. And my bottom line is not the issue, either, at least insofar as whether tools should be validated/certified.
I pay a lot of money for the tools that I use now and, sure, I'd like to have some confidence in their accuracy and repeatability. Ideally, the device vendors would provide a forensic method for acquiring raw data and parsing it, accurately.
But I'm not going to stop using existing tools until something comes along to perform this function. I am going to check my results as much as possible and, where possible, use independant methods to verify my conclusions.
And that option, I think, best expresses my view of the state of the art, what is achievable, today, and what are my standards of practice.
It is possible to have a Certified/Validated scheme in our field of distinction.
A lot of things are possible but not always available. Your poll left no room for healthy skepticism but left alot for cynicism. There was no option with which I could totally agree and no room to say "None of the above".
Greg,
I happen to agree that certification/validations is a goal that should be sought after. That being said, I hear about polls on a daily bases and we all know about how to skew poll responses.
Q1 Do you believe we should spend half of our GNP to aid in world peace.
Q2 Or are you a war mongering baby killer who doesn't want to help out.
If you want to get real results from a poll, you need to leave out the commentary. The last two questions insert value judgements with your response.
Eric
Greg - if you're going to offer "Who cares, we've been getting away with it so far" as an option you can expect to take a bit of flack. Let it go.
Mike, Eric - welcome to the site.
Jamie
Greg - if you're going to offer "Who cares, we've been getting away with it so far" as an option you can expect to take a bit of flack. Let it go.
Mike, Eric - welcome to the site.
Jamie
Jamie in fairness I have explained the purpose for both ends of the scale Poll Questions. You will also note I didn't reply to seanmcl's and Eric's last comments so that they could end on a note that they had fully aired their views.
Here are two threads at my webblog that set out some of the issues in the UK which are highly relevant to this thread. I have included the Statutory provision and the categories underpinning the provision. I have not identified the case law that supports the categories but can do so if that would prove helpful.
Mobile Phone is not a 'Closed Container'
http//
Mobile Phone is not a 'Closed Container' Part 2
http//
At the end of the day mobile phone examiners who produce or give evidence should be able to do so with total confidence about the tools they use and the data. I would rather we deal with hard ball questions amongst our peer group and be seen to be doing that.
Greg,
It's not a question of avoiding hard ball questions, I simply happen to agree that the poll could have been worded more appropriately. That, in itself, isn't too much a big deal - it's just a forum poll which has sparked some interesting discussion after all - I'm just not sure where the hostility to Mike's position comes from, on the face of it he seems to be making a reasonable point? Are there grounds for doubting the integrity of the .XRY software?
Jamie
I know there is massive debate about if we should have certified tools. I can't see any reason why an examiner would not want certification on their tools, it makes sense!
Instead of discussing poll options why not discuss how we can go ahead and create documentation and certification for tools. The vast majority of phone examiners will use un-certified tools most days in the course of their job.
I agree with Greg that we need to strive to get at least the mainstream tools certified and then look at possibly creating forensic versions of flasher boxes or put pressure on vendors to incorporate pass code recovery/reset and dumping into their (certified) products.
The article I posted on page 4 does go into depth about verification of flasher boxes and discusses the possible ways to measure results. It could be a good base to start from for ideas.
Greg,
You mention on page 2 that you're on the trail of something. Is it something which addresses the concerns surrounding the feasibility of certification expressed so far? i.e. addresses the issues of the growing number of handsets/OSs, lack of detailed documentation, time constraints etc.
Jamie
I'd like to ask what people mean when the say "validation" of the tools?
So taking Encase as an example does it mean that the tester has checked that the application opens an image file and parses the MFT correctly in NTFS, displaying all the dates, filenames and other metadata correctly. Then goes on to check each and every function of the application to see that it works correctly in every respect in every different OS and every different iteration of the OS?
If the "validation" does not cover some particular aspect of the functionality of the program that might be key to any case can this software be deemed to be validated?
Would it be possible in practical terms to test every function of a tool?
Indeed what is the point of validation of a tool if the testing is not comprehensive?
Would a better approach be to have a requirement to validate any evidence that is proposed to be used in a case. Again what that validation would amount to needs to be defined, e.g. does it mean that a second different method has been used to corroborate the evidence, or that a second person has verified the evidence (perhaps the opposing expert)?
H