ISO 17025 for Digital Forensics – Yay or Nay?

Trying to explain that we have several methods for imaging and it was up to the examiner to decide the most appropriate was not acceptable and they required that the use of different methods and reasons should be documented as part of the procedure.

Leaving them undocumented is clearly not according to 17025 requirements. (At least that's how I interpret what you are saying, and what 17025 says.)

Sorry I possibly wasn't clear here. We have several documented, validated imaging techniques. We allow the examiner to decide which of these to use when imaging a device.
The auditors wouldn't allow this approach as it wasn't part of a documented procedure to decide which documented, validated procedure to use. Presumably once we document this we will then have to validate the decision making process that we have just documented.

Very much in line with
http//dilbert.com/strip/2005-07-02

There was no question about the processes we had or the documentation of the methods, but allowing an examiner to use their experience to determine the best method to use isn't "ISO 17025".

"Except that during more than 1 accreditation visit at different locations across the UK, people have been asked how their environment protects against solar flare activity and other such things."

I don't want to think that this is true but, if it is, it's pretty scary.

We have several documented, validated imaging techniques. We allow the examiner to decide which of these to use when imaging a device.
The auditors wouldn't allow this approach as it wasn't part of a documented procedure to decide which documented, validated procedure to use.

OK. I'd think that's a excessive, as long as it was documented to be left to the examiner, as well as requiring the examiner to document his choice, and possibly also the reason for it.

Again, I think I can understand the reasoning a lab technician should not have to do that kind of decision. But if the assessors don't see a difference between a DNA lab and a DF lab … they're likely to stay with what they consider safe.

Just as some ISO 9001 assessors once required software consulting companies who tried to get certified to produce their raw material purchasing and quality validation documentation. Well, the standard said it had to be there. Might still do, for all I know.

Can you choose assessors, or do you just get assigned some randomly? If you can choose, it might be an idea to go with assessors who have done, say, a dozen DF lab jobs.

Just as some ISO 9001 assessors once required software consulting companies who tried to get certified to produce their raw material purchasing and quality validation documentation. Well, the standard said it had to be there. Might still do, for all I know.

Think there is still something in there about "critical supplier" lists. Same kind of logic, standard says….

Think there is still something in there about "critical supplier" lists. Same kind of logic, standard says….

As far as I remember, there was also something towards the end of the 9001 text saying that that standard could also be applied to other situations, not just manufacturing industry.

17025 has an Annex B that says something along those lines, but I think it's more developed here, and I'm not entirely sure what they mean by 'application'. B.4, however, seems to be possible to use for 'an entire technical field', but of course that means that technical field need to get together to produce the required complementary text.

(Added a pity that the original didn't proviode for 'ISO17025 extended with application-specific requirements'.)

Might want to ask assessors if they have taken those guidelines into consideration … -/

The Appendix B appears to allow for definitions for a field or industry but gives no scope to "ignore" irrelevant areas of the standard.

For my own part, the standard HDD operation requirement that stipulates that you must not drop, bump or jar the drive when it is operating could be a bit of a problem. Having to prove that I didn't do either might be tricky – a Maxtor requirement I've noted was less than '30 G during 2.0 ms'. Perhaps there's some 'fast' G-force sticker I could use to show I didn't jar, bump or drop worse than that?

As a side note, and JFYI, by using "G" you already violated ISO 17025 😯 , the unit is g (and the Japanese have some advantages wink ), see for some fun considerations
https://msfn.org/board/topic/141874-gs-and-gs-and-internet-noise/

But we have no actual "calibration data" about the actual amount of acceleration a hard disk can stand, so the 300 or 350 g that Seagate state (and that are already ten times what you posted) so once again "don't drop the disk drive" is a "common sense" or "good pratice" recommendation, the "enact suitable measures to avoid exposing the device to an acceleration in excess of 300 g or of 30 g for a time in excess of 2.0 ms" is instead something that some good guy verifying your ISO 17025 compliance is very likely to introduce/add, based on a vague and unproved/untested recommendation of the hard disk manufacturer and extremely hard to put in practice and document.

jaclaz

IMO,

These latest comments are the very discussions that are in the weeds because of the need to work out policies in compliance with this inappropriate standard that the accrediting body (or UK Regulator in this case) has to either overcompensate for or create endless exceptions and explanations to make things "fit."

On the flip side, I would like to offer a thought. Considering that many commenting in this thread are in the UK, have any of you considered becoming an assessor? This would likely be the most effect means of making change by being the very people who perform these assessments and debate the legitimacy from the inside. I don't know how it is there, but here my lab is accredited to 17025 by ANSI-ASQ National Acceditation Board (ANAB) which is the main game in town since merging with ASCLD/LAB. One can be a contract assessor for them, which is ideal if you're experienced in a particular subject matter. If you all have such an option, I would suggest considering it. Since things are already set in motion, its a route that could be taken by those interested before things get even more cumbersome.

http//parliamentlive.tv/Event/Index/7767e1b9-0e44-4de3-8627-baf9d091f487

06 February 2018

The Science and Technology Committee holds an evidence session on the Biometrics Strategy and Forensic Services with the Minister for Countering Extremism and the Forensic Science Regulator.

Watch Parliament TV Biometrics Strategy and Forensic Services
Inquiry Biometrics Strategy and Forensic Services
Science and Technology Committee

Witnesses

Tuesday 6 February 2018, Grimond Room, Portcullis House

At 9.30 am

Dr Gillian Tully, Forensic Science Regulator

http//parliamentlive.tv/Event/Index/7767e1b9-0e44-4de3-8627-baf9d091f487
The Science and Technology Committee holds an evidence session on the Biometrics Strategy and Forensic Services with the Minister for Countering Extremism and the Forensic Science Regulator.

At 9.30 am

Dr Gillian Tully, Forensic Science Regulator

Thanks for posting this pbeardmore. It well worth watching.

Below are some notes I took regarding what Dr. Tully stated. (not double checked or verbatim).

+++++++++++++++

- Forensic Services Providers are charging too little for their services.

- 50% of the decision to go with a bid is based on price, so good companies are bidding low to just get the contract no matter how good their lab is as many purchasing services are not worrying about ensuring the lab is accredited.

- Need stability within the marketplace

- One issue is when a forensic service goes out of business, where does that evidence (HDD's, blood samples, DNA, etc) go? Who stores it?

Difference between Police and Commercial Providers for accreditation and standards.

Large Commercial Providers
"Extremely compliant with the standards and have been for many years."

Police
Acceptance in principle that they will move towards meeting the standards. However, substantially behind where they need to be.

For Digital Forensics - many failed to meet the deadline in October 2017.

Of the approx. 46 legal entities in policing, only 17 had any sort of accreditation by the time the deadline passed and most of those, it wasn't the full scope that was required.

Good reason for missing the deadline?

Dr. Tully clarifies that they still must work towards accreditation and that the ones not compliant must disclose non-compliance in any statements they make to the court under the criminal procedure rules.

Question Do you think this will encourage them to meet the standards?

Dr. Tully Yes, I very much hope so. It is not that they haven't made any progress and I don't underestimate the challenge that this has been because digital forensics has grown massively, the only area of forensics that has grown over the last few years. And the volumes of data has grown massively.

Process of what Dr. Tully would call validation which is making sure that your methods are fit for purpose, that they do what you think they do, and they do what you say they do and you know what their limitations are, has been a very very steep learning curve in the Digital Forensics community and not something they have engaged in before.

But this is extremely important because the courts need to know what the limitations of a method are. They need to know that some sorts of files may not have been found when a phone is downloaded.

Question Do you think this lack of compliance will change the outcome of some court cases?

Summary For those that are on the path to accreditation, they will be able to provide the courts with information to show their validation and testing processes to enable the court to decide whether or not that their evidence is reliable and to be used as expert evidence.

They will also need to show/prove that their staff is competent and do they have properly calibrated equipment.

For those that are long on their way to accreditation, they will be able to provide the courts enough information to mitigate any issues. For those that have made little to no progress, they won't have that information to provide to the court.

In the commercial sector (not large providers), progress has also been slow. For Digital Forensics providers, many of them are smaller. Many are 1 or 2 person providers working from home and using tools that they may not have validated.

Dr. Tully states that Digital Forensics is the sector that is of most concern with compliance. She says she has spoken at many conferences where people have stated to her that she has no power to make sure labs are accredited.

She says that there are services to help police organizations reach accreditation, but very little for the small providers.

For the four (4) commercial providers that are compliant, they have a fair complaint that some of the contracts that have been awarded, that went past the deadline, didn't have the requirements that the providers would be compliant and therefore missed out on those contracts to cheaper providers that are not compliant or investing in quality standards.

Cost of Compliance

Question Heard that compliance can cost from $7,000 to $17,000 which doesn't seem like a lot.

Dr. Tully That is just one aspect of compliance. The total bill will be higher depending on how close an organization is to meeting standards.

Accreditation runs on a 4-year cycle. The first year is the most expensive.

For large commercial providers, they estimate that it is about a 15-20% overhead. Dr. Tully believes it is substantial but critically important. For smaller providers, especially 1-person providers, the costs are proportionately higher, but she doesn't see how she could have a different standard for those providers since they are still providing evidence to the criminal justice system.

She is working to see if they can find a way for smaller organizations to work together to achieve the standards but reduce the overall costs.

Dr. Tully believes she needs statutory powers to meet full compliance. Without statutory powers, she believes that only 80-90% compliance can be expected.

Until companies or organizations properly adopt standards, they are not looking for failures or issues and they are not reporting those failures.

One of the problems we come up with when we set standards in a new area is, everybody goes..Oh, there is no problem with this area. What's the problem? And that's been very much the issue with Digital Forensics. And that is one of the problems until people starting working with the standards, looking for the issues, they are not seeing the problems if they are not looking for them.

++ end of summary ++