ISO 17025 for Digital Forensics – Yay or Nay?

"Has anyone (UK) given any thought to next steps in the wake of these events?"

Having members of the Commons Science and Tech Committee who have a decent basic grounding in the issues would be good?

PS
https://cyfor.co.uk/forensics-expert-places-view-recent-criticism-digital-forensic-labs/

As a side-side note, meanwhile in Canada
http//reason.com/blog/2018/03/05/motherisk-hair-testing-program-was-junk
https://www.attorneygeneral.jus.gov.on.ca/english/about/pubs/lang/

I might be wrong, but between the lines of the actual report, I seem to be reading (in bold, red letters)
"This wouldn't have happened if the laboratory had been certified according to ISO 17025".

Q.E.D.

jaclaz

As a side-side note, meanwhile in Canada
http//reason.com/blog/2018/03/05/motherisk-hair-testing-program-was-junk
https://www.attorneygeneral.jus.gov.on.ca/english/about/pubs/lang/

I might be wrong, but between the lines of the actual report, I seem to be reading (in bold, red letters)
"This wouldn't have happened if the laboratory had been certified according to ISO 17025".

Q.E.D.

jaclaz

I'm not sure it is the worst idea for 'wet' forensics, which is probably why the FSR has regulated it.
It's just the sweeping generalization where digital is put in with 'other' forensics that makes it a problem.
That said, all I have to say is 'Randox' to show its not going to stop miscarriages of justice.
Actually, our closest counterparts, forensic accountants are excluded from ISO 17025

I'm not sure it is the worst idea for 'wet' forensics, which is probably why the FSR has regulated it.

Actually I am pretty sure that ISO17025 is a good thing for "wet forensics", AND NOT a "bad thing" for "digital forensics"[1], the issue (as I see it) is only about the false assumption
certified ISO17025=surely no sloppy work
or
certified ISO17025=solution for all issues

jaclaz

[1] If it was properly applicable to it, which isn't.

And now, for no apparent reason, a couple instructive videos on Vimeo 😯
https://vimeo.com/243645429
https://vimeo.com/234514957

jaclaz

Please tell me by "instructive" you were being sarcastic?? (I'm not sure, if not that's ok, I just want to make sure I read things correctly.)

Its kind of funny that in these videos they stress the "mandatory" by the Regulator when that very same regulator went before the committee and noted that she didn't have the authority to enforce anything, hereby rending the work "mandatory" wholly inaccurate.

Its a sales pitch. No more, no less. Someone was going to cash in on government mandates sooner or later…

Please tell me by "instructive" you were being sarcastic?? (I'm not sure, if not that's ok, I just want to make sure I read things correctly.)

Sure D , sarcasm is my middle name. wink

Its kind of funny that in these videos they stress the "mandatory" by the Regulator when that very same regulator went before the committee and noted that she didn't have the authority to enforce anything, hereby rending the work "mandatory" wholly inaccurate.
Its a sales pitch. No more, no less. Someone was going to cash in on government mandates sooner or later…

Not only, whenever a standard is "mandatory" the whole point is that it is "mandatory", so it is not that the compliant laboratory A is "better" than compliant laboratory B, it is only that non-compliant laboratory C is not authorized anymore, so it is IMHO a terrible sales pitch
"look, we are good (exactly like all the rest of people authorized)"

The doubt I have is more about the intended target audience.

I mean, the Crown Prosecutor and Police, and more generally the Government should be already well aware of the issue, I doubt that defendants are (or ever will be) so the only possible target is defense lawyers?

jaclaz

mandatory
ˈmandət(ə)ri/
adjective
adjective mandatory

1.
required by law or mandate; compulsory.
"wearing helmets was made mandatory for pedal cyclists"
synonyms obligatory, compulsory, binding, required; More

I check in on this, once in a while, to see where people are at with this.

Everything's already been said so have little to add (I'm on the side of those saying it's a poorly designed standard, bodged from other disciplines, that is pointless, time consuming, and won't achieve what it sets out to do).

I do however note that in the Forensic Science Regulator's own "Legal Obligations" document (issue 5) that it acknowledges that the ISO standard is not a legal requirement for evidence.

7.4 Accreditation
General
7.4.1 There is no general legal requirement for an organisation, or individual, to be accredited to any national, or international, standard before results they generate are admissible as evidence.
7.4.2 The lack of accreditation, in an area where such accreditation might be expected, could contribute to material being ruled inadmissible upon consideration of the requirement for reliability (see paragraph 7.1).
7.4.3 Lack of accreditation could also have an impact on the weight which is attached to evidence. Lack of accreditation has been noted by the courts (see R v. Reed & Ors. [2009] EWCA Crim 2698 at paragraph 105).
7.4.4 Accreditation may be preferred or specified by those instructing the expert. See the discussion on quality standards at 7.5 below.
7.4.5 Accreditation may be a relevant matter when considering the reliability of the evidence. See 8.12 below.
7.4.6 The European Union has adopted Council Framework Decision 2009/905/JHA on the accreditation of forensic service providers carrying out laboratory activities. This requires that laboratories providing DNA analysis services to the CJS be accredited to ISO 17025 [5] by December 2013. Fingerprint laboratories, providing services to the CJS, must be similarly accredited by December 2015.
7.4.7 The UK indicated its intention not to be bound by the provisions of this Decision [6]. It subsequently opted-out of the provisions.
7.4.8 HM Government applied to join the Prüm Decisions and, as part of that process, rejoined the Framework Decision. By Commission Decision (EU) 2016/809 of 20 May 2016 the EU agreed to the application and, as a consequence, the provisions applied to the UK from May 2016.
7.4.9 The manner in which this obligation is implemented is yet to be determined.
7.5 Quality Standards
7.5.1 There is no specific legal requirement for work to have been completed in adherence to a given standard to be admissible as evidence. However, the general requirements for relevance and reliability have the effect of requiring the work to be done to appropriate quality standards. This can be seen in a number of judgments discussed in this document, the work of the Law Commission and the 2014 changes to the Criminal Practice Directions.
7.5.2 In the Scottish case of Thomas Ross Young v. HM Advocate [2013] ScotHC HCJAC 145 the Court, at paragraph 57 of the judgment, noted that the lack of agreed national or international standards for the type of evidence was a factor in deciding whether to admit the evidence.
7.5.3 In the case of R v. McClenaghan [unreported] the defence maintained that the quality standards appropriate to the work required accreditation to ISO 17025 [5] and, as a result, extensively cross examined the witness as to compliance with the provisions of that standards and the laboratory’s quality processes. The issue of maintenance of appropriate standards was also noted by the appellate court.
7.5.4 The failure to comply with expected quality standards would be a matter which should be disclosed under the requirements of the CrimPR and Criminal Practice Directions. This is discussed sections 8.10 and 8.12. 41
7.6 Validation
7.6.1 In R v. Hoey [2007] NICC 49 validation was described, in paragraph 62, as follows.
““Validation” is defined in those guidelines as “the process whereby the scientific community acquires the necessary information to
• Assess the ability of a procedure to obtain reliable results.
• Determine the conditions under which such results can be obtained.
• Define the limitations of the procedure.
The validation process identifies aspects of a procedure that are critical and must be carefully controlled and monitored.“
7.6.2 This appears to have been a quote from the Scientific Working Group on DNA Analysis Methods – a US based body. It was also quoted in R v. Duffy and Shivers [2011] NICC 37. If it is read to mean that those undertaking the validation exercise acquire the information set out through the validation (and that this information would allow suitably qualified persons to assess the performance of the method), then this is a definition the Regulator can endorse. If the wording is interpreted as requiring the achievement of acceptance in the general scientific community, then the definition is less supportable as wider acceptance only occurs over time. The imposition of such a requirement could delay the availability of new methods to the CJS which is not supported by a number of judgments (see Parts 7.10 and 7.13 herein).
7.6.3 In Hoey the need for a clear protocol for validation was highlighted – see paragraph 64.
7.6.4 The requirements for validation are set out at Part 20 of the Codes of Practice and Conduct [7] and further discussed in guidance [8].
7.6.5 There is no legal requirement for a technique to be validated before results generated by it are admissible as evidence.
a. R v. Harris & Ors. [2005] EWCA Crim 1980;
“270 … There is no single test which can provide a threshold for admissibility in all cases. As Clarke demonstrates developments in scientific thinking and techniques should not be kept from the Court. Further, in our judgment, developments in scientific thinking should not be kept from the Court, simply because they remain at the stage of a hypothesis. Obviously, it is of the first importance that the true status of the expert's evidence is frankly indicated to the court.”
b. R v. Reed & Ors. [2009] EWCA Crim 2698; -Thomas LJ;
“73 The Forensic Science Regulator … also made clear that he did not consider validation a necessary pre-condition for the admission of scientific evidence, provided the obligations under Rule 33.3(1) of the Criminal Procedure Rules were followed. In the light of the issues that emerged in these appeals and considerations set out in the next paragraph and paragraphs 111 and following, we see much force in that view”. 42 43
42 Part 33 of the Rules in effect before 2015 became Part 19 in the 2015 issue of the Rules.
43 The Regulator views validation of methods prior to use within the Criminal Justice System as a key feature of any quality system and an important safeguard. However, it is recognised there may be cases where, exceptionally, non-validated methods may be employed. In such cases the status of the method must be made clear.
7.6.6 The lack of validation could contribute to proposed evidence being ruled inadmissible upon consideration of the requirement for reliability (see paragraph 7.1).
7.6.7 Alternatively the lack of validation could affect the manner in which the court evaluates the evidence. In Lundy v. The Queen (New Zealand) [2013] UKPC 28 the Privy Council made the following comments.
“81 It is important not to assume that well established techniques which are traditionally deployed for the purpose of diagnosis can be transported, without modification or further verification, to the forensic arena where the use to which scientific evidence is put is quite different from that involved in making a clinical judgment. Put simply, evidence that can properly be used to reach a confident medical verdict may not measure up to the more stringent requirements that arise in the setting of a criminal trial. While, of course, it is not always required that an individual item of scientific evidence proffered in support of a specific proposition will establish its correctness beyond reasonable doubt, the overall context in which scientific evidence adduced by the prosecution is presented is that it should constitute part of a case that will prove to the criminal standard the guilt of the accused.
82 Scientific proof such as fingerprint or DNA evidence is customarily given against the background of its having been theoretically tested in, if not laboratory conditions, at least empirical survey. The novelty of using, in a criminal trial, the type of evidence offered by Dr [M], especially when its reliability has not been subjected to such laboratory or empirical research, does not necessarily make it inadmissible but it prompts caution as to its role in establishing guilt beyond reasonable doubt.”
7.6.8 Where a technique/method is not validated the limitations in the technique should be made clear to the court (see Part 8 herein).
7.6.9 As noted in Part 8.12 the 2014 version of the CrimPR introduced a requirement, preserved in the 2015 version, for the witness to provide information relevant to the reliability of the evidence. Clearly the extent of validation will be relevant information.

Frankly I think you could make a strong case that to become ISO17025 compliant you would either be doing a "botch job" to claim compliance, or if making a futile effort to follow it to the letter, be stuck in a never ending cycle of testing/validation/documentation meaning you never did any work at all, and if you did, have the effect of meaning you're often on out-of-date software, to some degree, which may well have known (and fixed) faults, and therefore increasing your chance of producing poorer results.

As an update
https://services.parliament.uk/bills/2017-19/forensicscienceregulator.html

If you're in the UK, might be a good time to write to your MP!