ISO 17025 for Digital Forensics – Yay or Nay?

As onsite response generally requires on-the-spot decision making, innovative and untested responses to handle new threats, violating a lab policy will be the norm.

If it is, it's the fault of that lab itself, and noone else.

As far as I know, no quality management framework (ISO 17025 is one) makes any requirements on location, business model, style of office wear or facial hair. They do tell you what is mandatory, and require you to live up to that, as well as the other rules and policies you create for yourself. But they do not do anything else.

If you create your own ISO 17025-based system that requires you to do your lab work in a designated location, you have made your decision about how you want or need to work. If you say 'we do *all* our work in our own localities' you have to live up to it – and can't collect any information in the field. But then you've painted yourself into that particular corner.

So … don't do that. Or don't do it in a way that makes it an issue for an ISO 17025 auditor.
Separate field collection of material from other work, for example, and make it clear what rules and policies apply for those situations, and what rules and policies apply when you're back in the 'lab'.

ISO 17025 is used by assayers, for example they, too, may have to collect samples in the field, and not always under ideal circumstances.

I still think it's extremely unwise to try to create a complete ISO 17025-compliant system from the start. Better do the absolute minimum that is needed, by the standard and by business-critical customers. It's usually a larger effort than imagined to get an organization to work smoothly under any new rule it is not good business sense to make that effort larger than absolutely necessary – and one large part of that is educational. And by that that I mean educational for those who write the rules as well as for those who are expected to follow them.

Sometimes, I wonder if the industry itself is partly to blame. The language that we choose to use around the industry is broadly in line with previously existing forms of forensics. By aligning digital forensics so closely to other forms of forensics, we have been "lumped in" to other lab based forensic practices whilst, in "the real World", we are very very different and really require our own standard.

The debate concerning mandatory standards pre-supposes that there is a relevant and meaningful standard available to apply (with quaified people to oversea the process).

I think this is an interesting question as I have observed confusion within our industry around the idea that we are a subset of regular forensics.

I've spoken with some that argue we aren't really a subset of forensics since we are so different in how data is acquired and analyzed.

Does this cause us more confusion and issues in working towards a standard since many people both within our field and outside (regulators) always try to put us with regular Forensics rather than seeing us as a different discipline?

PS ever since using 17025 was first mooted by the regulator years ago, I dont think I have met one colleague within the industry who has said "yay, 17025, great idea". Almost the exact opposite. So do I trust the mass consensus within the industry (including many voices that I have huge respect for) or do I trust the regulator?

Playing devil's advocate…(not that I disagree)
Who likes regulation? How often do you hear anyone affected by a law (or standard) say they are happy that the law exists?

Is there anyone working within an ISO certified environment that is happy with the additional regulations?

Regulations, 'red-tape', bureaucracy all exist. No one likes working in those environments, but they do exist for a reason and job satisfaction and happiness is not one.

As far as I know, no quality management framework (ISO 17025 is one) makes any requirements on location, business model, style of office wear or facial hair. They do tell you what is mandatory, and require you to live up to that, as well as the other rules and policies you create for yourself. But they do not do anything else.

This is interesting because I think many, myself included, see ISO regulations as being very rigid, but your comments suggest that this is not the case.

My purpose of playing devil's advocate is to ensure we have valid reasons against 17025, not because it simply makes our job harder. As the survey (referenced in the article) discusses, less than 25% of those surveyed had a high or clear understanding of the details involved in ISO 17025.

How many regulators have a clear understanding of the issues within Digital Forensics and can adapt the guidelines to meet our needs within the industry?

Is our lack of understanding the details around 17025 the true issue with this standard?

I think any regulation (ISO 17025, New Standard, existing standard) can and will make things harder to start until it becomes common practice within our industry.

For those working in ISO 17025 labs, can you discuss exact issues you've seen with this regulation that you can confidently state is an issue rather than a possible misunderstanding of the requirement either by your lab or by the regulator (who may not fully understand Digital Forensics)?

I still think it's extremely unwise to try to create a complete ISO 17025-compliant system from the start. Better do the absolute minimum that is needed, by the standard and by business-critical customers. It's usually a larger effort than imagined to get an organization to work smoothly under any new rule it is not good business sense to make that effort larger than absolutely necessary – and one large part of that is educational. And by that that I mean educational for those who write the rules as well as for those who are expected to follow them.

I fully agree with this comment. Creating a new standard from scratch will be a very time-consuming effort which will require lots of support from various organizations.

Hopefully, these discussions either show that we as a community are clear on what is needed or we find a way to adapt to existing standards while working closely with those that write the rules for these standards so that they fully understand our field.

I still think it's extremely unwise to try to create a complete ISO 17025-compliant system from the start. Better do the absolute minimum that is needed, by the standard and by business-critical customers. It's usually a larger effort than imagined to get an organization to work smoothly under any new rule it is not good business sense to make that effort larger than absolutely necessary – and one large part of that is educational. And by that that I mean educational for those who write the rules as well as for those who are expected to follow them.

That is funny, because compliance is a minimum level of conformity. It does not go deep into specific things that are mission critical and affect organisations differently in respect to their operating environment. It's basically a pair of cheap socks with a label that says "one size fits all" on it.

Shoving a generic standard down the throat of specialist organisations and expecting it to do magic is setting yourself up for a costly disappointment that does not amount to much, the paper people who are unable to achieve anything but paperwork should go away and screw with someone else's lives. I've lived in such a paper regime for years and it achieved nothing pragmatic or anything that increased security. There were still gaping security holes in systems and the only thing that was done to remain compliant was to write a piece of paper that acknowledged that "Yes we are aware of the problem and are not gonna do jack s**t about it".

This is what Schneier called CYA security (Cover your a*s security) so they can point to someone to fire if SHTF - it is not a measure of quality or security. Compliance people it's the kind of useless people you want to send to the front lines in a war.

Is there anyone working within an ISO certified environment that is happy with the additional regulations?

Yes … and no. I've worked with ISO 9001, and was part of getting the company I worked in ISO 9001-certified. ISO 9001 was written for manufacturing industry at that time, not for software developers and consultants.

Yes, I liked the result. After certification, we had a far better project platform than we ever had before, we knew roughly how to build on it, and in what direction we were going with it. We had control of project documentation for one thing …

No, I didn't like it – we made a lot of mistakes that took a long time to recover from. Changing a management system can be like making an oil tanker turn … But the mistakes were all ours, and in about 90% of the cases because we thought we could do better than the people who helped us thought we could.

Yes, I liked it. We got rid of lots and lots of local and historical variations how projects were managed, we got reasonably well written 'SOPs' that helped new employees get up to speed much faster than was possible earlier … as well as keep older hands on the straight and narrow instead of relying on memory and failing.

No, I didn't like it. We got the certification, but we didn't renew it. We were spent we couldn't operate at the level that we thought we could. And we didn't want to back down to make it easier on ourselves. (We had for some reason decided that we would get certified in one year. We did it, but at a price. If we had paced ourselves for a five- or perhaps a ten-year effort, I think it would have been better.)

In the end, though, I liked it more than I disliked it.

This is interesting because I think many, myself included, see ISO regulations as being very rigid, but your comments suggest that this is not the case.

See above. I started out with much the same flak spiel I see here myself. ("We're tech consultants and software engineers for crying out loud – what does a manufacturing standard have to do with us? Nothing!" and so on. )

Parts of ISO frameworks are rigid some things just got to be in place quality management, document control, etc. However, you're the only expert there is on your own business no auditor can say they know better than you in that area.

But you can (actually should) explicitly allow for exceptions to any rule. The management system is the map, and sometimes the real world changes. Can't realy on old maps when that happens. As long as you describe how those exceptions are decided on and managed.

ISO frameworks are just that frameworks. It's easy to create a poor quality management system, and it is quite possible to get it certified. ISO frameworks stops you from omitting important components of the management system, but they don't stop you from choking yourself with them. That one company is ISO 17025-certified does not say anything about the job they produce you have to study their management system to see if it addresses *product* quality. ISO-certification (sich as ISO27001, 9001, and I'm fairly certain also 17025) only tells you that they have a management system, and that they operate in the same manner that this system says they should they have process quality.

Don't confuse product quality and process quality.

For the individual 'lab', absolutely. But as long as the test concerns something that is not lab-specific, it is technically acceptable to make one test, and share the results. A bit like what NIST does with their tests.

The absence of such activity suggests a field of forensic activity that isn't so much a field as a crowd of individuals.

Except that the codes required (created by regulator) basically prevent any digital lab from using methods, even those validated by another agency, without performing further validation. Examples of stupid reasons why include
Do you use the exact same make/model of computer as they do? What about USB docks/writeblockers? If not how do you prove that changing those doesn't affect your method.
Even if you get by with that, they will then start with how can you validate it with your method. If your work instruction isn't exactly the same as that use by the validating lab, you will have to validate it again…and again and again….

As far as your other post about its your own fault if you violate your own policies, UKAS are not allowing people to add sections in like "the examiner should select a suitable method for xyz" as they want the decision method used to determine which method used documented and included as part of the validation.

Short of sausage factory push button forensics, I see no way to comply with the standard and do decent work.

I see no possible way to regulate the digital forensics "lab" without irreparably harming the field.

You cannot regulate where the evidence is found or where the evidence needs to be examined, as most always it is captured (imaged) in the field and many times examined onsite. Once a lab standard has been implemented, any examination outside the "lab" will be called into question simply because not having occurred in the lab, regardless of how perfect an examination was conducted.

The only thing that can be realistically regulated is that of the training/education standards of the examiner.

We have to consider that it is not the tools, nor the pristine four walls of a lab with accreditation plaques that will better the field.

If you have ever seen emergency medicine conducted in the field, then you have the best analogy to this discussion. A competent doctor performing an unexpected emergency surgery on a mountain trail will always surpass that of an incompetent doctor failing at a surgery using the best equipment in the best operating room. It is the person, not the lab or equipment that should be the focus.

I see no possible way to regulate the digital forensics "lab" without irreparably harming the field.

You cannot regulate where the evidence is found or where the evidence needs to be examined, as most always it is captured (imaged) in the field and many times examined onsite. Once a lab standard has been implemented, any examination outside the "lab" will be called into question simply because not having occurred in the lab, regardless of how perfect an examination was conducted.

The only thing that can be realistically regulated is that of the training/education standards of the examiner.

We have to consider that it is not the tools, nor the pristine four walls of a lab with accreditation plaques that will better the field.

If you have ever seen emergency medicine conducted in the field, then you have the best analogy to this discussion. A competent doctor performing an unexpected emergency surgery on a mountain trail will always surpass that of an incompetent doctor failing at a surgery using the best equipment in the best operating room. It is the person, not the lab or equipment that should be the focus.

And here is where the problem lies. These areguements (and indeed many more similar ones) were put forwards and ignored.

Unfortunately the decision and policy makers in the UK are mostly biologists and chemists who don't understand that digital evidence is different to their area of "expertise". They operate in a much more clear-cut world - I pour something into the in-spout on this fancy looking machine and a print out appears telling me that it is ethanol. There simply aren't the same degree of knowns, unknowns and variables in their world as in ours.

Also some of the larger private companies (who already had accreditation for their 'wet' forensics) spoke rather loudly in favour of ISO17025 when the initial consultation was held. They already had accreditation which they could "extend" more cheaply than their unaccredited competition or anyone in public sector could gain from scratch. Unfortunately these were the people who were listened to, over the advice of police practitioners and other public sector (and indeed some wiser private sector) practitioners.

https://www.thetimes.co.uk/article/police-foot-the-bill-after-collapse-of-forensics-firm-key-forensic-services-limited-bg5nbxkxt

Police have been forced into a multimillion-pound bailout of a private forensics company whose collapse jeopardised thousands of cases that include rape and murder, The Times has learnt.

https://www.thetimes.co.uk/article/police-foot-the-bill-after-collapse-of-forensics-firm-key-forensic-services-limited-bg5nbxkxt

Police have been forced into a multimillion-pound bailout of a private forensics company whose collapse jeopardised thousands of cases that include rape and murder, The Times has learnt.

But they were accredited to ISO 17025, how could anything possibly have gone wrong?

https://www.keyforensic.co.uk/police.aspx