ISO 17025 for Digit...
 
Notifications
Clear all

ISO 17025 for Digital Forensics – Yay or Nay?

Page 3 / 9
athulin
(@athulin)
Community Legend

As onsite response generally requires on-the-spot decision making, innovative and untested responses to handle new threats, violating a lab policy will be the norm.

If it is, it's the fault of that lab itself, and noone else.

As far as I know, no quality management framework (ISO 17025 is one) makes any requirements on location, business model, style of office wear or facial hair. They do tell you what is mandatory, and require you to live up to that, as well as the other rules and policies you create for yourself. But they do not do anything else.

If you create your own ISO 17025-based system that requires you to do your lab work in a designated location, you have made your decision about how you want or need to work. If you say 'we do *all* our work in our own localities' you have to live up to it – and can't collect any information in the field. But then you've painted yourself into that particular corner.

So … don't do that. Or don't do it in a way that makes it an issue for an ISO 17025 auditor.
Separate field collection of material from other work, for example, and make it clear what rules and policies apply for those situations, and what rules and policies apply when you're back in the 'lab'.

ISO 17025 is used by assayers, for example they, too, may have to collect samples in the field, and not always under ideal circumstances.

I still think it's extremely unwise to try to create a complete ISO 17025-compliant system from the start. Better do the absolute minimum that is needed, by the standard and by business-critical customers. It's usually a larger effort than imagined to get an organization to work smoothly under any new rule it is not good business sense to make that effort larger than absolutely necessary – and one large part of that is educational. And by that that I mean educational for those who write the rules as well as for those who are expected to follow them.

ReplyQuote
Posted : 27/01/2018 8:32 am
Merriora
(@merriora)
Junior Member

Sometimes, I wonder if the industry itself is partly to blame. The language that we choose to use around the industry is broadly in line with previously existing forms of forensics. By aligning digital forensics so closely to other forms of forensics, we have been "lumped in" to other lab based forensic practices whilst, in "the real World", we are very very different and really require our own standard.

The debate concerning mandatory standards pre-supposes that there is a relevant and meaningful standard available to apply (with quaified people to oversea the process).

I think this is an interesting question as I have observed confusion within our industry around the idea that we are a subset of regular forensics.

I've spoken with some that argue we aren't really a subset of forensics since we are so different in how data is acquired and analyzed.

Does this cause us more confusion and issues in working towards a standard since many people both within our field and outside (regulators) always try to put us with regular Forensics rather than seeing us as a different discipline?

PS ever since using 17025 was first mooted by the regulator years ago, I dont think I have met one colleague within the industry who has said "yay, 17025, great idea". Almost the exact opposite. So do I trust the mass consensus within the industry (including many voices that I have huge respect for) or do I trust the regulator?

Playing devil's advocate…(not that I disagree)
Who likes regulation? How often do you hear anyone affected by a law (or standard) say they are happy that the law exists?

Is there anyone working within an ISO certified environment that is happy with the additional regulations?

Regulations, 'red-tape', bureaucracy all exist. No one likes working in those environments, but they do exist for a reason and job satisfaction and happiness is not one.

As far as I know, no quality management framework (ISO 17025 is one) makes any requirements on location, business model, style of office wear or facial hair. They do tell you what is mandatory, and require you to live up to that, as well as the other rules and policies you create for yourself. But they do not do anything else.

This is interesting because I think many, myself included, see ISO regulations as being very rigid, but your comments suggest that this is not the case.

My purpose of playing devil's advocate is to ensure we have valid reasons against 17025, not because it simply makes our job harder. As the survey (referenced in the article) discusses, less than 25% of those surveyed had a high or clear understanding of the details involved in ISO 17025.

How many regulators have a clear understanding of the issues within Digital Forensics and can adapt the guidelines to meet our needs within the industry?

Is our lack of understanding the details around 17025 the true issue with this standard?

I think any regulation (ISO 17025, New Standard, existing standard) can and will make things harder to start until it becomes common practice within our industry.

For those working in ISO 17025 labs, can you discuss exact issues you've seen with this regulation that you can confidently state is an issue rather than a possible misunderstanding of the requirement either by your lab or by the regulator (who may not fully understand Digital Forensics)?

ReplyQuote
Topic starter Posted : 27/01/2018 3:34 pm
Merriora
(@merriora)
Junior Member

I still think it's extremely unwise to try to create a complete ISO 17025-compliant system from the start. Better do the absolute minimum that is needed, by the standard and by business-critical customers. It's usually a larger effort than imagined to get an organization to work smoothly under any new rule it is not good business sense to make that effort larger than absolutely necessary – and one large part of that is educational. And by that that I mean educational for those who write the rules as well as for those who are expected to follow them.

I fully agree with this comment. Creating a new standard from scratch will be a very time-consuming effort which will require lots of support from various organizations.

Hopefully, these discussions either show that we as a community are clear on what is needed or we find a way to adapt to existing standards while working closely with those that write the rules for these standards so that they fully understand our field.

ReplyQuote
Topic starter Posted : 27/01/2018 3:40 pm
MDCR
 MDCR
(@mdcr)
Active Member

I still think it's extremely unwise to try to create a complete ISO 17025-compliant system from the start. Better do the absolute minimum that is needed, by the standard and by business-critical customers. It's usually a larger effort than imagined to get an organization to work smoothly under any new rule it is not good business sense to make that effort larger than absolutely necessary – and one large part of that is educational. And by that that I mean educational for those who write the rules as well as for those who are expected to follow them.

That is funny, because compliance is a minimum level of conformity. It does not go deep into specific things that are mission critical and affect organisations differently in respect to their operating environment. It's basically a pair of cheap socks with a label that says "one size fits all" on it.

Shoving a generic standard down the throat of specialist organisations and expecting it to do magic is setting yourself up for a costly disappointment that does not amount to much, the paper people who are unable to achieve anything but paperwork should go away and screw with someone else's lives. I've lived in such a paper regime for years and it achieved nothing pragmatic or anything that increased security. There were still gaping security holes in systems and the only thing that was done to remain compliant was to write a piece of paper that acknowledged that "Yes we are aware of the problem and are not gonna do jack s**t about it".

This is what Schneier called CYA security (Cover your a*s security) so they can point to someone to fire if SHTF - it is not a measure of quality or security. Compliance people it's the kind of useless people you want to send to the front lines in a war.

ReplyQuote
Posted : 27/01/2018 5:47 pm
athulin
(@athulin)
Community Legend

Is there anyone working within an ISO certified environment that is happy with the additional regulations?

Yes … and no. I've worked with ISO 9001, and was part of getting the company I worked in ISO 9001-certified. ISO 9001 was written for manufacturing industry at that time, not for software developers and consultants.

Yes, I liked the result. After certification, we had a far better project platform than we ever had before, we knew roughly how to build on it, and in what direction we were going with it. We had control of project documentation for one thing …

No, I didn't like it – we made a lot of mistakes that took a long time to recover from. Changing a management system can be like making an oil tanker turn … But the mistakes were all ours, and in about 90% of the cases because we thought we could do better than the people who helped us thought we could.

Yes, I liked it. We got rid of lots and lots of local and historical variations how projects were managed, we got reasonably well written 'SOPs' that helped new employees get up to speed much faster than was possible earlier … as well as keep older hands on the straight and narrow instead of relying on memory and failing.

No, I didn't like it. We got the certification, but we didn't renew it. We were spent we couldn't operate at the level that we thought we could. And we didn't want to back down to make it easier on ourselves. (We had for some reason decided that we would get certified in one year. We did it, but at a price. If we had paced ourselves for a five- or perhaps a ten-year effort, I think it would have been better.)

In the end, though, I liked it more than I disliked it.

This is interesting because I think many, myself included, see ISO regulations as being very rigid, but your comments suggest that this is not the case.

See above. I started out with much the same flak spiel I see here myself. ("We're tech consultants and software engineers for crying out loud – what does a manufacturing standard have to do with us? Nothing!" and so on. )

Parts of ISO frameworks are rigid some things just got to be in place quality management, document control, etc. However, you're the only expert there is on your own business no auditor can say they know better than you in that area.

But you can (actually should) explicitly allow for exceptions to any rule. The management system is the map, and sometimes the real world changes. Can't realy on old maps when that happens. As long as you describe how those exceptions are decided on and managed.

ISO frameworks are just that frameworks. It's easy to create a poor quality management system, and it is quite possible to get it certified. ISO frameworks stops you from omitting important components of the management system, but they don't stop you from choking yourself with them. That one company is ISO 17025-certified does not say anything about the job they produce you have to study their management system to see if it addresses *product* quality. ISO-certification (sich as ISO27001, 9001, and I'm fairly certain also 17025) only tells you that they have a management system, and that they operate in the same manner that this system says they should they have process quality.

Don't confuse product quality and process quality.

ReplyQuote
Posted : 27/01/2018 9:28 pm
minime2k9
(@minime2k9)
Active Member

For the individual 'lab', absolutely. But as long as the test concerns something that is not lab-specific, it is technically acceptable to make one test, and share the results. A bit like what NIST does with their tests.

The absence of such activity suggests a field of forensic activity that isn't so much a field as a crowd of individuals.

Except that the codes required (created by regulator) basically prevent any digital lab from using methods, even those validated by another agency, without performing further validation. Examples of stupid reasons why include
Do you use the exact same make/model of computer as they do? What about USB docks/writeblockers? If not how do you prove that changing those doesn't affect your method.
Even if you get by with that, they will then start with how can you validate it with your method. If your work instruction isn't exactly the same as that use by the validating lab, you will have to validate it again…and again and again….

As far as your other post about its your own fault if you violate your own policies, UKAS are not allowing people to add sections in like "the examiner should select a suitable method for xyz" as they want the decision method used to determine which method used documented and included as part of the validation.

Short of sausage factory push button forensics, I see no way to comply with the standard and do decent work.

ReplyQuote
Posted : 30/01/2018 4:43 pm
bshavers
(@bshavers)
Active Member

I see no possible way to regulate the digital forensics "lab" without irreparably harming the field.

You cannot regulate where the evidence is found or where the evidence needs to be examined, as most always it is captured (imaged) in the field and many times examined onsite. Once a lab standard has been implemented, any examination outside the "lab" will be called into question simply because not having occurred in the lab, regardless of how perfect an examination was conducted.

The only thing that can be realistically regulated is that of the training/education standards of the examiner.

We have to consider that it is not the tools, nor the pristine four walls of a lab with accreditation plaques that will better the field.

If you have ever seen emergency medicine conducted in the field, then you have the best analogy to this discussion. A competent doctor performing an unexpected emergency surgery on a mountain trail will always surpass that of an incompetent doctor failing at a surgery using the best equipment in the best operating room. It is the person, not the lab or equipment that should be the focus.

ReplyQuote
Posted : 30/01/2018 5:37 pm
benfindlay
(@benfindlay)
Active Member

I see no possible way to regulate the digital forensics "lab" without irreparably harming the field.

You cannot regulate where the evidence is found or where the evidence needs to be examined, as most always it is captured (imaged) in the field and many times examined onsite. Once a lab standard has been implemented, any examination outside the "lab" will be called into question simply because not having occurred in the lab, regardless of how perfect an examination was conducted.

The only thing that can be realistically regulated is that of the training/education standards of the examiner.

We have to consider that it is not the tools, nor the pristine four walls of a lab with accreditation plaques that will better the field.

If you have ever seen emergency medicine conducted in the field, then you have the best analogy to this discussion. A competent doctor performing an unexpected emergency surgery on a mountain trail will always surpass that of an incompetent doctor failing at a surgery using the best equipment in the best operating room. It is the person, not the lab or equipment that should be the focus.

And here is where the problem lies. These areguements (and indeed many more similar ones) were put forwards and ignored.

Unfortunately the decision and policy makers in the UK are mostly biologists and chemists who don't understand that digital evidence is different to their area of "expertise". They operate in a much more clear-cut world - I pour something into the in-spout on this fancy looking machine and a print out appears telling me that it is ethanol. There simply aren't the same degree of knowns, unknowns and variables in their world as in ours.

Also some of the larger private companies (who already had accreditation for their 'wet' forensics) spoke rather loudly in favour of ISO17025 when the initial consultation was held. They already had accreditation which they could "extend" more cheaply than their unaccredited competition or anyone in public sector could gain from scratch. Unfortunately these were the people who were listened to, over the advice of police practitioners and other public sector (and indeed some wiser private sector) practitioners.

ReplyQuote
Posted : 30/01/2018 7:36 pm
pbeardmore
(@pbeardmore)
Active Member

https://www.thetimes.co.uk/article/police-foot-the-bill-after-collapse-of-forensics-firm-key-forensic-services-limited-bg5nbxkxt

Police have been forced into a multimillion-pound bailout of a private forensics company whose collapse jeopardised thousands of cases that include rape and murder, The Times has learnt.

ReplyQuote
Posted : 31/01/2018 8:58 am
minime2k9
(@minime2k9)
Active Member

https://www.thetimes.co.uk/article/police-foot-the-bill-after-collapse-of-forensics-firm-key-forensic-services-limited-bg5nbxkxt

Police have been forced into a multimillion-pound bailout of a private forensics company whose collapse jeopardised thousands of cases that include rape and murder, The Times has learnt.

But they were accredited to ISO 17025, how could anything possibly have gone wrong?

https://www.keyforensic.co.uk/police.aspx

ReplyQuote
Posted : 31/01/2018 10:05 am
pbeardmore
(@pbeardmore)
Active Member

By insisting that Police forces could only outsource to 17025 labs, I think it was used as leverage in an attempt to get labs to go down this route on a "market forces" route without the regulator having formal powers.

That strategy surely has to be questioned now as this is hardly a great advert. So, in future we have either 17025 as a legal requirement or a review of the situation? or just tread water for a few years?

ReplyQuote
Posted : 31/01/2018 10:25 am
bshavers
(@bshavers)
Active Member

https://www.thetimes.co.uk/article/police-foot-the-bill-after-collapse-of-forensics-firm-key-forensic-services-limited-bg5nbxkxt

Police have been forced into a multimillion-pound bailout of a private forensics company whose collapse jeopardised thousands of cases that include rape and murder, The Times has learnt.

What's the relevance with the article?

ReplyQuote
Posted : 31/01/2018 11:03 pm
Merriora
(@merriora)
Junior Member

I see no possible way to regulate the digital forensics "lab" without irreparably harming the field.

The only thing that can be realistically regulated is that of the training/education standards of the examiner.

If you have ever seen emergency medicine conducted in the field, then you have the best analogy to this discussion. A competent doctor performing an unexpected emergency surgery on a mountain trail will always surpass that of an incompetent doctor failing at a surgery using the best equipment in the best operating room. It is the person, not the lab or equipment that should be the focus.

I think this is a really good example which clearly shows that we must first worry about ensuring the examiner is competent and well trained before we worry about the environment they are working in.

As it has been stated here by many others, DFIR can happen in any location, not only in pristine labs that are accredited. Unless the ISO 17025 standard is flexible, which doesn't appear to be the case, very few labs will be able to operate effectively.

ReplyQuote
Topic starter Posted : 01/02/2018 4:37 am
Brevs11
(@brevs11)
New Member

https://www.thetimes.co.uk/article/police-foot-the-bill-after-collapse-of-forensics-firm-key-forensic-services-limited-bg5nbxkxt

Police have been forced into a multimillion-pound bailout of a private forensics company whose collapse jeopardised thousands of cases that include rape and murder, The Times has learnt.

What's the relevance with the article?

As far as I know, Key Forensics did not undertake traditional Digital Forensics rather, 'Forensic Video and Image Analysis Services'. According to the UKAS website they are not accredited for this type of work or more traditional digital forensics work either.

I do not see the link between the 'failure' of Key Forensics and whether or not ISO 17025 is an appropriate standard in digital forensics either.

Key Forensics are however accredited to ISO 17025 in a significant number of other forensic disciplines (21 page scope). I would imagine that this required a significant amount of time, money and resources as well as a huge amount of hard work to achieve.

I don't see any link between Key Forensics current difficulties and accreditation.

There will also be lots of employees at Key Forensics extremely worried about their jobs and livelihoods and I wish them all the best.

ReplyQuote
Posted : 01/02/2018 10:20 am
steve862
(@steve862)
Active Member

Hi,

I thought I would chirp in again.

In response to the location issue, 17025 isn't intended to be flexible in this respect. This is a principal difference between it and ISO 17020.

When I worked in a pharmaceutical lab, different substances used in tests had to be kept in different environments. We even had a special low humidity store for certain highly reactive substances. Environment was crucial and you couldn't conduct any meaningful tests outside of that physical environment. If equipment were moved, very sensitive moving parts might have been affected and the unit would need calibrating. As has been expressed in this thread, the environment isn't the issue for us.

17020 is being proposed for scene attendance work, with a date of 2020 to become compliant. Although we can work to '17025 methodologies' out in the field right now, it's a bit of a fudge to say that work is in any way compliant with 17025 because of what 17025 was written for.

Possibly there is room within 17020 for digital forensics but i still think we are looking at a lot of adaptations for it to provide tangible improvements.

Right now funding is the greatest threat to the quality of forensics in the CJS. Key Forensics is a concerning situation and other smaller companies have either fallen by the wayside already or pulled out of the CJS market entirely. Unlike fingerprint and DNA, there is a huge corporate market for digital investigations.

RTS shows us that ISO 17025 cannot guarantee quality or reliability but is costly for companies to attain. Clearly many companies have thought to themselves why spend this money if we don't have to?

Micro companies make up the bulk of digital experts who do defence work and they simply cannot afford to attain ISO 17025. I know a lot of people who used to do defence work but have decided not to accept any more CJS work, not at £72 /hr and certainly not at £72 /hr and having to fund 17025 from it.

The loss of these defence experts concerns me greatly and what will happen when a prosecution report is produced, the defendant is permitted legal aid but then can't find a defence expert? If they can, what kind of defence expert will they get at that price? One who cannot afford to fund forensic tools and/or training quite possibly.

Issues of disclosure are the current hot topic. Quantities and the complexity of data to assess is the biggest problem, followed by a lack of training for disclosure officers in dealing with communications and open source data and digital evidence. Training is also a funding issue.

I think the Regulator is going to find it harder to push for statutory powers when the vast majority of potential miscarriages of justice involving forensics were carried out by an accredited laboratory and when funding is causing forensic providers and police units to fail and in more than one way.

2018 will be an interesting year.

Steve

ReplyQuote
Posted : 01/02/2018 10:31 am
Page 3 / 9
Share: