ISO 17025 for Digit...
 
Notifications
Clear all

ISO 17025 for Digital Forensics – Yay or Nay?

Page 4 / 9
trewmte
(@trewmte)
Community Legend

Hi,

I thought I would chirp in again.

In response to the location issue, 17025 isn't intended to be flexible in this respect. This is a principal difference between it and ISO 17020.

When I worked in a pharmaceutical lab, different substances used in tests had to be kept in different environments. We even had a special low humidity store for certain highly reactive substances. Environment was crucial and you couldn't conduct any meaningful tests outside of that physical environment. If equipment were moved, very sensitive moving parts might have been affected and the unit would need calibrating. As has been expressed in this thread, the environment isn't the issue for us.

17020 is being proposed for scene attendance work, with a date of 2020 to become compliant. Although we can work to '17025 methodologies' out in the field right now, it's a bit of a fudge to say that work is in any way compliant with 17025 because of what 17025 was written for.

Possibly there is room within 17020 for digital forensics but i still think we are looking at a lot of adaptations for it to provide tangible improvements.

Right now funding is the greatest threat to the quality of forensics in the CJS. Key Forensics is a concerning situation and other smaller companies have either fallen by the wayside already or pulled out of the CJS market entirely. Unlike fingerprint and DNA, there is a huge corporate market for digital investigations.

RTS shows us that ISO 17025 cannot guarantee quality or reliability but is costly for companies to attain. Clearly many companies have thought to themselves why spend this money if we don't have to?

Micro companies make up the bulk of digital experts who do defence work and they simply cannot afford to attain ISO 17025. I know a lot of people who used to do defence work but have decided not to accept any more CJS work, not at £72 /hr and certainly not at £72 /hr and having to fund 17025 from it.

The loss of these defence experts concerns me greatly and what will happen when a prosecution report is produced, the defendant is permitted legal aid but then can't find a defence expert? If they can, what kind of defence expert will they get at that price? One who cannot afford to fund forensic tools and/or training quite possibly.

Issues of disclosure are the current hot topic. Quantities and the complexity of data to assess is the biggest problem, followed by a lack of training for disclosure officers in dealing with communications and open source data and digital evidence. Training is also a funding issue.

I think the Regulator is going to find it harder to push for statutory powers when the vast majority of potential miscarriages of justice involving forensics were carried out by an accredited laboratory and when funding is causing forensic providers and police units to fail and in more than one way.

2018 will be an interesting year.

Steve

Steve as always a very sensible and highly reasoned opinion from you. Some of the single-person advisors and micro business are still around but the CJS digital forensic market is much smaller. The impact of that is why case reviews of criminal cases is taking place due to inappropriate lack of evidential scrutiny.

ReplyQuote
Posted : 01/02/2018 10:42 am
pbeardmore
(@pbeardmore)
Active Member

Yes, Steve, spot on,

slight tangent but it's interesting that, on forums and wider, we talk about the CJS and the Corporate Market as if there is a clear dividing line with no crossover. We all know that crime exists in all walks of life and, in the real World, there is no dividing line with Corporate investigations uncovering criminal evidence, civil cases escalating to criminal, etc etc,

I'm interested in how any possible future legal requirement for 17025 would deal with this scenario.

ReplyQuote
Posted : 01/02/2018 2:50 pm
steve862
(@steve862)
Active Member

Yes, Steve, spot on,

slight tangent but it's interesting that, on forums and wider, we talk about the CJS and the Corporate Market as if there is a clear dividing line with no crossover. We all know that crime exists in all walks of life and, in the real World, there is no dividing line with Corporate investigations uncovering criminal evidence, civil cases escalating to criminal, etc etc,

I'm interested in how any possible future legal requirement for 17025 would deal with this scenario.

Indeed, this is probably something the Regulator hasn't thought of.

Even big due diligence projects where company A wants to buy company B could result in a criminal matter coming to light during an e-Discovery process.

I know a lot of people advising on corporate security where they started out in digital forensics within the CJS. They've tried to move outside of 17025's reach, but perhaps they haven't succeeded after all!

Steve

ReplyQuote
Posted : 01/02/2018 3:30 pm
pbeardmore
(@pbeardmore)
Active Member

"Indeed, this is probably something the Regulator hasn't thought of."

Not sure if you're are being ironic -)

How much time have they had? If 17025 does become a legal requirement, defence lawyers will jump on any occasion when data was captured. stored , viewed etc by non-17025 parties. And when exactly does a civil case become criminal? We all know it's a complex issue.

ReplyQuote
Posted : 01/02/2018 3:54 pm
bshavers
(@bshavers)
Active Member

There is literally, practically, and virtually no difference in "doing forensics" using a laptop in a conference room of a business on a remote island and that of "doing forensics" in an ISO certified lab in an underground bunker. The only real difference is that of the examiner, in that the examiner either follows or does not follow community accepted analysis guidelines. Even the guidelines are recommendations since every approach to a computer system may be different from the last and different from the next one, each requiring a slightly different response using different tools and methods. Even the goals to each response are different.

Electronic evidence is not like any other evidence; there is nothing even close to it. Electronic evidence is virtual. You can't touch it. You can't see it. When preserved, you can examine it, duplicate it, replicate it, and transmit ad infinitum without ever changing it one bit. We only see the interpretation of electronic data via software and hardware.

This is the crux of the matter.

Digital evidence is unlike other other evidence. Other evidence types need regulation to reduce spoliation and provide accurate tests results which rely on a controlled physical environment.

I fear that some in the DFIR field may want DFIR to be treated as if it were like scientific fields for the sake of notoriety (as in, "I am a scientist!") without understanding that regulating the physical work environment of digital forensics would be as effective as regulating the movement of clouds (real clouds, not computer servers…). You can make regulations, but they won't work because the DFIR field will be unable to comply with them.

ReplyQuote
Posted : 01/02/2018 6:11 pm
pbeardmore
(@pbeardmore)
Active Member

Great post IMHO. All the points are well made. And I don't think you have to be an experienced expert in the field to grasp these points. So why has the regulator bundled digital forensics with conventional forensics?

ReplyQuote
Posted : 02/02/2018 11:17 am
minime2k9
(@minime2k9)
Active Member

Great post IMHO. All the points are well made. And I don't think you have to be an experienced expert in the field to grasp these points. So why has the regulator bundled digital forensics with conventional forensics?

Because the people sitting on the "digital" board and advising have no experience in digital.
For example the lead for the Police is someone from Stafford shire who has no actual experience with digital forensics and their background in wet forensics.

The issue usually is, that in police forces, digital is relatively new and sits in the forensics department. The head is usually an ex CSI/Fingerprint/other form of wet forensics who has no idea. All the heads from across the country get consulted and go "Yeah seems like a good idea" without knowing anything.

Then when anyone else raises issues, they are just accused by the regulator of being 'difficult'

ReplyQuote
Posted : 02/02/2018 11:55 am
pbeardmore
(@pbeardmore)
Active Member

But does that mean that we, collectively, have failed to get the message across? or was it always a "done deal"?

ReplyQuote
Posted : 02/02/2018 12:05 pm
minime2k9
(@minime2k9)
Active Member

I think it means we have been ignored in favor of managers at a higher level who made the decision without useful knowledge.

ReplyQuote
Posted : 02/02/2018 12:33 pm
pcstopper18
(@pcstopper18)
Member

I have been following this discussion and I have not had time to generate a response and add to the conversation until now. I do want to encourage everyone here that this is a good discussion and commend everyone on the sincere and thoughtful responses thus far. Pardon the length.

Context for myself
I am in the US. I currently work in an ISO 17025 accredited, public sector forensic lab that is an LE agency but not a police department. It is an independent local government corporation, which is unique in this country. Prior to this, I worked for an LE federal lab under the DoD that was also ISO 17025 accredited. And lastly, prior to that I worked for a big 4 financial firm that was not accredited under ISO 17025.

My quick response to the question at hand - Is ISO 17025 adequate for use as an accreditation standard for digital forensics? I say NO, egregiously so. (If you don't want that rest, stop reading here ) )

This ISO was not designed, nor ever intended, for the discipline of digital forensics. If I was in the UK I would have put that position forward as much as I could. Here in the US, the National Commission on Forensic Science (NCFS), which has since been disbanded by the current US Attorney General, made some recommendations along these lines and there was plenty of public comment against it. Feel free to check it out for yourself comments and all - https://www.regulations.gov/document?D=DOJ-LA-2016-0002-0002

Having said that, I also submit there is nothing wrong with the concept of accreditation. The idea of accreditation is that you make policies for yourself/organization and someone other than you holds you to those policies and enforces accountability if you violate them. That idea should be welcome by everyone in the field. I would support a minimum standard that was actually designed from the ground up to be for digital forensics. It should be created by a cohort of practitioners and other stakeholders (not to outnumber the actual practitioners) be written in plain language, and free to obtain. Many also need training and support because actual accreditation is not as daunting some would purport, provided you do not paint your own organization into a corner with the policies you write, which is usually what happens. The cost will vary based on different factors such as organization size and services provided. That is the fact. The time and cost involved depends. It is not one-size-fits-all. Outside of this, only unchallenged government bureaucracy will make it worse…and has in many instances.

Now, the current bureaucratic nonsense that is the accreditation industry has taken advantage of this idea and we are where we are. We are having to combat that, as well as those who seem to want to have input into/control of a subject matter for which they have no direct knowledge, qualification, or background. The only way in my mind to combat this is through consensus among practitioners and major entities in the field and using the consensus to create a singular body, and push to gain formal recognition as a self-governing body. The consensus needs to be more than just complaining, or sitting idly by and not getting involved at least in some form. Too many people have an opinion to the contrary, yet have a stake in the status quo because consensus would dictate that they themselves have to adapt, just as unwanted accreditation would also have them adapt. With the truth becoming evident that they just don’t want to adapt at all.

I submit that the immediate focus should be on the certification of practitioners. Brett and others have promoted such an idea. If this can be accomplished, then ISO or national type standards will be easier to implement from the ground up because there will be a cohesive body behind them, one that holds all practitioners accountable for their work. In focusing on the practitioner, I believe a model akin to the American Medical Association or American Bar Association and their British/European equivalents be adopted. By addressing the practitioner directly, the largest hurdles to consensus can be mitigated, such as tool/training vendor dependence for example. With buy in from, and submission to, such an idea, others can see that they can keep what they have already invested in and still be a party to the larger whole, especially since they can’t avoid it in the end anyway. I heard someone say it this way, “If we don’t do it, the government will.” This is what can be seen in Britain with the Regulator and such.

As others have said, digital forensics is not like, nor can it be like, the other forensic fields. All efforts to make it so are ignorant and ill-advised in my opinion. Just stop it. You can support goals of common sense and even scientific rigor and it not be a carbon copy of disciplines like DNA or toxicology (a rant for another day).

Hope this adds to the discussion.

ReplyQuote
Posted : 02/02/2018 4:39 pm
Merriora
(@merriora)
Junior Member

This conversation is becoming a truly epic discussion on reasons for and mostly against ISO 17025. This will help other regions & countries come to a decision on how they should proceed with accreditation and the issues to consider.

I truly appreciate and thank everyone for taking part in this discussion and survey so far.

Prior to this discussion, I thought that ISO 17025 was perhaps just misunderstood which resulted in DFE’s fighting against the standard out of fear of the unknown. These discussions have shown that although there is a lack of understanding, this isn’t the fault of the DFE’s or labs, but rather in how the standard is written and the fact that it just doesn’t seem to fit with Digital Forensics.

Correct me if I’m wrong, but frustration by many practitioners is then further increased as a result of having to deal with Inspectors for adherence to the ISO Standard who are also unclear of how the standard fits within Digital Forensics when faced with new situations and questions.

It was suggested that ISO 17025 was chosen based on

1. Decision makers with experience in regular forensics, but no experience in Digital Forensics, sitting on boards and wrongly assuming that Digital Forensics operates in the same manner as ‘wet’ forensics.

2. Large Labs, already ISO 17025 accredited, pushing for this accreditation as they knew they would have an advantage in the marketplace due to how difficult it would be for non-accredited labs to attain ISO 17025 standards.

Voted ISO 17025

Out of 38 votes cast so far, only 3 believe that ISO 17025 is the correct standard.

For those that voted for 17025, would you be able to clearly address the issues and concerns presented so far in this thread?

- Can you analyze and acquire data outside of the lab?
- Do you need to test all software releases prior to use?
- If testing is involved, how much extra time does this take?
- Are you a private or public lab?
- How many people do you have in your lab?
- What key aspect is everyone missing that is voting against 17025?

Voted Standards are Not Required

So far, 6 people have voted stating that Standards are not required.

- Can you explain why you feel this is the case?
- If no standards exist either for the lab or examiner, how do we reduce the number of ‘experts’ with no training or experience?
- Why do you feel so strongly that standards are not required?

Hopefully, by the end of this discussion, we as a community, will have a clear idea of the best path for our profession.

ReplyQuote
Topic starter Posted : 02/02/2018 4:54 pm
Merriora
(@merriora)
Junior Member

Context for myself
I am in the US. I currently work in an ISO 17025 accredited, public sector forensic lab that is an LE agency but not a police department. It is an independent local government corporation, which is unique in this country. Prior to this, I worked for an LE federal lab under the DoD that was also ISO 17025 accredited. And lastly, prior to that I worked for a big 4 financial firm that was not accredited under ISO 17025.

My quick response to the question at hand - Is ISO 17025 adequate for use as an accreditation standard for digital forensics? I say NO, egregiously so. (If you don't want that rest, stop reading here ) )

This ISO was not designed, nor ever intended, for the discipline of digital forensics.

I also submit there is nothing wrong with the concept of accreditation. The idea of accreditation is that you make policies for yourself/organization and someone other than you holds you to those policies and enforces accountability if you violate them. That idea should be welcome by everyone in the field.

The consensus needs to be more than just complaining,

I submit that the immediate focus should be on the certification of practitioners. Brett and others have promoted such an idea.

I heard someone say it this way, “If we don’t do it, the government will.”

As others have said, digital forensics is not like, nor can it be like, the other forensic fields. All efforts to make it so are ignorant and ill-advised in my opinion. …

Thank you Preston for this very informative post. You hit on a lot of points that I strongly agree with (highlighted above).

ReplyQuote
Topic starter Posted : 02/02/2018 5:04 pm
athulin
(@athulin)
Community Legend

As it has been stated here by many others, DFIR can happen in any location, not only in pristine labs that are accredited. Unless the ISO 17025 standard is flexible, which doesn't appear to be the case, very few labs will be able to operate effectively.

Does ISO 17025 make any difficult or inflexible requirements? Sometimes 'ínflexible' just is another word for 'I don't want to do that'. It might be useful to ensure that that is not what we're discussing here.

In the following, I'm only concerned with environmental requirements.

ISO 17025 does require a working environment that does not affect the methods you're specified to follow. The actual limits of such environment conditions are up to each lab to set down. So as far as the standard is concerned, you need to show that you follow what you preach. Nothing more. (Unless your customer complains. In which case you need to listen to them, and possibly perform some corrective action. That's often a bore …)

Questions might perhaps be raised about extracting evidence, say, from hard drives in an environment that does not exceed recommended operating conditions … so perhaps the precautions and guidelines expressed by HDD manufactures (such as secure drive in a chassis, or keep on padded, grounded, antistatic surface, operate at a temperature of 5° C to 55° C, not above 10,000 feet – or below -200 feet – , and within 5% to 95% non-condensing relative humidity, etc.) may become important. (I would need my own weather station to get a written record of all that …)

But ISO doesn't require that you set your own limits, or possibly none at all (ponder point below). As long as you follow your own rules, you're fine. Well, yeah, … you have to listen to your customer as well.

For my own part, the standard HDD operation requirement that stipulates that you must not drop, bump or jar the drive when it is operating could be a bit of a problem. Having to prove that I didn't do either might be tricky – a Maxtor requirement I've noted was less than '30 G during 2.0 ms'. Perhaps there's some 'fast' G-force sticker I could use to show I didn't jar, bump or drop worse than that?

So … does digital evidence handling really differs from other evidence? Or is it just that these requirements are rarely difficult to fulfil – though proving them fulfilled may be tricky. (And accidents do happen …) Or is it that we don't really want to have to prove that we didn't exceed any of those limits? Perhaps we don't even want to have to think about how clumsy we occasionally can be? (I have at least one situation where a G-sticker on a HDD might have busted a job …) But then, do we have any research into the kind of errors to arise out of bumping and jarring? (Perhaps the HDD manufacturers do?) Or extracting an image in the rain? Or in a house at the Dead Sea? We do seem to know that relative humidity can be a disk killer in certain situations …

In the absence of any hard science, the method could easily specify that it's up to the examiner to decide if evidence collection can be performed without risk of environmental damage to data.

(Interesting point to ponder 'Digital Forensic Processing and Procedures', which I think claims to fulfill ISO 17025, does not seem to have any requirements on these lines – though I could easily have missed it, not having a 'grep' function handy. Hm … what could that mean?)

ReplyQuote
Posted : 02/02/2018 5:58 pm
minime2k9
(@minime2k9)
Active Member

In the following, I'm only concerned with environmental requirements.

ISO 17025 does require a working environment that does not affect the methods you're specified to follow. The actual limits of such environment conditions are up to each lab to set down. So as far as the standard is concerned, you need to show that you follow what you preach. Nothing more. (Unless your customer complains. In which case you need to listen to them, and possibly perform some corrective action. That's often a bore …)

Except that during more than 1 accreditation visit at different locations across the UK, people have been asked how their environment protects against solar flare activity and other such things.

Even specifying relatively straightforward requirements, such as the temperature shall be within tempertaures as you described in your post, then becomes an admin nightmare. How do you know the temperature is within that range? How accurate is the device you use to measure the temperature? How often is it calibrated?

In the absence of any hard science, the method could easily specify that it's up to the examiner to decide if evidence collection can be performed without risk of environmental damage to data.

Again assessments have shown they heavily push back on anything that isn't rigidly documented. Trying to explain that we have several methods for imaging and it was up to the examiner to decide the most appropriate was not acceptable and they required that the use of different methods and reasons should be documented as part of the procedure.

Whether you have had different assessors also seems to be a major factor, but the rosy "just do what your documents state and you'll be fine" scenario you seem to give seems massively false.

ReplyQuote
Posted : 03/02/2018 7:18 am
athulin
(@athulin)
Community Legend

Except that during more than 1 accreditation visit at different locations across the UK, people have been asked how their environment protects against solar flare activity and other such things.

I'd might guess that in their traditional calibration lab environment that is a real issue, so of course they'll ask it. Sensitive measuring devices would likely be affected by power and current variations produced by solar storms, so some form of protection against spikes, and other noise variations on power lines might be called for.

But computer-based operations? … I'd guess it's the power grid itself that's the big factor. If that goes, you're not likely to produce bad test results except by hand … in the dark. But how do spikes and other random power line noise or even direct radiation affect computation so that test result quality might suffer without anyone noticing? I'd like to see a study of that before I even bothered to implement any particular countermeasures.

Hm … I find a reference to a Google study (http//www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf), but it suggests that soft bit flips for any reason are so rare (i.e. uncorrectable) that ECC RAM may be the best way to detect that it has happened at all. Other factors, such as motherboards seem to be greater offenders. (Hard bit errors should be caught by other means… and yes, I have had one situation in which an image persistently failed until I memtest86-ed the system, and replaced the failed memory bank. But that was DOS-based imaging long ago.)

(There are some references to other relevant studies there "Alpha-particle-induced soft
errors in dynamic memories", for example. Looks like I have some reading to do.)

Using ECC memory might be a reasonable way to at least detect when errors happen. (How does one verify that it works, though? Get an alpha-particle source?)

ven specifying relatively straightforward requirements, such as the temperature shall be within tempertaures as you described in your post, then becomes an admin nightmare. How do you know the temperature is within that range? How accurate is the device you use to measure the temperature? How often is it calibrated?

You need to calibrate anything, you ask a certified calibration lab to help. They'll have recommendations for calibration ranges, number of data points, and frequency of calbration occasions, depending on how important correct temperature or humidity is.

However, if all you really need to stay within a safe range of 5-55 degrees centigrade, your thermometer does not need to be very accurate, as long as you add some margins – say use a range of 15 - 40 degrees C measured by a thermometer that may have up to 5 degrees centigrade error, say, when compared with calibration master. (I find one such instruction here, for example http//www.usf.edu/research-innovation/comparative-medicine/documents/sops/s1115-thermometer-hygrometer-calibration.pdf)

Trying to explain that we have several methods for imaging and it was up to the examiner to decide the most appropriate was not acceptable and they required that the use of different methods and reasons should be documented as part of the procedure.

Leaving them undocumented is clearly not according to 17025 requirements. (At least that's how I interpret what you are saying, and what 17025 says.)

Whether you have had different assessors also seems to be a major factor, but the rosy "just do what your documents state and you'll be fine" scenario you seem to give seems massively false.

Well, I was rather assuming that the mandatory part of the standards requirements were fulfilled, and the question was about what the indvidual methods or procedures actually said. But if the methods/procedures aren't documented in the first place, that doesen't matter you fail early.

ReplyQuote
Posted : 03/02/2018 11:02 am
Page 4 / 9
Share: