The sequence of bia...
 
Notifications
Clear all

The sequence of bias in the UK

23 Posts
5 Users
0 Likes
2,770 Views
(@amutimer)
Posts: 14
Active Member
Topic starter
 

Hi Jamie,

Yes, I am not a DFIR professional; my background is in inspection and testing; starting in a lab and ending my career as the quality director for one of five arms of the second largest international inspection and testing company in the world. I retired from that and became a contractor to UKAS. 

I have worked rather hard to gain a good knowledge of the relevant law, criminal procedures, PACE and police methods, the operation of digital forensic software, and of course I know the processes of inspection and testing well and have implemented 9001/2, 17020 and 17025 on large scales. I am also an expert in statistical techniques.

I hope nobody thinks I am boasting saying any of this. In the normal course of things I would not have said any of it. I hate being in situations where people throw their weight around. I feel I have had to say it because people simply dismissed the original post

So, why did I join here?

You most probably know that the UK Forensic Science Regulator's new Code of Practice is out for comment in its second draft and I have been asked by him to comment on it. I want those comments to be as useful as they possibly can be to the cause of justice. I came here because, having analysed the *systems* at great length I believe I understand the flaws in them - especially the flaws in what we could call the overview. The reason I hope(d?) for dicussion here is I need the knowledge of actual *users and operators* of those systems.

So, if I am unqualified to be here, or if my reasons for being here aren't good enough, or they are likely to take discussion in the group outside its proper limits please just say so and I will leave. I certainly won't stay here if I have the sense that I am being tolerated.

But, if you think what I have to say could be useful or interesting and falls within the proper limits of the group then I would be glad to lay out some ideas, BUT, if it is OK with everyone, not until I have explored them *individually*. My first post was waaaaay too long and contained waaaay too may contentious statements (regardless of their accuracy!) and could not really be anmswered properly. So my preference is to consider the ideas I have peicemeal, one at a time, in order, in separate posts. Will that work?

 

 
Posted : 18/07/2022 12:28 pm
(@rich2005)
Posts: 535
Honorable Member
 

amutimer, I'll speak bluntly, since you've already tried to claim I'm misleading people by suggesting your original post was wrong, and then doubled down on that by saying the next person with decades of experience is wrong too!

Instead of just telling everyone they're wrong, perhaps listen to them, and you might actually find that some of your concerns (about ISO17025 not making any difference) might well be shared (or find people who wish to go further in explaining many more reasons why it's poorly designed / unsuitable for digital forensics in its current form).

If you read back your original post you'd hopefully see that it's pretty disparaging to most digital forensic examiners and/or police officers involved in these cases. Even the thread title itself is hardly an encouragement to engage for those involved in the field and for the most part probably doing a conscientious/diligent job day in day out.

That's not to say that there's never been a case done to a pretty weak standard by someone and made it to court. Whilst more of my work has been for the prosecution over the years. I've been instructed as an expert for the defence in just such a case, in this type of work, and seen it dropped in the first moments of the trial because the evidence would simply not bear any scrutiny.

If you want people to explain their main issues with the state of digital forensics in this country (and/or ISO), I think you'll actually find a lot of people wanting to give their two pennies worth, although they might want to do it do a "work" email address of yours (so they know they're not wasting their time talking to someone with a grievance about DF), and they might want to do it anonymously (so that they can speak more freely and tell you what you really want to know).

 
Posted : 18/07/2022 4:38 pm
Jamie
(@jamie)
Posts: 1288
Moderator
 
Posted by: @amutimer

So my preference is to consider the ideas I have peicemeal, one at a time, in order, in separate posts. Will that work?

It will not. Should you wish to solicit any further responses which meet the requirements I referred to previously you should do so succinctly and in this topic only. If that is unacceptable for any reason, it may be that this is not the most appropriate forum for your enquiries.

 
Posted : 18/07/2022 4:48 pm
(@amutimer)
Posts: 14
Active Member
Topic starter
 

@rich2005 

Hi Rich, can I call you that?

Yes I am very interested to hear about the views of people here about 17025.

I first realised there was doubt about the applicability of the standard when I read the consultation paper that was put out before the decision to give the FSR statutory powers. In addition to the statistics the writer produced quotes from people who had given input. One said that digital forensics was unlike other branches of forensics and warned that a one-size-fits-all approach would be a mistake. I think - I don't know - that was a reference to ISO 17025.

My fear is that the FSR may be running down the wrong road and there may be much cheaper ways to get improvements and get them much quicker.  

So Sir, please say, what are your misgivings about 17025?

Please understand, if you tell me that you don't think UKAS can do the job, you won't offend me in the least - I am concerned about exactly the same thing!   

As to going to email, I have no UKAS email address because I am a contractor not a staffer. The only email address I have is my own - the one I gave the group. But, if it is possible I would like to keep it here. The question is whether the mod will allow it.

Regards

Adrian

 

 

 
Posted : 18/07/2022 5:29 pm
(@amutimer)
Posts: 14
Active Member
Topic starter
 

@rich2005 

Hi Rich, can I call you that?

Yes I am very interested to hear about the views of people here about 17025.

I first realised there was doubt about the applicability of the standard when I read the consultation paper that was put out before the decision to give the FSR statutory powers. In addition to the statistics the writer produced quotes from people who had given input. One said that digital forensics was unlike other branches of forensics and warned that a one-size-fits-all approach would be a mistake. I think - I don't know - that was a reference to ISO 17025.

My fear is that the FSR may be running down the wrong road and there may be much cheaper ways to get improvements and get them much quicker.  

So Sir, please say, what are your misgivings about 17025?

Please understand, if you tell me that you don't think UKAS can do the job, you won't offend me in the least - I am concerned about exactly the same thing!   

As to going to email, I have no UKAS email address because I am a contractor not a staffer. The only email address I have is my own - the one I gave the group. But, if it is possible I would like to keep it here. The question is whether the mod will allow it.

Regards

Adrian

 

 

 
Posted : 18/07/2022 5:30 pm
(@rich2005)
Posts: 535
Honorable Member
 

I'll type this relatively quickly as it's currently 9000 degrees...

In short, I think I can summarise the "issues" with standards/regulation in digital forensics as follows:

  1. Almost everyone would agree there is some degree of need for some kind of standards/regulation in digital forensics (although often it has seemingly arisen as a result of knee j**k responses to issues that never related to standards or tool testing - rather things like untrained officers "dabbling" in digital forensics - and other incidents I'm too hot to remember right now!)
  2. Historic (and failed) attempts have been made to address this (CRFP) and more recently ISO17025 (not popular)
  3. The issues can broadly be broken down into
    1. Competence of examiners
    2. Time given to examiners to do the job vs rigid process following
    3. Competence/reliability of tools
  4. Competence with examiners isn't an easy thing to assess, everyone will have a different view on it, but there are undoubtedly many highly competent, examiners, some with a degree, some without, some with a load of tool/training provider certificates, some without. Experience is an important factor in this field and few in the field would assess their colleagues' ability based on any of these and certainly not as a limiting factor in most cases. It might be the case that there should be a required certification for examiners in the future, or levels of, to perform certain roles. However this probably isn't the most pressing factor and I'd say most labs, public or private, allocate people reasonably well based on the ability they see. It's not perfect but is far less important than points 2 or 3 above.
  5. Time given to examiners to do a job is critical. This probably varies the most in the industry and the more attempts are made to "streamline" activities, or cut costs, it inevitably will be heading towards pushing an examiner down the simplest path to the "quick win" or a process following yes/no answer. Where you're wrong above is that in most instances that examiner will likely be desperately trying to be as fair as possible within the time available, not only for their own conscience' sake, but because nobody likes a grilling in court, should things not stack up with their evidence. This inevitably comes down to a resource issue. With digital forensics being prevalent in almost every instance of crime yet is still not really resourced as such and standards lumped on it from people with backgrounds in other areas of forensics (seemingly with the mindset of we can get a simple testing regime set up, pump data through tools, and output the reliable results - anyone who knows anything about digital forensics, knows how far from realistic this is - due to the variety of data types/artefacts/how often they change/interact/and so on and so forth).
  6. Which leads onto 3) and the whole tool testing argument of ISO17025. The giant, and false, assumption is that the giant expense of this will make tools materially more reliable, and simultaneously that by rigid process following, you will somehow avert miscarriages of justice. I think you could make a pretty strong case that by spending all this money in a largely futile effort to test tools, you divert money away from examiners spending more time examining a case properly, and the process, in its current form, will not make those tools within a million miles of being extremely reliable. Virtually every forensic tool I use, from all the major providers, I am constantly finding problems with, and go into the queue(s) of development waiting to be fixed. All of these tools being subject to endless testing by ISO17025 following labs. Some of the testing might find an error. Much of it may not and will rely on the examiner "paying attention". The reality of this testing (with vast amounts of duplication involved - and more than a smattering of testing for the sake of testing to pass certification) is that it is not meaningfully improving tool reliability and this should be acknowledged as an extremely difficult task, and not the solution to the reliability of evidence in digital forensics. It is quite simply something that should be addressed at a national level, ideally in conjunction with tool vendors, against real-world data sets, by dedicated teams of people performing the task for the country (or world) as a whole, funded in the (many) millions. It is this important...however the way it's done currently is bluntly, in my view, a giant waste of money spread across many many labs. Even then, it would not lead to the mana from heaven, of a "find evidence button", as you say, but at the very least it would start to cut down of time wasted for a forensic examiner, trying to work round a tool not doing what it should.

A footnote to that would be that digital forensics knowledge and guidance is poorly shared across law enforcement and industry. This is reasonable if it's a sensitive or covert tactic. However there's many things, like the legal basis for cloud/remote collections, as an example, where law-enforcement guidance exists (to a degree) but even that is very far from clear both for what LE actors should do and even more so for non-LE.

As I can feel the heat getting to me I won't waffle any further but suffice to say it's my view that ISO17025 is entirely a waste of money in its current form. It should be rethought from the perspective of what problem is trying to be solved and think of the best way of doing it. Rather than we need to apply a load of standards to everything and try to justify why later (it's hard to bar saying testing might find a bug). If you think of it in this way, things are still extremely challenging, but become simpler.

If none of that made any sense...I blame the heat...for the third and final time.

 
Posted : 18/07/2022 6:18 pm
(@amutimer)
Posts: 14
Active Member
Topic starter
 

@rich2005 

Rich, despite the heat you gave a great answer. 

I am a UKAS man so you should expect that I think 17025 is the dog's todger, and in some situations it surely is, but as we stand right now in digital forensics in the UK, 17025 is not going to take us anywhere good. Actually, I see the potential for a real disaster ahead if we stick with it. I will explain more about what I think I see ahead if you want me to. 

Just as you say, the first thing is the money - the police/labs are simply not given enough of it - and no amount of talking or paper writing or intellectual superstructures like 17025 are going to change anything unless the money changes. The pressure must be taken off the technicians.  

As to tool testing - validation - again I believe you are right. The tools do fail, you only have to look on this group to see technicans tearing their hair out because they narrowly avoided a screw-up caused by the failure of the tools. This does not happen to anything like the same extent in most other areas of forensic science because for many tests validation can be a once-for-all-time thing. But applications and programmes are changed constantly. My answer is the same as yours. The validation should be done super-nationally and it should be done by the manufacturers. In this regard I have pressed Magnet AXIOM to talk about this issue with me, but they won't because I am not part of a government agency. Maybe they would talk to you?...The thing is, the FSR CoP document will probably become Bible by November, so if good input is to be given, now is most definitely the time. In case you don't know, you can give commentary directly to the FSR on the Code - anyone can...the email address to do it is at the end of the Code itself.

Like you, I am dying here. If you are on good terms with God could you ask him to turn the thermostat down? He seems to have forgotten this is Britain - we don't do 40 degrees here...

Rich, when it cools down I will put to you and anyone else who is still with this thread an idea I have to take the pressure off the labs.

And, if you can, please forgive my tone in my original post. I have just reread it and I am embarrassed. No excuses.

A 

 

 

 

 

 

 

 
Posted : 18/07/2022 9:09 pm
(@amutimer)
Posts: 14
Active Member
Topic starter
 

I am aware there is a rule here about answering your own posts. But, mod, please recognise, all I am doing here is extending this thread and with new information. I am not trying to bump.

So, Rich, and anyone else who is still with this. Will the following idea work?

The essential background to this is to understand a very odd conversation that goes on in the offices of solicitors. They get a phone call from Mr X. whose property has been searched and whose devices have been confiscated and he is looking for representation in a case of making of indecent images. The solicitor will typically act professionally and appear to believe every word he hears, but actually, although he notes it all down for future use, he does not believe what he hears. The reasons are several, but the short story is the solictor knows that people who commit this crime nearly always lie to him. So when he asks how many indecent images are there, the solictor will hear "there aren't any images", or "one, or two...I think....maybe". When he asks, "did you ever search for this kind of material?" he hears "no, never". etc. At that point the lawyer will say, "OK, we will need to wait for a copy of the evidence file from the CPS before I can advise you".

When the file arrives, quite concievably a year later, there is another conversation. In this conversation the lawyer says "Mr X. you told me there was was only one image - but there are 500...You told me there were no searches but we can see 20 of them". At this point he asks if Mr. X can explain what the solicitor can see and almost always MR. X will start to tell the truth, at last some realism enters the conversation and the lawyer can give some well founded advice. Most typically Mr. X wil hear "your best bet is to save your money, save your time, save your agony waiting for the court, keep your penalty credit and plead guilty".

The problem with this scenario is that by then the police, the lab and the CPS have put in a lot of time and effort and a lot of public money has been spent. Can the system be short-circuited?

My suggestion is that the lab works in two stages. In the first stage all the lab does is mount the drive(s), extract all the live images (not the ones bound to software, of course), their metadata, file paths and the searches and copy the lot to a cloud server. Bespoke software is made avialable to do all this efficiently. A key is then made available to the defence lawyer. You can see that the lawyer will now be able to have the second, realistic conversation first, and he will be able to do so before he spends more time, before the lab has to put in any serious thinking, before the interview under caution, before the report to the CPS, before the CPS review it all and prepare the evidence file, before the court registrar is bothered etc. Everyone in the chain is worked less, or not at all. 

The effectivenes if the idea rests on a set of ratios that will be apparent to readers. But I suggest those ratios are very favorable because about 95% of the time the initial intelligence that gave rise to the search is indictive of real crime. The main work will only happen in say 5-10% of cases.   

Now consider the case of the 5% who genuinely aren't guilty. In these cases the evidence from the lab's first stage will not be conclusive and in this case the lawyer gets to start exploring alternative theories with his client. So let us say the evidence is of 5 images along with a set of .lnk files with dates that span the last five years and with incriminating target titles but with actual targets unfound. The position is not clear. The lawyer can reasonably ask his client "did you ever perform a backup of some one else's machine?"..."did you ever buy a second hand drive and fail to wipe it? etc. If Mr X and the lawyer happen upon some possibilities they can arrange a meeting of experts as envisaged in 19 of the Criminal Procedure Rules and in the case of agreement between the experts, the system is short circuited. In this meeting the most likely thing is that the MAC addresses embedded in the .lnk files will be examined and tell both parties which machine made them. If Mr. X is innocent not only will this emerge, but a trail of evidence leading to the real perpetrator can be followed.  

It's not just the savings in time and money; justice is much better served this way. The guilty confess early and the innocent get to the truth...all before the trial, which trial will probably never take place because there will, I suggest, be very few occasions when the two experts do not agree - maybe none...      

What are the faults with this? Could this idea work in relation to other crimes?

 

A    

 

 
Posted : 19/07/2022 8:58 am
(@jerryw)
Posts: 56
Trusted Member
 

@amutimer Who is the defence lawyer and how are they paid, if the subject hasn't even been interviewed?

How are defence lawyers expected to be able to understand the technical implications of the material provided to them? While some are very technically knowledgeable, many are not (to put it mildly).

Your suggestions seem to overlook the accusatorial nature of the criminal justice system.

While most on here are happy to consider beneficial specific technical changes to working practices, your proposals appear far too wide-reaching and more suited to an academic thesis than brief discussion points. You may be in a better position to evaluate your thoughts once you have received and disseminated the results of your Freedom of Information requests.

It may be that these are discussions better held with your employer, with regard to your concerns over the efficacy of current accreditation.

 
Posted : 19/07/2022 1:05 pm
(@amutimer)
Posts: 14
Active Member
Topic starter
 

@jerryw

HI Jerry

See interspersed.

Who is the defence lawyer and how are they paid, if the subject hasn't even been interviewed?

[ADrian - If a person is actually guilty, he is likely to hire a lawyer after the search. He *knows* he's in big trouble. The innocent guy probably won't. But he forms only 5% of the suspects.] 

How are defence lawyers expected to be able to understand the technical implications of the material provided to them?

[ADrian - After the first pass by the lab the information is not yet very technical. The lawyer will have the images, their file paths (and a diagramatic file schema to make sense of them), their metadata and the searches.]  

While some are very technically knowledgeable, many are not (to put it mildly).

[Adrian - Yes. I have found the technical knowledge of lawyers who are happy to tell you they are competent in relation to thsi crime is often realy bad. But, they *can* understand an indecent image when it is set before them...]

Your suggestions seem to overlook the accusatorial nature of the criminal justice system.

[Adrian - Could you expand on this?] 

While most on here are happy to consider beneficial specific technical changes to working practices, your proposals appear far too wide-reaching and more suited to an academic thesis than brief discussion points.

You may be in a better position to evaluate your thoughts once you have received and disseminated the results of your Freedom of Information requests.

It may be that these are discussions better held with your employer, with regard to your concerns over the efficacy of current accreditation.

[Adrian - Believe me, I have tried...]

[Adrian - Guys, having had the input I have and having read back in the group some, I have formed the view that my stuff probably needs a different group. If you know of one I'd love to hear from you. What I am looking for is a place where some of the members are technically competent in the lab, know 17025 etc. but where the range of discussion can take in the entire process of the UK CJS. 

Best A]     

 

 
Posted : 20/07/2022 7:41 am
Page 2 / 3
Share: