Evaluating Results from Automated Systems in Forensic Science

Welcome to this presentation on how to evaluate the results from automated system in forensic science. I’m Timothy Bollé, a PhD student at the University of Lausanne, and Eoghan Casey is my supervisor.

Automated systems are, including a rule-based system and machine learning and deep learning systems, are more and more used in forensic activities and in digital forensic. And one of the questions that’s raised is how to critically evaluate the output of those automated system. Last year, we published a paper where we presented three level of evaluations in order to do this.

  • The first one, the performance evaluation aims at being sure that the system is fit for purpose in regard of the question at hand and on the data.
  • The second level, the understandability evaluation, aims at being sure that the user understand the meaning of the output.
  • And finally, the forensic evaluation, aimed at drawing conclusions from the output.

In the extended abstract, we present some use case scenario and give some detail on how to perform those levels of evaluation, but here I will directly jump to the recommendations.

The first one is that transparency is important, but not only the transparency on how the system works, but also the transparency on how it was tested. Then we realized that users have different level of expertise, meaning that their understanding of a system will be subjective. But there is a need for communication between the developers of the system and between the users to be sure that the needs of the users can be addressed by the system.

Then we wanted to insist on the importance of the forensic evaluation, because it will help formalize the reasoning that will go from the output of the system to the conclusion. And if this reasoning is based on some assumptions, it will help checking those assumptions. At the end, formalizing the reasoning and checking assumptions will result in more robust conclusions.


Get The Latest DFIR News

Join the Forensic Focus newsletter for the best DFIR articles in your inbox every month.


Unsubscribe any time. We respect your privacy - read our privacy policy.

Finally (and maybe this is the most important point of this presentation) is that currently the responsibility of these critical evaluation of outputs is on the user. If the user does not have a sufficient level of expertise, the consequence could be that this critical evaluation is not performed at all, or it’s performed incorrectly. And this could lead to conclusion that are not robust enough, that are easily challenged by decision-maker or lawyers etc. And in the long term, it could lead to a loss of trust and confidence in those automated systems.

Now, if the systems support those level of evaluation that we presented, it will take some of the responsibility from the user and put it on the system. And it will ensure that some critical evaluation is performed systematically and in the long term, it will lead to more robust conclusions and more robust decisions.

Thank you for your attention and feel free to contact us if you have any questions.

Leave a Comment

Latest Videos

Digital Forensics News Round Up, February 28 2024 #digitalforensics #dfir

Forensic Focus 29th February 2024 4:58 pm

Digital Forensics News Round-Up, February 21 2024 #digitalforensics #dfir

Forensic Focus 21st February 2024 6:19 pm

Alan Platt, Professional Services Consultant at MSAB, discusses his experience as a former UK police officer working in digital forensics. He talks about the different levels of digital forensics capabilities within police forces and how MSAB products like XAMN and XEC Director are used by frontline officers versus lab analysts. 

The discussion covers how MSAB partners with law enforcement to develop custom workflows for mobile device acquisitions that facilitate ISO compliance. Alan explains MSAB's managed service offering, where approved MSAB staff can remotely access a customer's XEC Director server to assist with software updates and troubleshooting. He emphasizes the strict data segregation policies enforced by customers to prevent MSAB from accessing any sensitive case data.

Looking ahead, Alan mentions MSAB's new CEO and hints at some exciting developments coming down the pipeline. He spotlights recent enhancements to XEC Director's speed and database functionality for managing large estates of networked Kiosks. Alan also plugs the new XEC Director training he created to help users fully leverage the platform's capabilities.

00:00 – Introduction to Alan Platt
07:00 – Training
12:00 – Workflows
17:20 – Ensuring a secure environment
19:45 – Customer training
20:35 – Helping customers comply with ISO accreditation
25:00 – Validation and verification
27:30 – ISO standards
30:00 – MSAB’s pipeline plans
32:40 – XEC Director 
43:45 – Privacy of user data

Alan Platt, Professional Services Consultant at MSAB, discusses his experience as a former UK police officer working in digital forensics. He talks about the different levels of digital forensics capabilities within police forces and how MSAB products like XAMN and XEC Director are used by frontline officers versus lab analysts.

The discussion covers how MSAB partners with law enforcement to develop custom workflows for mobile device acquisitions that facilitate ISO compliance. Alan explains MSAB's managed service offering, where approved MSAB staff can remotely access a customer's XEC Director server to assist with software updates and troubleshooting. He emphasizes the strict data segregation policies enforced by customers to prevent MSAB from accessing any sensitive case data.

Looking ahead, Alan mentions MSAB's new CEO and hints at some exciting developments coming down the pipeline. He spotlights recent enhancements to XEC Director's speed and database functionality for managing large estates of networked Kiosks. Alan also plugs the new XEC Director training he created to help users fully leverage the platform's capabilities.

00:00 – Introduction to Alan Platt
07:00 – Training
12:00 – Workflows
17:20 – Ensuring a secure environment
19:45 – Customer training
20:35 – Helping customers comply with ISO accreditation
25:00 – Validation and verification
27:30 – ISO standards
30:00 – MSAB’s pipeline plans
32:40 – XEC Director
43:45 – Privacy of user data

YouTube Video UCQajlJPesqmyWJDN52AZI4Q_ifoHVkjJtRc

How MSAB Is Managing The Digital Forensics Challenges Of Frontline Policing

Forensic Focus 21st February 2024 3:07 pm

This error message is only visible to WordPress admins

Important: No API Key Entered.

Many features are not available without adding an API Key. Please go to the YouTube Feed settings page to add an API key after following these instructions.

Latest Articles