Evaluating Results from Automated Systems in Forensic Science

Welcome to this presentation on how to evaluate the results from automated system in forensic science. I’m Timothy Bollé, a PhD student at the University of Lausanne, and Eoghan Casey is my supervisor.

Automated systems are, including a rule-based system and machine learning and deep learning systems, are more and more used in forensic activities and in digital forensic. And one of the questions that’s raised is how to critically evaluate the output of those automated system. Last year, we published a paper where we presented three level of evaluations in order to do this.

  • The first one, the performance evaluation aims at being sure that the system is fit for purpose in regard of the question at hand and on the data.
  • The second level, the understandability evaluation, aims at being sure that the user understand the meaning of the output.
  • And finally, the forensic evaluation, aimed at drawing conclusions from the output.

In the extended abstract, we present some use case scenario and give some detail on how to perform those levels of evaluation, but here I will directly jump to the recommendations.

The first one is that transparency is important, but not only the transparency on how the system works, but also the transparency on how it was tested. Then we realized that users have different level of expertise, meaning that their understanding of a system will be subjective. But there is a need for communication between the developers of the system and between the users to be sure that the needs of the users can be addressed by the system.

Then we wanted to insist on the importance of the forensic evaluation, because it will help formalize the reasoning that will go from the output of the system to the conclusion. And if this reasoning is based on some assumptions, it will help checking those assumptions. At the end, formalizing the reasoning and checking assumptions will result in more robust conclusions.


Get The Latest DFIR News

Join the Forensic Focus newsletter for the best DFIR articles in your inbox every month.

Unsubscribe any time. We respect your privacy - read our privacy policy.


Finally (and maybe this is the most important point of this presentation) is that currently the responsibility of these critical evaluation of outputs is on the user. If the user does not have a sufficient level of expertise, the consequence could be that this critical evaluation is not performed at all, or it’s performed incorrectly. And this could lead to conclusion that are not robust enough, that are easily challenged by decision-maker or lawyers etc. And in the long term, it could lead to a loss of trust and confidence in those automated systems.

Now, if the systems support those level of evaluation that we presented, it will take some of the responsibility from the user and put it on the system. And it will ensure that some critical evaluation is performed systematically and in the long term, it will lead to more robust conclusions and more robust decisions.

Thank you for your attention and feel free to contact us if you have any questions.

Leave a Comment

Latest Videos

Digital Forensics News Round-Up, June 12 2024 #dfir #digitalforensics

Forensic Focus 12th June 2024 5:51 pm

Digital Forensics News Round-Up, June 12 2024 #dfir #digitalforensics

Forensic Focus 12th June 2024 5:39 pm

Internal investigations and eDiscovery face rising challenges in the data collection landscape. There is an urgent need to preserve and analyze data; rising costs for server infrastructure and overhead and the increasing complexity and volume of data from emerging sources is overwhelming. Laptops, computers, phones, tablets, cloud sources, and messaging applications – data is stored anywhere and everywhere with employee communications being the riskiest data sources.

The scope and specific challenges of data collection affect organizations and law firms differently, presenting a need for a variety of solutions to best fit their needs. With Cellebrite’s suite of SaaS (Software-as-a-Service) cloud-based collection solutions, corporate investigators and eDiscovery practitioners can close investigations and get to review faster.

Cellebrite's market-leading SaaS based solutions minimize business disruption and save organizations money by:

- Eliminating the need for large upfront costs and maintenance expenses
- Minimizing overhead costs without hosting the solution, no hardware shipping, and no technical calls for assistance
- Minimal and predictable data collection costs, allowing you to scale your usage according to your specific needs and budgetary considerations
- Stay up to date with continuous updates to data sources with updates pushed to the Cellebrite cloud
- Close investigations and review discovery faster with cloud-based innovation
- Manage customer requests and provide transparency throughout your organization across the globe

Watch Cellebrite's webinar where Monica Harris, Product Business Manager, showcases how Cellebrite’s range of SaaS-based solutions have you covered whether you need remote collection across all devices, including computers, cloud sources, chat applications, and mobile devices or full-file system advanced collection capabilities across the widest range of mobile devices and applications.

Internal investigations and eDiscovery face rising challenges in the data collection landscape. There is an urgent need to preserve and analyze data; rising costs for server infrastructure and overhead and the increasing complexity and volume of data from emerging sources is overwhelming. Laptops, computers, phones, tablets, cloud sources, and messaging applications – data is stored anywhere and everywhere with employee communications being the riskiest data sources.

The scope and specific challenges of data collection affect organizations and law firms differently, presenting a need for a variety of solutions to best fit their needs. With Cellebrite’s suite of SaaS (Software-as-a-Service) cloud-based collection solutions, corporate investigators and eDiscovery practitioners can close investigations and get to review faster.

Cellebrite's market-leading SaaS based solutions minimize business disruption and save organizations money by:

- Eliminating the need for large upfront costs and maintenance expenses
- Minimizing overhead costs without hosting the solution, no hardware shipping, and no technical calls for assistance
- Minimal and predictable data collection costs, allowing you to scale your usage according to your specific needs and budgetary considerations
- Stay up to date with continuous updates to data sources with updates pushed to the Cellebrite cloud
- Close investigations and review discovery faster with cloud-based innovation
- Manage customer requests and provide transparency throughout your organization across the globe

Watch Cellebrite's webinar where Monica Harris, Product Business Manager, showcases how Cellebrite’s range of SaaS-based solutions have you covered whether you need remote collection across all devices, including computers, cloud sources, chat applications, and mobile devices or full-file system advanced collection capabilities across the widest range of mobile devices and applications.

YouTube Video UCQajlJPesqmyWJDN52AZI4Q_SE7Cl5jkigk

Maximising Data Collection With SaaS Innovations

Forensic Focus 10th June 2024 12:42 pm

This error message is only visible to WordPress admins

Important: No API Key Entered.

Many features are not available without adding an API Key. Please go to the YouTube Feeds settings page to add an API key after following these instructions.

Latest Articles