Evaluating Results from Automated Systems in Forensic Science

Welcome to this presentation on how to evaluate the results from automated system in forensic science. I’m Timothy Bollé, a PhD student at the University of Lausanne, and Eoghan Casey is my supervisor.

Automated systems are, including a rule-based system and machine learning and deep learning systems, are more and more used in forensic activities and in digital forensic. And one of the questions that’s raised is how to critically evaluate the output of those automated system. Last year, we published a paper where we presented three level of evaluations in order to do this.

  • The first one, the performance evaluation aims at being sure that the system is fit for purpose in regard of the question at hand and on the data.
  • The second level, the understandability evaluation, aims at being sure that the user understand the meaning of the output.
  • And finally, the forensic evaluation, aimed at drawing conclusions from the output.

In the extended abstract, we present some use case scenario and give some detail on how to perform those levels of evaluation, but here I will directly jump to the recommendations.

The first one is that transparency is important, but not only the transparency on how the system works, but also the transparency on how it was tested. Then we realized that users have different level of expertise, meaning that their understanding of a system will be subjective. But there is a need for communication between the developers of the system and between the users to be sure that the needs of the users can be addressed by the system.

Then we wanted to insist on the importance of the forensic evaluation, because it will help formalize the reasoning that will go from the output of the system to the conclusion. And if this reasoning is based on some assumptions, it will help checking those assumptions. At the end, formalizing the reasoning and checking assumptions will result in more robust conclusions.


Get The Latest DFIR News!

Join the Forensic Focus newsletter for the best DFIR articles in your inbox every month.


Unsubscribe any time. We respect your privacy - read our privacy policy.

Finally (and maybe this is the most important point of this presentation) is that currently the responsibility of these critical evaluation of outputs is on the user. If the user does not have a sufficient level of expertise, the consequence could be that this critical evaluation is not performed at all, or it’s performed incorrectly. And this could lead to conclusion that are not robust enough, that are easily challenged by decision-maker or lawyers etc. And in the long term, it could lead to a loss of trust and confidence in those automated systems.

Now, if the systems support those level of evaluation that we presented, it will take some of the responsibility from the user and put it on the system. And it will ensure that some critical evaluation is performed systematically and in the long term, it will lead to more robust conclusions and more robust decisions.

Thank you for your attention and feel free to contact us if you have any questions.

Leave a Comment

Latest Videos

Magnet Forensics' Matt Suiche on the Rise of e-Crime and Info Stealers

Forensic Focus 12th January 2023 3:00 am

Just like your current holiday shopping for last minute presents a lot of the good stuff has gone off the shelves already. You reach to the back and find the toy nobody really wanted but it’s the thought that counts, you stare down at Si and Desi’s Holiday Special 2022 podcast. 

Please join these two as they lament over the year that was, discuss all the things they didn’t do but promise they will do them next year, query whether putting a NAS in the storage of a roller door is a good idea, and finally arrive at what they’re looking forward to bringing you in the new year.

Show Notes:

Arduino PLC IDE - https://docs.arduino.cc/software/plc-ide
Mycroft Mark II (open source Alexa) - https://www.kickstarter.com/projects/aiforeveryone/mycroft-mark-ii-the-open-voice-assistant
Christa’s new blog - https://christammiller.com/
Si’s holiday reading - https://amzn.to/3iJyGrR
Desi’s holiday reading -  https://inteltechniques.com/
Strange event for the end of the year - https://www.reuters.com/world/europe/25-suspected-members-german-far-right-group-arrested-raids-prosecutors-office-2022-12-07/
Si’s wishful thinking - https://www.youtube.com/watch?v=GXnRgXclLd0
Si’s list to do before the EOY - https://intrepidcamera.co.uk/products/intrepid-4x5-camera
Desi’s list to do before EOY - https://www.wired.com/story/how-to-reset-your-phone-before-you-sell-it/
“Cleaning your office” - https://www.manfrotto.com/uk-en/vintage-collapsible-1-5-x-2-1m-ink-sage-ll-lb5720/
Conference recorder - https://amzn.to/3UBmre5
Desi’s blog - https://www.hardlyadequate.com/

Just like your current holiday shopping for last minute presents a lot of the good stuff has gone off the shelves already. You reach to the back and find the toy nobody really wanted but it’s the thought that counts, you stare down at Si and Desi’s Holiday Special 2022 podcast.

Please join these two as they lament over the year that was, discuss all the things they didn’t do but promise they will do them next year, query whether putting a NAS in the storage of a roller door is a good idea, and finally arrive at what they’re looking forward to bringing you in the new year.

Show Notes:

Arduino PLC IDE - https://docs.arduino.cc/software/plc-ide
Mycroft Mark II (open source Alexa) - https://www.kickstarter.com/projects/aiforeveryone/mycroft-mark-ii-the-open-voice-assistant
Christa’s new blog - https://christammiller.com/
Si’s holiday reading - https://amzn.to/3iJyGrR
Desi’s holiday reading - https://inteltechniques.com/
Strange event for the end of the year - https://www.reuters.com/world/europe/25-suspected-members-german-far-right-group-arrested-raids-prosecutors-office-2022-12-07/
Si’s wishful thinking - https://www.youtube.com/watch?v=GXnRgXclLd0
Si’s list to do before the EOY - https://intrepidcamera.co.uk/products/intrepid-4x5-camera
Desi’s list to do before EOY - https://www.wired.com/story/how-to-reset-your-phone-before-you-sell-it/
“Cleaning your office” - https://www.manfrotto.com/uk-en/vintage-collapsible-1-5-x-2-1m-ink-sage-ll-lb5720/
Conference recorder - https://amzn.to/3UBmre5
Desi’s blog - https://www.hardlyadequate.com/

YouTube Video UCQajlJPesqmyWJDN52AZI4Q_BhrBg5_sAKo

Si and Desi Holiday Special 2022

Forensic Focus 16th December 2022 12:00 am

This error message is only visible to WordPress admins

Important: No API Key Entered.

Many features are not available without adding an API Key. Please go to the YouTube Feed settings page to add an API key after following these instructions.

Latest Articles

Share to...