Christa: Digital video is everywhere, but the complexities and challenges of using it as evidence in legal proceedings continue to advance. This week, Simon Biles and I talk to Martino Jerian, Founder and CEO of Amped Software. This is the Forensic Focus Podcast. Welcome, Martino, thanks for taking this time today.
Martino: Hi, Christa. Hi, Simon, and thanks for having me.
Simon: Hi. No, I’m personally really excited. We had a chat at the Euro Expo and I think this product is fascinating and I am deliriously happy to sit and listen to you talk about it more. So this is a definite opportunity for me. I’m thrilled.
Martino: Great. Thank you.
Christa: So I’m going to start off with, my first question is, there’s a lot of educating the general public, and I think, you know, not just on your blog, but just in general about video evidence, including issues around manipulation, deepfakes are the big thing in the news right now obviously.
You’ve also mentioned misattribution and speed estimation and just so many different variables. What are some awareness challenges that are actually unique to policy makers and other high-level stakeholders?
Martino: I think at the root the biggest issue, which basically it’s at the foundation of all the other issues, is a kind of misunderstanding. Most of the people just think that video is easy, just because everybody can hit play on YouTube, look at vacation pictures, I mean, who needs technical skills to analyze a video?
But at the end, digital images and videos are a kind of representation of matrices and what’s behind them, it’s pretty complex mathematical concepts. So if you don’t understand that you can get things very wrong, especially on the forensic side, as long as you’re speaking about photos and movies, I mean, little details don’t matter, but can make a huge difference in court.
And especially, I think the issue is that power of a video. And because if you think nowadays between videos on social media, CCTV, pictures from a mobile phone there are very few cases when there is not a video or an image. And from an image or video, often you can reply to the various questions of the five W investigation model: who, where, why, when, how, and and so we have a lot of responsibility to handle this kind of evidence.
Christa: So, how does that translate to like, I mean, there’s educating sort of I guess, like I said, the general public that might end up on a jury, for example. But then there’s also the policy makers and others that are, I guess, judges would be a part of that group that need to maybe have different understandings, right? So what’s kind of unique, I guess, to those groups?
Martino: I think, again, it’s creating a culture at a wide level, the problem, because as a society, we give a lot of power to images and videos. And finally the people also start to recognize deepfakes and other stuff that you mentioned before that we cannot always trust. I think for the younger generation, they don’t trust images and videos as much as the older generation does.
Christa: Yeah.
Martino: And so it’s nowadays maybe is not, I mean, it’s like writing a text, they see a video on Instagram, say, “Yeah, maybe it’s real, maybe not, who cares? It’s nice.” This is the point.
So when we started creating this awareness, we started with training many years ago experts, okay? Users of let’s say high-hand software. And so these people had a technical background and we were kind of missing the gaps with the training.
Then we started expanding our user base with products like Amped Replay which is still a software for, let’s say working on video evidence, but for investigators, for responders. So people who have less technical ground. So we kind of also expanded our training and our language. And we have a training called “Investigating Video Evidence,” which is teaching the basics. I mean, what you need to be aware of, then if the case is complex, you need to call an expert of course.
And then we realized that there’s much more. I mean, there are not just people who are working on the case, there is the jury, the public opinion, there is the judge, the attorney, and very often even judge, prosecutors, they really lack understanding of the issues surrounding this word. So we thought we needed to take one more step and go wider. And this is a lot of the activities that I do nowadays.
Christa: Yeah, I remember I interviewed some prosecutors for some research that I was doing not that long ago, but one of the things that they raised was, you know, that sort of balance between explaining to a jury what was going on and making sure they understood the science behind it versus putting them to sleep. So there was definitely that sort of strategy that they needed to have the technical background to be able to decide how far to go with it, really.
Martino: Yeah. It’s, you know, the challenge is that often you need to explain pretty complex mathematical or technical concepts in a simple way. And then if you make it too simple, you can get it wrong. And then the other side can say, “No, you said something wrong.”
And this is the difficulty also because very often in court, let’s say this is a typical situation which happens. And maybe there is one expert who does a very technically good and reliable analysis, which oftentimes unfortunately is inconclusive because sometimes you don’t have data.
Then there is the other side who gives a much stronger and different opinion. “Yes, we can see it’s him or it’s not him” or whatever. And maybe the second guy who technically didn’t do a very nice job is just saying what his customer wants to say. Maybe he is much better at speaking and so he can convince the other side, because, I mean, again, judges, they don’t have the technical background, so they cannot judge how to rely more than their gut often. And this is a big issue for our legal systems.
Christa: Well, it reminds me also of there was the line in the the National Institute of Standards and Technology, the NIST report that came out recently that also was that two completely different examiners can arrive at completely different results and both be right.
So there’s that sort of wrinkle as well as in addition to what you’re describing. So kind of on that note, I mean, one of the blog posts you wrote recently mentioned international guidelines for video evidence. What are some of the barriers to the adoption of those guidelines?
Martino: Again, awareness probably. Knowing that they exist. And again, misconceptions about the kind of video. Other kinds of evidence, other kinds of stuff which are being worked on in forensic labs. Think about DNA, but even, I mean, mobile forensics, computer forensics, there is a more standardized workflow.
With video and images. let’s say intuitively, let’s say, ah, every case is different. One time I have to do one thing, another time or another thing, which is true, but overall, the issues and the methodology that you should adopt should be coherent. You should have a predefined workflow or at least a framework for the different situations that can happen because it’s the scientific method that we should follow.
So it should be accurate as much as possible, free from bias errors and stuff like that. It should be repeatable, so if I do the same processing analysis in one week, only one error should be the same, and should be reproducible, and other experts should get the same result as I do, at least from the technical level, of course. And because otherwise it’s not scientific evidence and it’s very, very hard to bring it in court and support it.
Simon: What’s unique about video evidence in that regard, though? Because what you’ve just said is brilliant best practice for handling any evidence that’s going to court. You know, that’s what we do on the digital front. That’s what one would hope people do when they’re dealing with blood or whatever wet stuff they do in that disgusting world.
So what is it that’s unique about video evidence or image evidence that might subtly alter the way that one would understand the scientific process? Are there any particular gotchas, any particular things that make it unique?
Martino: I think again, maybe I’m a bit boring, but I think it’s the main problem, as I told you, awareness. Just think about, I mean, CSI kind of fiction. From the DNA lab arriving with the white lab coat, very professional. And when there is to work with video, a random investigator just screams at the screen, “Hey, zoom in on that. Crop.”
So I think this is very meaningful of the attitude. In a similar fashion, I mean, we do a lot of training and sometimes it’s a problem to find the budget for the training, even though they don’t have issues finding a budget for forensic training, which is actually much more expensive than ours, just because they think for that they need a tool and the training and the people, but for video, again, everybody can hit play. That’s always the basic issue.
And if you go into something let’s say more technical, I think probably it’s a slightly more technical yeah, a kind of different approach to authenticity and integrity of data, okay? So these are two concepts which are often confused, but they are quite different.
So integrity or originality means that some data is an original error and already since the time of acquisition, okay? Authenticity means that it’s a truthful representation of what it purports to be.
And images and videos, sometimes they are not the same because you may have a video which has been, I don’t know, sent via WhatsApp or Facebook or whatever. So it’s not the original anymore, but it’s still, hopefully we could have the original and would be the best opportunity, but you can consider it even if it’s been downloaded from social media and it’s the only kind of evidence. So it’s not the original, but still good.
And then there is the concept of authenticity. Sometimes you have an unoriginal image which does not represent the truth. So for example, when you’re using it out of context, in a fake profile, in taking a picture from a previous event, and this also goes into another aspect, which relates to, let’s say, the enhancement of images and videos, which is something which we often do.
Let’s say, so you process images because you change the brightness, you correct the blur, you distort the image. So, the image is not original anymore in a sense, but what we are doing with a scientific process is obtaining an image which is more authentic in a sense.
I’ll make you a practical example. When you take a picture with a wide-angle lens, you have walls, which are straight lines, which become curved. I think you have seen it many times. So if you want to obtain a more thoughtful representation of the scene, you need to correct the lens distortion, okay?
The image is not original, but it’s more authentic in a sense, and this is underpinning the entire process of enhancement and analysis of images and videos, because you always want to get something which is more accurate than the original image.
Simon: Okay, yeah, that makes sense. Yeah, so I mean, the CSI/zoom enhanced thing is a real issue. And it’s the same across all of the forensic industry, as far as I can tell there seems to be this disillusion between what the police think is possible and what is actually possible. So I have sympathy for you.
With regards to sort of your training or more to the point, qualified people, I mean, in other areas of the industry, we’re seeing a severe lack of trained people. I assume it’s exactly the same for you, possibly worse. Are there, you know, less qualified or less trained examiners than there should be or are required?
Martino: Yeah, I definitely think so. And again, some people are put there without the right, let’s say, preparation, unfortunately. But I also see an improvement on that side and I mean, there are a lot of people in our community which is really great. They do an amazing job, I must say, but still we need more.
Simon: Okay. I mean, you guys do training and, you know, I’m looking forward to, I have an opportunity to come on some of your courses later this year, and we’ll talk about that after that fact and we’ll be talking about that on Forensic Focus and talking about your training.
But if somebody wanted to come along and get into the industry, what would your recommendations be for being able to do it? I mean, obviously your training courses and those are the best, but apart from that, how should people start to look at this?
Martino: I think it’s probably a bit dangerous to start real cases straight away. There is a lot of information available online, of course. If you look at our blog, I mean, I started it like 14 years ago or so, we have almost 600 articles, I think. I mean, we tend to not, I mean, there are software updates and usual let’s say specific stuff, but also much, let’s say, general topics, even for people who are not our customers, they want to understand a bit more the issues and stuff like that.
I think it’s important to go side by side on the, let’s say, forensic side, the image processing theoretical side and the practical side, because, I mean, using the tools, just tweaking a few sliders, I mean, it’s not bringing you anywhere or even bringing you in the wrong directions, doing just theory without the practice is useless and of course you need to put the forensic factor inside it.
So you should go in parallel in these three lines. And I think it’s important to give, I mean, the field is very wide because we have, let’s say, the digital media evidence side more related to video formats like video coming from DVRs and different formats. So starting playing with FFMPEG or reverse engineering byte codes from different systems.
There is the image processing side, so enhancements, the blurring, frame rate, stuff like that. There is the image authentication side, image and video authentication, understanding JPEG formats, deepfakes, noise and stuff like that. There is the metrology side. So measure how tall a person is from a video, how fast a car was going.
So there are many different aspects and the risk, especially at the beginning, is that you go into a rabbit hole in one of these and you never come out, because they are so wide. And I mean, even if you look at the papers in literature, there are so many that it’s easy to get lost. So I think it’s important to have a kind of 360-degree view at the beginning, and then focus on what’s more important or you enjoy the most.
Christa: That seems like it would be almost challenging to figure out in a way, just because of the nature of the legal system right? I mean, Si and I have been having this kind of conversation offline about the legal system, just the amount of evidence.
It almost seems like you would have to know what to, I guess, in the US, we have a saying, “What’s going to get the most bang for the buck?” right? What’s going to be the most important or influential skill to build over time that is going to have the most impact? And deciding what that’s going to be must be very challenging.
Martino: Yeah, I think it really depends on the country and in the kind of sub field or unit that you are, because if you do kind of more on the side of digital forensics, mobile computer, maybe the authentication side, understanding where a picture is coming from is more interesting. If you work on collision investigation, kind of crash reconstruction, maybe the speed analysis is more interesting. If you are on a video unit, of course, different video formats from any kind of crazy system is your bread and butter. So I think the problem will find you.
Christa: So, with that in mind, like, what kinds of feedback, or what kind of traction do you feel like you’re getting from your awareness building efforts?
Martino: No, I think it’s going very well. First of all, from our, let’s say, historical users, especially the most expert, they are making up very strongly because we are kind of helping them to solve the issues that they have. So they are very happy.
From the stakeholder point of view, we have done a meeting at the European Parliament, we replied to an inquiry from the UK Parliament, and we are doing a lot of meetings with institutions at high level in Europe and outside. And the feedback has been really great.
With respect to other, let’s say, industries I hear that people have been much more responsive. Kind of once we introduce the issue and the challenges and what could be improved, people are really interested and they follow up with more calls and want to learn more.
So what we kind of are planning to do now, it’s outline some general principles to kind of jumpstart in a more practical way these supporting and awareness activities with the institutions.
Christa: How so?
Martino: Simply try to define because the problem is that as I say, there are people who are very, very good at understanding and solving this problem. And there are people at the top who are not aware of this problem. So we kind of want to define a relatively short list of principles that should be considered and taken into account.
But at a very high level, we’re not expressing them in, let’s say, in very technical terms, just explaining from stakeholders or even, I mean, people in law enforcement, but who are not technical people, understand what are the main steps you need to do and things to take into consideration to work well.
And only once you set this kind of, again, someone calls them “guidelines,” I think “principle” is more appropriate because something that then everybody should kind of deploy into their guidelines, standard operating procedures, but kind of a general idea of things that you should not forget.
Christa: I guess that, I’m sitting here thinking it sounds like the sort of work that the Scientific Working Group on Digital Evidence does. And I know, you know, you’ve been blogging lately about key questions about video evidence, which it sounds like this is kind of going down that path.
Martino: Yeah, yeah. It’s all on the same line. Yeah.
Christa: How do the issues that you’re raising or those principles factor into efforts to standardize digital forensics practice? I mean, particularly I think Si is more probably integrated with this than I am, but you know, standardizing in the UK which is an actual effort that’s underway. How are your principles factoring into those kinds of efforts?
Martino: Yeah, again, I hope this kind of principle would help the adoption of these guidelines with, I mean, various institutions and the MC and OSAC are doing an amazing job and of course, much bigger, much better than we could do it alone.
But I think they should be more widely adopted, and at all levels they are very, I mean, for example, SWGDE they have a kind of document for any kind of activities nowadays, even very specific, and we want to help the adoption of this because I think the entire industry can benefit from that.
And so I hope this is kind of doing a bit of advertising for those, because there are various guidelines and I think some go more, I mean, the weekly ones are quite practical, the MC ones are in my opinion a bit more rigorous from a formal point of view, but slightly more abstracted. At the end, they more or less agree.
And I think it is more important than adopting a specific guideline is using one, rather than doing every time kind of artisanal work and say, “Oh, every case is different.” Yes, it’s different. But again, you can follow a procedure.
Simon: It’s the scientific rigor that you need to maintain, isn’t it? And as long as you’re doing it in one of an acceptable range of ways, it doesn’t really matter which one you’ve chosen. I saw a beautiful cartoon the other day, which was one of the xkcd ones that I think we are all very familiar with in the IT industry of course, but it was like, “Oh, you know, there’s 15 standards for this. Let’s create a unifying standard.” And then the next frame there’s 16 standards for this. So we need to be careful to make sure that we’re not reinventing the wheel, rewriting it all.
Martino: Absolutely.
Simon: I think you’re right, you know, having a good standard that whatever it is that you are repeating is definitely the way to go.
Christa: Well, I feel like that goes back to what you were saying earlier, Martino, about how you could have an image that is not original, but it’s more accurate. You would still need to be able to do that in a way that the expert can certify didn’t change the original.
Martino: Yep.
Christa: Yeah.
Simon: I think we’re in the, and I’m going to sort of push you to a slightly more technical conversation, as opposed to the thing is that we’re in a position now where we’re seeing the rise very much of things like computational photography, where you’ve got Apple iPhone that is taking an image. Is that presenting a challenge to you in the software way that you are to perhaps reduce the beautifying functions of some, you know, selfie software to get back to the original image?
I mean, obviously to a certain extent, everything that’s being applied in that is a mathematical process to alter your matrix of numbers to make you look better, you know, as we do, and if, you know theoretically the algorithm, you could, I guess, reverse it, but like, there must be data loss in the process of doing that. Is this something that is challenging at the moment?
Martino: Yeah, absolutely. I actually wrote, I think probably five or six years ago, a blog post about the challenges with computational photography and the originality of pictures, of course, and even more challenging nowadays because all this stuff is done with AI.
And another topic I brought up quite a lot is the use of AI for forensic use of video evidence. We have quite a long blog post on that and the gist is that for some kind of application like enhancement, it is very dangerous.
And basically, I won’t go too long, but basically for two factors: one is that AI with respect to traditional algorithms, it’s kind of a black box, at least the algorithms that we are using now, not all but most, and the intrinsic bias introduced by the training data set.
So if you train, for example, the network with the people of a certain ethnicity and gender, then if you apply it to another kind of people, then you can get quite wrong results, of course.
So coming back to your original question, yes, this computational photography is a big problem, and we actually published a paper on the issues that we have with modern smartphones on the use of PRNU. PRNU stands for photo response non-uniformity, and it’s sensor noise which is used to identify which camera has taken a picture just from the picture itself without metadata or anything else.
And traditionally it was very reliable. Even if you sent pictures through social media, processes them to an extent, this kind of noise fingerprint was still staying there, but with the pictures, especially with the pictures in kind of portrait mode, this is not working very well anymore.
And we published a scientific paper, which has been quite popular, let’s say, and at least we can recognize them. We can recognize when it doesn’t work anymore. But a solution which works, let’s say, in a robust way, not just in synthetic tests in a university lab, it’s still not found. We are still working on that. But it’s one of the main issues which is coming with the new technologies, you know?
Simon: Yes, things that make our lives better and forensic analysis harder. And I probably imagine this isn’t too much of an issue, but are you finding that size of images is being a factor? Because obviously we’ve gone from, well, early cameras with two megapixels to you know, I have cameras which have 50 megapixels. My phone has, well, I don’t know how many, but we’re getting up to 4K 5K 8K video. Is just sheer volume of data being an issue in terms of video analysis, or is it just throw more processor at it and you can still do everything that you want to?
Martino: There are two aspects. One is the raw power that you need that you need more of course. And in fact, a big kind of project that we were doing on the development side this year was to better exploit the hardware and go faster because there is more data we need to go faster.
But the other is that for most of our practical application is a kind of fake improvement. So we see this surveillance video, which improves in resolution on paper, but then they are much more heavily compressed. So you have maybe eight times the number of pixels than before, but then when you zoom in, it’s all kind of blocks of the same color or that are artifacts.
So in practice, I don’t often see benefits. It’s better to have a more performance sensor, less noise, better configuration, or less compression than having this 8K video with blocks like this.
Christa: A lot of the issues, Martino, that you’re describing kind of factor into, it feels like the general public, as well as marketing teams, like to talk about how rapidly technology advances and then how hard it can be for vendors to keep up. And I’m sitting here, you know, listening to you thinking, like there’s got to be a tipping point at some point where you know, it’s just going to go too rapidly and vendors may not be able to keep up at some point.
But on a practical level, how does that tie to issues? I think deepfakes is the biggest example. But I mean, tools like Amped Authenticate. How are you incorporating the latest research that’s coming out? You know, you mentioned PRNU before. How are you incorporating research?
Martino: Yeah, we do a lot of research. And also let’s say for fun and experimentation, not, I mean, the real research is that the more risky one when it’s not directly tied to let’s say a product, because if it’s successful 100% of the time, it’s not research, it’s development.
So we do a lot of experimentation and that, and we’ve been researching deepfakes and gun images. Gun images are those like synthetic faces like thispersondoesnotexist.com, this kind of stuff, for a few years now. For gun images, actually, you have a quite nice solution which I think we will release soon. We actually participated in a gun detection competition with other research groups and we got the second spot. So it was quite a nice result.
And for deepfakes we are still working on that. And because we believe that it’s still a kind of cat and mouse game, it’s working nicely. But then, I mean, there are a few systems that you can play with online and they work in easy cases, but I mean, in real life cases, when maybe something is heavily recompressed on YouTube, or you just flip the video or stuff like that, they stop working.
And I hope next year we’ll be ready to launch some additional deepfake-specific tools in Authenticate. There are some of the existing fields, which you work in some situations with deepfake, but it’s a specific, deepfake detection.
On the other hand, very often it’s the opposite that you are interested in, not much showing that it’s a deepfake, but proving that it is not a deepfake. Because maybe you have someone bringing the evidence saying, “No, it’s not me. It’s a deepfake.”
And that is much easier because you can look at the format of the file to see if it’s an original coming from a device and not computer-generated video or image. We have video tools, again, like the pyramid or the BPF, which is detecting, let’s say, double compression on videos. Also in this case, it can help determine if a video is original or not. So maybe you cannot take for now from one side that you can take it from the other.
Simon: So do people approach you as a company directly for doing some of this analysis or are you purely a vendor for products? Do you do consults at all?
Martino: No, we do the software and provide the training, but we don’t do services. I did it on a personal basis also, let’s say as a company and also several of the people in our team did it. Someone still does it on a personal basis from time to time, but I think at a certain point you need to decide what to do.
Also because let’s say that the conflict of conflict of interest if you do a lot of case work is a bit high, in my opinion. What if I go to court and testify against one of my customers that I trained?
Simon: Yeah, no, that’s fair. So in that regard, how are you making sure that you are delivering what your customers want? I mean, you know, you have a robust feedback process from your customers, or are you looking at just what’s moving in the industry as a whole, or you know, how are we making sure that Amped Five is actually, you know, a useful tool to apply to the tasks at hand?
Martino: Yeah, there are different factors. The problem is not having ideas, it’s prioritizing them, because if we look at our, I mean, database of feature requests and improvement requests, I think we have more than 3000 items between the various products.
But we are in continuous contact with our users. I think probably one of the most valuable resources that we have now is a Discord server that we have. We launched it at the end of last year and we have several hundred members already that exchange ideas, opinions, and suggest when there is something that in their opinion should be improved or some big function that they would want.
In addition to this also a lot of ideas come when we do technical support. As I mentioned, several of the people in our technical team are former users of our tool, mostly from law enforcement and military. So they know very well what’s needed, what works and what’s needed to improve. So there is so much to do that the problem is really understanding what to do first.
Simon: Yeah, that’s fair. And at the moment you are currently a Windows platform software, aren’t you?
Martino: Yeah, all of our tools are desktop software. Let’s say, Windows-based.
Simon: Are there any plans to perhaps migrate that out to Mac or Linux? Because one, you know, obviously you are heavily algorithm-based and I would’ve thought that your software would translate well potentially to (he says kindly about Windows of which he’s not a terribly big fan), to something like Linux, whereby you could build more powerful clustered computers in the way that that might leverage some of that extra processing power. Is that something that you are, in that roadmap of 3000 requests that you’re considering at the moment?
Martino: No. Actually it was in the past, but I think for our kind of users nowadays, the market share I mean, it’s hugely in favor of Windows, so there is so much more important stuff to do.
Simon: Fair enough.
Martino: But the technologies that we are using are all let’s say mostly cross-platform. We are not bound to Windows. So, I mean, in the future potentially we could do it with a bit of effort, but it is not impossible.
Christa: So, you mentioned your Discord server, you’ve got your blog, YouTube, I think you also do user days, right, to connect with customers and others?
Martino: Yeah. We did it for a few years in person then COVID came and everything went online, which to an extent is also good because, not the COVID, but the online because we had more many participants last year. We are repeating it this year. It’s actually in two weeks. So if any of our users will listen to this and are not registered yet, they can register on our website.
We are also doing again a survey which is actually open to anybody working in our industry and working even, I mean, occasionally with video, it’s not just for our user, but it’s just to understand the perception of the various issues. And we will present the results of the survey in the user day and on our blog with analysis of what are the main issues and what is the perception of the evolution of the industry and the field of video evidence in general in the past few years.
Christa: I can’t wait to read that. That’s going to be really interesting, I think.
Simon: Yeah, completely agree.
Christa: Well, Martino, thank you again for joining the Forensic Focus Podcast. It’s been a really, really interesting conversation.
Martino: My pleasure.
Si: Yeah, thank you.
Christa: Thanks also to our listeners. You’ll be able to find this recording and transcription along with more articles, information and forums and www.forensicfocus.com. Stay safe and well.
