Si: So, friends and enemies, welcome to the Forensic Focus Podcast. Today we are honored to have Marco from Amped Software. We’ve talked to him before, and we … there’s several reviews on Forensic Focus, but they specialize in image and video forensics software.
And I personally am a big fan. So, I’m really excited to have Marco on today to talk about Amped Authenticate. But before we get on with that, can you tell us who are you and what did you do to get you this life sentence of coming to talk to us?
Marco: Sure. Thank you for hosting me. And yes, I’m Marco and I’m a computer engineer. I graduated in University of Florence here in Italy, and then I earned my PhD in another university (still in Italy), University of Sienna. And I’ve been working with images and video through all my career, to be honest. Worked on biomedical images, watermarking during my thesis.
And then my PhD was specifically on multimedia forensics, so techniques for detecting, splicing in images, videos, and even something about audio tracks. And then as a part of our research project we were doing as university, I was a postdoc at the time. I got in touch with Amped, we collaborated in a research project funded by the European Union. And after that, you know, we liked each other, and so eventually I joined Amped.
And I really like the possibility to transfer the academic research (I’m still trying to publish papers through Amped) into the practical world, you know, because that’s the main issue, sometimes the research remains in the lab in not very realistic settings. Bringing it on the field, it’s very challenging and very interesting to me.
Si: Oh, that’s fantastic.
Marco: Yeah.
Desi: Yeah. That’s really awesome to hear. Because I know it’s something that Si and have discussed before, is that challenge between getting academic research into the practical field and into the hands of people who under need it most. So, it’s really awesome to hear that.
Si: And it’s interesting also that you say that you’re still trying to publish, because Desi and I were having a conversation about that the other day! About getting commercial stuff put into published papers. So, perhaps you and he can have a chat later about how we can do it. No, that would be cool. Oh, that’s fantastic.
So, today we’ve asked you specifically to come and talk to us about Amped Authenticate. We’ve talked with Martino about Amped Five at the end of last year. Authenticate: totally different product, totally different, sort of, set of inputs. What…can you give us an overview, an elevator pitch for Authenticate?
Marco: Sure. So, Amped Authenticator, well basically lets you investigate in a very complete manner, the integrity and the authenticity of images. And we are also starting to work on videos. We already added some features to work on video now, and we are investing a lot of time to support more videos. And this is very important for many two reasons.
The first one is that an image can contain far more information that you can see from pixels, okay? So, you may have information about the author of the image, the originating device, the place and time where it was captured, and even more, which is all very useful, of course for intelligence and investigations.
And the second reason is that, well, deep fakes are everywhere now, but even before then, since the very beginning of photography, fake pictures existed. And so today, seeing is no longer believing, actually, and you need to assess the trustability, okay?
Of something before admitting it as evidence, or even just using it to advance your investigations, you should rather check twice if you should trust what you see. And this is something for which you really need Authenticate. Even compared to other products that we have, Authenticate focuses on the integrity and authenticity verification.
Si: You said that it now handles video a bit more. Is that particularly difficult in comparison to doing image work? I mean, a standard JPEG seems fairly straightforward, really. How does it work? Do you just pull a JPEG from the video, or is it more complicated?
Marco: No, actually we usually recommend users not to do that during the training that we deliver, because videos are compressed completely different. So, there are some techniques that you can use both for images and screenshots of videos.
To cite one, is the shadows analysis. So, everything that is based on some geometrical inconsistency, well, of course you can use it for a video. You can also use it after you scan an image, actually, because if the shadows are not right, they stay that way. But most of the things you can do in Authenticate leverage the compression traces, okay?
And the coding properties of images, which are very different than those videos. And so taking steels and running it through all the filters in Authenticate is not really scientifically sound, okay? Because some filters have been designed to take into account how images are compressed, which is different, okay? That’s why we need to develop dedicated tools for videos.
And this is more challenging, first of all because you know that video compression is usually more aggressive, and so it will tend to act as a counter forensic, an intentional counter forensic technique, let’s say, that will remove details, snd so remove traces of manipulations. And it also is computationally intensive. And so to give something which works practically, you need a lot of effort because you cannot have users, wait, I don’t know, 20 hours to process a five minutes video.
Desi: Yeah. So, I know you mentioned just before when you were talking about the images themselves and what you’re checking for. Maybe you can explain a little bit the difference between authenticity and integrity when we’re talking about images. Like, what are the differences between those two words?
Marco: Oh, thanks for this question, because it’s a very important one and often misunderstood. So, integrity means that an image is complete and unaltered since the time of acquisition on the device, okay? So, it means that I capture one picture with a smartphone, okay? The first version of the image that is stored on the device, or on the camera, okay? That is the file which has integrity. There is not an adjective for integrity in English. So we say original file, okay?
An original file is a file with integrity preserved. The moment you do something, whatever, to that file, the integrity is basically lost, okay? So even if you just, I don’t know, perhaps you rotate it with a computer software, which will edit something in the metadata, okay? You didn’t really touch pixels because it’s just changed the metadata, but integrity is lost, basically the hash would not match.
Authenticity is very different. It means that the picture is a true and accurate representation of what it reports to be. So you can trust the semantic content of that image. So, of course you can have an image which is authentic, but without integrity and vice versa.
So, for example, I take a picture with the smartphone, I send it to, I don’t know, Facebook, upload to Facebook, will cause a lot of issues. (We’ll probably talk about this later!) But it’ll not alter the semantic content of the image. So, the authenticity is still there, right?
Desi: Right. Yeah.
Marco: On the other hand, I may create a fake scene, perhaps with actors to mimic that something is happening and take a picture. That picture is not authentic because what’s happening there is not the truth, actually, okay? But still the image has perfect integrity because everything looks fine.
Desi: Okay. Awesome.
Si: So, I mean, obviously we are moving into a world of (I hate to use the term, and Desi knows this) of artificial intelligence. And I’ll, if I’m on video, I’ll be doing the air quotes. And computational photography is obviously being rolled out, left, right, and center, in particular on smartphones. How…what sort of impact does this have on what one might consider the integrity…? (Hang on. Yes.)
The integrity of an image, because what’s coming into the lens comes out different as to what’s going onto the phone? You’re already applying some sort of level of alteration before it hits it. How are we handling that in Authenticate?
Marco: This is very true and there’s not much we can do to handle that because we actually discussed this with other experts in the field. What should we consider as the first version of the image inside the smartphone, which is doing so much stuff on it? Okay.
One possible interpretation is that as long as you use, let’s say, the default app of the smartphone, what is produced as the first accessible file from the app could be considered the original file. Because we cannot access the steps in between because they are protected of course, they are industrial secrets!
And so there’s really no way we can know. And they have potentially a very strong impact in authentication because of course, the generation model, which was a simple and plain once; so you had the lens, the color filter array, and the demosaicking algorithm, it was all pretty clear, okay?
Yes, there were some things which were still proprietary, but more or less they were clear. Now they’re very hidden and very powerful, and they…I’ve seen that some producers are also considered to embed artificial intelligence inside the sensor, which means basically, I mean, you only have the lens which belongs to the old world, and then whatever else is AI powered.
So, we still work thinking that this kind of AI, yes, it can do something important to the content of the image, but in general, okay, it will try to represent the scene as its best, okay? Let’s say. And we’ll try to detect what happens later than that. That’s the idea.
Si: Is there anything that we can sort of take from the color science to perhaps re…like, we would do white balancing in a normal sense? Is there something that we can apply equivalently to a computational photography edited thing to try and bring it a little more in line with what we might consider to be reality? Or is that…is that a manipulation too far for court, do you think?
Marco: Yeah, I think so. I mean, when you’re doing authentication, you should not change the input signal, okay? So, that is what you have and you have to do analysis on that, but you should not bring it back. That is not longer part of authentication, I would say. It becomes more part of an enhancement technique. Actually as time evolves, okay, the artificial intelligence techniques are becoming very good in mimicking all the finest details of true natural images, okay?
So, perhaps once you could detect some strange patterns in the Fouier spectrum of the image when they were created by a neural network or processed by an neural network. But the more they train these networks and the more they will try to mimic also the properties of very natural pictures. Yeah, so it’s very challenging. It’s a very challenging subject.
Si: Okay. Yeah, no, absolutely. And I saw…there’s been some…I’m sure you’ve seen it, the press recently around the Samsung moon shots adding in that level of detail that absolutely 100% doesn’t exist because it’s not a picture of the moon.
I did notice when I was looking through the material on Amped; I’ve got a trial version here as well, which I haven’t really had enough time to play with before today, so I do apologize. But that you have a tool for detecting whether a image has been taken from a monitor or not, because that was one of the sort of early anti-forensics to evade the, sort of, the metadata being embedded was to take a photo from a high resolution monitor and then that would appear to come without modification. But I understand that there’s a filter that you can apply to detect that.
Marco: Yes, that’s true, indeed. What you say it’s called recapturing in the field. So, it’s a very common…well, common, I don’t know, but very effective counter forensic technique, if you have the tools to do it properly, of course, because if you just point a smartphone, it will be full of artifacts, okay?
But if you use proper settings, you can get a very good picture where, of course the integrity is preserved, because you’re taking actually a new picture. The filter that we recommend is the…is just a simple Fourier analysis, actually. So, you are just looking at the Fourier spectrum of the image, and we provide a detection system to detect and show you peaks in unexpected places, okay? Of this Fourier spectrum.
And this is because even if using a very good setting, you could still have (and you’ll most often have) an interference between the pixels of the monitor and the pixels in the camera sensor, which brings you to some kind of moiré effect, okay? So, some aliasing, which probably you cannot see by the naked eye if the settings is professionally made, okay?
But they could often be revealed by a Fourier analysis, an analysis in the frequency spectrum. We have been trying to implement even more automated techniques that would flag the image automatically, but it turned out that they were very sensitive to the training material.
So, we prefer it to keep it simple, very easy to explain even in a court, because Fourier, I mean, for someone in the field, it’s pretty basic knowledge. And it’s also very easy to explain why you would see these peaks because you would’ve these periodic artifacts due to the interference, which makes it, yeah, pretty honest. Yep.
Desi: So, for someone not in the field…so my background is incident response, I guess. So, it’s not so much doing this stuff day in, day out. For people who aren’t in the field, is the point of doing that analysis to just determine whether a photo is taken from a screen, is that the idea? To flag that you may need to look at it, or if you are in a court case, you could say that someone has taken this photo of a screen?
Marco: Well this is often what happens in authentication, is that you observe something, you cannot trace it specifically to one source, okay? So, you see these peaks in the Fourier spectrum, and you have to provide justifications for that. So, when we train to use this filter, we explain that there are other things that may bring these peaks into the spectrum. One is JPEG compression, but then those peaks would be placed in some specific positions, okay?
Another one could be an artifact, which is called JPEG dimples. (This is very technical and very recently discovered.) But once again, those peaks are in specific regions. While when you have this kind of recapture attack, the peaks are more spread all around.
And if you see them in the image and there is not any content in the image that explains them, so a fence or something very periodic that could be the responsible for that, then a reasonable hypothesis is that there is something like this interference between the monitor and the sensor, that could be the explanation for that. So, it acts as a red flag, okay? But it cannot guarantee you that you have a recapture. It’ll be up to your judgment to give a final evaluation.
Desi: Okay. Awesome.
Si: I think one of the other filters that’s distinctly interesting and that, you know, would still work with this, is your camera ballistics, which is attempting to identify (or probably very successfully identifying) specific…not even specific makes and models, but you do that as well, don’t you?
Marco: Yes. So we…there are two different kinds of analysis. They’re both related to the source device, but you may be interested in understanding what was the model of the camera. And we can provide information about that, of course, with the metadata, which is very straightforward. But also we have a very rich data set of JPEG quantization tables. So, you know that when an image is compressed to JPEG, you have to specify these tables of coefficients that are a property of the compression.
They needs to be into the image file, otherwise you cannot decode it. And sometimes devices use standard JPEG tables. And so in that case, of course, it could be whatever the source, okay? They’re just the standard. But most of the time they will use proprietary tables.
So, for example, Apple has their own set of proprietary tables and other producers as well…and just saying some very pamela’s names from producers, of course. And…so, we have built a database with over, I think we are reaching 20,000 soon, quantization tables, okay.? And so we read this information, which stays there, even if you remove the metadata, okay?
Because it’s a property of the compression and we match it against database, and we can provide you a list of compatible devices. Of course, it all depends on keeping this database updated, but it can be very useful.
A different kind of analysis is the camera ballistics. In this case, you’re not interested in the model, but in the specific exemplar that capture the picture, just like you do with the bullet and the gun. So, you may have an image and you have three smartphones of the same brand, same model, consecutive serial numbers on your desk, and you want to trace that image to the one that captured it. You can do that using this PRNU, Photo Response Non-uniformity Noise analysis.
It’s actually quite simple from a technical point of view to do. And it has shown impressive performance in the years. So, it has been used in court and, of course it is a very powerful technique, especially in cases where we have child abuse, and you want to understand whether someone produced those images, which is of course quite an important charge.
Unfortunately though, we discovered together in a project (in a research project with University of Florence, actually where I came from) we discovered that for recent smartphones (so I’m talking starting 2020 on), these internal processing that they’re making, probably AI-based (we don’t know, okay, what is the exact source). They are introducing some…we think they are non-unique artifacts.
So, some kind of shared properties into the image that will make this detection system fail, giving you some false positives, which are very dangerous. They were considered extremely unlikely before. They are becoming much more easy to find now, okay? And so, we have published a paper straight away and we published an article on our blog to warn the scientific community and our users, first of all, about that, we’re saying this in the training, “please be cautious, okay?”
Because you may have, even in a controlled setting where you have the smartphone, you can take the pictures, you can do whatever, but if the smartphone is using this kind of technology inside, it is probably no longer safe to use this technique, okay? And we want to be very clear about that because we know that this tool can change someone’s life for the worst!
And so, yeah, we have published it, and we published it actually as a call to action. We released the dataset that we use to prove this, hoping for other researchers to work on that. We still didn’t have an answer from the research community. So, there is not a technique which solves the issue, but we are working on that. I know that some other universities in Italy have been working on that, trying to detect whether images can be used and they are suitable to be used for a PRNU analysis or not. So, it’s an open challenge and it’s a very interesting one.
Si: Sounds it, sounds it.
Desi: So, going back to the…when you were talking about the coefficients that are in a JPEG, because that sounds super interesting, and having that data set would be super useful to try and determine the model numbers. Do you see within a manufacturer, so say Apple as an example, who are known to just change things every time they release a new phone, that they’ll change the internals. Do they change those kind of tables, or is the coefficients that is proprietary, you see them be quite stable for different makers?
Marco: That’s a very good question. Indeed, I think Apple has been quite consistent in the last years because honestly, there is not a huge advantage you can get by changing these tables, okay? Some models have adaptive tables. They are very challenging of course, because in this case it makes little sense to build the data set because the table will be determined based on the properties of the image, okay?
So, different images taken two seconds apart will have different tables just because the content is different. When this is the case (and we published an article about that on our blog), you should make sure that this is not the case when you’re doing your analysis. So, you should get more images from the same device that you are hypothesizing is the source, and make sure that it’s not using one of those technique for adaptive QT. If it’s not using that, then using the database makes sense. Yeah.
Desi: Okay.
Marco: So, there are some producers which tend to update. For example, Facebook has been changing the quantization tables that they’re using, and they also use a bit of adaptive tables when you upload images, while other producers are more loyal to their old tables. Yeah.
Si: That’s an interesting segue. I mean, I was aware that, you know, when you upload an image to something like Instagram or to Facebook, it then does its own processing on it, but it actually goes as far as changing the tables for the JPEGs as well. I mean, you’ve got a…there are some plugins, I believe, that are handling sort of the web content stuff. Can you sort of tell us a bit more about that?
Marco: Sure. So, when you upload images to social media, most of them (and there are several papers about researchers about this), they will do something to the image. Most commonly they will strip all the metadata and sometimes add some new metadata that they use for internal processing. They will downscale the image if it’s a larger resolution image, even considerably, okay? And recompress the image almost always, okay? (There are very few exceptions.)
And when they recompress the image that means change in the quantization tables, because that’s the main point of compression. You…when you do JPEG compression, you can…okay, you can do chroma sub sampling, which is…means throwing away some of the color information. But this is usually already done by capturing devices. But then to reduce the size of an image, you will use different tables, because that is what brings the real saving in terms of memory, okay?
They will use tables with larger numbers, which will reduce the file size. And so while for devices, it’s hard to find images that are native and with a quality less than 90% (if we put it on a scale from 0 to 100), when you download them from Facebook and this kind of…it could be 70, 75, so you can see that someone had to reduce the size, of course. So, it’s a quite aggressive processing that images go through, even if you send them through messenger apps, and this kind of thing.
Si: So, what are we left with at the end of that that we can still work with? I mean, the ballistics, is that still a functioning thing? Is the, you know, the moiré analysis, is that still available or is it, you know…are we really suffering from effectively a big muddy footprint over the top of the data we’re trying to recover?
Marco: This is a very good question also. The camera ballistics, we made experiments about that. Actually we published another dataset with the university,…I think that was when it still was at the University of Florence, which is called “Vision”. And it has hundreds of citations because it’s very rich. We collected images from more than 30 smartphones, recent and less recent at the time.
And we uploaded these images to Facebook and we sent them through WhatsApp, and we tested the camera ballistics on them, and we kept working. You could see a slight drop in the performances, but yes, it kept working. But of course then this needs to be constantly reevaluated, because what Facebook does to today is different than what was done few years ago.
The implementation of the apps change, okay? So, it has the potential to stay there because PRNU is quite a robust fingerprint compared to other stuff that we rely on. But you have to test on case by case. There is hope to find something, though. There have been…I’ve seen papers about linking social media profiles, okay? Which could be very interesting. For example, for counter-terrorism, you may have one guy which is using the same smartphone and uploading pictures to two different accounts, the everyday account and the terrorist group account, you know, and if you can link those images with the noise, well that could be an interesting thing.
Desi: So, that was gonna be, I guess, a question that I was thinking of as you were just, I guess, discussing that was like linking the two photos together between the two accounts on Facebook. But also when you mentioned when it gets uploaded to Facebook, it, kind of, imprints its own metadata for internal use.
Do you see any of that, say if it went from Facebook to WhatsApp, are you…is there any metadata to link it back if you got the photo from WhatsApp to link it back to Facebook? Or does every time it gets uploaded, it’s like the platforms are just restamping their own metadata over the top?
Marco: I would need to check because I didn’t try this, but I’m pretty confident that they will strip the existing metadata and put the new one on. Yeah. They don’t really care about preserving the original metadata usually.
Desi: Yeah.
Marco: Actually they say that they strip it to preserve your privacy, but usually when someone uploads to Facebook, privacy is not the…!
Desi: Yeah.
Marco: Yeah.
Desi: I’m pretty sure Zuckerberg had a Senate hearing about privacy that probably contradicts that, right?
Marco: I think so.
Si: I mean, with sort of, again, building on from the idea that we are dealing with images that have been messed around with by someone and what can we regain from it. Plenty of cases that I’ve done as a digital forensic analyst…we do file carving and we get fragments back of JPEG images. Is that still something that, you know, you can work with? If there’s only half a JPEG, is that sufficient? Because I know you can render half a JPEG and when you, you know…it was quite happy to do it. Is that…does that still give you enough to work with? Or is…do you need the full thing?
Marco: I would say no. We didn’t design Authenticate to handle partial files. It would be an interesting thing. But to be honest, we don’t receive many requests in this sense from our users. And so it’s not up in the product roadmap because it’s a feature that is not often asked for, this idea. But it would be very interesting.
And I know there are researchers that work on that. I know some of them. It’s very challenging though. So, these are quite challenging already with complete files if you start dropping stuff! Yeah, it gets even worse. We have a tool for extracting jpeg images from inside larger files. So, could be, for example, a PDF. So it’ll take images out of a PDF file or if they’re embedded in some different file, you can extract JPEGs, thanks to the magic numbers of course. But still, they’re not meant to work for data carving, let’s say.
Si: Okay.
Desi: How does Authenticate go with…so, some of your customers would obviously have huge image databases. So how does authenticate go handling, like, large volumes, not just in images, but also large volumes of potentially cases side by side that they’ve got to, kind of, process through?
Marco: Sure. So there are two ways that Authenticate can save you lots of time. One is the batch processing features that is there since the beginning. So, you can configure a set of filters that you want to run through your images. All of them is the default, but you can choose a subset if you think that’s better.
And you leave it there running and you go away, you go home, you come back the day after, the week after, depending on the amount of images, and you will find report file with the results for all the filters and the cached version. So, you can use the interface of the software and it will load the cache, so you don’t have to wait for computation to happen. The issue here is that of course, you still have to go through all of them one by one, okay? So, the software is not doing the work in the place of you.
Then we have the SMART report. This is a newer feature that will do a, kind of, triage for pictures. So it’ll first scan the metadata of images, and if metadata are found and nothing suspicious is detected in them, like, for example, different EXIF, create data, modified data (to cite one). Or for example, traces of software in the active EXIF software field.
So, you see Photoshop there, this is a red flag, okay? If no red flags are found, the image is marketed with a green light. It doesn’t mean that it is certainly authentic, but it means that the integrity seems okay, and so it is kept, okay? If something strange is found, it’s sent through some filters that are chosen based on the kind of image.
So, if it is a JPEG, there will be some filters, if it’s a Bitmap, different set of filters. And some of these filters as an automatic detection system that will turn a red flag when something suspicious is found. And so, if the image goes through and nothing suspicious is found, it’ll get a yellow light means that, okay, integrity seems broken, but we didn’t find any clear trace of tampering.
And if something suspicious is found gets the red light. And so in the end you will get a summary and say, “look, we have hundred images with red light”. I would start my analysis from those ones. It has the ability to focus your attention. That said, Authenticate is a tool that is made for the human to work on, okay? So, it will uncover lots of statistical properties of the image, lots of potentially compelling results, but it’s up to the human to take the decision, and we think this is very important in forensics. That’s the human takes a responsibility for the decision. Yeah.
Desi: Yeah.
Si: In regard to all of those sort of filters and statistical things that you can apply when triaging a large data set, is there anything in there that does things like skin tone detection and percentage for potentially triaging CSAM or IIOC out of a large data set?
Marco: No, we don’t do this. We believe this is more on the side of analysis than authenticity verification. So, there are other tools, competitive tools on the market that do this kind of work and they don’t do authentication. So, we think they are quite two distinct things. And so…
Si: Yeah, a tool to do a specific job. You know, I come from a Unix philosophy: one tool, one job, chain them together when you’re happy and it’s all good! So no, I’m absolutely fine with that. Not a problem at all.
Desi: So, what…I guess that’s another good segue into, I guess…what does Authenticate do that we can’t do in other tools? Or, like, what’s it’s, like…you’ve already kind of explained to us the elevator pitch on it’s doing the authenticity and integrity, but what kind of other features can, kind of, we expect?
Marco: Sure, thanks. This is a good question, indeed. I think there’s an important point to clarify. When you want to establish the trustworthiness of a picture, you have many possible ways to go, okay? What we do in Authenticate is blind content analysis and container analysis. It means that we start from the file, we have some internal databases, okay? But we work with the file that we are given. You are not supposed to do any prior work on the file.
Other approaches are active authentication approaches like watermarking and blockchain technology, I don’t know. So, in this other paradigm, you will embed information at creation time, okay, to, you know, preserve the integrity of the picture. Of course, this is a much easier settings later for the analyst because if the envelope is preserved and you find all the data and the hash and everything, okay, it’s much easier to say, “we can trust this image”.
The issue of course, is that you have to have the source to be compliant with that, which is a pretty strong requirement. While in Authenticate we work on the other hand side where you have a passive approach. So, you are given a content and you work on that. Second point, which is very important, we don’t limit to container analysis, okay?
So, there are other solutions that will do an excellent job, but they will just check the container. So, basically, like if we stop it after checking the quantization tables, let’s say. No, we go down to the content, so we process the pixels, basically we provide forgery localization maps to help you understanding whether some region is suspicious inside the image, whether some region seems to be cloned from one region to the other inside the image. So, this is computationally intensive, but it’s also very important. And I think there is nothing more complete than Authenticate for that, to be honest.
Desi: Okay, cool.
Si: Yeah, I mean, I think it’s a fascinating field, isn’t it? Because Photoshop gets better and better by the week. And Desi and I were talking the other day about the new, is it Nvidia…eye..?
Desi: …eye things.
Si: Yeah. So what we are seeing on a day-to-day basis is out there and and I know that there’s, sort of, been a number of of truth in advertising laws passed as well, and, you know, to do with certain body shapes and, you know, one’s mental health about the fact that some of us certainly aren’t in any sort of fit shape to wear swimwear on the beach. But…and for everyone. Liquefy and that kind of thing, is that, I mean, I assume that’s really easy for you guys to detect in Authenticate, or is it the software…I mean, exclude the metadata side of stuff, because we all know that I can rewrite JPEG metadata without any trouble at all.
Marco: Of course, of course.
Si: EXIF tool is brilliant. I love it. I, you know, you can read, write, change. I’ll open up to my dishonesty on this one: I used to do a photography thing, which is a 365 project, okay? And the idea is that you take a photo every day over a year, and you share it with people. It’s an online site. And the site is really good in so much as it checks the metadata of the JPEG you’re uploading to tell you whether you’ve taken it on that day or not. And I wasn’t all that accurate getting my days right.
And I would just change the metadata so it would let me upload! So it looks like I have a 365 day project, and I don’t, I’ve got a 350 day project with 15 fakes, but we won’t talk about that anymore! But anyway, metadata is easy to alter. But I assume that you can detect things like Liquefy and this eye movement stuff really quite easily without the metadata flagging it…?
Marco: Okay. So eye movement looks like more a video thing. So, we are still working to detect things in videos. I will answer about Liquefy in images. Yes, usually we can detect that, but it really depends on what happens later, okay? So the Liquefy, per se, it will change pixels quite dramatically, okay? So, the traces that it leaves from a statistical point of view are quite compelling.
And so if you just open an image in Photoshop, use the Liquefy tool and save it to a decent quality, let’s say, and send it to me, we should be able to find…not because we trained anything on the Liquefy. We usually have tools that work on the other way around. So, we detect which pixels have NOT been changed because, for example, the images started as a JPEG, you change it something inside, and then you save again as a JPEG, usually.
And so the original pixels show some statistical traces, which are called double quantization traces, because they went through JPEG twice. The modified pixels will not show this because they’ve been changed in between. So, whatever you do to modify those pixels, even the most advanced AI stuff, okay, that change that region, we don’t care what you did because we actually detecting the unchanged region and that the changed region will stand out as a suspicious, regardless if it was Liquefy or some content well feeling or other kind of stuff.
This is very powerful because it frees you from challenging, you know, all these new filters that are released and it’s an effective approach. That said, if you use the Liquefy and you export the image and you save it to a bad quality, and then perhaps you upload it to Facebook and then download it, you will end up concealing the traces, okay? And so you cannot longer use this kind of statistical methods. You may still have luck with different kind of analysis, but it becomes much more challenging. So, it really depends much on what happens after the manipulation.
Desi: Sorry. Is that just because of the…so when you’re saying you’re uploading it to Facebook, if the statistical analysis doesn’t work because compressions happening on the image and you are losing, I guess, data points to then do the statistical analysis on it. Is that why?
Marco: It is often a combination of resizing and compression.
Desi: Okay.
Marco: So, from a statistical point of view, this is quite an attack. I mean, because you are resampling pixels and then you are compressing…so quantizing them again. So, of course if something remains visually inconsistent, you will still be able to see it, of course. But in terms of statistics, okay, this is no longer as suspicious as it was before.
Desi: Yeah. So, it’s the reverse of the ‘zoom in and enhance’ that we see on crime shows all the time.
Marco: Yeah, exactly!
Desi: Criminals do the reverse, they can get away with Liquefy, is what we’re getting at. Yeah, awesome.
Marco: Yes.
Si: Oh dear, I was wondering if that would creep in.
Desi: It has to!
Si: My children wind me up with that. We were sitting on the sofa the other day and there was something on TV and they were like, “zoom, enhance!” I was like, “no, it doesn’t work like that!” So…
Desi: Two hands on the keyboard work even better to hack things as well, you know?!
Si: Something you said was interesting, actually, in that a lot of the detection that you are talking about is because of JPEG to JPEG…
Marco: Some of those, I mean, we don’t only do that.
Si: No. You don’t only do…and this is where I was trying to get you to go, is that actually I take all of my photos in RAW and then process them, and then resave them as JPEG. So, I wouldn’t see that double compression feature. Yeah. Okay. So the, you know, that’s a…
Marco: You could still detect other kind of anomalies, like, just to mention one, we have a noise level consistency tool, okay? So, if the RAW image starts with a certain amount of noise and then you process pixels, and when you process pixels, you often interpolate them just because you rotate or you compute some pixel from the neighbors.
You may find inconsistency in these maps where you see that the amount of noise is uniform everywhere else, but in a specific region, right? Things like that. These do not depend on compression. Actually, it’s even better if you don’t have compression here. And it’s a good way to go.
Si: And I assume you can pick up the cloning of areas equally as well because…
Marco: Yes, we have…
Si: …exactly the same in two different places!
Marco: We have two dedicated tools for that, actually. One which is more dense, so it will work even if you clone a region of the sky over the sky. So, it’s without any real content, you usually will be doing that to hide something. And then we have one which is based on key points. So, when you’re cloning objects, even if you rescale them and rotate them and place it in another place, so you do some quite dramatic attack on it, it is usually quite effective in detecting them. Yes.
Si: Okay. Oh, really cool stuff. So, where…what are you actually…I mean, apart from the video stuff what, what are you working on at the moment to…where is Authenticate going next? What’s the next exciting…? Or is this commercial secrets? Don’t, I mean, don’t say anything…!
Marco: No, of course. No worry. I came prepared! No, no, jokes aside. We recently released the first deep fake detection filter in Authenticate, which is made for images, okay? To detect GAN generated images, which are those…I don’t know if you ever visited ‘this person does not exist’ website. So, these hallucinated, or synthetically generated faces are very compelling.
There are papers where it is shown that humans cannot tell them from the real, actually, they tend to say that real images are more fake than the synthetic ones, because synthetic faces are very average faces. They don’t have strange stuff like sometimes you find in real people’s face, no? And so, we worked a lot to create the tool based on AI. (We had to AI for this because it was nearly impossible to fight it without it.) And we published it in our paper.
We took part to an international challenge. We ranked very good, we had more than 90% accuracy in their validation. We didn’t do the validation. They did it, of course, on that dataset. And so we published it and we included it in the software. And now we are trying to push a lot for video, okay?
So, we are working a lot to expand the authentication abilities to video. We can already use the camera ballistics for videos in Authenticate, that’s already available. You have a tool which is based on MacBook analysis to do double encoding detection for videos. So, trying to understand whether the video’s been reencoded, which is a threat to integrity of course. But we plan to do much more in the future.
This is where our effort is going. And of course in deep fake detection, because this is really needed, okay? Whatever we do, we try to do it in a way that has chances to work in the real world, okay? So, we try to validate on videos that have been downloaded from YouTube and recompressed, okay? That’s why it takes time because we are quite demanding on the performance of the system before we send it out.
Si: No, I was going to say, I’ve been fortunate enough to trial…to have access to trial versions of Amped tools and I’m not doing anything terribly strenuous, but I am running them in a virtual machine and they still run astonishingly quickly. So, your optimization is clearly working very well, I have to say! So, well worth it.
Desi: It would be…just, it made me smile thinking about the fake human photos. Because I recently created…I’ve been creating a CTF and I’ve made, like, a fake LinkedIn profile as part of it. But thinking a more broader sense about like foreign intelligence services that are using fake LinkedIn profiles to try and connect with people, there’s potentially a use case for companies like that where you’re meant to have a real representation of yourself on the platform and using that kind of technology to scan through profile pictures of everyone and, kind of, flagging which ones are potentially fake humans, which could be like really useful at a, like, a national security level.
Si: Yeah. It certainly extends beyond the courtroom as a concept, doesn’t it? No, really cool stuff. Cool.
All right then.
Desi: I guess…I’m probably at the end of my questions. Are you…you’re at the end of yours as well?
Si: I think I am, certainly for anything that’s going to be of interest to listeners! Without getting into huge amounts of mathematics and questions around exactly how things work, I think it’s probably best that we leave it here. Maria, thank you so much for coming on. Really, really enjoyed it. Really fascinating. I’m really excited to play more with the product and let’s see if I can afford it later! And yeah, absolutely. You got anything exciting that you are looking forward to outside of work and then upcoming? You know…or I was gonna say, you said earlier that there was somebody who…downstairs who might be dividing…taking up some of your attention. So, I assume you have other things to do!
Marco: Definitely absorbing all of my time, because this is the second daughter, so I’m here: one wife, two daughters, and that’s already quite of a challenges.
Si: Yes. Don’t tell my daughters, but they are quite a challenge. I couldn’t agree more! One of them probably within earshot now, so I’m gonna be in trouble later.
Desi: Yeah, you’re in trouble, Si, that’s for sure.
Marco: Thanks a lot for all these very insightful questions. Much appreciated.
Si: No. Yeah, our pleasure. Thank you very much for coming on. It’s been great to talk to you…and Desi’s dog! There we go. All right. Fantastic. Thank you very much. You take care.
Thank you very much for listening, everybody who’s joined us today. Links and the transcripts of this will be available on the forensicfocus.com website. And we look forward to speaking to you again sometime in the future. Have a great time, keep safe and we’ll see you soon.
Desi: See you next time.
