Si: Hello everyone, and welcome to the Forensic Focus Podcast. Today (as was being briefly discussed before I pressed the record button) I have the first forensic married couple that I have ever interviewed. And we’d like to welcome Heather back, who’s been with us before, and Heather and Jared Barnhart, who are both from Cellebrite, and who have come on today to talk to us about some work that they’ve been doing with regard to AI and image analysis on the basis of forged and fraudulent images created by AI.
So, first of all, thank you very much for coming back. Obviously, we didn’t manage to put you off last time, so that’s a good sign! And also, thank you for coming to talk to us about something which is incredibly topical at the moment. I mean, I was away last week at a conference in…I got to go to Europe. I went to Luxembourg. It was fun for me.
But the conference on AI and the law and criminal cases and certainly the idea of bringing deep fakes and things into the courtroom is a huge worry and concern to lawyers and in the criminal law academic community and we’ve seen some cases already whereby…perhaps not necessarily if the actual material has been present, but people are sort of starting to try and use the AI defense as opposed to the Trojan defense, and we’ll touch on that in a minute. But you know, obviously working together and being married wasn’t enough for you, so you decided to go off and do some research. What prompted this? What brought you onto this?
Heather: I would say my whole RSA talk last year on sextortion and then the Taylor Swift deep fakes. That was kind of a huge thing in our house. We have three little girls and Taylor Swift is queen and you don’t do wrong to the queen. So I think just the fact that we started generally…AI is huge. Everyone is like, “oh, you must talk AI.” And this is sad, but true. I knew if we threw an AI topic into techno security, we’d get accepted. So it was like the guarantee in, but then we didn’t know what to do with the research because we truly didn’t know what we would find.
Jared: Yeah. I think in the last 12 months or so, we’ve sort of lived this fun, watch a video and think, “is that real?” You know, just that casual feeling and of course it’s real. And I was like, “no, that’s definitely not real. Look at the…”, and then you start to pick apart visually some of these things. And now it’s, I guess happened so quickly that it’s almost a zero trust of…I can’t even believe what I see. And so it just kind of worked, I guess.
Si: I mean, obviously it came at a very pertinent time. And, you know, for listeners, this is being recorded the week after the results of the US election were announced. And we won’t go into politics, because that’s something that it’s best to steer clear of, I think, for various assorted reasons. But, you know, there were suggestions on all sides that created media was being there. And the fact that you…I mean, we’re professionals and we look at something and we have natural skepticism built into us! But it is getting so good now that the lay person isn’t going to pick up on it straight away.
So, you know, raising public awareness is definitely a thing. So, I mean, has that sort of tainted your view of it as well? Or did you finish off on…or I was going to say the Taylor Swift stuff was clearly related to the election itself anyway. So some of it, some of it other stuff. So what do you think the biggest sort of risk is now? Is it that people are so unaware or is that it’s so good? Or is it just…where is the actual problem? Is it that we are so trusting of media or is it that we are not, you know, not critical enough. Or what do you think?
Jared: We actually had this talk in the last few days of…as consumers, we get our own flavor of the internet and flavor of whatever we are choosing to participate in. So her feed on a certain social media platform is strikingly different than mine. And as we would discuss, “I didn’t see that, or I didn’t hear that.” And it’s sort of got us thinking about not just trust of, like, is this one thing real, but that we’ve fallen into this, I guess funnel as consumers.
And some of the truth that’s out there will never even hit my platform that I’m choosing to live on. So it’s not just true or not, but which version am I getting of what’s out there and then the thing that I see is that thing even believable? So it’s become quite complicated as individuals to understand what truth is. And I think that is going to…
Heather: And our kids (so think of the next generation), our kids range from 7 to 12. They believe everything they see. So it’s like, “oh, I saw this thing, this is true.” Or “did you see…?” It could be…they love Benson Boone too, it could be Benson Boone broke up Travis Kelsey and Taylor Swift and they saw a video, so it is the truth. And now all things must occur. But kids have no idea. And I think that is also…oh, I say kids: my dad, my dad has no idea. I have no idea half the time.
Jared: Yeah, I mean, less technical people or those who maybe aren’t as skeptical as some of us in the forensic cybersecurity arena, they just accept that what hit the screen is a lot of times a fact. And it’s a wild environment right now as to how different those things are.
Si: I was going to say, it’s interesting because it’s a sort of a bit family folklore, but my youngest daughter, I mean, she’s 18 now, so she’s well past…well, she’s not well past receiving media…that’s very true, but she’s…I’ve built a healthy skepticism into her over the last 18 years that at least makes her question things. But once upon a time, you know, for her, she went with her mum to a cash machine and tried to swipe on the screen on a…so an ATM. Because for her, touchscreen was such a natural interface.
Do you think we’ll get to a point where the next-next generation perhaps are so media savvy that they don’t believe anything? Or do you think it’s curable? Do you think it will reintroduce skepticism? When they realize that, you know, what they see isn’t necessarily true?
Heather: I personally fear it will get worse. I think that they don’t even take pictures and share photos unless it has a filter. They’re immediately changing everything about themselves. Like our two oldest have phones. They will rarely take a picture that is not filtered or something added to it nonstop. So that is their reality. Like, they never had to develop film and wait for doubles and…remember all that? Like, it was crazy you get one shot and then you get two copies and that’s it .
Si: Yeah after waiting a week as well! That’s the other thing, yeah.
Jared: I think that we’re definitely trending in a direction where we will perhaps focus instead of on maybe like large networks of, “I prefer this network versus that network” that we will actually tune into individuals. And I think that’s been a thing for decades of a certain host of a show that you engage with better than someone else.
But I know for me in the last few years of a lot of sort of misinformation, disinformation, obvious campaigns to lead people in one direction or another, whether true or not, that I have latched on to certain people in the media that I believe that I can trust as, you know, doing due diligence and things like that. I do worry that the generation that we…ahead of us is all in on the network and the generation behind us, I have no idea. I mean, what will they…what is true to them?
As Heather said, we have a Snapchat generation that every…the first picture they ever took, they thought it would be fun if my face looked better. You know, and that’s, you know, a terrible thought. But not just better, but now., “I want to, you know, have some other feature.” And so everything has been some sort of AI spun improvement or change. And that’s the normal, much like you’re swiping at the ATM screen. Like, her normal is that that screen is supposed to behave with her finger and it’s, you know, it just didn’t so…
Si: Yeah, I think it’s interesting because I think you’re absolutely right. And also, I think there’s the sort of the insidious back end of that, which is the computational photography that’s built into certain devices. It infuriates me. I have some fantastic cameras. I still have some film ones. I still develop film. So I’m very aware of that. I do it myself and it’s fun to do of a weekend, but my wife will pick up her phone, an Apple phone and we’ll take a far better photo than I do with my thousand pounds worth of camera because it has the computational photography built in that does that. But obviously that’s not 100% of true representation of…or is it? I mean, that’s a…it’s a philosophical debate as much as anything else really.
But what I’ve certainly seen people asking is that where people are using phones to take pictures of injuries that are then being cited in domestic abuse cases, is that is that a real bruise or is it enhanced by the fact that the color has been, you know, jumped up, popped, to make it pop? And, you know, that sort of issue is definitely prevalent as well as the, you know, deliberate attempts to distort or to auto enhance, but…to make things look that way.
So, you know, in terms of the research that you’ve done so far, how are you getting on with detecting these sorts of things, especially once it’s been put into something like Snapchat or Facebook or Instagram, or one of those where they’re actually removing a lot of the metadata from the imagery during their processing so that when, you know, when somebody else is able to look at it, a lot of that sort of, you know, EXIF data is vanished?
Heather: You want me to start? All right, so on the originating device, if I take the photo and then share it with you or put it on Snapchat or whatever, Android or iOS, there are databases that know everything about the images. So on iOS, you have photos that SQLite on Android external.db R2 that will tell you the application that was used to create or take the photo.
So if it was a AI generated picture, it’s actually tracked in that database. The issue is it doesn’t just carry into the EXIF. So even if you had EXIF and you were just looking quickly, it’s not going to be with the image, it’s going to be in the database that has information about the image, which makes it a little bit more complicated because the tools don’t do a good job saying, “I know this from over here, I know this from over here.” So it requires manual forensics.
Si: And I mean to be fair, you’re both representing Cellebrite today so is that something that you’re looking at in terms of device forensics? If I plug in the phone, is it going to…is that something that’s that you’re either able to do or going to be able to do soon?
Jared: Yeah, so one of the one of the features that we did somewhat recently, it was called Media Origin, and it took, I would say, one of the most commonly asked questions and simplified it, but the simplification was actually extremely difficult. “Was this photo taken with this device or not?” And, “oh, well, just go, go look at the DCIM folder and you’ll know.” Well, no, that’s not how that works, because if you send me a photo and I save it to my gallery, it hits that same folder, right.
And then we’re digging into metadata to understand like which type of camera and as you describe being a photographer of sorts, if you take this pristine photo somewhere in the world, you don’t want to post it and then have someone say, “look at this picture that I took, it’s mine, it’s not his,” right? And so they’re the metadata and that signature of me, the photographer matters. I’m not really sure that we have a clear answer on the easy button.
So Media Origin was fun as it solved the problem of like, “hey, this…taken by this device,” that’s really important to my investigation, a really fast filter, or not, right? And so to say what, what things may have been received from someone else that are resident on this thing and filtering that big chunk in or out can make analysis quite fast. An easy button for detecting AI is, I think, even more complicated than the previous, but it is definitely something that we’re working on. One of the really common things that we see as we dig into the file system and understanding how these different applications are behaving is the file path, like path to file, as simple as it sounds, like file names, file paths. A lot of the applications that we tested created some pretty grotesque file names.
Heather: Like whatever you typed in…
Jared: Like your prompts plus, you know, dot web P or whatever the file type was, right? So some obvious indicators that can get you there quickly. I think the arena where we have the biggest problem with this is not necessarily the slow on device forensics. If you really have to figure it out, you can probably dig in and take, you know, a week to be sure of something. But we’re making this analysis up from online platforms where it’s, “real or fake? Real or fake?”
And truly the thing that was posted, so taken with whatever application, modified with AI, and then posted where everything gets stripped off to the internet, and then law enforcement or whoever is going to seek that file and say, “give me that, that’s the bad thing, I think.” And when they get it, they have this really disconnected version of where it started. So it really always brings you back to that originating device. And so I think that investigative approach is very solid to determine what happened on this phone versus another, and then file paths and file names are sort of a quick win. There’s some other stuff coming for sure, but there’s like the staple quick wins, I think that we’ve found so far.
Heather: And I think from the investigator perspective, similar images. So if you found…if we walk back through what Jared just said, and you find the AI one in Snapchat, but then there’s two other iterations of it, similar images in things like Pathfinder, could also help a lot.
Si: Yeah, no, I mean, again, you step outside of the technical sphere and into the investigation sphere at that point, don’t you? It’s like, you as a human being are able to say, “okay, you know, I found three variations of this particular type of image obviously being created and using a similar prompts to get to this point. Therefore, this one is probably not…or probably like that.” So yeah, no, I appreciate that. I mean, there’s been a lot of talk about the AI companies. And it’s in the AI Act in the EU as well about putting watermarks into imagery. What are your thoughts on that? I’ve got some fairly strong thoughts on that, so I’d interested to hear what your opinion is.
Heather: One: they should have to do it. But you can pay your way out of it from what we’ve seen so far for like $6 US dollars. You flip a switch and you pay them money and they take their watermark off. So that has to stop. Like, if it’s a forced watermarking to keep people safe, that’s not good enough. You shouldn’t be able to pay your way out of a label.
Jared: Yeah. I think philosophically, I love that obligation placed on those generating the content. It clears up the whole dispute of real or fake. Like, “if it came through here, we modified it in some way. Just letting everyone know.” And that would just clear up so many things. At least to have a flag or, you know, a skepticism mark. “This isn’t the original thing.” And, you know, if my face on this video, you know, has less lines up here than it really does in real life, we all click that button to say, “please, just fancy me up a touch.”
Heather: I don’t know where that button is on here though, so if you want to give us that button, I’ll press it!
Si: Yeah, sorry, this isn’t Zoom, it doesn’t have the smoothing effects and there are no Snapchat filters for here. If we talk to Zoe nicely, she might do something for us in post. I, unfortunately, I had some dental work and I can see now in this bloody video the gap in my teeth far worse than I’d noticed in the last two weeks when it was sorted out. But anyway, so yeah, we might be having words about that! I think, certainly from my side, I think watermarking is fascinating, but I could…my…it strikes me that it’s almost, I mean, apart from that, I didn’t…I wasn’t aware you could pay to get out of it. I mean, I think that’s horrific.
But it’s almost impossible to enforce because I can download open source software that will allow me to generate stuff and a) it doesn’t come with watermarking anyway, and b) even if it did come with watermarking, it’s open source. I can recompile it, or somebody competent could recompile it without the watermarking included in it. Leica put out a camera that I can’t afford that has watermarking built into it to generate a watermark for genuine images. Do you think that’s something that we might potentially see rolled out into things like Apple devices and Android devices, whereby instead of authenticating things that have been created by AI, we actually authenticate things that are genuine? Is that a concept you think that holds water?
Jared: Okay. So…but I think I would just challenge even your foundation of that question with what you said before, of that what the device is doing for me as I press the button to take that photo, and is it real, is that the real thing? And you mentioned the domestic assault injury. And I think back to being in law enforcement and looking at that arm with a bruise or the red mark, and then trying to capture that with the cheapest digital camera that was assigned to me. But think of even just the simple thing of lighting. Too much light on that thing makes it look like it didn’t even happen. Not enough light makes it look like, “I can’t really tell.”
And it would have to be some version of perfect. And we hope that when the flash goes and that it captures that it’s the correct thing. I think from what you said, I would love a feature as a Apple iPhone user that just puts even the simplest small Apple emblem to basically say, “the native camera on this device took this picture.” I don’t know how they would police…I now take that picture to the next thing and modify it at an auto strip off the watermark because it’s been changed now. I don’t know. But I like the idea. And they very proudly have commercials and everything to say, “filmed by the iPhone 6 Pro”, right? Like, they’re proud of it, but I don’t know about the visual so that everyone else can say, “that was just simply taken with the camera.”
Heather: And it’s in the metadata, but think about how many people don’t look at that. When they’re just scanning images quickly. If you have…if you’re working CSAM investigations and you’re looking at hundreds of thousands of images, you’re not looking at the metadata of every one. You’re trying to see how it exists. So I think that’s tricky too, for the push button examiners. We’ll have to call them that. We’re going to call them…
Si: Yes. I think…I mean…I think it’s that…I think what…you call them push button examiners. I mean, there’s obviously various schools of thought about how one approaches this and on the one hand, you want people to be as thorough and as detailed and as technically knowledgeable and as, you know, evidentially sound as they can possibly be when you’re presenting evidence that obviously is of critical importance. Conversely, we also know that there are hundreds of thousands of phones in the world and there aren’t that hundreds of thousands of police officers to look at them.
And therefore some of these police officers aren’t going to be as highly trained as others. And that’s where tools such as Cellebrite that can do this consolidation piece to say, “okay, this is…there’s enough evidence to suggest that this is AI versus not AI.” Not that it makes any difference in this country, certainly on CSAM anyway, because, you know, it’s an offense to have either generated or real.
Heather: Yeah.
Si: Slightly different offenses, but you know, it’s still charged very similarly. Do you, I mean, I believe if I remember correctly…and I did read the…I haven’t read obviously your original work, because that was in…you presented that at the conference. But did you touch on, sort of, sensor noise as well? Is that something that Cellebrite can look at? So image ballistics, as other people have referred to it as image ballistics, where, you know, a given electronic sensor creates, you know, known noise patterns. And also those noise patterns are reflected across multiple devices of the same type. So, you know, there are similarities between the sensor noise on one Apple and one and another Apple.
Jared: Yeah. So I think our approach…I haven’t…a while ago, I heard someone say media ballistics and they were sort of talking about what we did already with Media Origin. And so Media Origin allows you to say, “this device or another device or an attributed account of the owner of this device.” So Jared’s Snapchat username is this, and we see that that’s the logged in account. So these photos are also attributed to this person. That’s sort of an approach. Not necessarily the way you’re describing as the media ballistics, but I think as we move forward with our AI detection, it will fit in nicely with the Media Origin.
To say, you know, we’re checking all these other, the semantics of all these other things, and now we think that taken with this device, but also this one here, label it as AI, gen AI, modified AI. I think another difficult part is: was there an original thing that was touched up or was this a prompt that became a file? Right? And that’s a big thing to tackle because there’s so many avenues on the device for one or both of those things to happen.
Heather: And we actually…we just finished a CTF at Cellebrite, and it was our biggest one ever. It’s always fun to create. But we had a little mishap where I was two people at the same time on a Disney cruise and I left the other phone on the cruise. And I was like, “I just ruined our entire timeline, our entire scenario.” But Disney shipped it back to me. I filled out a lawsuit and I got the phone back. I was like, “excellent!” But in our chats, we had already kind of murdered him. So we’re like, “he’s dead. Now what?” So I was the girl he was dating.
So I was like, “you know what? I’m going to use AI and I’m going to become him.” And this was like to generate images, to put Facebook posts, like all the things. And then Physical Analyzer parsed all of it as a chat. So you could literally see what Jared was just saying. If it’s gen AI created, if I said, “create a picture of a man with headphones on a podcast,” it showed exactly what I asked for in the response as a chat, and I was actually kind of annoyed because I didn’t want it to be that easy. I was so impressed. I was like, “I guess it…”, and then I was kind of put off on it’s not a conversation, but it is. You are talking to an AI chat bot asking for something and it’s giving you the return.
Si: You’re getting back into the semantics and philosophy of AI as a whole. I remember…I’m was going to say, I am older than both of you, I suspect, both by a reasonably large amount. But you know, if you may have come across Eliza as the therapist chatbot purely because you’re in IT, and therefore you may have come across it. But the idea of the Turing test of, “am I talking to something?” is obviously one of the…talking to a computer or to a human being?” is one of the fundamental pre premises of AI as a field. So to hear you say that you’re considering these things to be a conversation, it is quite fascinating, really. I was going to say that I totally lost my train of thought, which I really like because it’s always fun. Totally different rabbit hole! Oh, yes.
So, in terms of AI detection. Okay. I mean, obviously we’re looking at things like metadata and we’re, you know, the sensor noise is another option, but quite a lot of people are leveraging other AIs against images to get results. So effectively adversarial neural networks that are going, “is this a real image or not?” Is that something that you’re using and Cellebrite at the moment, or is that still a stage away in terms of you’re looking at the hard data rather than throwing it into another black box to ask it questions that it doesn’t necessarily know how it’s answering.
Jared: So, I’ll talk not Cellebrite for a second. What you just suggested is I need to ask AI to tell me if something is AI. How would you…what if…so, and I think this is the foundational challenge that we have right now in digital forensics is…you alluded to it before: volume of devices, volume of data, not enough trained people to do the work.
So we are certainly on a path of how fast can we get the job done? And we have this conversation somewhat frequently of saying, “look, the speed of that job can only be as fast as we can do it while being forensically sound.” We can’t just pipe straight to an end product and everyone just throws their hands up and says, “I don’t know. I didn’t really look at it, but that’s it.” So to ask AI to do something for us, I think, we certainly have a lot of things that are being considered. We have to do it safely, right? We’re not just gonna jam stuff in that’s not carefully being used. But to increase efficiency, I think there’s a lot of things, and we already have some. So our Cloud Solution Guardian.
So, storage of evidence, but also a SAS based analysis platform. So instead of your hardware powering everything, AWS is behind, you know, full throttle. So sort of a faster way to look at data. And within that we see things like chat summary. So to say, “here’s a thread and within this thread, we see a conversation about eluding the police while discussing potential drug use and weapons.” Something like that, right? Now, if that thread is one of 1000 threads in the queue and on the other side of it is a chat with my mom and down here with my wife.
And that one in the middle is the one that I truly need to find as an investigator. I’ll take that. I’ll take that summary that points me in the right direction. That doesn’t put someone in jail. It doesn’t output an auto report of saying, “here’s why we think this person’s guilty.” Like, none of that. But it got me to what I’m ultimately going to be looking, and so trying to find ways to integrate safely into product AI, it’s definitely happening. But with that, we are definitely faced with a challenge of: what is AI to the court? Like, I always throw this example: when I was a police officer in Maryland, they said, “texting and driving is really, really bad. We’re going to make it illegal.”
And the first thing that hit the books was you cannot send an SMS text message while operating a vehicle. Well, that’s great. But as I sit here and watch the phones glow and the driver’s hands going by, I have no idea if they’re surfing the web, on YouTube, like all the things that you didn’t make illegal. So you put me in a pretty tough spot, right? We’re sort of at that moment with AI, of the courts are now going to start to see the use of AI that led to something that hits the court. And the analysis of that is sort of something that we’re going to be waiting for at least for the next few months.
And we’ll start to see initial cases that potentially deal with the issue. But we’re at the very beginning of what this looks like. And as a company, we’ll have to react a bit to what is palatable by those triers effect.
Heather: It will require a human. That’s what I think everyone has to realize. You can’t go from AI to court. You need human eyes on it in between.
Si: I think essentially what you’re saying is that there’s a big difference between an investigative lead that’s generated by a tool that helps you to find the evidence that you’re looking for that you then examine, and pressing a button that gives you answers to everything that you then presented court, not actually having reviewed it in any way, shape or form. I think…I mean, I agree with you. I think we’re at a fairly critical stage in…somebody sort of said the other day in this conference I was at, is that we sort of we had, you know, various revolutions over time of, you know, stone age, iron age, whatever…industrial revolution.
This is probably one of the larger revolutions that we’re going to face, possibly even more so than the digital revolution itself. You know, the information revolution happened thousands of years ago at the Library of Alexandria before it burned down and then, you know, okay, so we’ve made it a bit quicker, but fundamentally there’s not been a huge amount of change in some things.
This will generate comments if nothing else does. But this lack…this point where we are effectively handing over understanding or deep understanding of what’s going on inside the black box and then allowing it to make decisions is an interesting turning point in the world and therefore is something to be watched with a great degree of interest as to the way the courts will judge it.
And I mean, it certainly has come up in…there’s a couple of US cases that were cited the other day, whereby somebody had upscaled and enhanced a video and then asked if they could admit it as evidence, which obviously (I’m not going to say, obviously), fortunately, the judge decided was a bad idea. But he only decided it was a bad idea because he couldn’t explain how it works, not because it in and of itself was clearly obvious to anybody thinking about it that he’d just created a bunch of stuff and stuck it in a video.
So I think we will see some interesting case law that happens over the next, like you say, months, years, couple of years to set the standards on that. So, I mean, obviously, this is a not insubstantial piece of work, and on the one hand, it’s never going to be finished because every time you build a better mousetrap, the…nature built a better mouse. So your AI hunt will continue ad infinitum. But what is the next step for you guys on this one?
Heather: So we…when we did techno security, I feel like we left it as, “you must look at the metadata you understand and then focus on the gaps” on the ones that just are missing stuff, which is terrible. It’s not good enough. And then we just did this presentation again. And that’s when we found photos that SQLite and external DB tracking the app that was used to create it.
But I think, honestly, testing the tools that are starting to release AI assistance (that’s what I’m going to call it), AI assistance, because it’s not the answer, will be next. I plan to (and it was actually Jared’s idea), for RSA this year, I’m going to submit the idea of doing the misinformation/disinformation, but even have different countries, like where you are and what’s created and try to get people thinking on what’s real and what makes those things target you in different ways. So I think all of that is going to keep us really busy personally.
Jared: Yeah. The…as you said before, the volume of data that is, you know, everyone is facing today, we’re going to see AI applied in a good way. As everything adjusts and allows it to be used. But at the same time, the speed with which people can do things that is creating more data is amplified in a great way. “Hey, write this email for me with this tone,” and it’s done in seconds, right? So the faster we’re creating data, we’re still not dedicating the proper resources to all the police units and the people that have to deal with it. And I think that’s one of the biggest challenges that I’ve seen in the digital forensics community over time, is very few places have paid attention to the volume and resources around digital forensics.
Everybody’s happy when you find a deleted message that incriminates someone, or prove the case with whatever picture, video that you find. Like, everyone is super happy for that, and they expect, you know, this, “give me everything, make it fast, and be able to explain it perfectly later in court.” But we’re not scaling those units as we would everything else if it saw the same volume. Any crime that happened, if it started happening 10 times more within 3 years, you would have that many more investigators applied to go investigate it. Digital forensics has every single case doesn’t matter, big, little, everyone has some sort of digital footprint and it has not been prioritized by resources. So I think we’re hopefully going to see that challenged, that fixed, because we’re headed for a really tough few years of the volume is going to go crazy big.
Si: Yeah. And on sort of just again, taking what you’ve just said and going off on a complete tangent…because I’m good at that, and Desi’s is not here to keep me in check today. I’ve heard an interesting…so I’ll do a background story on this one first. In the UK, we had a huge problem several years ago with something called mad cow disease. Okay, mad cow disease is a disease called bovine spongiform encephalopathy. There you go. That’s random things that I know.
But actually what it is to do with is the fact that we were feeding cows cows. Okay. They were…we were taking dead animals, turning them into food, and then feeding them back to vegetarian animals, which is not a good way to go. It’s the way of horror movies. We should have seen it coming. If you just watched a couple of things, you’d know this is a bad idea. We did it anyway. We weren’t alone in it, but we did it anyway. But there’s this concept that we’re getting a whole bunch of AI generated imagery, putting it onto the internet, and then the internet is scraping that AI generated imagery and using it to train AI generators. What do you think might be the outcome of this in the long run?
Jared: So we have a, I’ll say a great colleague, Dan deBeaubien, that works at the SANS Institute, and he’s very good at explaining AI, because it’s complicated. But also pushing forward some of the proper guardrails that need to be applied to some of the models and how it behaves.
And so I think surface level consumers, I think of AI as: I open up a prompt, I type and it does something for me. But as we start to apply it to commercial products and enterprise level things, I think if we’re extremely careful about what it’s intended to do, we can avoid what you were just describing, which was a digital forensics mad cow disease. But you’re right, you know, if the thing is supposed to go out and scrape everything and just know, if what’s been scraped is already terrible and wrong, then it’s going to start to produce that as the correct thing.
And that’s why Dan’s name popped into my mind is because I’ve heard him speak about, you know, there is a good proper way to do this, and to do it safely. And I know that in the US there’s some government directives around AI and making sure that as it starts to be applied to lots of different things, that it’s done correctly because we don’t want, well, the targeting system decided based on the decision tree that this thing had to explode.
And well, there was a mistake. We wish that there was a human there to make that decision. And there wasn’t. So there’s a lot of things in the balance. And, you know, when you’re talking wartime decisions, it’s very different than digital forensics. But I like what you said, because you’re right. If there’s a junk out there and it’s the only thing we train it on, then it’s going to produce junk, and that is unfortunately…
Heather: Yeah, but the negative side of that…yours was very positive: your AI response was…
Jared: I killed the wrong person!
Heather: The negative is what you said earlier: it’s open and you can have smart programmers do whatever they want and change it and still provide access. And that’s what we have done AI generated CSAM talk several times and people were like, “isn’t it protected? Aren’t there things in place to prevent it?” But you can’t prevent someone smart downloading it, training it on what CSAM looks like, and then it’s out there. Now other people can search for it and use it. So we will have nice things that people will always do terrible things with. That’s what happens, right? So you will, like, have a little bit of a mad cow in there.
Jared: Yeah, sure. I mean, it’s…I think as a company, we’ll do the opposite, right? So one approach that we have was training a model based on a large amount of child sex abuse material to basically say, “if we can understand what these images and videos are, we can flag potential hits for our users.” Much as what I described before of that investigative speed of not saying, “hey, this is illegal,” but saying, “hey, take a look at this one specifically because we think that it might be.” Yeah, and if you…
Heather: …and preserving mental health.
Jared: Yeah, right. Like, I have to scan through, you know…my screen’s disgusting all day long, but if I can save myself a bunch of time and be efficient with viewing some of the disgusting stuff, then I think that we definitely preserve some mental health along the way, which a lot of times gets lost in the speed and demand.
Si: Yeah, I mean, I couldn’t agree more in that regard and…I mean, the counterargument to this is always, “well, what if it misses something?” But it’s not as if human examiners are devoid of…especially when you are scanning…I mean, I’ve certainly done cases where I’m reviewing somebody else’s work and I go through their stuff and I find another five images because they didn’t find them. Or something like that. It’s not an unusual thing for a human to miss it as well.
So there is some argument for a degree of false negatives to be allowed. I think the problem is always around false positives and false negatives whether it’s, you know, human detection or not. And I think the area where I personally have the biggest concern is the idea of…somebody told me the other day that they have a…oh, actually, a couple of things…one came up, which was to do with…it’s the largest supplier of body worn cameras in the US, actually, it’s Axon. They now have AI technology that writes police reports on the basis of what it sees in camera…
Heather: My goodness!
Si: …for you. So, to speed up the reporting process, so that scares me slightly. I mean, again, if it speeds it up and somebody then actually properly reviews it and goes, “well, actually, you know, that’s not correct, or it needs this added,” I’m okay with that. But we all know what happens to process when time is on the line and money isn’t. And there’s that.
But the other one is the aging of CSAM because, you know, it is a subjective process anyway. And I don’t understand how an AI can have any subjective idea of what that is, in my head. But you know, that’s just me and my nightmares. So that’s fine. And then…but the idea of anything that could help out mental health is definitely a good thing. Now, while we’re on mental health, I’m just going to be completely personal and ask you: is it easier for you to be able to talk to each other about it?
Heather: I think so.
Jared: We have a really healthy relationship. We try really hard to keep this solid. So, and, you know, we’ve had a lot of personal stuff coming at us in the last few days. And we’ve sort of, you know, taken a moment to say, “glad we have a solid foundation for this one. We’re great.” But you know, we take a lot of action and efforts to take care of us first.
So I think, from my time in law enforcement, it would have been taboo for me to raise my hand and say, “I feel…I’m feeling (really anything) like I’m feeling like, I’m struggling to do this work anymore,” or like that would have been…I don’t know that that would have been laughed at. I had an amazing organization and administration, but it wasn’t normal then. Mental health wasn’t a conversation. Barely, within ICAC units, the crimes against children specific people. I think we’ve come a long way since I left law enforcement in 2018, come a long way for mental health in that arena, for sure.
But also we’re talking about it, right? There’s platforms to…right here. We’re speaking about it, that it is okay to not be okay. And that was again, not something that was ever really said. Law enforcement in general shows up. They see terrible scenes, not just crimes against children’s stuff, but just terrible, terrible things that they’re supposed to walk into their house later that evening, kiss your husband, wife, kid and pretend like everything’s fine.
And then tomorrow the same thing can happen again, right? Like, it’s…what would shock some and, you know, send them crying for a week, law enforcement is expected to deal with on the fly and to keep making good decisions no matter what. So, definitely a challenging task. I’m glad that things have changed a good bit to be able to prioritize mental health and I know a lot of…when things happen now, it’s like an automatic task of, “you experience that you’re now going to go…you don’t have to say a word, but you have an opportunity to, and you actually have to go,” and then, you know, “if you need more, we’ll take care of you.” But it’s better, but I’m sure not great. And I’m sure there’s many that still struggle quietly. There’s one person units of “I look at CSAM all day.”
Heather: Yeah.
Jared: And no one…no, you know, sounding board, no mandated walk outside to clear the air, right? Like, I don’t know.
Si: No, it’s back to that issue of there’s more and no more resources being allocated. And, you know, even if we assume that it’s just more people and the percentage of CSAM doesn’t increase or the AI doesn’t generate more images that you have to work through, it still means that there’s just more. So yeah, no, you’re absolutely right. So, you know, correct funding is obviously a hugely important thing. Is that (and not getting too political), but is that something that the new administration has any interest in? Improving is funding for law enforcement. It struck me that perhaps that was something that they might invest into.
Jared: I don’t know so well what is coming next. I believe that the early sound is that I think law enforcement funding specifically (and I don’t want to go down a whole bunch of political rabbit holes, I’ll just stick with it as we’ve been talking) I do believe that law enforcement funding will be solid or increased, but they’re not going to see a defunding moment here. Where…how that gets prioritized, at least for the United States, unfortunately…you know, Heather took the stage at RSA this year and talked about sextortion and about, you know, some of the resources that are available to everyone to educate your kids and to make it normal to talk about very uncomfortable things.
As we come out of the election cycle, I think the only thing that everyone’s screaming about is border security and, you know, sextortion, as reported by the National Center for Missing and Exploited Children, has seen a phenomenal increase, which is terrible. The numbers are massively increasing and we don’t hear enough, in my opinion, conversation from those who are choosing where the money goes. Ultimately that money needs to go to digital forensics, it needs to go to regulatory functions around social media to mitigate some of the activity that ultimately causes children in their teens to be preyed on, and entrapped by these people from wherever in the world whose whole job is to fish for that next victim. Without consequence. So while law enforcement may be funded adequately in this next cycle, I worry that the priorities may be missed with that money.
Si: A very good answer. Thank you. I appreciate your candor. It’s interesting because in the UK, we changed government recently, earlier this year, and they have done some things to the legal system that have not necessarily made a lot of sense. But one of the things they’re mooting at the moment is the idea of banning social media for under 16s. It’s just been done in Australia. And they’re looking at that now to see whether that’s something that we could roll out here. And that would probably have a huge impact on quite a lot of this, this as a concept because if you can’t do sextortion, you can’t get, you know, images, you can’t do grooming of somebody who’s under 16 because they’re just not on social media. Obviously that makes a huge difference, but we’ll, you know, we’ll see what happens.
Heather: You’ll have to be in the video games too, because they’ll get them through those chats.
Si: Yeah, absolutely. There’s…something you can rely on is that criminals will be very inventive about ways of carrying out their crimes!
Jared: I think we’re trending in the other direction. Instagram just released, like, Instagram for teens. So very specifically targeting, “here’s a platform just for…”, and you’re targeting a group that doesn’t have a photo ID or some qualifier to say, “I’m definitely not an adult.” You know, maybe they do a great job. And I don’t mean…I have a tremendous amount of respect for most social media companies trust and safety groups and what they try to do while participating in the obviously bigger machine that is the platform, the ads and all the other stuff. But to say, “let’s sign up all the 13 year olds and let them share pictures and have chat…”
Si: Yeah! How to create problems for yourself? Yeah, absolutely.
Jared: And will very surely generate just more content, which is bad.
Si: Yeah, it’s bad. If nothing else, it’s just purely more volume for everybody. Yeah. So, thank you so much for your candor and your honesty and doing the research because obviously it’s an incredibly important area that we need to address and, you know, nobody really wants to do it, if we’re brutally honest! It’s really nice when it’s a theoretical pictures of fluffy ducks, and it’s a lot less pleasant when it’s…
Jared: We have development teams that are focused on doing a good job with this. Like, it’s a task that is assigned and being worked. Don’t take just our, “hey, we want to look into this and we want to determine, you know, what’s the easy button? Is there some really easy way to detect AI as people start looking at pictures and seeing things, you know, every day in their analysis?” We have other things that are really actively being worked to do better, faster. But this…sort of our first splash in 2024 was let’s just figure out what we can see. And it just so quickly went down these rabbit holes of every application behaved a little differently and so it wasn’t a red flag, “just go look for this thing”, which we hoped for. It wasn’t.
Si: Yeah, that would have been nice, wouldn’t it? But I mean, it’s really important because you’re…I mean, you know, you work for a commercial company, and you are out talking about it. You’re coming here to talk about it. You’re talking at events. You’ve got, you know, publications online that talk about this. You’re not keeping it to yourselves. And I think that’s a big mark of your integrity and shows the integrity that you’re operating within. That it’s not just about, you know, doing it better for Cellebrite, it’s about just doing it better. And you know, I’m hugely grateful to you for your work on that. And, you know, thank you very much. And it’s fascinating. And it’s an absolute pleasure to talk to you both.
Jared: You as well. Thank you for this.
Si: And, no, it’s been wonderful. And hopefully Desi will be back with us the next time we have this opportunity, if you’re willing. I mean, Heather’s done this once and you’ve now done this twice.
Heather: It’s great, it’s fun!
Jared: I mean, I love this.
Si: So you might come back another time, maybe with two headphones, maybe with one. I think this has worked really well! We’ll see how it goes, but thank you so much for coming on. Everybody who’s been listening to the podcast, thank you for joining us. I hope you found this as interesting and as exciting as I did. You can find the podcast on (and I wish Desi was here because he knows all of these), it’s on Spotify, YouTube, you can pick it up on the Forensic Focus website. There are other things, Apple…Apple i…podcasts. I don’t know.
But anyway, thank you very much for joining us. We hope that you’ll be back soon in the near future to listen to the next interesting thing that we come up with. But again, thank you so much for both joining us. And I’ve thoroughly enjoyed this and come back in six months and tell us what the next great thing is. Because I’m really looking forward to it. But in the meantime, thank you so much. Honestly, and, I’ll say goodbye, goodnight and stop the recording at this point.
Heather: Thank you.
Jared: Thanks.
Si: Pleasure.
