Desi: Welcome, Forensic Focus listeners. We are back again with Marco from Amped. We’re really excited. We were just talking about having an anniversary with Marco, because we had him on the 4th of April, and today is the 12th, currently? Yeah, 12th of April. So, it’s nice to have old friends back on the podcast. And as always, we’re joined by Si.
Before we kick off, we want to point this out, because we always point it out at the end, but you can get the Forensic Focus Podcast from all your favorite podcast apps, and we also have it on our website and YouTube, where you can watch the video and see our talking heads, and we may chuck some stuff up. But thanks everyone for joining us, thanks Marco, and welcome.
Marco: Thank you. I’m happy to be here again.
Si: Yeah, great to have you back. It’s always a pleasure to have people over from Amped. You guys, A: are incredibly articulate and know your stuff incredibly well; and B: your products are fun. So, I’m really excited about hearing about this new one today.
So, this is a brand-new, spanking, shiny, fresh-out-of-the-box toy that you’re bringing to us that we can share with people. So, it’s called DeepPlate. So, do you want to give us an overview and what it’s about and what it does and what we’re aiming to get from it?
Marco: Sure. So, DeepPlate, as you can guess maybe from the name, it’s made of deep and plate. So it’s a deep learning system, so it’s an AI-based tool to read the license plates, okay? It will not give you an enhanced image of the license plate, which is what you normally do with Amped FIVE and our products. This one is for reading the license plate. So it’s something new. It’s a different kind of thing that we do with this system.
And it is completely designed for license plates affected by the common issues introduced by surveillance systems, okay? So. of course, you know, reading license plates is something which is considered basically solved in the state of the art when you have a proper automatic number plate recognition system. So you have a high-end camera installed in a certain way, for example, in the entrance of a parking lot, right? Or even at the supermarket sometimes, they will read the license plate printed on your receipt and everything. Okay, that’s one thing. DeepPlate is not made for that. It’s made for more challenging scenarios where the license plate is almost unreadable for a human and so we need some AI to do it.
Desi: So, when I think of number plates, they vary quite a lot and it would probably vary across different countries. I know within Australia they vary across different states, even. So is there a certain level of coverage that DeepPlate has?
Marco: Correct. We currently support eight countries, okay? I can list them if you want: they’re France, Germany, Italy, the Netherlands, Spain, Sweden, the UK, and the United States. And so, as you can guess, there are a lot of variabilities, especially for the United States where you have 50 states and all their various flavors of those licenses that they have in each individual state.
So we created a separate model for each country, okay? We can go more in detail if you want, but basically when the user uses DeepPlate, they are asked to select the country and for the United States, they can also select the state if they want, if they know which is the presumed state of the license plate.
So this is some kind of knowledge that you have to know beforehand. We don’t have at the moment a general purpose network because, as you said, it would be very hard to design because the font changes, the organization of the character changes, so you have to know what state you’re looking for, basically.
Desi: So, my geography is terrible, and this is an abstract question, but does it use all the countries’ models to try and look at them, or do you have to select a country? So, I’m thinking, if you’re in Europe, and say you were in a country that was selected, but you had another car from another country that was supported was driven across the border, would it detect that it’s that other country’s one, or would you still need to select it?
Marco: No, presently you need to select it.
Desi: Okay.
Marco: Yeah, we don’t automatically detect the country, because sometimes it could be very hard, because technically speaking for some countries, you have very similar licenses. The only difference would be a tiny character in the blue banner, and if that’s not really visible, you can hardly detect. So at the moment the user has to do it.
Desi: Yes, we’ll wait until Amped does 192 countries, and then we can expect the automatic country recognition, once you’ve got 100% coverage.
Marco: Yeah, correct.
Si: So, in day-to-day use, I’m going to use America probably as a prime example. Although, given that there are states in America that are about three times the size of the UK, it’s probably not a great example. But, I’m going pick the only American state that springs to mind, which is Texas, which is actually six times the size of the UK or whatever it is, but if you were looking for a plate, would you pick your state and then a selection of states around it? Can you choose more than one to work with or is it just one at a time and you would go around aiming to get the best read possible?
Marco: So, the way it works is that you upload your image, you select the country and the state, and once you select the state you can also select the format of the license plate optionally, okay. So, you can either say just the United States and that’s it, I don’t know anything else, and we will use a model trained with all the license plates from the United States without any rules about the first character should be a letter, the second should be a number and so forth. Or, optionally, you can also select the state and also the format of the license plate for that state.
If you do that, the neural network is basically the same, but we will add the information about where numbers or letters are expected, and this will of course influence the results. So if you can select the state and the format of the license plate, this will probably help the network giving you the best results. But if you have no idea, because maybe they travel a lot across, so you just don’t know, you can just say United States, and that’s it, and the system will do it with the general network.
Si: So, is the refinement in the search done not by the base data set, so it’s not going to give you an additional set of license plates trained in California, it’s just going to say that California starts with CA, and therefore that’s what we should expect to find? Is that the way I understand it?
Marco: Yeah, not exactly meaning that depending on the state you may have the first character of the license plate must be a letter, or maybe must be a number. So these are the kind of rules that are fed to the neural network. That’s basically it.
Si: Okay. So, I mean, that is immensely cool. So, in the UK, for example, we’ll take the UK, we actually have quite a wide variety of number plates now, going from, I mean, A1 is still technically a valid license plate, an incredibly expensive valid license plate, up to our modern standard, which is two letters, two numbers, and then three letters, I can’t remember.
Marco: So, for Europe, it’s a bit different, meaning that the moment you select the country, you will see a list of the supported formats and the picture of it, okay? And so you know that the network has been trained on all those formats at the same time. The network has seen all those variations, and it will automatically interpret it, okay?
So you don’t have to know in advance which one of the specific formats for that country you want to use, okay? That’s up to the network. So it’s easier in some sense because you just have to make sure that the license plate you’re submitting follows at least one of those because otherwise the results could be incorrect, suboptimal, let’s say, because the network has never seen a license plate of that format.
We don’t cover customized license plates, just to be clear. So for countries where you can buy, “I am the best license plate” and stuff like that, no, this is not supported because of course it would be nearly impossible to devise the rules for these kinds of license plates.
Desi: So, the formats you’re taking are essentially from the roads and traffic authority within each country to go, okay, this is the format that the standard license plate comes out as?
Marco: Yes, if you have exceptions like diplomatic license plates and stuff like that, this is not used to train because it’s not really representative of the large population of license plates. They are exceptions in some sense.
Si: And they’re immune from prosecution as well because they’re diplomatic. Anyway, so in the UK we have to adhere to plates being within the format that that’s allowed, but sometimes people mess around with the spacing on them to make them look like other things. The typical one is changing the color of a screw cover to make it look as if it’s a different letter slightly to make a word or something. How does that get handled? Or does that count as a sort of a personalized place in that regard?
Marco: So, colors are factored out, meaning that we convert the images to grayscale, okay? And that was a good choice with hindsight because now that you have electric vehicles, I’ve seen that you may have green license plates for electric vehicles. And we did some testing and this is not affecting the performance of the license plate at all because colors are removed. Also because colors, as you know, in surveillance systems get killed a lot. So the contribution we’re getting for color was mostly noise in many cases.
And also for tiny details like the color of the screws, I mean, if you can really see the screws, you don’t need DeepPlate likely. You see what I mean? The kind of images you would use with DeepPlate you really cannot even figure out there was a screw there. So if the personalization is very minimal, like slight spacing, things like that, I don’t expect it to affect the result as much. If they are strong personalization, then we are moving toward a customized license plate, and so we cannot guarantee the support for that.
Desi: Okay. That makes sense.
Desi: So, with the introduction of DeepPlate, how is it that a forensic video analyst would use that as part of their workflow when they’re looking at license plates? So, obviously this would solve a challenge for them, so it’d be interesting to just hear how that’s solving it and how they would use it.
Marco: This is a very important question, I think, because we strongly believe that all these kinds of AI systems are not there to replace the work of the analyst. Absolutely not. This is here to give you a second opinion, basically. So, the workflow we recommend is that you take your image, you process it in Amped FIVE, you enhance it as you would normally do, the best way possible, possibly integrating together multiple frames if they are available, so you get the best image you can get of the license plate. You do your own assessment of that. So if you are in charge of reading it, because that’s not always the case, sometimes you are just asked to announce it. If you’re asked to read it, you do your reading, and then you can go to DeepPlate and have a second opinion from the AI tool.
Maybe we will have a chance to discuss this later, but we implemented some bias mitigation techniques because if you look first to the output of DeepPlate and then you do your work, you would be biased likely, okay? So, you have already seen the output of the network and you may be tempted, unwillingly, to adjust the processing so that the result is the same. That’s why we recommend using the previous workflow, just do whatever you were doing before, the best you can, and then you have a second opinion tool, like that one, which is an objective tool, but of course, it can fail. It can easily fail, of course.
Desi: Yeah, so I guess, from an outsider perspective, not being a video forensic analyst, but the ideal case would be: you went through your workflow, came up with an answer, ran it through DeepPlate, it came up with the same answer, and that’s, you can’t say a 100% sure that you’re correct, but it’s just a nice little extra validation to help the analysts along that way.
Marco: Yes, perfect. Yeah.
Si: So, I mean, in that regard, how do you see DeepPlate being used for court? Do you see it making it there at all? Or do you see it as purely a forensic tool in the lab and then every other piece of evidence is presented as that?
Marco: Yeah, this is an important question, indeed. In our mind, DeepPlate is an investigative tool. It’s not really made to go to court, but to give you a lead and maybe using that lead, you can find other sources of evidence to find the culprit. And that’s it: you’ve done your job and we are happy to have helped.
It really depends on the national regulations, whether AI tools are applicable to court or not. I think there is a lot of evolution recently about this, about the EU AI Act recently released, and as far as I understand, the tools for law enforcement based on AI to do identification are considered high-risk, because of course they have a lot of implications. And in the US they are also discussing recently whether AI enhancement should enter the court. So it’s a hot topic.
It’s important to say that we don’t do AI enhancement. I mean, we will not give you an enhanced version of the license pattern, enhanced pixels. We will give you a reading. So this is not an enhancement tool. This is an analysis tool: I give you an image, you give me some letters, okay?
So, this is an important difference. It’s up to the analyst how to use this, but we recommend using it as a decision support system for investigative use. This is also why it’s not inside Amped FIVE. On the other hand, Amped FIVE is made for code, okay? We want you to be able to do your processing in Amped FIVE, create your report, and be safe to go to court, and that’s, I mean, agreed science in the field. I mean, if you know what you’re doing, you are using the correct tool to do it. That’s why the DeepPlate is outside; we offer it as an online service outside of the product. Does that answer your question?
Si: Yes, I think so. I’ve got two questions to follow on from that. We’ll come back to how we actually use an online service and how that’s licensed and who has access to it and stuff in a second. But actually, I mean, knowing Amped and knowing Amped, I’ve been familiar with Amped for a couple of years, I can’t claim to be as in-depth as all that, but you seem to have actually been quite anti-AI as a company for quite a long time. Why the decision now? Why are we going into AI now?
Marco: We are not really changing anything about our position on AI. We have been saying that you should use AI in a very controlled way, and we still stand by our position that the current methods for AI are not suitable to do image enhancement and video enhancement, because at the moment there is a lot of hype about the super solutions you can do with some solutions you can find online, but they are mostly creative solutions.
They don’t do anything wrong, okay? They’re perfect if you want to get pictures from the seventies and make it look like they were taken with a 40-megapixel camera. There’s nothing wrong with that. This is not forensic, though. They are halfway between enhancement and deepfakes, because they are introducing new details based on something that’s been learned through millions of images. So they are in some way contaminating the image with some external thing. Perfectly good for your holiday photos, not good for forensics, in our opinion. And we still stand by this position.
We are not anti-AI, although I can understand why you say that, because we have been very skeptical in the use of AI in some scenarios. We just think that the fact that some technique is available doesn’t mean it’s good to be used everywhere. With the current requirements for a trial, we think everything should be explainable, reproducible, and an expert should be able to tell how you reach those results.
This is not true for current deep learning methods; there is a lack of explainability in them. There is a lot of research on this, so maybe two years from now everything will be different, and we will update our position. But at the moment, the position is the same. So, this tool we developed, we developed as a second opinion tool for investigations. So if you go back and look at what we have published on our blog over the years, you will see that we always say that you can use AI for assisting the user. You have lots of tools for forensics that will try to prioritize your work based on something like finding the red cars in the video. I don’t see anything wrong with those. But this is different from sending someone to jail because the shape of the nose is the same and that nose has been enhanced 300 times by an AI system. There’s a lot of difference.
Si: So, in that regard though, I mean, you’ve got a deep learning model that you’re putting into DeepPlate. Are you updating that on a fairly regular basis? And if so, would I expect to get the same results out if I put the same plate in two consecutive runs? Because that’s one of the things that we’ll find is that as you change your model data, your results will change. How are you handling that? Because I could go to it one day and it could give me one number plate and I could go to it the next day and it would, having been updated, might give me another number plate.
Marco: Absolutely, yes. Actually, the answer is yes, we plan to update the model. We’ve already been doing this before releasing the service. It went through a lot of development and improvements. And we will maintain a changelog, so you will know when we update something. But as for reproducibility, actually, even now, if you go there two times, since you have to click on the four vertices of the license plate every time, if you click on a slightly different position, maybe you could get the same reading, but the confidence score associated to each character will most likely change. I mean, it could be 99 one time and 97 the second time because you click on a slightly different pixel.
We believe this is normal. And, since it’s not for evidential use, it’s not an issue, to be honest, in our opinion. But we are open to hearing feedback from users about that, for sure. So the answer is yes, we plan to update it. Yes, the result could change as a consequence of the update or as a consequence of multiple runs, even with the same model.
Desi: So, I’m interested to know with it being an online service, it’s all trained on just the formats of the license plate, there’s no integration or scraping from the web of, these are all the active license plates in California at the moment?
Marco: So, this is a very good question that you made and a very important point. So, what kind of data is used to train the system? I think this is something that people must be aware of because depending on how you choose your training data, first of all, you may be violating someone’s privacy, of course. Because you cannot scrape license plate images from the web and use them to train a model. At least, I don’t think this is legal in the EU, to be honest. I would have to check, but we didn’t go down that way.
And the second reason why you have to know the training dataset is because of bias. So, for example, if I only trained it on license plates that are, I don’t know, perhaps yellow, now, I told you that color doesn’t matter much, but just to give you an example, what happens when I will send it a white license plate? So, the way we designed our training system is published research. So there is a published paper about DeePplate. I can provide you with, of course, the link.
And we entirely trained the system on a synthetic data set. So what we did was to look at how licenses that are made in a country, recreate scientific images with the same font, same spacing, same everything, same banners. And once we create hundreds of thousands of these possible combinations of license plates, only valid license plates, meaning that if the country is made of letter-letter-number-number-number-letter-letter, we will only create license plates of that format. We will not put characters in the proper position, it doesn’t make sense. Then we do very strong data augmentation to mimic the same deterioration of a surveillance system. So we know that when you have a license plate taken by a surveillance system, it will probably have some blur, some perspective distortion, some compression. All these kinds of defects, okay?
And we mimic all these defects and we do doubt augmentation, reaching millions of license plates for each country, okay? And we train the system on these millions of license plates, which are synthetic data. We didn’t download anyone’s data from the internet. That’s the idea.
Si: That is fascinating, I have to say. That is really cool. So, and I assume as well as adding noise and blur, you’re distorting them from angles and all of the things you would expect?
Marco: Yes, correct.
Si: And then how have you tested that in a real place? Have you all driven around with your cars and handed over your speeding tickets and stuff like this to verify?
Marco: So, we did two kinds of testing. One was to test also on synthetic images that were not part of the training dataset. Then we tested, even at the time we published the paper, which is a couple of years ago now, on some real images for which we knew the ground truth. Because for testing, I mean, you can use real images because you are not embedding the data into the model, you know? So basically you just test and that’s less sensitive, let’s say.
And then we also run a research project together with an Italian university where they had access to lots of real license plate images and their ground truth and we provided them with access to DeepPlate to run it automatically,and they carried out large validation that now has provided some results and we are about to publish these results. So we’re writing a paper with this validation and hopefully we will have published it in some scientific journals.
Si: Okay. And I don’t know if you can share the results with us before the paper is published, but how did that pan out?
Marco: We were quite happy. I cannot share the results now because we have to polish them. You know, you have to do some data analysis. You just don’t put the percentage values, but the interesting part is that the license plate images were split into five levels of difficulty from one, where a human could easily read it, more or less, and there, of course, you are well above 90% accuracy for each character, of course, up to level five, where it takes a miracle, basically, because the license plate is completely white. And there we are much lower, of course, but still better than random guessing. But the intermediate level, which is where you can really hardly guess what’s written there, but still, the network performed decently well. I will let you know when the results get published.
Si: Yeah. We’ll definitely have to put that one out as an addendum at a later date so that people can catch up with it. Because, I mean, that sounds like an absolutely fascinating way of managing what is quite a challenging problem, and perhaps one that could be scaled to other areas of training AI for various tasks, so that is quite cool.
So, let’s go back to the actual service itself. It’s an online service, it’s a web service, so you are running the cloud and the back end of this. How do I get into it? What do I upload? And then what do you do with my data once I’ve uploaded it?
Marco: Perfect. So, to be able to access the server, you have to be an Amped FIVE user with an active SMS plan, support maintenance service. So either you can have a perpetual license with an active SMS plan or subscription license of Amped FIVE, and if the agency you work for is in one of the supported countries, otherwise, of course, it doesn’t make sense to allow you the access because it will be misleading, you can access it from your support portal.
So every user is invited the moment they are associated with a license to subscribe to the support portal from where they can download the installers. They can ask for support from our team. And you will now have a badge at the top, which is called DeepPlate, you click on that, and yeah, for each seat of your license, the agency is entitled to 50 users per month shared across all the users of the license. And so every time you use the software, you will consume one usage. This comes at no extra cost
Si:. Is one usage one login for an hour or two while I process the plate?
Marco: No, no, it’s one processing of a license plate.
Si: So one processing of a license plate.
Marco: Yes. And this is coming at no extra cost now, right? So, it’s an addition we do to the service provided when you subscribe to the SMS. And so once you log in, to answer the question, you will be asked to upload your image because of course we need it, you select the country and you select the license plate over the image because you may have an image with multiple license plates, we need to know which one you want to read. You click on the four vertices and you start the processing, we process the data, we compute the results, we bring you to the results page. And at the moment when you download your results, everything is deleted, basically. So we don’t retain your image, we don’t retain the results, we don’t retain anything. So after a couple of minutes, I think, just the technical time of processing it, everything is deleted. And indeed, we recommend the user to download the PDF version, because if they come back the day after, they don’t find their data anymore, of course. Because for data protection, we prefer not to retain it.
Si: So, what is the actual infrastructure that you’re building this on? Is it Azure that this is what you’re setting up?
Marco: No, we prefer it to hold the data on our systems at the moment.
Si: So this is all in-house, but obviously in your own data?
Marco: Yes, because we believe that this could be sensitive data sometimes. So, we just set the plate safe. Then of course it will depend on the scale this gets. Of course, we may need to change this, but at the moment we are keeping the data on our systems.
Si: So, they’re not sent to the cloud.
Marco: Yeah, to the cloud or other external services.
Si: I was going to say, 50 plates strikes me as actually relatively few, especially if you might want to verify your verification, because, like you said, if you’re looking at something that’s quite small and you might want to adjust where you’ve put your reference points, is it at this point possible to buy more verifications or request more verifications or get more verifications or get more verifications?
Marco: At the moment, you can buy additional seats for your license if you want, and you will be given more usages. This could change, of course. It’s part of how we commercialize the tools. We are open to changing this depending on how the demand grows. That’s basically the idea. For launching it we prefer to keep it free included with the SMS plan and see how it goes and get feedback from users. Because at Amped we love to get feedback from users and get good ideas and also monitor how much usage they actually need every month, because it doesn’t make sense to create a huge system then you see that a simpler solution is already good, right? So we are still in this monitoring stage about the way to make it available.
Desi: And so we’ve talked about some of the limitations and the concerns around bias and before I jumped on more, I was reading through the blog that you wrote, Marco. What are some other limitations and considerations that need to be made when using an AI-based tool like this as part of, I guess, a forensic workflow?
Marco: Yes, I think there are. So, we already talked about bias, so do not look into the results before you do your own assessment. And we try to help this in the service because the moment you reach the results page, you don’t see the results. There is a button which says ‘show results’. Now, this seems very simple, but it’s also very effective because there is a red box telling you, ‘before clicking this button, make your own evaluation. After you’ve seen the results, you are biased.’ That comes from all the guidelines about bias mitigation.
And those in the PDF, you download the PDF, the results are never on the first page, they are on the second page because, on the first page we tell you, make your own assessment first and then go to the second page. This is point number one.
Point number two: we explain very clearly in the results, at least we tried to, how to interpret those results. So the confidence score you have from a network does not give you the probability that character is correct.
So let me explain with a simple example. I put a license plate and they get the first character is an A with 99% accuracy or 100% accuracy, does that mean I can be completely sure it’s an A? No. It means that the network is very, very sure that it’s an A. But if, for example, that license plate is a custom license plate and it has a letter where a number was expected, well, the network could be easily wrong because the network has never seen a license plate with that character there and so the conclusion will be wrong. You see what I mean? So, you have to consider it kind of like another human giving you an opinion, but even if he’s very sure it doesn’t mean that that’s the truth.
Desi: So, this is interesting. Does it also give you a confidence interval on, let’s say you selected America, you selected Texas and then it was a custom license plate, but maybe you hadn’t realized or it was so obscure that you couldn’t see it, does it give you confidence that it is a Texas license plate as well or only on the characters themselves?
Marco: No, the confidence is only for characters. Hopefully, if you’re using the wrong pattern, you will have a combined low confidence overall.
Desi: Because it might not match the pattern for the state.
Marco: Yes, correct. And so hopefully the network will say, “Well, maybe if I have to say it’s an A, but I’m not very confident about that.” That’s the idea. In the preliminary results I’ve seen from the validation that I’ve told you, it seems that the confidence levels do actually make sense, meaning that when the network is very confident there are lower chances that it’s wrong. Well, that hopefully makes sense. But still, we make it very clear, the network could be very sure about something and still be wrong about it. And this will be for every AI system you see around. So if people are giving you an AI system and they’re not telling this to you, they’re lying, which is quite severe, of course, in some settings.
Desi: Yeah, I can say to all our listeners, definitely go read the blog because it is really well written in providing the limitations and explaining the percentages and how it can still be wrong, which you don’t see a lot with AI tools from a lot of other providers, I think. They’re not as transparent with how stuff’s put together.
Marco: Yeah. I mean, it really depends. You have to know where you’re working. I mean, if I’m doing a home assistant, okay, and I ask for, I don’t know, the recipe for doing a cake, and if you give me the recipe for making bread, that’s probably tolerable. It’s not so serious, right? If you’re doing something for investigations and forensics, I mean, you have to be very honest, I think, and open.
Si: I mean, I think the questions that sort of spring to mind are given that this is clearly working both from an academic perspective and potentially from a user perspective, is there anything that Amped is considering using AI for in other areas at the moment?
Marco: Well, there is an open discussion ongoing about this. I would say I cannot share much of that for now. But yes, I think what we want to do is to be very loyal to our principles. So, for example, just to mention, for deepfake detection in Authenticate, we already have the filter, which is based on AI, and this is clearly stated in the help of the filter and the blog. Because it seems from the scientific research that for detecting some kind of fakery, you really need to use data-driven approaches. So, there could be places in image analysis where we start using AI or keep using AI. We just want to be very clear about how we train those and the performance and whatsoever. So that’s the idea.
Si: No, that’s fantastic. Okay, cool. Well, I mean, I think that does wrap up for us, doesn’t it, Desi? I think it’s a brilliant product by the sounds of it and I look forward hopefully to getting an opportunity to play with one or two of somebody’s 50-seat license at some point and seeing how well it actually pans out. Well, I mean, as always it’s been a pleasure having you, Marco. Thank you and we’ll see you in a year’s time, if not sooner I’m sure.
Desi: hopefully sooner. It’s always good to have Marco.
Si: And we’ll just wrap up again to say thank you very much to the listeners and to you Marco and to Amped for being such a good friend of the podcast and always always bringing us interesting things to talk about. It’s always always such fun. Always enjoy it. And therefore just left to us to say, everybody keep safe, keep well. You can, as Desi pointed out at the top of the show, find us on all sorts of podcasty type things, and there will be videos and links available on the Forensic Focus website. Otherwise we’ll close it there. Thank you very much, everyone.
Desi: Thanks all.
Marco: Thanks, guys. Bye bye.
