By Massimo Iuliani, Forensic Analyst at Amped Software
When deepfakes first emerged, they exhibited evident visual inconsistencies that could typically be detected through simple visual inspection. However, in 2021 a scientific research highlighted that people could not visually distinguish between natural and synthetic images anymore!
In the last three years, with the advent of diffusion models, there has been a noticeable improvement in the tools for the synthetic generation of images. Amped Software presents a selection of images below. It can be quite challenging to visually discern whether they are natural or synthetically generated.
Before moving on, it is important to clarify that deepfakes can be generated by different technologies. For instance, they can be created through Generative Adversarial Networks (GANs) or text-to-image models (also known as “diffusion models”, such as Dall-E, Midjourney, Stable Diffusion and Flux). You can find a brief introduction to the differences between them in an article on the Amped Software blog. However, diffusion models are gaining prominence due to their ability to convert textual prompts into photorealistic images.
One of the first technical approaches to detect synthetic content is the analysis of metadata. Indeed, tools used for generating synthetic images may leave traces within the image container by embedding specific metadata.
For instance, a synthetically generated image’s metadata may disclose the prompt used in the text-to-image process or include AI-related tags.
Below, Amped Software presents a subset of the metadata extracted from an image generated using diffusion models.
The value “AI generated image” is clearly visible in the “ActionsDescription” field, which is very self-explanatory.
However, metadata analysis has prominent limits and can hardly be considered a definitive solution.
First, some AI images are generated in .png or other formats with no relevant metadata. Below, you can see the metadata extracted from the first image shown above (depicting two soldiers in the desert in front of a refugee camp and a tank).
It can be observed that the image is described with less than 10 very general metadata fields. It is difficult to determine the origin of this image from these field values.
Second, even when some AI-related metadata is attached to the image during the generation process, it is generally removed in the next steps of the image life cycle. The most common situation is when the image is uploaded to a social network: most of the metadata is stripped, thus making the analysis useless for detecting deepfakes.
Last but not least, metadata can be manually modified by a user. This process, although technical, is feasible by a skilled user, thus limiting the reliability of metadata-based analysis in various contexts.
Luckily, recent research has shown that diffusion models leave identifiable traces in the generated images. Notably, one of the latest studies demonstrated that diffusion model traces can be detected using CLIP (Contrastive Language-Image Pre-training) features extracted from the image [1]. This research exhibited excellent performance both in the scientific paper and Amped Software’s internal validation. As a result, Amped Software has decided to integrate this specific deep-fake detection filter into Amped Authenticate.
In this first release, Amped Software trained the system with a diverse set of real images, as well as images generated by popular text-to-image methods, including Midjourney, Stable Diffusion, and Dall-E. When you process an image with the filter, the output will be displayed in a table format, showing the confidence score assigned by the classifier to each class. The scores sum to 1, and for user convenience, the class with the highest score is always reported in the “Predicted Class” row.
Let’s see the output of the tool on the same PNG image with missing metadata:
In this case, the filter shows that the image is strongly compatible with a synthetic generation process. More specifically, the image statistics are highly compatible with Dall-E (confidence 0.903).
As illustrated in the example above, the filter also provides a score for the “No Known Diffusion Model” class. An image with a high score in this class is deemed compatible (according to the detector) with other synthetic generation methods, as any other unpredicted image life-cycles. In this case, however, the filter is only confident that the image is not compatible with one of the trained diffusion models.
Note that the detection method is based on machine learning and the classifier can get it wrong for several reasons:
- There are numerous generation methods available. The filter may get confused when analyzing a synthetic image produced by an unknown model. For instance, an image generated with Fooocus can be misidentified as a Dall-E image. However, in this case, the filter at least provides a hint that the image is synthetically generated.
- Generative models are frequently updated, which means the filter may not recognize images generated by new, previously unseen versions of known models.
- Several generative models allow a deep-setting customization. It is unfeasible to predict how all available parameters can influence the traces left on the generated image.
- The filter cannot be trained on every possible image life cycle which means its behavior cannot be entirely predicted when analyzing an image that has undergone a very unusual processing history.
As a consequence, a high confidence score does not guarantee that the predicted class is correct.
On the other hand, however, the filter detects traces that are generally robust to compression. This means that the filter has chances to work effectively even when the image is exchanged through a social network. For instance, if we exchange the previous image example (depicting two soldiers) through WhatsApp, the filter still predicts Dall-E with high confidence (over 0.9).
In the end, Amped Authenticate’s Diffusion Model Deepfake filter is a valuable tool for countering deepfakes, especially in common situations where metadata is unavailable or unreliable.
[1] Cozzolino, D., Poggi, G., Corvi, R., Nießner, M., & Verdoliva, L. (2024). Raising the Bar of AI-generated Image Detection with CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4356-4366).
