We find ourselves analyzing new surveillance videos almost every day, and in most cases we can either solve the problem very quickly or understand (even quicker) that there is no information to recover in the video. In special cases though, where something very specific and strange happened, or the problem is very complex, it can take a lot of time.
As always… Pareto principle: you solve 80% of the cases in 20% of the time, and, well, 20% of the cases takes 80% of the time. In our own work, the right numbers are probably 95% to 5%, but the idea still holds.
With our experience in working on several thousand cases, we can estimate whether an image or video contains some information and is worth processing, or not, very quickly. In this article, we will describe some of the tests that can be done to quickly tell if you can get that license plate!
THE BIG PICTURE
Let’s work on the most common case: enhancing a license plate. The most common reasons why we can’t see a license plate (or anything else useful) are:
- low resolution
- lossy compression
- optical blur
- motion blur
- low brightness / contrast
In most situations you have all of the above combined together. But if one of these problems alone is strong enough, it may cause your license plate to be unreadable.
It is important to note that these problems are strictly related to the image quality and that other problems also exist that may be corrected much easier: for example a bad perspective or interlacing can immediately be solved, if the overall quality of the image is not too bad.
Let’s now show in detail all of these issues and what can be done to correct them. Of course, we can’t show you images from real cases so we’ve generated some samples for you.
A good quality license plate
The idea is pretty simple: the resolution determines the number of points you have in an image and if you don’t have enough points for drawing a digit, you won’t be able to see it.
How to recognize it: Zoom in on the license plate and count how tall it is in pixels. If the license plate is, for example, 3 pixels tall, just think how impossible it is to draw a number with just three points (maybe two because it will also have the plate border).
How to solve it: If the problem is very significant there is not very much that can be done. In some situations a simple digital zoom (the filter Resize in the software Amped FIVE) can be helpful; although it does not actually add information to the image, it will make characters and details better recognizable to the human eye. If you have multiple frames, applying techniques such as frame integration (Frame Averaging) or super resolution (Super Resolution) can be effective in some situations.
With digital zoom
Lossy compression is the main drawback of digital surveillance systems if compared to the good old VCR. Very often, the resolution is actually pretty good but the compression makes the digits unreadable. Videos are compressed in order to save bandwidth in transmission and space in the storage systems.
How to recognize it: The typical example is when the picture looks like you can see the license plate if you zoom, then you zoom and… you still can’t read the license plate.
The problem appears in two ways:
- You have enough pixels, but many of them appear to have the same value (tile effect), so in practice you have a much lower actual usable resolution than the native resolution of the image file format.
- You have artifacts, which means details that do not belong to the image but are introduced by the compression (typical blocking artifacts of JPEG files).
How to solve it: With a deblocking filter you can noticeably improve the quality of the image from a visual point of view. It is important to ensure you do not apply a filter’s effect too much as that will also cut out useful detail. Also in this situation, if you have multiple frames, applying techniques such as frame integration (Frame Averaging) or super resolution (Super Resolution) can be very useful.
The optical blur happens when an image is not perfectly in focus. Usually CCTV optics are configured to have a very big depth of field, everything is more or less in focus, but it’s difficult to have something perfectly in focus.
The basic idea is that every pixel of an ideal image in focus is being transformed into a circle with a radius depending on the strength of the blur.
How to recognize it: The border and the digits of the license plate are (guess what) blurred. If you basically just see a white area it is often a blur together with a lack of contrast.
How to solve it: If you have enough resolution, optical deblurring filters can be very effective to recover even license plates that are not visible at all. Amped FIVE provides the Optical Deblurring and Blind Deconvolution filters: the first works very well on big blurs (even some tenths of pixels), while the second works more effectively on little blurs (less than 10 pixels usually).
A big problem is that the deblurring won’t work very well in images that present a strong compression, since it introduces some non-linearities that the algorithm is not able to manage.
Image out of focus
The motion blur is the effect that appears on a subject that moves too fast in relation to the shutter speed. Even if the effect looks very different from the optical blur, the nature of the problem is very similar: every point becomes a line (instead of a circle like in optical blur).
How to recognize it: Usually it’s pretty noticeable since you can distinguish a kind of trace that follows the points in the images.
How to solve it: If the kind of motion is linear and uniform, usually we can take some reference point that we know should be a point (like the angle of a license plate or a shimmer), but in the image it clearly appears as a segment. We can estimate the features of the blur through the length and the angle of this segment. In Amped FIVE you can use the Motion Deblurring filter or Nonlinear Deblurring if the motion is not straight but follows a curve.
If the resolution is not too bad and the compression is not exaggerated we may be able to correct even very strong blurs.
LOW BRIGHTNESS / CONTRAST
Different light conditions or characteristics of the cameras may lead to images that are either too dark (for example in night shots), too bright (when we have a light that points in our direction) or simply lacks contrast (pixel values are too similar).
How to recognize it: You can look at the image histogram. If the histogram is very shifted to the left, the image is too dark (most of the pixels will have values close to zero). If the histogram is shifted to the right, the image is too bright (values close to 255). If the histogram is very thin the problem is in the contrast (too many pixels have the same value).
How to solve it: There are many tools and the use of these can vary from case to case. Two good starting points are that you can automatically equalize the histogram or manually adjust the brightness (filter Curves or Contrast /Brightness). In order to improve the appearance of the details you can use one a sharpening filter (Sharpen group).
An area of the image is saturated when all the pixels have values of zero or 255. If the image is saturated (or, in general, all the pixels have the same value), there is nothing to do, since all useful information is gone.
With histogram equalization
The noise is a random variation of the pixels in an image and it appears like a grain that is usually evident in pictures taken with low light. In fact, the noise becomes very visible usually when we brighten a dark video.
How to recognize it: There are random spots on the image that modify the correct value of the pixel. Many times it’s difficult to precisely separate noise and compression artifacts.
How to solve it: If we have only a single image we can improve the overall quality with a smoothing filter. However, on a single frame this is a risky approach because many times the filters can take away the details together with the noise. If we have multiple frames of the same license plate in the same position, a technique that gives amazing results is frame integration (Frame Averaging filter). If the license plate is not in the same position it must be aligned with a stabilization filter (Stabilization or Perspective Registration).
One noisy frame
THE BOTTOM LINE
Real cases are often more complicated than these examples, since most (if not all) of these problems and many others will be present at the same time. It’s very important to keep these basic problems in mind in order to always start with the proper filters and algorithms. It is even more important to understand how to determine when an image does not contain the minimum amount of information for feasible analysis.
This article is an edit of the post that appeared on the Amped Blog on 2011-06-15.