By David Spreadborough – Forensic Video Analyst at Amped Software
There are several common components within a CCTV Video file. This article from Amped Software is going to dive into the data that links the image to the time when the captured event occurred: the timestamp.
Most video surveillance systems have a method of linking the imagery with the date and time of the recordings. However, as is common in the world of CCTV, nothing is simple, and every manufacturer attempts to reinvent the wheel and do their own thing.
Looking back in history, this has always been the case. In the days of VHS tapes, many CCTV systems would use a hidden data area on the tape to record a barcode. Invisible when played back normally, but decodable as a timestamp by their equipment.
The first method we will look at in the digital era is the naming convention of the exported file. It is for this reason that filenames are so important and must not be changed during handling, automated processing, or digital evidence management. The filename may be the only reference to date and time.
Ch02_20220619111254080_20220619112811160.mp4
So, from this filename, I am expecting a single camera of footage from 19th June 2022, with a duration of 15mins, 17 secs, 080ms. I have calculated the duration thanks to the Time Calculator tool in Amped FIVE.
The Time Calculator is such a handy tool, allowing you to not only work out a duration between two times but also a new time if an offset is known. All the data within all the timestamp filters and the Time Calculator is copy-pastable for flexibility.
We will be using the Time Calculator tool in Amped FIVE more a bit later but first, let’s look at pixel-encoded timestamps.
This information is written onto the video at the time of original encoding.
If the camera has access to a date and time system, either via internet access or from a network video recorder, the timestamp can be written immediately to the recording at time of camera capture.
When, instead, an analog video signal is transmitted to a Digital Video Recorder, the timestamp data from the device can be merged with the signal and then written directly onto the recordings.
There are a few other methods but they do not commonly relate to original encodings, and would probably take us deep down a rabbit hole so we had better move on!
If the date and time have been written onto the original recordings, it is worth saying that they cannot be removed to better observe an event happening in that same area. Yes, that is a common request! The pixels themselves have been changed, it is not simply an overlay that can be moved.
Turning the filename information or pixel encoded timestamp into usable data is very simple in Amped FIVE.
The Add Timestamp filter can be found within the Presentation category.
The filter has several uses, with the first tab in the Filter Settings dealing with the time itself.
You can set two timestamps based on your available data. You do not have to enter the first and last timestamp, you could select 2 adjacent frames. The filter will interpolate the duration in between those two times to the entire range. You can also see that the filter has an option to allow for the automatic adjustment of frame rate.
You may only know a single timestamp, or have several that are spaced randomly in the file.
This is where the List tab comes in.
Known frame times can be entered into the list, and no automatic interpolation occurs; however, you will be able to use these timestamps later on with the Adjust Timestamp filter.
The Formatting tab allows you to select display options, including the ability to store the data but disable the rendering of it onto the video footage. We will look more at this a little later.
Let us deal now with timestamps that are inside standard subtitles.
There are many different subtitle formats and when manufacturers use them correctly, they can make things a little simpler during an investigation, but the files must be handled carefully. As they are simple text, they can be easily edited, which may impact the integrity of the data.
This category of timestamps is split into two, and they differentiate by the way that they are exported from the CCTV or video recording system.
The first subtitles are those that are in a separate file. The system will export a video file, and also a subtitle file. They will commonly have the same filename but note the different file extension.
The other type of subtitle is the one placed inside the multimedia container, rather than stored as a separate file. This is good, as it removes the possibility of the two components being separated and the time data being lost.
Multimedia containers, such as .mp4, .mkv, and .avi, can contain video, audio and subtitle streams and their presence can be identified in FIVE within the File Information Tool.
These subtitle files can often contain other non-timestamp related data, such as camera name or recording system.
Before we start working with the timestamps, let us finish off and look at the most common type which, unfortunately, is also the most complicated.
To understand this type, let us go back to the original data on the video surveillance system. When a user wants to search for an event, they use a Date and Time Index. They don’t scan for a frame number or a duration into the video. They want to look for 3 PM last Saturday, as an example.
All the thousands of different systems have a different way of creating, managing, and exporting this index data at the time of evidential acquisition.
Some will be in a separate proprietary data file that, upon initial analysis, contains no human-readable data. Most however, will place the data within their proprietary multimedia file. It could be at the end of the file or interspersed with the video. Some may only place in the first or selected timestamps and their proprietary player will interpolate the rest dynamically upon playback.
And when it comes to the data, some store it in a standard Unix format, whereas some feel that their own way is better. (Hint: It’s not!)
There are so many methods that it would be a challenge to list them and the identification of the timestamp and subsequent decoding is often harder than either the video or audio.
This is where the Amped Conversion Engine comes in.
When new file formats are identified we get to work on correctly separating the video, the audio, and the timestamp. This ‘cleaning’ recipe then gets placed into the engine so you can correctly decode all the components, and the timestamp will be in a format that you can interrogate and use effectively.
Without cleaning, if you attempt to decode the video, you may suffer from errors in one or several of the multimedia components.
Visible errors caused by no data cleaning prior to decoding.
Often the problems are not easily seen, and may result in you missing entire frames.
When a video is decoded, either immediately or after analysis and cleaning within the Amped Conversion Engine, if a timestamp is identified the video will load followed by the Load Timestamp filter.
Upon loading the timestamp, the data will be linked into the Player window with the start and end timestamps displayed at either end of the Player bar, and the current timestamp position displayed next to the running time.
This data is actionable with a right-click, allowing you to either copy the timestamp or paste / type in a timestamp to navigate directly to a certain time.
Lastly, when a timestamp is loaded, you will be able to playback using the timestamp to control the video speed.
Loading the timestamp as a separate filter allows you to deal with the data more effectively.
You are able to override any standard subtitle positioning and even change the formatting. You may prefer to show the date in a certain order such as Day, Month, Year.
You will notice again the Disable Rendering checkbox.
There will be occasions where you may want to have the data stored and usable, but you do not want it to display over the video until further in your filter chain. You may also need to adjust your timestamp prior to it being displayed.
Let us take a look at this now, along with viewing the timestamp in your default text editor.
After loading in a new file, FIVE has detected a proprietary container and the Amped Conversion Engine has been used to clean and correctly decode the video and the timestamp. The result is a .time file inside the same directory as the video.
We can take a look at this file in any text editor as it is a standard open format.
The first column is frame number, the second is frame type, the third is the decoded timestamp.
This manufacturer only stores the I frame time within the file, which is actually fairly common.
If you want to learn more about I frames, P frames and B frames, take a look over on the Amped Blog at the post relating to macroblocks.
The filter settings are fairly simple, but it’s important to highlight the timestamp format options.
You can display the data any way you require, and although the dropdown list has some predefined options, you can enter your own.
As we have seen from viewing the .time file, a timestamp is only displayed on every I frame and we can see an error at the start, with two duplicated timestamps. Consequently, it’s required to adjust the timestamp.
After checking the Disable rendering checkbox, it’s time to select the Adjust Timestamp filter, found again within the Presentation category.
The formatting tab is similar to both Add and Load Timestamp filters but the next tab deals with a common requirement, the shifting of a timestamp.
CCTV Systems rarely display the correct time and this tab allows for the offset to be entered and the correct time displayed. Remember, offsets can be calculated using the Time Calculator tool.
The Shift tab can also be used for timezone adjustments.
It is the next tab however that allows us to add in the missing timestamps, and we have several options on how to achieve this.
In our case, where we only have timestamps on I frames, and we have an error at the start, we will duplicate to get a timestamp on every frame. However, each one will simply be a copy of the previous until an original different timestamp is detected. Each will therefore have .000 ms, and of course, the error will remain. If we used “Linear”, instead, the missing timestamps would have been computed from existing ones by evenly splitting the time between intermediate frames.
The next tab is where we can fix that: Refine.
It is now possible to use a linear interpolation of the duplicated values to increase the precision.
We can thus say that while the Interpolate tab is for adding timestamps to frames that don’t have them yet, Refine is for increasing the precision of existing timestamps (e.g., adding milliseconds).
You may notice that both the Interpolate and Refine tabs allow you to use the PTS.
The Presentation Timestamp is a controlling timeclock found in many video files and several systems use this data to control and move their date and time data. If you have an understanding of this data and find that your files data is reliable then using this can ensure very high accuracy in your timestamp adjustment.
If you want to learn a little more on video timing, take a look at this blog post on frame rate identification.
This brings us neatly to a connecting issue. Working with the timestamps often links with changing the frame rate, so it is worth taking a structured approach to correcting the timestamps, changing the frame rate, and then presenting that data on the video.
Although you can use the timestamp filters to display the information on the video, you don’t have to. You can disable rendering and then use the time and date macros within any of the text presentation methods.
Summarizing, here’s what you have in Amped FIVE for dealing with timestamps:
- Load Timestamp lets you load time data from an existing file (supported types are .time, .smi, .sri);
- Add Timestamp lets you manually enter time data for one or multiple frames, which is useful when timestamps are printed over pixels;
- Adjust Timestamp provides several functions. The Shift tab is for correcting date and time (e.g. adjusting the timezone or fixing the clock); the Interpolate tab is for adding timestamps to frames that don’t have them yet; and the Refine tab is to increase the granularity of timestamps (adding milliseconds).
Extracting and then using the data provided in timestamps has been made so much easier thanks to the Amped Conversion Engine, the Time Calculator, the Add Timestamp filter, the Load Timestamp filter, and then the Adjust Timestamp filter.
They have all been designed to help you deal with time.
One thing they can’t do is find you more time, but we are working on that!
