File Signature Analysis - Tools and Staying Current

I was curious to hear what tools other people are using in order to do file signature analysis?

Given what file signature analysis entails, I'm curious to what you might have found on Google to help you with this, and what you might have found lacking amongst those tools.

Also, how you keep the "database" of file signatures up to date.

From my own perspective, the easiest way is to simply add the signature to the database or flat text file being used. With my own tools and processes, I include the ability to identify file signatures and/or extensions that are not already in the database.

We use EnCase in some instances for this, but their file signature list is a bit old so we're looking to improve upon it.

I think that this was covered in the EnCase user forums, not long ago.

Given what file signature analysis entails, I'm curious to what you might have found on Google to help you with this, and what you might have found lacking amongst those tools.

I'm still looking on Google, I was just reaching out to see if there was anything else that other people were using besides the common ones (filext.com, filesig.co.uk, etc).

From my own perspective, the easiest way is to simply add the signature to the database or flat text file being used. With my own tools and processes, I include the ability to identify file signatures and/or extensions that are not already in the database.

I was thinking more on the lines of something that I could sync on a regular basis with a central directory on my forensic machine. Or update a file and have the examination tools just point to that directory.

I'd be interested to hear about the process you are using.

I think that this was covered in the EnCase user forums, not long ago.

I was looking on the forums, and I remember the discussion about the file sigs being out of date, but I didn't see anything on what people do to update them in this case.

Thanks,
Tom

Tom,

I'm still looking on Google, I was just reaching out to see if there was anything else that other people were using besides the common ones (filext.com, filesig.co.uk, etc).

Sorry, I misunderstood…I thought you were looking for tools, like file or missidentify.exe.

I was thinking more on the lines of something that I could sync on a regular basis with a central directory on my forensic machine. Or update a file and have the examination tools just point to that directory.

Sure, you can do that. Simply update your storage resource.

I'd be interested to hear about the process you are using.

Sorry, thought I'd covered it somewhat already.

I use my own tool/process to scan drives and perform file signature analysis. Essentially, when a file is found that has a signature that isn't in my db listing, I have my code tag it so I can review it and possibly add it. I don't rely exclusively on external third-party collections, because I can't verify the credibility of the information.

h

Tom,

Basic file signature analysis is available in tools like TrID (http//mark0.net/soft-trid-e.html) for free, unfortunately you get what you pay for. Their accuracy is low and unproven, and they don't provide support or timely bug fixes. EnCase is great as a platform to perform analysis on mounted disk images, but they have put very little effort into their signature analysis. They only provide weak identification of the most common 250 file types.

Some people choose to create their own solutions, because they know exactly what it does and can add signatures as they need to. This is similar to creating your own virus scanner. Your solution can be very accurate on files (or viruses) that you have already encountered, but it will not be able to identify file types that you haven't encountered and difficult file types that use complicated signatures or no signatures.

My company provides signature analysis (file identification APIs) for the big players in the industry like FIOS, LexisNexis, KPMG, CACI, etc.. We provide an investigator application called FI TOOLS. We are the only vendor that focuses solely on the internal file formats of files to identify and extract data from 3,400+ file types. You're welcome to download a trial version from (http//www.forensicinnovations.com/fitools.html). While we welcome sample files and format specifications to add, from our customers, we do not provide access to edit our signature databases. Controlling our databases gives us the advantage of providing a solution tested by the big players as well as individual investigators. Open Source and home made solutions can't provide this level of accountability and dependability.

Rob

Gary Kessler from Champlain College maintains a fairly extensive and relatively up to date (last update 6/30/09) File Signature Table
http//www.garykessler.net/library/file_sigs.html

This is not an app to maintain the database, but perhaps Keydet89's suggestion to write your own would be more advantageous for the reasons he specifies.

Mark

Sorry, I misunderstood…I thought you were looking for tools, like file or missidentify.exe.

My bad, probably could have phrased the question a bit better.

Sorry, thought I'd covered it somewhat already.

I use my own tool/process to scan drives and perform file signature analysis. Essentially, when a file is found that has a signature that isn't in my db listing, I have my code tag it so I can review it and possibly add it. I don't rely exclusively on external third-party collections, because I can't verify the credibility of the information.

h

Ah ok, I didn't gather that from your previous response. So how do you validate that a new signature you find is legitimate? Do you compare multiple instances of the same file extension/type?

Thanks,
Tom

Basic file signature analysis is available in tools like TrID (http//mark0.net/soft-trid-e.html) for free, unfortunately you get what you pay for. Their accuracy is low and unproven, and they don't provide support or timely bug fixes. EnCase is great as a platform to perform analysis on mounted disk images, but they have put very little effort into their signature analysis. They only provide weak identification of the most common 250 file types.

Yes, I'm looking at TrID, and the EnCase statement is how the initial discussion came up internally here.

My company provides signature analysis (file identification APIs) for the big players in the industry like FIOS, LexisNexis, KPMG, CACI, etc.. We provide an investigator application called FI TOOLS. We are the only vendor that focuses solely on the internal file formats of files to identify and extract data from 3,400+ file types. You're welcome to download a trial version from (http//www.forensicinnovations.com/fitools.html). While we welcome sample files and format specifications to add, from our customers, we do not provide access to edit our signature databases. Controlling our databases gives us the advantage of providing a solution tested by the big players as well as individual investigators. Open Source and home made solutions can't provide this level of accountability and dependability.

I've heard of your tool as well (your company recently announced the ability to identify TrueCrypt files if I'm not mistaken). I'm looking more for a "database" along the lines of what Harlan is doing himself that I can utilize in the existing tools I have. Once central repository that I can point EnCase, X-Ways, whatever at.

Thanks,
Tom

Gary Kessler from Champlain College maintains a fairly extensive and relatively up to date (last update 6/30/09) File Signature Table
http//www.garykessler.net/library/file_sigs.html

This is not an app to maintain the database, but perhaps Keydet89's suggestion to write your own would be more advantageous for the reasons he specifies.

Mark

I looked at his page earlier. But that's another good resource. I'd have to look at how to parse the information out. The site in my browser is a bit "funky".

Thanks,
Tom

A file signature is more than just looking at say the first 10 bytes. Although this approach can often work well, it can cause many false positives. For many file types it is necessary to look at other points in the file to determine exactly what type of file it is.

For instance, Word and Excel both start 0xD0 0xCF 0x11 0xE0 etc. All new DOCX and XLSX files start "PK"

Building up your own database is probably best, and keep looking for false positives. It is common to detect short signatures within a 'random data' file, eg image, or compressed/encrypted data.