One other thought which is, don't forget that if you wind up working civil or criminal cases your tools are going to get challenged in court. Open source tools are more vulnerable to attack by the opposition than commercial tools that are standardized. I don't know if the tools you seek are for learning or if you are building a lab to work for yourself.
Are they really? I would suggest that that really isn't the case at all, largely because it appears that what gets "attacked" isn't the tools but the analyst's abilities and processes.
Also, there is one important factor that is not mentioned or apparently considered when the subject of going to court is brought up; that is, as an analyst, you don't simply walk into court one day and get on the stand and testify. The fact of the matter is that you're a witness for one side or the other, and you will not be brought in to testify unless the attorney you're working for or with is completely comfortable with your knowledge, your skills, and your ability to testify in support of their case.
Therefore, if you're supporting the prosecution, and there's any question about what you found because you used open source tools instead of a commercial product, and that question cannot be addressed by the prosecution, you very likely won't be put on the stand.
There are a couple of other issues at hand here…one is that there are things you can do with open-source tools that you simply cannot do with commercial tools. The last PCI assessment I did while at IBM involved me using my timeline creation tools to build a more comprehensive picture of what happened on the system than was available with any commercial tool. And because I knew exactly what I was looking for, I didn't run into the problem you see on the EnCase user forums all the time…"I pushed the button and nothing happened…why?" I did a more complete and comprehensive analysis of the system than was available solely through the use of a commercial application, and was able to minimize the window of exposure for the customer.
Finally, what does it matter what tool was used? I've used open source tools to find things, completely documented my procedures and findings, and turned that over to someone else, that then validated my findings via their commercial toolset in order to present this information in a court of law. The information contained within an acquired image is nothing more than a stream of bits. If I can verify the integrity of the data through the use of a checksum, then what does it matter how I go about finding evidence? If the bits are there, and they are not changed, who cares if I use a backhoe, a shovel, or a toothbrush to extract that data?
The point should REALLY be that regardless of the tool used, the process should be completely and thoroughly documented.
Honestly, what I think this really comes down to is that for many of the more common tasks, it's much easier for the majority of analysts to use the commercial tools.
The last thing I'll say here is that yes, I've used many of the commercial tools mentioned…EnCase, FTK, XWF, even MacForensicsLab. Like any other tool, they have their uses. For example, when doing the PCI forensic assessments, our team used EnCase, and custom EnScripts…we HAD to go custom because at the time (as of June '09, to my knowledge), the built-in function that GSI used to determine whether or not a credit card number was "valid" did not cover all of the card brands that were considered valid by PCI. Therefore, certain card numbers would be found, even in track data, and the hit would be considered invalid by the built-in function…I got help from someone really knowledgeable in EnScripting to write the necessary code to replace the built-in function. My point is simply that if you're using a tool simply because it's easier…maybe that isn't the right answer.
I recently just passed my GCFA and was curious as to what software is good for analyzing data/memory, indexing files in allocated and unallocated space?
I realize everyone is going to say FTK or Encase but keep in mind, I am buying this with my own proceeds not the companys so what software program can you recommend?
Actually, I use neither for the uses that you mention. I have found that there are a number of open source and/or inexpensive tools that will assist in the tasks that you mention for far less money than any of the big name tools.
I believe that Harlan has a blog entry dealing with open source tools and you'll find many examples of code that can perform specific tasks as well as or better than commercial tools.
The main reasons that I use commercial tools are
1. Familiarity. I have worked with EnCase and FTK long enough that I can perform focussed tasks very quickly. For example, FTK's use of the dtSearch engine makes ad hoc queries very fast (at the expense of a lot of up front processing). On the other hand, neither tool has the ability to do searches based upon Perl Compatible Regular Expressions which I find much more powerful than GREP and, carefully crafted, are much less likely to return false positives.
2. To confirm what I find using other tools.
But, with few exceptions, there is nothing that I can do with commercial tools that cannot be done with open source tools (and a bit of programming/scripting knowledge), if you are willing to get under the hood.
One other thought which is, don't forget that if you wind up working civil or criminal cases your tools are going to get challenged in court. Open source tools are more vulnerable to attack by the opposition than commercial tools that are standardized. I don't know if the tools you seek are for learning or if you are building a lab to work for yourself.
Can you give us an example of how open source tools are more vulnerable. Just hearing it is from someone doesn't count. An example from a case where you can show that open source tools are more vulnerable.
Greetings,
Another thing to keep in mind is how few cases end up in court. One approach is to use open source tools and back them up with commercial tools. If you start approaching court, rework the case with the tools that are less likely to be challenged.
There is an enormous amount of corporate internal investigation forensics analysis work that never gets anywhere close to court. Tools that get the job done quickly and efficiently are far more valuable than ones that are "court approved" but require more time and effort to use.
-David
Here is the link seanmcl referred to on Harlan's blog
http//
Can you give us an example of how open source tools are more vulnerable. Just hearing it is from someone doesn't count. An example from a case where you can show that open source tools are more vulnerable.
I doubt that such examples exist. In fact, if you use EnCase, you might be asked "What kind of training and certification do you have in the use of EnCase?" whereas if you use TSK, Autopsy, PhotoRec, etc., what can they ask that you can't answer with "There is no such thing as TSK certification."
My point being that the use of CF tools for which there are corresponding training and certfication paths which you do not have may raise more questions.
In either case, what is required is that you used the tool correctly, and that you understand the tools limitations and capabilities and how it helped you to arrive at your conclusion.
THanks again for all of your posts. I am looking into the software packages as well as open source. Based on what I have done so far, there isn't really 1 tool that does everything perfect so it will probably be a mix and match of different tools!
Agreed -)
Can you give us an example of how open source tools are more vulnerable. Just hearing it is from someone doesn't count. An example from a case where you can show that open source tools are more vulnerable.
I doubt that such examples exist. In fact, if you use EnCase, you might be asked "What kind of training and certification do you have in the use of EnCase?" whereas if you use TSK, Autopsy, PhotoRec, etc., what can they ask that you can't answer with "There is no such thing as TSK certification."
My point being that the use of CF tools for which there are corresponding training and certfication paths which you do not have may raise more questions.
In either case, what is required is that you used the tool correctly, and that you understand the tools limitations and capabilities and how it helped you to arrive at your conclusion.
>>snip
Open source tools are more vulnerable to attack by the opposition than commercial tools that are standardized. I don't know if the tools you seek are for learning or if you are building a lab to work for yourself.
A good read on the issue of open source is Brian Carrier's paper found on his web-site. He argues that open source tools are less vulnerable to attack because they are vetted by users due to access to the code and this transparency makes them more suitable for Daubert type challenges.
Sorry guys, I didn't mean to kick open the hornet's nest. Harlan is absolutely right that if multiple tools can get the same evidence it doesn't matter which one you use as long as the results can be validated. I made a fatal error when I assumed that the tools he is be learning on now will be the ones he will be submitting evidence with later, which is why I was asking for context.
I was taught Encase by two people, one who has his own forensic company doing mostly civil cases for theft of intellectual property as there is a lot of industrial espionage in the area we live in. The person teaching with him is the lead investigator at the cybercrime unit of the local PD. I was taught by them that your tools must be repeatable, certifiable and licensed. Open source is usually none of these things. If you want to use open source to get the data that is fine but you better validate with a certified commercially licensed program when it comes time to submit your evidence. It is not likely that you will have to testify but you do need to certify the results you submit to the lawyers as accurate. Now I will end my involvement in this fracas with the submission of a pretty good article about this very subject for your perusal.
Created April 4, 2002
Updated Feb 3, 2009
Author Steve Hailey
Here is an excerpt from the end of the article.
Stemming from the Federal Rules of Evidence, came the Daubert (Daubert vs. Merrell Dow Pharmaceuticals, 1993) reliability test. The Daubert reliability test requires special pretrial hearings for scientific evidence and special procedures on discovery. The Supreme Court in Daubert declared that the more flexible Federal Rules of Evidence had completely replaced the Frye test in determining whether an expert's testimony was admissible, and that the Frye test would no longer be used in federal courts.
In its basic form, Daubert says that experts must use objective methodological principles in their work, and that they should also be qualified to testify as a true expert in their field. Federal trial judges were granted the right to screen an expert's qualifications and test the reliability of the expert's methodology.
A number of reliability factors can enter into the Daubert reliability test
Whether the expert's technique or theory can be or has been tested – that is, whether the expert's theory can be challenged in some objective sense, or whether it is instead simply a subjective, conclusory approach that cannot reasonably be assessed for reliability
Whether the technique or theory has been subject to peer review and publication
The known of potential rate of error of the technique or theory when applied
The existence and maintenance of standards controlling the technique's operation
Whether the theory or method has been generally accepted by the scientific community
Individual states and even jurisdictions within these states have their own rules of evidence, and you'll find many are based on the Federal Rules of Evidence
States accepting Daubert
Connecticut
Indiana
Kentucky
Louisiana
Massachusetts
New Mexico
Oklahoma
South Dakota
Texas
West VirginiaStates accepting Frye
Alaska
Arizona
California
Colorado
Florida
Illinois
Kansas
Maryland
Michigan
Missouri
Nebraska
New York
Pennsylvania
WashingtonStates with their own tests
Arkansas
Delaware
Georgia
Iowa
Military
Minnesota
Montana
North Carolina
Oregon
Utah
Vermont
WyomingIt is our belief that as a digital forensics expert, you should be aware of the Federal Rules of Evidence, as well as those for your state or jurisdiction. You'll want to go over these with the legal counsel you are working with.
As you begin to work on a case and process the evidence, you'll want to ask yourself and legal counsel if your methods will survive the rules of evidence tests for your particular situation. All of your work will mean nothing if the evidence you recover is not admissible.