Join Us!

Thoughts on Tools/P...
 
Notifications
Clear all

Thoughts on Tools/Processes  

  RSS
keydet89
(@keydet89)
Community Legend

I am responding to a post to another thread…

What is the practical suggestion?

  1. Do not use any third part software and write your own tool.
  2. Use multiple softwares and delve deeper if there are differences between results.
  3. Test as deeply as you can all available software, then choose one and only that one, because you already know which "quirks" it may produce.
  4. Other (please specify).
  5. [/listo]

    If #1, then all software houses could close down, and each forensic examiner will have to write, validate and eventually "defend" in court his/her tool against the findings of the expert witness of the other party. who also wrote his/her own tool and has exactly the same issue.

    If #2, then all forensic investigators should test the data/SIM/whatever against *all* available tools, freeware or Commercial and "hope" that no inconsistencies are found. (as one of the tools may be "right" in one specific point but "wrong" on another)

    If #3, the risk of a peculiar artifact not having being tested or tested properly seems to me rather BIG.

    I would say that all three approaches above are either very problematic or very non-productive.

    What is option #4? ?

I don't think that any of the options is necessarily "practical". #1 is simply untenable, as is #2. #3 is simply not practical, either. I also don't think that a solution of any kind can necessarily be lumped into a single, absolute option.

A while ago, I was asked for my assistance…the request came in this manner "The tool you gave me doesn't work." It took a bit of time and exchange of emails to find out that this analyst had been given Windows XP Event Logs by another analyst, and ran a tool I had written against them, and not gotten any output. I was assured that the files were indeed Windows XP Event Logs. I asked for a sample, and within seconds of it arriving in my inbox, I opened it in a hex editor and found that what was called "secevent.evt" was not, at all, a Windows Event Log.

I don't expect every analyst to write their own tools, nor to test every possible tool against each data set. However, there are solutions

First, understand the data structure you're looking for/at, as well as what the tool you're using actually does with respect to the data. If you ask me for my MBR parsing script, and then run it against a memory dump, please don't then contact me with, "your tool doesn't work."

Does this mean that every analyst has to memorize all of the various possible data structures? No, not at all…it's hard enough as it is just to keep track of Windows Event Logs…now throw formats of SMS messages on the different platforms, and that would simply be too much. Instead, share information. Document what you find. Make use of sharing sites such as forensicswiki.org. Crowd-sourcing your question by simply posting it to a forum can prove beneficial to some, but in the end, how many of us are actually sharing what we learned back with the community?

Engage with other analysts. Not just in the general sense, but directly, as well.

I know that not everyone has the time nor the interest to do this sort of thing…but I've seen quite a few analysts spend way too much time on something, all for the sake of figuring out for themselves. If you spend 8 hrs on something that you should have been able to complete in 15 minutes, then it should be pretty clear why you don't have time for other things.

I once talked to an analyst who spent over three months working on something, trying to figure it out themselves, before asking for help. After three months, they finally decided to ask a question, and in less than 20 minutes were provided an education and a tool to help them with their task.

Overall, I'm concerned about the level of engagement within the community, in that a great deal of information is being lost. Let's say someone has a need or question…they go to ForensicsWiki.org, do a search, and don't find what they were looking for. If they simply stop there, so much is lost. Did they search for the right thing? Did they ask for assistance from someone else? Did they let anyone know that what they were looking for wasn't there, so that someone can then research it and provide the information, i.e., fill in the gap?

I didn't read Yunus' paper, in short because I don't deal with SMS messages and mobile devices at the moment. However, I have seen comparisons of tools before, and one of the things that has concerned me is the evaluation itself. I've seen tools be evaluated for being "scalable to the enterprise" and fail, when they were never designed to be scaled to the enterprise. There needs to be some peer review of the information in general, whether it's the description of a data structure, or the evaluation of a particular tool to parse and display that data structure, and a wiki is a great way to go about that.

HTH

Quote
Posted : 09/05/2013 11:17 pm
mscotgrove
(@mscotgrove)
Senior Member

I think option 2 is the closest to a possible answer.

However, ultimately the answer must be some knowledge to recognise relevant data structures from a hex dump.

Finding help can be difficult - but Google / newsgroups / forums can often help set one in the correct direction.

ReplyQuote
Posted : 10/05/2013 1:51 am
jaclaz
(@jaclaz)
Community Legend

Just for the record the original thread is this one
http//www.forensicfocus.com/Forums/viewtopic/t=10575/

jaclaz

ReplyQuote
Posted : 10/05/2013 3:08 am
Bulldawg
(@bulldawg)
Active Member

I've only been in the field for a few years, and I'm just now getting enough work that it's a full time job. Just consider that in my reply.

I find there are too few active online communities of digital forensics examiners. I don't know why this is, but I suspect too many of us work or used to work in an agency where secrecy was important. Coming from that environment, sharing is not encouraged outside the agency, even if it's just techniques, best practices, or problems encountered. The only way we can grow the knowledge base of the community is to share as much as possible.

As someone new to the field, I crave information. I have a whole library of books, most of which are excellent resources but grow stale quickly in this environment. There are a number of excellent blogs. However, this isn't enough without active engagement from experienced examiners. There are a handful on here that I think fall into this category. When I've posted questions about specific problems, I've received great responses. The threads with specific questions seem to be the most popular, but they are few and far between. In a perfect world, this forum should be much more active with 1,000+ new posts daily. We have 24,102 members are the moment. Where are they?

I'm going to CEIC in a couple weeks, and I hope to meet some examiners there. (BTW, if you're going to be there let me know. I'll buy you dinner in exchange for picking your brain a bit.)

ReplyQuote
Posted : 10/05/2013 3:40 am
Passmark
(@passmark)
Active Member

Slightly off topic,

We have 24,102 members are the moment. Where are they?

If it is anything like the forums I run, 98% of the sign ups are from automated spam bots and not from real people.

Mainly using XRumer spamming tool.
http//en.wikipedia.org/wiki/XRumer

Might be just my bad luck however. One of my sites is #5 worldwide in a list of good sites to spam, http//www.thetechfizz.com/high-pr-do-follow-forum-list-2013/ cry

ReplyQuote
Posted : 10/05/2013 5:50 am
Adam10541
(@adam10541)
Senior Member

I find there are too few active online communities of digital forensics examiners. I don't know why this is, but I suspect too many of us work or used to work in an agency where secrecy was important. Coming from that environment, sharing is not encouraged outside the agency, even if it's just techniques, best practices, or problems encountered. The only way we can grow the knowledge base of the community is to share as much as possible.

The big problem is the "elitist academics" on these type of boards, you'll spot them and they basically discourage question asking and knowledge sharing. So forums like this that used to be very active, become less active because people get sick of dealing with them. )

ReplyQuote
Posted : 10/05/2013 6:33 am
trewmte
(@trewmte)
Community Legend

As someone new to the field, I crave information. I have a whole library of books, most of which are excellent resources but grow stale quickly in this environment. There are a number of excellent blogs. However, this isn't enough without active engagement from experienced examiners. There are a handful on here that I think fall into this category. When I've posted questions about specific problems, I've received great responses. The threads with specific questions seem to be the most popular, but they are few and far between. In a perfect world, this forum should be much more active with 1,000+ new posts daily. We have 24,102 members are the moment. Where are they?

A very fair appraisal of what happens and is happening.

ReplyQuote
Posted : 10/05/2013 7:17 am
jaclaz
(@jaclaz)
Community Legend

It seems to me like we are drifting away form the original set of (mostly rhetorical) questions.

In the original thread Yunus kindly shared a report where he tested again the SAME data three different tools.

Of these tools
a) "lost" a line of a multi-line SMS
b) considered the line as a "separate" message
c) parsed the SMS as a whole

I can see nothing more, nothing less in the report.

Then it seemed to me like everyone jumped at Yunus' throat 😯 taking exceptions at the way he made the report, did the test and failed to check RAW data.

The (BTW nice) comparison keydet89 made with event log files does not stand.

Here there is "same" (proper) data parsed/analyzed by three different tools.

Everybody seems concerned (very understandably) with finding out manually what was the RAW data and which tool did the best job and why.

I am seeing it a little differently.

Given a same set of data three different tools (of which at least two are commonly used in investigations) provided different results.

Even without seeing the RAW data and determining why this happened, it is evident that two of the three tools provided incorrect results.

And seemingly this happened in a fairly "normal" case (a longish SMS) involving not any "proprietary" or "phone make/model specific" coding, etc.

The questions I asked were more - indirectly - about the lines of
How many people normally check the same data with three (or more) different tools?
How many people additionally check manually the RAW data?
How can it happen that these bugs (different bugs in two different tools) were not already made public?
How many cases there can be where a first line of a multiline SMS has "slipped by" because the single tool used missed it by accident?

While related issues might be mitigated by a better (or more "open") exchange of info among the professionals using these tools ) , isn't also the "duty" of the Authors of the tools to validate their tools against such data? (as said seemingly the data the report is about seems a lot like "common")

jaclaz

ReplyQuote
Posted : 10/05/2013 4:06 pm
jhup
 jhup
(@jhup)
Community Legend

Without the raw data it is not "evident that two of the three tools provided incorrect results".

All three could be wrong. Or, possibly all could be correct, but because of user, or technical errors reporting different results.

How many people normally check the same data with three (or more) different tools?

At least two, and more if there is discrepancy.

How many people additionally check manually the RAW data?

Depending on the type of data, I do.

How can it happen that these bugs (different bugs in two different tools) were not already made public?

I do not know for sure if these discrepancies are bugs. It could be user error, configuration error, or indeed bugs. It would not be the first time when a moth takes down a whole region.

How many cases there can be where a first line of a multiline SMS has "slipped by" because the single tool used missed it by accident?

Do not know.

As for "everyone jumped at Yunus' throat" maybe a wee bit stretching it, eh? wink

ReplyQuote
Posted : 10/05/2013 6:10 pm
jhup
 jhup
(@jhup)
Community Legend

[…]
There needs to be some peer review of the information in general, whether it's the description of a data structure, or the evaluation of a particular tool to parse and display that data structure, and a wiki is a great way to go about that.

HTH

I have thought about this quite some time - for almost a decade now.

I have some ideas regarding this, business plan in my head, copious amount of notes, appropriate domain names, software lined up, competition researched, market size, etc.

I also have some $ put away for just this project.

Question is how do I kick it off? This thread might just push me over the edge to start it.

ReplyQuote
Posted : 10/05/2013 6:18 pm
jaclaz
(@jaclaz)
Community Legend

As for "everyone jumped at Yunus' throat" maybe a wee bit stretching it, eh? wink

Sure ) , that was intentionally overstressed.

Still, you start by assuming that Yunus testing procedure was failed somehow.

I start assuming that running three different tools on the same set of data should provide same results (or compatible results).

I also assume that (difficult to say since it's in Turkish) the "whole" message makes sense while the same message without first line appears like a truncated message and as well that the single line by itself makes no sense.

I simply refuse to believe that one of the three tools "invented" out of thin air a line (in Turkish) that makes sense as first line of another message and by "sheer luck" merged it to the "right" message.

So, the tool that provides a "monolithic" message seems "right",
The one that misses a line "wrong".
The one that finds the line but as a separate message also "wrong".
The above is the most probable situation (even before and besides checking the Raw data).

But still, the point I was trying to make is not about which tool (if any) is "right" and which is (are) "wrong" (which is something that should be investigated and ascertained, but that is "specific"), only that the found discrepancy seems related to a "rather common" situation (a multi-line SMS) and that seemingly noone encountered it before (or didn't make it "public" or didn't notify about it the Authors of the various tools).

This latter aspect is IMHO what - as keydet89 understood - prompts - maybe - to re-analyze processes and procedures, while you might be personally testing each and every phone dump with two (or more) tools, and additionally check the RAW data I believe that this is not the "most used" or "standard" procedure. ?
The specific report is about a very small set of messages, less than 30, how many messages do you find on average on an examined phone?
Tens, hundreds or maybe thousands?
Wouldn't this make possible that *something* slips by and goes unnoticed?

jaclaz

ReplyQuote
Posted : 10/05/2013 6:55 pm
jhup
 jhup
(@jhup)
Community Legend

This latter aspect is IMHO what - as keydet89 understood - prompts - maybe - to re-analyze processes and procedures, while you might be personally testing each and every phone dump with two (or more) tools, and additionally check the RAW data I believe that this is not the "most used" or "standard" procedure. Question
The specific report is about a very small set of messages, less than 30, how many messages do you find on average on an examined phone?
Tens, hundreds or maybe thousands?
Wouldn't this make possible that *something* slips by and goes unnoticed?

Agreed, and it dovetails well into why Mr. Carvey started this thread.

(edited to properly reference this thread.)

ReplyQuote
Posted : 10/05/2013 8:09 pm
jaclaz
(@jaclaz)
Community Legend

Agreed, and it dovetails well into what Mr. Carvey points to in the other thread he started.

….which is this thread here …. roll

jaclaz

ReplyQuote
Posted : 10/05/2013 8:12 pm
jhup
 jhup
(@jhup)
Community Legend

. . . oops I have about three dozen tabs open. An excuse, but inexcusable.

ReplyQuote
Posted : 10/05/2013 8:14 pm
Share: