Greetings,
I'm doing a lot of log file analysis work, often combined with looking at shell histories, and would like to find a tool to assist with these efforts. At a minimum, a tool that can take an arbitrary collection of logfiles and merge them together by time and adding a tag for each line showing which file it came from would be very helpful.
Ideally it would understand a lot of logfile formats - syslog, secure.log, xferlog, wtmp, etc - and work on a variety of platforms - Linux, Mach, Solaris.
There are a lot of log file analysis tools here
http//
I'm slowly working my way through them but I haven't found what I need yet.
Worst case, I'll write one ….
Thank you.
-David
Will Zeitline help?
http// projects . cerias .
Good morning,
It might at that. Looks promising, thank you.
pyflag (available in Helix) looks interesting, too.
They're both "heavier" than I'd like for a first pass, but if they get the job done, I'm happy!
-David
PyFLAG's actually pretty cool for that kind of analysis; thanks for refreshing my memory.
In the past I have used a variety of programs to do log file normalization and analysis, all with varying results. I came across this tool (IDEA by CaseWare) while working with our forensic accountants normalization some accounting data. The program is not free and geared towards accountants but it is worth every cent. To sum up IDEA it is like Excel, over dosed on steroids and then some… I have been using it for a few months and there is not a lot which it can not handle. Below are some key details of IDEA
- Keeps a Forensic log file of every change you make to the duplicate of the original file which can be replayed later if you need to reproduce results, you can also produce templates from this code, e.g. Apache log templates, Event Log templates etc.
- Back end scripting similar to VB for the "hardcore" geek in all of us
- Allow setting of masks for date/time formats
- Great built in search criteria if statements, finds which days are days of the week, date and time formulas, afternoon and morning formulas, NOT statements, it seriously has some cool built in search/criteria statements.
- Append, join, compare and some database functionality
- Instant stats
- Duplicate Detection
- Gap Detection (Need I say more)
- I have used IDEA on upwards of 2 millions rows/records on a average PC and IDEA hasn't once faltered or slowed down.
By far the best feature of IDEA is its import ability and quick normalization with templates. If the log file resembles some form of column/row combination IDEA will pick it up
- CSV, TXT, XLS, a lot of accounting packages
- Inform you if any "bad data" could not be import according to the rules/wild cards you specify and put them in a separate worksheet.
- My favourite import feature is the ability to specify your own column/rows using something similar to a regular expression/layering, this feature also lets you put a repeating sentence at the beginning of each row, construct layers around importing of the data. This feature is hard to describe but is brilliant to say the least.
- Export results/finding in CSV, TXT, XLS, accounting database, XML
On a side note the CaseWare/IDEA [
Greetings,
Further research
- Splunk might be interesting - http//
- Also LogLogic - http//
As far as I can tell, neither of these has been tested in court. IDEA, as a tool for forensic accountants, probably has been to court. I'm drifting away from open source here, but this may be an area where I need to.
-David