I would like to open a discussion for a particular problem that many of us might have faced at one point in time or other.
How do we go about log analysis if we have tons (maybe in trillions) of logs from lets say tcpdump (raw logs) or some firewall (like netscreen or pix)?
What would be the best way to normalize and analyze these logs in the shortest possible time?
Import them into a database? Use a commercial application like arcsight? loglogic? simple text editor like editplus?
What is the best solution for manazing and analyzing logs from various sources in different formats and timezones?
Any suggestions/comments would be appreciated.
Regards,
Chetan
One of the things I recommend is Marcus Ranum's "artificial ignorance", particularly when dealing with web server logs. When I get an image containing a web server, I have the logs and the actual web site itself…from there, I can cull through the logs and extract all of the stuff I know is supposed to be there, and I'm left with the stuff that isn't.
Now, with regards to the approach you're looking for, it depends on what you call "best". Is "best" least expensive? What are you looking for? Is a database solution acceptable, and if so, do you have anyone capable of administering the database?
Maybe accessing the log-analysis list would be a great way to go, as I think the address this sort of situation quite regularly.
One of the things I recommend is Marcus Ranum's "artificial ignorance", particularly when dealing with web server logs. When I get an image containing a web server, I have the logs and the actual web site itself…from there, I can cull through the logs and extract all of the stuff I know is supposed to be there, and I'm left with the stuff that isn't.
Now, with regards to the approach you're looking for, it depends on what you call "best". Is "best" least expensive? What are you looking for? Is a database solution acceptable, and if so, do you have anyone capable of administering the database?
Maybe accessing the log-analysis list would be a great way to go, as I think the address this sort of situation quite regularly.
Harlan,
By best I meant the one that can provide the solution quickly and relaibly. Cost is something that is not on my mind right now.
A database solution is definitely acceptable. Do you have any one in mind?
BTW, is "artificial ignorance" a commercial solution or free one?
I have tried analog.exe for web server logs but am looking for something which is highly customizable and reliable.
I know this question is best suited in the loganalysis list but I thought the experts at this forum would also definitely have some thing to add to my knowledge.
A fried of mine suggesed ACL. I tried it with some logs but dint quite like it. I was told is that it can handle logs in GBs easily so thought may be worth trying….Has anyone tried it for logs (in human readable format)?
Chetan
> A database solution is definitely acceptable. Do you have any one in mind?
Not specifically, no. I know that Perl can be used to integrate with almost any db via the DBI interface, and that mySql may be an acceptable solution.
> BTW, is "artificial ignorance" a commercial solution or free one?
It's a concept more than anything else. Take a look at Marcus Ranum's stuff and you'll see how he's espoused this approach to get folks over their paralysis.
I use Splunk. It is the best search tool for large amounts of logs in various formats.