Tool to analyze hundreds of GB of Pcap traffic

I'll second netwitness - its an excellent tool for profiling large datasets (such as getting a list of all services, hosts, requests made, etc). Id also take a look at netminer, an open source tool with similar functionality.

jpgauvin,

Thanks for the post. Unfortunately Wireshark is not good at analyzing 300GB+ of PCAP data.

Thanks

Sorry, I thought it could help )

But just out of curiosity, how much time did you spent sniffing packets for getting a 300 GB of PCAP data ?

You could do worse than to write your own if you are looking for specific things - that way you could possibly get better speed results with such a large data set.

Try NetPCAP in Perl, call libpcap directly from C/C++/ObjectiveC or try one of the other implementations listed on here.

You could also have some fun with Linux and tethereal which would do the translation to human readable for you - you could then play with sed, awk, grep, wc etc. to get the statistics that you are after.

Hi,

do you know deft?

http//www.deftlinux.net/

regards
k.w.

agree, deft linux has xplico preinstalled on it.
give it a try.

One of the functions I am looking for is to be able to summarize all the types of traffic, X number of HTTP connections, X number of FTP, mail, IRC etc.

A late response, but here is my take on the tools mentioned

Xplico do that "summarize" part with a simple to use GUI, unfortunately, it does not feature as many protocols as say Wireshark and is rather focused on "normal" protocols. NetworkMiner also do a good job summarising sessions, but both X and NM has very limited data presentation capabilities.

For more advanced features, your best bet is Netwitness Investigator, there is a basic version is available free, has to be renewed/activated every year though. If you want to go dig down into the capture files, Netwitness Informer is an additional piece of software you could look into. Unfortunately, there is no freeware version for Informer, but i can say from a live demo that it adds some cool features.

Another thing you could do to save money on expensive license fees is to focus on certain times/dates or IP addressees and dump the PCAP files as text with packet data, then parse the files manually with your favourite tools. Its not a fast solution though.

I try E-Detective Decoding Center before to analyze pcap files. I think it suits your requirements. try it out at www.ed-systems.sg
Its quite easy to use web based tools. It categorizes all the packet based on its protocols. You can have http, ftp etc traffics to analyze.

@mdshukri
E-Detective it is a good system but it do not respect the GPL. In fact now they have removed the ISO trial version from all their sites.
Their system (all applications FTPParser, GTalkParser, MsnParser, OpenRAW, …) uses the libnids library which is GPL.

@MDCR
You are right, Xplico is a good tool but provides few protocols and with few details. One thing that is missing is a timeline view.

Try this.

http//networkminer.sourceforge.net/

This might be a good one to read.

http//forensicscontest.com/2010/07/09/puzzle-6-winners

There were some custom tools written to parse pcap files.

I'd second Netwitness, the freeware version is pretty awesome (only limitations are the enterprise features, like taking captures live from network appliances)

I'd also look at Netminer - open source tool with similar functionality.