This is another of my posts originally posted this on http://networking-forum.com/blog/.
Recently we needed to analyze packet flow through a router which had roughly 1000+ DSL customers running through. Initially we used Wireshark, but Wireshark has a nasty habit of crashing when trying to analyze so many packets and sessions. In fact this bug is a known bug and can be viewed here: http://wiki.wireshark.org/KnownBugs/OutOfMemory
Our capture server was initially running Windows 2003 with only 1GB of ram. This lasted roughly an hour or less capturing before dying. We upgraded the ram to 4GB but the server would still crash, albeit after about 2 hours now.
We then decided to use TCPDUMP. We’ve had TCPDUMP running for many days without a single crash and it hardly uses any resources at all:
You can see TCPDUMP has currently been running for almost 138 hours. CPU usage is 2.3%, memory usage is negligible, and remember it’s currently capturing packets for over 1000 users!
In order to use TCPDUMP you will of course need to have a Linux box (any distro will do) and TCPDUMP. TCPDUMP comes standard with many distros but if not you’ll be able to install it from your repo’s.
You’ll need 2 network card if you’re going to be managing this remotely. One to log into the server itself and the other which will be capturing packets.
This is currently how I’ve set mine up:
You’ll need to configure the router itself to mirror all traffic through the port you want to monitor to your Linux box. Basically it’s simply sending the same packets out both interfaces.
On the Linux box you need to configure the capture NIC with no IP address and your access NIC with it’s normal usual IP address (So you can still log onto it) Each particular distro will be different but on RedHat type systems have a look in /etc/sysconfig/network-scripts
On my particular server I have eth0 as my access NIC with a regular IP address and eth1 as my capture NIC. The config for the capture NIC in the ifcfg-eth1 file is as follows:
As you can see no IP address is configured although it is initialized on boot.
I personally like to use screen to run TCPDUMP as it allows me to log off the server and log back on and connect to the same process. If you’ve never used screen before I suggest this link: http://linux.die.net/man/1/screen
First you will need to create a folder for your dump files. I personally created one in /tcpdumpfiles/ – You then need to make this writable by all by doing a:
chmod +x /tcpdumpfiles/
How many files you dump is up to your needs and space available on the server. This is the command I used:
screen tcpdump -i eth1 -C 100 -s 100 -w /tcpdumpfiles/dsldump.dmp -W 500
Screen will open up TCPDUMP in the screen emulator.
Tcpdump will start up the process itself.
• -i eth1 is telling TCPDUMP to capture packets on the eth1 interface.
• -C 100 tells TCPDUMP to create files of 100MB in size.
• -s tells TCPDUMP to capture the first 100 bytes of the packet only (We only actually need the headers, no data)
• -w /tcpdumpfiles/dsldump.dmp tells TCPDUMP to create a *binary* file to the dump location we created earlier (More on this later) and not just output live data to screen
• -W 500 tells TCPDUMP to create 500 files and then start rotating, writing over the first file created and so on to prevent this server from simply running out of space.
This particular configuration will capture 50GB worth of headers and then start rotating. Feel free to change the values to suit you.
In the previous section we specifically created binary files instead of regular text dumped to screen. Why is this? Well TCPDUMP’s binary files are in the same format as WireSharks. Therefore in order to analyze it’s now easy to copy a dump file to a Windows box and just open it in Wireshark. This way we get the best of both worlds. Capturing using TCPDUMP which is extremely solid and unresourceful, as well as being able to view the captured packets in Wireshark with it myriad of filters and colours.
Capturing to a binary file also allows you to analyse the dump files with TCPDUMP itself or by using a script, but that will be another lesson in itself.