Troubleshooting 101 – Can I get back?

On August 13, 2010, in JNCIE, by Darren

Rule #1 – Just because Host 1 can get to Host 2, does NOT mean that Host 2 can get back to Host 1!

I can’t tell you how many times I’ve seen this out in the field. Here is a prime example.

This is quite a simply common topology. OfficeA and OfficeB have a WAN connection running between them. RouterA and RouterB are under control of the ISP and are running OSPF. RouterC is a customer-owned router, and so doesn’t run any OSPF with RouterA or RouterB.

OfficeB has an internet connection, and RouterB is injecting a default route to OfficeA via OSPF.

101routeback1 Troubleshooting 101   Can I get back?

RouterB has a default route to RouterC. So let’s think about traffic flow now. A host connected to RouterA needs to send traffic to the internet. It will send it’s traffic to it’s default gateway, RouterA. RouterA has a default route injected into OSPF from RouterB and so sends traffic to RouterB.

101routeback2 Troubleshooting 101   Can I get back?

RouterB has a default route and hence sends that traffic out to RouterC. Which then goes out to the internet.

101routeback3 Troubleshooting 101   Can I get back?

Traffic then flows back from the internet to RouterC. The return address will be an IP that belongs in RouterA and RouterB’s routing table, however RouterC has no knowledge of that subnet (as it’s not participating in OSPF). RouterC will just use it’s default route and send that packet back out to the internet.  Eventually the TTL will kill that packet.

101routeback4 Troubleshooting 101   Can I get back?

This can be fixed by putting a static route on RouterC to let it know that RouterA’s ip range needs to be sent off to RouterB instead.

101routeback5 Troubleshooting 101   Can I get back?

A similar thing will happen if we add a server to SwitchA. That server’s default gateway will most likely be RouterC. If a host in OfficeA send a PING to that server, that server will then send traffic off to RouterC. If RouterC does not have the static route added above, it’ll send it out to the internet.

I’m well aware that the design in the picture is pretty bad, but I used it to illustrate a point. That point being that just because router’s know how to get from A to B, it does NOT mean they know how to route that traffic back. Make sure you understand this!

Tagged with:  

Using TCPDUMP to Capture and Analyse Packets

On October 15, 2009, in CCIE, by Darren

This is another of my posts originally posted this on http://networking-forum.com/blog/.

Recently we needed to analyze packet flow through a router which had roughly 1000+ DSL customers running through. Initially we used Wireshark, but Wireshark has a nasty habit of crashing when trying to analyze so many packets and sessions. In fact this bug is a known bug and can be viewed here: http://wiki.wireshark.org/KnownBugs/OutOfMemory

Our capture server was initially running Windows 2003 with only 1GB of ram. This lasted roughly an hour or less capturing before dying. We upgraded the ram to 4GB but the server would still crash, albeit after about 2 hours now.
We then decided to use TCPDUMP. We’ve had TCPDUMP running for many days without a single crash and it hardly uses any resources at all:

screenshot Using TCPDUMP to Capture and Analyse Packets

You can see TCPDUMP has currently been running for almost 138 hours. CPU usage is 2.3%, memory usage is negligible, and remember it’s currently capturing packets for over 1000 users!

Prerequisites

In order to use TCPDUMP you will of course need to have a Linux box (any distro will do) and TCPDUMP. TCPDUMP comes standard with many distros but if not you’ll be able to install it from your repo’s.
You’ll need 2 network card if you’re going to be managing this remotely. One to log into the server itself and the other which will be capturing packets.

This is currently how I’ve set mine up:

monitoring Using TCPDUMP to Capture and Analyse Packets

You’ll need to configure the router itself to mirror all traffic through the port you want to monitor to your Linux box. Basically it’s simply sending the same packets out both interfaces.

On the Linux box you need to configure the capture NIC with no IP address and your access NIC with it’s normal usual IP address (So you can still log onto it) Each particular distro will be different but on RedHat type systems have a look in /etc/sysconfig/network-scripts
On my particular server I have eth0 as my access NIC with a regular IP address and eth1 as my capture NIC. The config for the capture NIC in the ifcfg-eth1 file is as follows:

DEVICE=eth1
BOOTPROTO=static
HWADDR=
ONBOOT=yes
TYPE=Ethernet

As you can see no IP address is configured although it is initialized on boot.

Using TCPDUMP

I personally like to use screen to run TCPDUMP as it allows me to log off the server and log back on and connect to the same process. If you’ve never used screen before I suggest this link: http://linux.die.net/man/1/screen
First you will need to create a folder for your dump files. I personally created one in /tcpdumpfiles/ – You then need to make this writable by all by doing a:

chmod +x /tcpdumpfiles/

How many files you dump is up to your needs and space available on the server. This is the command I used:

screen tcpdump -i eth1 -C 100 -s 100 -w /tcpdumpfiles/dsldump.dmp -W 500

Screen will open up TCPDUMP in the screen emulator.
Tcpdump will start up the process itself.
• -i eth1 is telling TCPDUMP to capture packets on the eth1 interface.
• -C 100 tells TCPDUMP to create files of 100MB in size.
• -s tells TCPDUMP to capture the first 100 bytes of the packet only (We only actually need the headers, no data)
• -w /tcpdumpfiles/dsldump.dmp tells TCPDUMP to create a *binary* file to the dump location we created earlier (More on this later) and not just output live data to screen
• -W 500 tells TCPDUMP to create 500 files and then start rotating, writing over the first file created and so on to prevent this server from simply running out of space.
This particular configuration will capture 50GB worth of headers and then start rotating. Feel free to change the values to suit you.

Analyzing

In the previous section we specifically created binary files instead of regular text dumped to screen. Why is this? Well TCPDUMP’s binary files are in the same format as WireSharks. Therefore in order to analyze it’s now easy to copy a dump file to a Windows box and just open it in Wireshark. This way we get the best of both worlds. Capturing using TCPDUMP which is extremely solid and unresourceful, as well as being able to view the captured packets in Wireshark with it myriad of filters and colours.
Capturing to a binary file also allows you to analyse the dump files with TCPDUMP itself or by using a script, but that will be another lesson in itself.

Tagged with:  

© 2009-2014 Darren O'Connor All Rights Reserved