September, 2014 | Darren's Blog

python interface-checker

I’d like helpdesk to be able to enable and disable switchports without requiring them to know the underlying OS. My plan is to have a webpage with a list of devices. When you click on a device it will check the interfaces status via SNMP and display administrative and operational status of all interfaces on the device. Then via a few click it’ll use NETCONF to enable or disable a port.

As a first step I’ve created a quick python script to SNMP into a device and get it’s output. For the moment this just outputs to the CLI in a table. Long term the front end will use this to get the interface status when a device is clicked.

The next step is to get NETCONF working.

I’ve tested this script on Juniper, Cisco, and Brocade and all data is retrieved correctly.

Here is an example from a Juniper MX80:

Darren-MBP:interface-checker darren$ ./ic.py router-name
+-------------------------+-----------------+--------------+--------------------+
|        OID Value        |  Interface Name | Admin Status | Operational Status |
+-------------------------+-----------------+--------------+--------------------+
|  1.3.6.1.2.1.2.2.1.2.1  |       fxp0      |     DOWN     |        DOWN        |
|  1.3.6.1.2.1.2.2.1.2.4  |       lsi       |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.5  |       dsc       |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.6  |       lo0       |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.7  |       tap       |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.8  |       gre       |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.9  |       ipip      |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.10 |       pime      |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.11 |       pimd      |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.12 |       mtun      |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.16 |      lo0.0      |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.17 |       em0       |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.18 |      em0.0      |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.21 |    lo0.16384    |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.22 |    lo0.16385    |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.23 |       em1       |      UP      |        DOWN        |
|  1.3.6.1.2.1.2.2.1.2.33 |       me0       |      UP      |         UP         |
|  1.3.6.1.2.1.2.2.1.2.34 |      me0.0      |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.501 |      demux0     |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.502 |     lc-0/0/0    |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.503 |  lc-0/0/0.32769 |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.504 |       cbp0      |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.505 |       pip0      |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.506 |       pp0       |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.507 |       irb       |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.508 |     xe-0/0/0    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.509 |     xe-0/0/1    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.510 |     xe-0/0/2    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.511 |     xe-0/0/3    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.512 |     ge-1/0/0    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.513 |     ge-1/0/1    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.514 |     ge-1/0/2    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.515 |     ge-1/0/3    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.516 |     ge-1/0/4    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.517 |     ge-1/0/5    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.518 |     ge-1/0/6    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.519 |     ge-1/0/7    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.520 |     ge-1/0/8    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.521 |     ge-1/0/9    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.522 |     ge-1/1/0    |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.523 |     ge-1/1/1    |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.524 |     ge-1/1/2    |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.525 |     ge-1/1/3    |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.526 |     ge-1/1/4    |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.527 |     ge-1/1/5    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.528 |     ge-1/1/6    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.529 |     ge-1/1/7    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.530 |     ge-1/1/8    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.531 |     ge-1/1/9    |     DOWN     |        DOWN        |
| 1.3.6.1.2.1.2.2.1.2.532 |    pfh-0/0/0    |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.533 |    pfe-0/0/0    |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.534 | pfh-0/0/0.16383 |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.535 | pfe-0/0/0.16383 |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.547 |    ge-1/1/0.0   |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.552 |    ge-1/1/1.0   |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.553 |    ge-1/1/2.0   |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.554 |    ge-1/1/3.0   |      UP      |         UP         |
| 1.3.6.1.2.1.2.2.1.2.555 |    ge-1/1/4.0   |      UP      |         UP         |
+-------------------------+-----------------+--------------+--------------------+

You can find it here: https://github.com/mellowdrifter/interface-checker

Debian/Ubuntu PMTUD & uRPF

I originally started my PMTUD posts using Ubuntu 14.04. Halfway through the post I simply could not get Ubuntu to change it’s MTU on receipt of ICMP fragmentation needed messages. I then tried Debian and it worked. Windows also had no issues changing it’s MTU.

Wanting to finish off the post I switched to Debian and then would investigate the fault later.

Let’s remind ourselves of the original topology:

Swap out Debian for Ubuntu in the above image.

When I initially started to test, I dropped the MTU between R1 and R2 to 1400. The link between R2 and R4 was kept at 1500. If the user requested a file from the server at this point, Ubuntu would attempt to send at 1500 and get it’s packet dropped at R1. R1 would send a Fragmentation Needed packet back to the Ubuntu server, which would adjust it’s MTU and then send at 1400.

When I changed the MTU between R1 and R2 back up to 1500 and dropped R2-R4 down to 1400, it no longer worked. Debian and Windows did work. I ran tcpdump on Ubuntu and confirmed that it was definitely getting Fragmentation Needed packets. Ubuntu was only acting on Fragmentation Needed packets if it came from it’s default gateway, R1. Any router further along in the path was getting it’s ICMP packets ignored.

In order to understand what the problem is I need to show more about the topology. While the above diagram shows how thins are connected for the most part, it is missing a couple of things. All the devices are running inside virtualbox linked to GNS3. eth1 of all the servers are connected to the above topology, while eth0 was connected via NAT to my host PC so I could install software:

Each device had a static route to 192.168.0.0/16 to go out eth1 while their default route was out eth0. Some of you may be sensing what the issue is already…

The point to point links between the virtual routers are using the 10.0.0.0/8 space.

If Ubuntu received an ICMP packet from 192.168.4.1, it’s local default gateway on R1, there were no issues. If it received a packet from R2 or R4’s local interfaces, the packet was dropped. Debian and Windows both didn’t have problems, even though they are configured the same way.

sysctl.conf

I’ve touched on sysctl.conf before the the PMTUD posts, but there is an important difference in the defaults of Ubuntu and Debian. Take a look at this.
Debian:

Ubuntu:

uRPF

Ubuntu has Unicast Reverse Path Forwarding on by default. Debian has it off by default. In sysctl.conf on both machines, the required configuration setting is commented out:

R2 was originating it’s ICMP packets from it’s local interface, 10.0.12.2 in my example. Ubuntu did receive that packet, but it failed the RPF check and so was ignored. To confirm I tested this in two different ways:

Add a static route to 10.0.0.0/8 out eth1
Disable uRPF check on Ubuntu

Each test individually allowed the original PMTUD to work. What’s odd is that the sysctl.conf file in Ubuntu says that you need to uncomment the lines to turn on uRPF, but it’s on by default. Uncommenting the lines and setting the value to 1 is the same as leaving them commented. In Debian the default is to disable uRPF. In that distro you would need to uncomment the uRPF lines and set the value to 1 to turn the feature on.

Conclusions

If a server is multi-homed, PMTUD could break if the ICMP message arrives on an interface that the server is not expecting.

If you do have a server multi-homed, it would probably be best to turn off uRPF

Fundamentals – PMTUD – IPv4 vs IPv6 – Part 2 of 2

This is a continuation of a post I started back here. Please read it first before starting below.

RFC 4821

Another workaround we can use is Packetization Layer Path MTU Discovery – RFC 4821. The RFC enables a host to mainly acts in one of two ways:

Use regular PMTUD. If no acknowledgments are received and no ICMP messages are received, start to probe.
Ignore regular PMTUD and always probe.

Probing is where the host will send a packet with the min MTU configured and then attempt to increase that size. If acknowledgements are received on the larger size, then try increase it again. Option 1 will wait for a timeout so on broken PMTUD paths it starts a bit slow. It will however use regular PMTUD whenever it can so it’s a lot more efficient. Option 2 simple probes all the time. It starts a bit quicker on smaller MTU paths, but the server is also sending smaller packets to ALL paths in the beginning. Much less efficient.

I’ll configure this in Debian and then go through Wireshark to show what’s going on. Add the commands net.ipv4.tcp_mtu_probing = 1 to /etc/sysctl.conf then reload sysctl:

[email protected]:~# sysctl -p
net.ipv4.tcp_mtu_probing = 1

Start the transfer and what does Wireshark show us:

After the standard 3-way handshake, the server sends a number of 1514 byte packets. ICMP has been blocked and as such there are no ICMP fragmentation needed messages coming from R2. After 5.3 seconds the server sends a number of 578 byte packets.

These get ACK’d correctly:

0.5 seconds later the server sends a single 1090 byte packets and fill the rest of the window with 578 byte packets. As soon as the ACK for that big packet comes back, the server sends all of its packets at 1090:

A couple of things to note about this setting in Ubuntu 14.04 and Debian 7.6.0:

The system does not cache the MTU of the path found through PLPMTUD. This does mean that if you have a host making multiple TCP connections to your server over a small MTU path, each one of those are going to need to wait for the timeout.
There is no net.ipv6.tcp_mtu_probing setting in sysctl.conf. However if you enable this setting for IPv4 then IPv6 has the same behavior as IPv4:

Windows can also be configured for PLPMTUD but I’ll leave it up to the reader to figure out how to do that.

PMTUD Cache

I showed in part 1 that the server will cache an entry if the MTU is lower than the local link. By default, Debian will cache this entry for 10 minutes. This time is adjustable via sysctl.conf:

[email protected]:~# sysctl -a | grep mtu_expires
net.ipv4.route.mtu_expires = 600
net.ipv6.route.mtu_expires = 600

As soon as a value is cached, the timer starts. This timer counts down even if there is an existing file transfer. The reason is because paths can change. While the transfer is going on it could move to a path which has no MTU issues. We would want the server to then increase it’s MTU. Doing this too quickly can cause more traffic to drop and so the suggestion is to cache the MTU for 10 whole minutes and then try to increase. I’ve started a file transfer which is ongoing and then checked the cache entry on the server. You can see the timer going down:

The client has then finished downloading and disconnected from the server. At this point the server still keeps that cache entry. This ensures that if the client connects again shortly it will start with an MTU of 1400:

I’ve started a new download within the cache time above and we can see the server immediately starts sending packets with the correct MTU:

What should happen when the cache times out is that the server should try to send a larger MTU packet, up to the local MTU. I don’t see that with Debian though. I started the test with the lower MTU cached on my server. When the cache was about to expire above I started the test again and as expected the session starts with the lower cached MTU. I then changed the MTU between R2 and R5 back up to the regular MTU:

R2(config)#int fa0/1
R2(config-if)#no ip mtu 1400
R2(config-if)#end

The odd thing is, when the cache entry timed out, Debian carried on sending packets with an MTU of 1400 and cached the entry again. That’s not supposed to happen.

I then tried the same test again, this time manually clearing the cache on Debian:

[email protected]:~#ip route flush cache

This time the server immediately started to send larger packets:

IPv6 has roughly the same broken behavior. At first the cache is created and starts to count down. I started a transfer when it was about to expire. This time it again stayed at 1400, but the timer jumped into a huge number:

8590471 seconds is roughly 99 days. Not sure if this is a bug or what exactly.

Clearing the IPv6 cache on the other hand had the required effect:

If the MTU matches the outgoing interface, there is no need for the system to cache that entry taking up more resources on the server. Wireshark shows the jump in MTU:

Conclusions

Blocking the required ICMP packets breaks PMTUD completely.
There are alternatives to PMTUD, but they are slower initially.
Test your OS’s behavior. I mainly tested with Debian and I ran into a number of ‘odd’ scenarios. Mainly to do with the cache.