Tag Archives: design

SPF Delay – CCDE

SPF timers are usually one of those things that engineers don’t bother with. Hello/Dead timers are often adjusted, but not actual SPF timers themselves.

Different vendors, and even different platforms within vendors, can have dramatically different timers. Micro-loops can be even more pronounced when different vendors/platforms are involved.

SPF Timers

In OSPF, SPF is only run when certain conditions are met. One of those conditions is when a router originates a new type-1 LSA. If a router interface goes down, it will originate a new type-1 to let other routers in the area know about it. How soon after the interface goes down does the type-1 get sent? Once another router in the area receives that type-1, does it run SPF straight away? Does it flood the LSA before or after it runs SPF?
Micro-loops form when router’s FIBs do not agree on where the best path is. Two routers will bounce a packet backwards and forwards to each other until those routers agree on the forwarding path and have that path installed in their FIB.

The best way to understand this is to show the loop forming.

Let’s consider the following topology of five routers. The OSPF costs of each link is also displayed:
SPF Timers SPF Delay   CCDE

Most router interfaces have a cost of 50, while R3 has a second slower link with a cost of 200.

Under normal circumstances, any traffic from R1 to R5 with go through R2-R4.
SPF Timers2 SPF Delay   CCDE

R1#traceroute 10.0.0.5
Type escape sequence to abort.
Tracing the route to 10.0.0.5
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 12 msec 32 msec 16 msec
  2 192.168.24.4 44 msec 56 msec 16 msec
  3 192.168.45.5 68 msec 48 msec 48 msec

When the link between R2 and R4 fails, traffic should traverse the R2-R3-R4 links:
SPF Timers3 SPF Delay   CCDE
There are a number of milliseconds where this will not be the case.

In order to show how a micro-loop is formed, I’ll first need to artificially increase my SPF timers. This is because it’s very difficult to show an actual micro-loop simply with traceroute.
On R3 I’ll increase the wait time to run SPF after it receives an LSA to 10 seconds:

R3(config)#router ospf 1
R3(config-router)# timers throttle spf 10000 10000 10000

I’ll now break the link between R2 and R4 and run another traceroute from R1 to R5:

R2(config)#int gi2/0
R2(config-if)#shut
R1#traceroute 10.0.0.5
Type escape sequence to abort.
Tracing the route to 10.0.0.5
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 16 msec 16 msec 12 msec
  2  *  *
    192.168.23.3 36 msec
  3 192.168.23.2 40 msec 36 msec 68 msec
  4 192.168.23.3 44 msec 60 msec 60 msec
  5 192.168.23.2 56 msec 64 msec 60 msec
  6 192.168.23.3 100 msec 80 msec 80 msec
  7 192.168.23.2 80 msec 80 msec 84 msec
  8 192.168.23.3 80 msec 104 msec 104 msec
  9 192.168.23.2 100 msec 104 msec 100 msec
 10 192.168.23.3 128 msec 124 msec 124 msec
 11 192.168.23.2 132 msec 116 msec 124 msec
 12 192.168.23.3 152 msec 148 msec 148 msec
 13 192.168.23.2 144 msec 144 msec 148 msec
 14 192.168.23.3 152 msec
    192.168.45.5 112 msec 84 msec

Because R3 is delaying it’s SPF run until 10 seconds after it receives a relevant LSA, it still assumes the best path is through R2. R2 has run it’s SPF and it assumes the best path is through R3. This is the reason the packet bounces between both routers. The packet get to it’s destination only when R3 has run SPF and CEF updated.

Of course in the real world we don’t wait 10 seconds. But what are the actual timers? That depends a lot on which vendor and platform you’re running:

Vendor OS Initial SPF Delay (ms)
Cisco IOS & IOS-XE 5000
Cisco IOS-XR 50
Cisco NX-OS 200
Juniper Junos 200

The above list is of course not exhaustive.

The timers between vendors and platforms can be dramatically different. Even in an environment in when you are not cared about rapid convergence, it’s still important that your IGP routers all agree on their timers. Connecting an ASR1k to an ASR9k with default timers could cause traffic to loop for almost five seconds if left to the defaults. I would suggest you ensure all OSPF routers in an area, or all IS-IS routers in the same level, have identical timers.

Another option is to ensure the initial SPF delay run timer is set high enough so that LSA/LSP reaches all edges of the area/level. That way all router can run SPF at the same time and update their FIBs at the same time. The problem with this approach is that each router receives the LSA at different times. Even if they did receive them at exactly the same time, we are relying on the fact that all routers have 100% identical SPF and FIB-Update run times.

Further Reading

RFC 5715 – A Framework for Loop-Free Convergence
RFC 6976 – Framework for Loop-Free Convergence Using the Ordered Forwarding Information Base (oFIB) Approach

Reliable static routing without the need for the data license

Sometimes it’s required that you have a number of static routes on a router, maybe for management or some other reason. If the static route point to a next-hop, but the exit interface stays up, there is no way for the router to know that it’s sending traffic down a black hole. Let’s show the following diagram as an example:
reliable static Reliable static routing without the need for the data license

R2 is a CPE on site. It has a primary link on fa0/0 connected to both R3 and R4 through a switch/VPLS. R2 is running OSPF with R3 and R4 and also has a floating static default route to R4′s fa0/0 interface. If this link goes down, the floating static route should come into play and take over. While the link is up we have a static route on R2 that sends our management traffic (10.0.0.0/24) to R4′s fa0/0 interface.

R3 is originating a default route via OSPF.

But does this actually work? Let’s configure it up quickly first and then break R2′s primary link.

R3:

interface FastEthernet0/0
 ip address 192.168.234.3 255.255.255.0
 ip ospf 1 area 0
!
router ospf 1
 default-information originate always

R4:

interface FastEthernet0/0
 ip address 192.168.234.4 255.255.255.0
 ip ospf 1 area 0
!
interface Serial0/0
 ip address 24.24.24.4 255.255.255.0

R2

interface FastEthernet0/0
 ip address 192.168.234.2 255.255.255.0
 ip ospf 1 area 0
!
interface Serial0/0
 ip address 24.24.24.2 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 24.24.24.4 200
ip route 10.0.0.0 255.255.255.0 192.168.234.4

Let’s have a look at R2′s routing table:

R2#   sh ip route | begin Gate
Gateway of last resort is 192.168.234.3 to network 0.0.0.0

C    192.168.234.0/24 is directly connected, FastEthernet0/0
     24.0.0.0/24 is subnetted, 1 subnets
C       24.24.24.0 is directly connected, Serial0/0
     10.0.0.0/24 is subnetted, 1 subnets
S       10.0.0.0 [1/0] via 192.168.234.4
O*E2 0.0.0.0/0 [110/1] via 192.168.234.3, 00:01:22, FastEthernet0/0

All looks good. I have my OSPF default route and I also have my management range route to R4.

Now for some reason, R2′s primary link fails. The fa0/0 interface stays up however.The link is dodgy, r there is a mess-up with the VPLS, it doesn’t really matter. What happens then?

R2 loses it’s adjacency with R4, but what about our management traffic?

R2#   sh ip route | begin Gate
Gateway of last resort is 24.24.24.4 to network 0.0.0.0

C    192.168.234.0/24 is directly connected, FastEthernet0/0
     24.0.0.0/24 is subnetted, 1 subnets
C       24.24.24.0 is directly connected, Serial0/0
     10.0.0.0/24 is subnetted, 1 subnets
S       10.0.0.0 [1/0] via 192.168.234.4
S*   0.0.0.0/0 [200/0] via 24.24.24.4

The problem is that we are still sending management traffic off to R4. This is the problem with a static route, it’s static! R2 has a next-hop of 192.168.234.4 – It’s interface in this subnet is still up, and so the router is trying to ARP for 192.168.234.4. Of course R4 never responds but the router will continue to try. It’ll never fail over to the backup.

Now with reliable static routing you are able to generate an IP Sla object which consistently pings another interface. If you get no response you cause the track object to go down and hence the static route goes down. The problem with this is that you need an expensive data license for the privilege of doing this.

But track objects can track a lot more than just IP Sla objects. You can also track routes. So why not track the default route considering we are learning that through the primary link? If the primary fails and OSPF times out, we will remove the OSPF default. Let’s try and see what happens:

R2:

track 1 ip route 0.0.0.0 0.0.0.0 reachability
!
ip route 10.0.0.0 255.255.255.0 192.168.234.4 track 1

Some of you may see a problem here, but bear with me.

Let’s see if this has fixed the problem:

R2#   sh ip route | begin Gate
Gateway of last resort is 24.24.24.4 to network 0.0.0.0

C    192.168.234.0/24 is directly connected, FastEthernet0/0
     24.0.0.0/24 is subnetted, 1 subnets
C       24.24.24.0 is directly connected, Serial0/0
     10.0.0.0/24 is subnetted, 1 subnets
S       10.0.0.0 [1/0] via 192.168.234.4
S*   0.0.0.0/0 [200/0] via 24.24.24.4

Hmm, we are still sending traffic to R4′s fa0/0 interface. Why is this?

R2#sh track 1
Track 1
  IP route 0.0.0.0 0.0.0.0 reachability
  Reachability is Up (static)
    1 change, last change 00:04:22
  First-hop interface is Serial0/0
  Tracked by:
    STATIC-IP-ROUTING 0

The problem is that our floating static route went live. As soon as it did we had a default route again and hence the track object is now UP.

But you don’t HAVE to track a default route. Why don’t we simply inject a phantom prefix on R3? One that will simply be used for tracking?

R3:

interface Loopback1
 ip address 3.3.3.3 255.255.255.255
 ip ospf 1 area 0

R2:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#no track 1
R2(config)#track 1 ip route 3.3.3.3/32 reachability

R2 is now tracking the loopback route from R3:

R2#sh track 1
Track 1
  IP route 3.3.3.3 255.255.255.255 reachability
  Reachability is Up (OSPF)
    2 changes, last change 00:00:03
  First-hop interface is FastEthernet0/0
  Tracked by:
    STATIC-IP-ROUTING 0
R2#   sh ip route | begin Gate
Gateway of last resort is 192.168.234.3 to network 0.0.0.0

     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/11] via 192.168.234.3, 00:00:24, FastEthernet0/0
C    192.168.234.0/24 is directly connected, FastEthernet0/0
     24.0.0.0/24 is subnetted, 1 subnets
C       24.24.24.0 is directly connected, Serial0/0
     10.0.0.0/24 is subnetted, 1 subnets
S       10.0.0.0 [1/0] via 192.168.234.4
O*E2 0.0.0.0/0 [110/1] via 192.168.234.3, 00:00:24, FastEthernet0/0

R2 now loses OSPF adjacency:

R2#
*Mar  1 00:38:51.919: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.234.3 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R2#
*Mar  1 00:38:55.239: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.234.4 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R2#
*Mar  1 00:39:05.307: %TRACKING-5-STATE: 1 ip route 3.3.3.3/32 reachability Up->Down
R2#
R2#sh track 1
Track 1
  IP route 3.3.3.3 255.255.255.255 reachability
  Reachability is Down (no route)
    3 changes, last change 00:00:11
  First-hop interface is unknown
  Tracked by:
    STATIC-IP-ROUTING 0
R2#   sh ip route | begin Gate
Gateway of last resort is 24.24.24.4 to network 0.0.0.0

C    192.168.234.0/24 is directly connected, FastEthernet0/0
     24.0.0.0/24 is subnetted, 1 subnets
C       24.24.24.0 is directly connected, Serial0/0
S*   0.0.0.0/0 [200/0] via 24.24.24.4

R2 loses the track route, removes the static and install the floating static route. All is good :)

I know there are better ways of doing the above. As in advertise management ranges via OSPF or running BFD, but not all of these are always available, especially over back up links.

MHSRP & OSPF design

I’m going to start blogging about various design stuff that I do these days at work. Maybe you can get some ideas from these types of posts.

I’m going to run this simple topology:
HSRP Secondary1 MHSRP & OSPF design

The client has 2 networks at the first site – 192.168.1.0/24 and 10.10.10.0/24. These are both connected to R1 and R2 (via a switch not shown) – R1 and R2 are running MHSRP, with R1 being the master of 192.168.1.254 and R2 being the master of 10.10.10.254. Both routers are the backup of each others group.

I want the link from R1 to be used for 192.168.1.0 and I want the link from R2 to be used for 10.10.10.0. I also want return traffic to take the same path. i.e. if R3 needs to send traffic to 10.10.10.0 – it needs to prefer the direct path to R2.

Another constraint is that I need convergence to be fast, therefore I’m not going to run HSRP on the WAN side of R1 and R2 and have to wait for the IGP to re-converge.

The final constraint is that I don’t want to have to use route-map’s on R3 itself. Let’s pretend there are going to be a lot more routers connected to OSPF area 0. I want to make it as simple as possible when newer routers come into the area. Therefore all this routing needs to be controlled by R1 and R2 themselves.

So far, pretty simple. So let’s get cracking!

Let’s do R1 first, this is the relevant config:

interface FastEthernet0/1
ip address 10.10.10.1 255.255.255.0 secondary
ip address 192.168.1.1 255.255.255.0
standby version 2
standby 1 ip 192.168.1.254
standby 1 priority 110
standby 1 preempt
standby 1 track FastEthernet0/0 50
standby 2 ip 10.10.10.254
standby 2 preempt
standby 2 track FastEthernet0/0 50

And now R2:

interface FastEthernet0/1
ip address 10.10.10.2 255.255.255.0
ip address 192.168.1.2 255.255.255.0 secondary
standby version 2
standby 1 ip 192.168.1.254
standby 1 preempt
standby 1 track FastEthernet0/0 50
standby 2 ip 10.10.10.254
standby 2 priority 110
standby 2 preempt
standby 2 track FastEthernet0/0 50

Now let’s set up OSPF. Remember that I want 192.168.1.0 traffic to go via R1 and 10.10.10.0 to go via R2. There are a number of ways of doing this.

First let’s try and set up regular OSPF with a metric:

router ospf 1
 R1(config-router)#network 10.10.10.0 0.0.0.255 area 0 ?
  cr>
 R1(config-router)#network 10.10.10.0 0.0.0.255 area 0

OSPF will not allow you to specify a metric under the network command!

There is another way. Run 2 OSPF processes and redistribute with a higher metric.

router ospf 1
 redistribute ospf 2 metric 50 metric-type 1 subnets
 network 1.1.1.0 0.0.0.255 area 0
 network 192.168.1.0 0.0.0.255 area 0
!
router ospf 2
 network 10.10.10.0 0.0.0.255 area 0

Let’s check R3′s routing table:

R3#sh ip route
     1.0.0.0/24 is subnetted, 1 subnets
C       1.1.1.0 is directly connected, FastEthernet0/0
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.50.0 is directly connected, Loopback0
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/51] via 1.1.1.1, 00:00:39, FastEthernet0/0
O    192.168.1.0/24 [110/2] via 1.1.1.1, 00:00:39, FastEthernet0/0

It works, but I think its messy. We could also use route-maps which would give us a lot finer control over newer subnets that may be added (without having to run more OSPF processes)

Let’s remove that OSPF config from R1 and do the following:

router ospf 1
 redistribute connected metric-type 1 subnets route-map 10_HIGH
 network 1.1.1.0 0.0.0.255 area 0
!
ip prefix-list 10NET seq 5 permit 10.10.10.0/24
!
route-map 10_HIGH permit 10
 match ip address prefix-list 10NET
 set metric +100
!
route-map 10_HIGH permit 20

Now we do this on R2:

router ospf 1
 redistribute connected metric-type 1 subnets route-map 192_HIGH
 network 1.1.1.0 0.0.0.255 area 0
!
ip prefix-list 192NET seq 5 permit 192.168.1.0/24
!
route-map 192_HIGH permit 10
 match ip address prefix-list 192NET
 set metric +100
!
route-map 192_HIGH permit 20

This will ensure R1 redistributes the 10.10.10.0 network with a higher metric to R3, and vice-versa for R2. R3 will have a possible route in it’s database, but won’t use it until the preferred link is down/

Let’s have a look at R3′s routing table:

R3#sh ip route ospf
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/21] via 1.1.1.2, 00:00:28, FastEthernet0/0
O E1 192.168.1.0/24 [110/21] via 1.1.1.1, 00:00:28, FastEthernet0/0

Excellent, just what we want. but do the backups work correctly? Let’s shut down R1′s interface to the customer LAN and see what happens:

R1(config)#int fa1/0
R1(config-if)#shut
R1(config-if)#
*Mar  1 00:34:20.711: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 1 state Active -> Init
*Mar  1 00:34:20.723: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 2 state Standby -> Init
*Mar  1 00:34:22.723: %LINK-5-CHANGED: Interface FastEthernet1/0, changed state to administratively down
*Mar  1 00:34:23.723: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0, changed state to down

What does R3 see?

R3#sh ip route ospf
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/21] via 1.1.1.2, 00:00:18, FastEthernet0/0
O E1 192.168.1.0/24 [110/101] via 1.1.1.2, 00:00:18, FastEthernet0/0

Both are now going via R2, which is exactly what we wanted. Let’s now reconnect R1 and disconnect R2′s WAN link:

R1(config-if)#int fa1/0
R1(config-if)#no shut
R1(config-if)#
*Mar  1 00:35:47.775: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 1 state Listen -> Active
*Mar  1 00:35:47.859: %LINK-3-UPDOWN: Interface FastEthernet1/0, changed state to up
*Mar  1 00:35:48.859: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0, changed state to up
R2(config)#int fa0/0
R2(config-if)#shut
R2(config-if)#
*Mar  1 00:36:16.179: %TRACKING-5-STATE: 1 interface Fa0/0 line-protocol Up->Down
*Mar  1 00:36:16.187: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.3 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
*Mar  1 00:36:16.187: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.1.1 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
*Mar  1 00:36:18.179: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Mar  1 00:36:18.231: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 2 state Active -> Speak
*Mar  1 00:36:19.179: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down

What does R3 see this time?

R3#sh ip route ospf
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/101] via 1.1.1.1, 00:00:03, FastEthernet0/0
O E1 192.168.1.0/24 [110/21] via 1.1.1.1, 00:00:03, FastEthernet0/0

Great. Let’s just make sure it all works properly again if all links are up:

R2(config-if)#int fa0/0
R2(config-if)#no shut
R2(config-if)#
*Mar  1 00:37:53.291: %TRACKING-5-STATE: 1 interface Fa0/0 line-protocol Down->Up
*Mar  1 00:37:54.235: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 2 state Standby -> Active
*Mar  1 00:37:55.287: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar  1 00:37:56.287: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
R3#sh ip route ospf
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/21] via 1.1.1.2, 00:00:06, FastEthernet0/0
O E1 192.168.1.0/24 [110/21] via 1.1.1.1, 00:00:06, FastEthernet0/0

Let’s see what happens when a client is pinging a network directly connected to R3 over the 192.168.1.0 network. I’ll be using a router as a client, so it’ll be a ping flood.

R4#ping 172.16.50.1 repeat 1000

I’ll quickly shut R1′s LAN interface. Wait a bit, and then no shut it. What happens to R4′s pings?

Sending 1000, 100-byte ICMP Echos to 172.16.50.1, timeout is 2 seconds:
!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (998/1000), round-trip min/avg/max = 1/44/160 ms

I lost only 2 pings! Once when I shut the interface, and the second when it came back up again. Pretty good!

It wont be as fast if the WAN link goes down though. R2 will take over for sending packets out, but return traffic will still go to R1. Why? Well remember R3 still thinks R1 is there. Unless they were directly connected, or you wait for OSPF to realise the peer is down, it’ll continue to send traffic that way. You can speed this up by reducing your OSPF hello timers though. This will need to be balanced on how much OSPF traffic you want going across your WAN link.