Always check the forwarding table – IOS, Junos, Netiron

Most bigger routers these days use a distributed system. One of the bigger differences is the separation on the control and forwarding plane. When troubleshooting or verifying it’s essential to view both. Too many engineers simply show the control plane output. While these should match, they don’t always. Note that the forwarding table doesn’t have to be distributed to different hardware.

For the examples below I’ll simply be viewing a default route learned through OSPF. The router in question will always have two equal costs out of the network so you would expect to see two routes.

IOS

First we check the routing table:

R1#sh ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
  Known via "ospf 1", distance 110, metric 1, candidate default path
  Tag 1, type extern 2, forward metric 2
  Last update from 10.0.12.2 on GigabitEthernet2/0, 00:00:33 ago
  Routing Descriptor Blocks:
  * 10.0.13.3, from 10.0.24.4, 00:00:33 ago, via GigabitEthernet1/0
      Route metric is 1, traffic share count is 1
      Route tag 1
    10.0.12.2, from 10.0.24.4, 00:00:33 ago, via GigabitEthernet2/0
      Route metric is 1, traffic share count is 1
      Route tag 1

Two ways to get to 0.0.0.0 – What does the forwarding table show? For this I’ll choose an IP that would follow the default route:

R1#sh ip cef 4.2.2.1
0.0.0.0/0
  nexthop 10.0.12.2 GigabitEthernet2/0
  nexthop 10.0.13.3 GigabitEthernet1/0

Both control plane and data plane agree.

Netiron

Routing table:

[email protected]#sh ip route 0.0.0.0
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       0.0.0.0/0          10.0.0.1        eth 15/1      110/110       O1   1h22m  -
        0.0.0.0/0          10.0.0.2        eth 16/1      110/110       O1   1h22m  -

In order to show the forwarding table you use show route x.x.x.x detail. Note that I’m executing this command on an XMR16 and I will get the forwarding entry for every single module. I’m going to only show the output for the first module:

[email protected]#sh ip route 4.2.2.1 detail
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       0.0.0.0/0          10.0.0.1        eth 15/1      110/110       O1   1h24m  -
        0.0.0.0/0          10.0.0.1        eth 16/1      110/110       O1   1h24m  -
        Nexthop Entry ID:65540, Paths: 2, Ref_Count:707/712

D:Dynamic  P:Permanent  F:Forward  U:Us  C:Connected Network E: ESI VLAN
W:Wait ARP  I:ICMP Deny  K:Drop  R:Fragment  S:Snap Encap N:CamInvalid

Module S1:
      IP Address         Next Hop        MAC              Type  Port  Vlan  Pri
      0.0.0.0/0          10.0.0.1       0012.f293.a802   PF    16/1   1     0

      OutgoingIf  ArpIndex PPCR_ID   CamLevel   Parent  DontAge Index Is_trunk
      eth 16/1    5        1:1       31              0               0 0

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000               0     0x0005ffff (L3, right)  2

        CAM Entry Flag: 00000001H
        PPCR : 1:1 CIDX: 0x0005ffff (L3, right) (IP_NETWORK: 0x56000)

        pram_index_programmed: ppcr[0] 0x0000014c

The output is a little cryptic so I’ll highlight the important bits. First the paths show as two:

Nexthop Entry ID:65540, Paths: 2, Ref_Count:707/712

But the actual next-hop is only showing a single:

     0.0.0.0/0          10.0.0.1       0012.f293.a802   PF    16/1   1     0

This is a cosmetic error. The most important bit is here:

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000               0     0x0005ffff (L3, right)  2

The hardware path count is two, which is what we expect.

Junos

Finally Junos. First up we look at the route table:

[email protected]_SRX6> show route 0.0.0.0

inet.0: 32 destinations, 32 routes (29 active, 3 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[OSPF/150] 00:00:12, metric 0, tag 0
                      to 172.30.0.17 via ge-0/0/4.126
                    > to 172.30.0.89 via ge-0/0/4.146

Two routes, our forwarding table should match?

[email protected]_SRX6> show route forwarding-table destination 4.2.2.1
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            user     1 0:c:29:86:21:55    ucst   584    13 ge-0/0/4.146
default            perm     0                    rjct    36     5

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   534     1

Well no, it doesn’t. While the route table shows two routes, only one is being used by the forwarding table. Junos will not install multiple next-hops into the forwarding-table unless you tell it to:

[email protected]_SRX6> show configuration policy-options policy-statement BALANCE
then {
    load-balance per-packet;
}
[email protected]_SRX6> show configuration routing-options forwarding-table
export BALANCE;

Let’s check again:

[email protected]_SRX6> show route forwarding-table destination 4.2.2.1
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            user     1                    ulst 262142     7
                              0:c:29:25:21:57    ucst   612    11 ge-0/0/4.126
                              0:c:29:86:21:55    ucst   584     9 ge-0/0/4.146
default            perm     0                    rjct    36     5

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   534     1

This time we have both in the forwarding table. Note that while the policy states load-blance per-packet, it’s actually doing per-flow load-sharing.

Conclusion

I have seen routers disagree as to what they think they are doing compared to what they are doing. You need to check both tables above to note what both are doing. This could help immensely when a router is dropping packets it’s supposed to be forwarding, due to your FIB having no entry. I might write a bit on this as I’ve seen it happen more than once.

EDIT – 04/11/13

I’ve since found another way to verify this on the Brocades. If you rconsole onto the line card itself you can see a bit more:

[email protected]#rconsole 1
Remote connection to LP slot 1 established
Press CTRL-X or type 'exit' to disconnect it
LP-1>en
LP-1#sh ip network 0.0.0.0
D:Dynamic  P:Permanent  F:Forward  U:Us  C:Connected Network
W:Wait ARP  I:ICMP Deny  K:Drop  R:Fragment  S:Snap Encap N:CamInvalid
      IP Address         Next Hop        MAC              Type  Port  Vlan  Pri
      0.0.0.0/0          10.0.0.1*    0012.f293.ad02   PF    15/1*  1     0

      OutgoingIf  ArpIndex PPCR_ID   CamLevel   Parent  DontAge Index Is_trunk
      eth 15/1    4        1:1       31              0               0 0

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000  0x00000001   0     0x0005ffff (L3, right)  2

        CAM Entry Flag: 00000001H
        PPCR : 1:1 CIDX: 0x0005ffff (L3, right) (IP_NETWORK: 0x56000)

        pram_index_programmed: ppcr[0] 0x0000014c
use_index: 0
IP-nh-Pram 0: 0x2ebeec10, ref_count 1
n_paths = 2, type = ECMP_PHY_VE, is_default  = 1, vrf_index = 0
  path[0]: FORWARD, out_intf eth 15/1, nh 10.0.0.1, out_port 15/1, is_trunk 0
  path[1]: FORWARD, out_intf eth 16/1, nh 10.0.0.5, out_port 16/1, is_trunk 0
Pram info: alloc_count 2 use_count 2
  pram[0]: idx 0, pram_idx[0] 0x0000014c
  pram[1]: idx 1, pram_idx[0] 0x0000014d

The top half still shows a single port, but down it shows this:

n_paths = 2, type = ECMP_PHY_VE, is_default  = 1, vrf_index = 0
  path[0]: FORWARD, out_intf eth 15/1, nh 10.0.0.1, out_port 15/1, is_trunk 0
  path[1]: FORWARD, out_intf eth 16/1, nh 10.0.0.5, out_port 16/1, is_trunk 0

n paths is the number of paths. The router is also doing ECMP. It then shows which ports outbound it’ll send traffic.

On a route with only a single hop the bit above are shown as so:

n_paths = 1, type = NON_ECMP, is_default  = 0, vrf_index = 0
  path[0]: FORWARD, out_intf eth 1/20, nh 10.0.0.8, out_port 1/20, is_trunk 0

RIB, FIB, LFIB, LIB etc

It can be quite confusing to work out what all these terms are exactly referring to. I’ll try and put a concise answer for all of them here with an example.

RIB – Routing Information Base
This is the route table. i.e. When you do a show ip route, the RIB is what you see

AR1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/32 is subnetted, 1 subnets
C       10.255.255.5 is directly connected, Loopback0

A router may have many separate RIB’s. If you’re running vrf’s with customer, then each vrf will have a separate RIB:

AR1#sh ip route vrf CUS1 | begin Gateway
Gateway of last resort is not set

C    192.168.1.0/24 is directly connected, FastEthernet0/0

AR1#sh ip route vrf CUS2 | begin Gateway
Gateway of last resort is not set

     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.1.0 is directly connected, FastEthernet2/0

FIB – Forwarding Information Base
The FIB is an optimised version of the RIB. Or more correctly it’s the table a router looks at when deciding where to actually forward traffic. In Cisco land, the CEF table is a FIB.

AR1#sh ip cef
Prefix              Next Hop             Interface
0.0.0.0/0           drop                 Null0 (default route handler entry)
0.0.0.0/8           drop
0.0.0.0/32          receive
10.255.255.5/32     receive
127.0.0.0/8         drop
224.0.0.0/4         drop
224.0.0.0/24        receive
240.0.0.0/4         drop
255.255.255.255/32  receive

Like the RIB, there can be multiple FIB tables on a router:

AR1#sh ip cef vrf CUS1
Prefix              Next Hop             Interface
0.0.0.0/0           drop                 Null0 (default route handler entry)
0.0.0.0/32          receive
192.168.1.0/24      attached             FastEthernet0/0
192.168.1.0/32      receive
192.168.1.1/32      receive
192.168.1.255/32    receive
224.0.0.0/24        receive
255.255.255.255/32  receive

AR1#sh ip cef vrf CUS2
Prefix              Next Hop             Interface
0.0.0.0/0           drop                 Null0 (default route handler entry)
0.0.0.0/32          receive
172.16.1.0/24       attached             FastEthernet2/0
172.16.1.0/32       receive
172.16.1.1/32       receive
172.16.1.255/32     receive
224.0.0.0/24        receive
255.255.255.255/32  receive

LIB – Label Information Base
The LIB is an MPLS table. This is the place where the router will keep all known MPLS labels. To take a look, you just need to use show mpls ldp bindings:

AR1#sh mpls ldp bindings
  tib entry: 10.0.0.0/30, rev 18
        local binding:  tag: 21
        remote binding: tsr: 10.255.255.1:0, tag: imp-null
        remote binding: tsr: 10.255.255.2:0, tag: imp-null
  tib entry: 10.0.0.4/30, rev 16
        local binding:  tag: 20
        remote binding: tsr: 10.255.255.1:0, tag: 32
        remote binding: tsr: 10.255.255.2:0, tag: imp-null
  tib entry: 10.0.0.8/30, rev 44
        local binding:  tag: 34
        remote binding: tsr: 10.255.255.1:0, tag: 31
        remote binding: tsr: 10.255.255.2:0, tag: 32
  tib entry: 10.0.0.12/30, rev 42
        local binding:  tag: 33
        remote binding: tsr: 10.255.255.1:0, tag: imp-null
        remote binding: tsr: 10.255.255.2:0, tag: 31
  tib entry: 10.1.0.0/30, rev 5
        local binding:  tag: imp-null
        remote binding: tsr: 10.255.255.1:0, tag: imp-null
        remote binding: tsr: 10.255.255.2:0, tag: 30
  tib entry: 10.2.0.0/30, rev 4
        local binding:  tag: imp-null
        remote binding: tsr: 10.255.255.1:0, tag: 30
        remote binding: tsr: 10.255.255.2:0, tag: imp-null
  tib entry: 10.7.0.0/30, rev 36
        local binding:  tag: 30
        remote binding: tsr: 10.255.255.1:0, tag: 25
        remote binding: tsr: 10.255.255.2:0, tag: 26
  tib entry: 10.8.0.0/30, rev 34
        local binding:  tag: 29
        remote binding: tsr: 10.255.255.1:0, tag: imp-null
        remote binding: tsr: 10.255.255.2:0, tag: 25
  tib entry: 10.255.255.1/32, rev 30
        local binding:  tag: 27
        remote binding: tsr: 10.255.255.1:0, tag: imp-null
        remote binding: tsr: 10.255.255.2:0, tag: 23
  tib entry: 10.255.255.2/32, rev 10
        local binding:  tag: 17
        remote binding: tsr: 10.255.255.1:0, tag: 23
        remote binding: tsr: 10.255.255.2:0, tag: imp-null
  tib entry: 10.255.255.3/32, rev 28
        local binding:  tag: 26
        remote binding: tsr: 10.255.255.1:0, tag: 22
        remote binding: tsr: 10.255.255.2:0, tag: 22
  tib entry: 10.255.255.4/32, rev 26
        local binding:  tag: 25
        remote binding: tsr: 10.255.255.1:0, tag: 21
        remote binding: tsr: 10.255.255.2:0, tag: 21
  tib entry: 10.255.255.5/32, rev 6
        local binding:  tag: imp-null
        remote binding: tsr: 10.255.255.1:0, tag: 20
        remote binding: tsr: 10.255.255.2:0, tag: 20
  tib entry: 10.255.255.7/32, rev 24
        local binding:  tag: 24
        remote binding: tsr: 10.255.255.1:0, tag: 18
        remote binding: tsr: 10.255.255.2:0, tag: 18
AR1#

LFIB – Label Forwarding Instance Base
The LFIB is another MPLS table. This is the table that the router uses to forward labelled packets going through the network. Much like the RIB uses the FIB to forward traffic, so the LIB uses the LFIB to forward traffic. This is how you view it:

AR1#sh mpls forwarding-table
Local  Outgoing    Prefix            Bytes tag  Outgoing   Next Hop
tag    tag or VC   or Tunnel Id      switched   interface
17     Pop tag     10.255.255.2/32   0          Se1/2      point2point
20     Pop tag     10.0.0.4/30       0          Se1/2      point2point
21     Pop tag     10.0.0.0/30       0          Se1/2      point2point
       Pop tag     10.0.0.0/30       0          Se1/0      point2point
24     18          10.255.255.7/32   0          Se1/0      point2point
25     21          10.255.255.4/32   0          Se1/2      point2point
26     22          10.255.255.3/32   0          Se1/0      point2point
27     Pop tag     10.255.255.1/32   0          Se1/0      point2point
29     Pop tag     10.8.0.0/30       0          Se1/0      point2point
30     25          10.7.0.0/30       0          Se1/0      point2point
33     Pop tag     10.0.0.12/30      0          Se1/0      point2point
34     32          10.0.0.8/30       0          Se1/2      point2point
       31          10.0.0.8/30       0          Se1/0      point2point