Always check the forwarding table – IOS, Junos, Netiron

Most bigger routers these days use a distributed system. One of the bigger differences is the separation on the control and forwarding plane. When troubleshooting or verifying it’s essential to view both. Too many engineers simply show the control plane output. While these should match, they don’t always. Note that the forwarding table doesn’t have to be distributed to different hardware.

For the examples below I’ll simply be viewing a default route learned through OSPF. The router in question will always have two equal costs out of the network so you would expect to see two routes.

IOS

First we check the routing table:

R1#sh ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
  Known via "ospf 1", distance 110, metric 1, candidate default path
  Tag 1, type extern 2, forward metric 2
  Last update from 10.0.12.2 on GigabitEthernet2/0, 00:00:33 ago
  Routing Descriptor Blocks:
  * 10.0.13.3, from 10.0.24.4, 00:00:33 ago, via GigabitEthernet1/0
      Route metric is 1, traffic share count is 1
      Route tag 1
    10.0.12.2, from 10.0.24.4, 00:00:33 ago, via GigabitEthernet2/0
      Route metric is 1, traffic share count is 1
      Route tag 1

Two ways to get to 0.0.0.0 – What does the forwarding table show? For this I’ll choose an IP that would follow the default route:

R1#sh ip cef 4.2.2.1
0.0.0.0/0
  nexthop 10.0.12.2 GigabitEthernet2/0
  nexthop 10.0.13.3 GigabitEthernet1/0

Both control plane and data plane agree.

Netiron

Routing table:

[email protected]#sh ip route 0.0.0.0
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       0.0.0.0/0          10.0.0.1        eth 15/1      110/110       O1   1h22m  -
        0.0.0.0/0          10.0.0.2        eth 16/1      110/110       O1   1h22m  -

In order to show the forwarding table you use show route x.x.x.x detail. Note that I’m executing this command on an XMR16 and I will get the forwarding entry for every single module. I’m going to only show the output for the first module:

[email protected]#sh ip route 4.2.2.1 detail
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       0.0.0.0/0          10.0.0.1        eth 15/1      110/110       O1   1h24m  -
        0.0.0.0/0          10.0.0.1        eth 16/1      110/110       O1   1h24m  -
        Nexthop Entry ID:65540, Paths: 2, Ref_Count:707/712

D:Dynamic  P:Permanent  F:Forward  U:Us  C:Connected Network E: ESI VLAN
W:Wait ARP  I:ICMP Deny  K:Drop  R:Fragment  S:Snap Encap N:CamInvalid

Module S1:
      IP Address         Next Hop        MAC              Type  Port  Vlan  Pri
      0.0.0.0/0          10.0.0.1       0012.f293.a802   PF    16/1   1     0

      OutgoingIf  ArpIndex PPCR_ID   CamLevel   Parent  DontAge Index Is_trunk
      eth 16/1    5        1:1       31              0               0 0

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000               0     0x0005ffff (L3, right)  2

        CAM Entry Flag: 00000001H
        PPCR : 1:1 CIDX: 0x0005ffff (L3, right) (IP_NETWORK: 0x56000)

        pram_index_programmed: ppcr[0] 0x0000014c

The output is a little cryptic so I’ll highlight the important bits. First the paths show as two:

Nexthop Entry ID:65540, Paths: 2, Ref_Count:707/712

But the actual next-hop is only showing a single:

     0.0.0.0/0          10.0.0.1       0012.f293.a802   PF    16/1   1     0

This is a cosmetic error. The most important bit is here:

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000               0     0x0005ffff (L3, right)  2

The hardware path count is two, which is what we expect.

Junos

Finally Junos. First up we look at the route table:

[email protected]_SRX6> show route 0.0.0.0

inet.0: 32 destinations, 32 routes (29 active, 3 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[OSPF/150] 00:00:12, metric 0, tag 0
                      to 172.30.0.17 via ge-0/0/4.126
                    > to 172.30.0.89 via ge-0/0/4.146

Two routes, our forwarding table should match?

[email protected]_SRX6> show route forwarding-table destination 4.2.2.1
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            user     1 0:c:29:86:21:55    ucst   584    13 ge-0/0/4.146
default            perm     0                    rjct    36     5

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   534     1

Well no, it doesn’t. While the route table shows two routes, only one is being used by the forwarding table. Junos will not install multiple next-hops into the forwarding-table unless you tell it to:

[email protected]_SRX6> show configuration policy-options policy-statement BALANCE
then {
    load-balance per-packet;
}
[email protected]_SRX6> show configuration routing-options forwarding-table
export BALANCE;

Let’s check again:

[email protected]_SRX6> show route forwarding-table destination 4.2.2.1
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            user     1                    ulst 262142     7
                              0:c:29:25:21:57    ucst   612    11 ge-0/0/4.126
                              0:c:29:86:21:55    ucst   584     9 ge-0/0/4.146
default            perm     0                    rjct    36     5

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   534     1

This time we have both in the forwarding table. Note that while the policy states load-blance per-packet, it’s actually doing per-flow load-sharing.

Conclusion

I have seen routers disagree as to what they think they are doing compared to what they are doing. You need to check both tables above to note what both are doing. This could help immensely when a router is dropping packets it’s supposed to be forwarding, due to your FIB having no entry. I might write a bit on this as I’ve seen it happen more than once.

EDIT – 04/11/13

I’ve since found another way to verify this on the Brocades. If you rconsole onto the line card itself you can see a bit more:

[email protected]#rconsole 1
Remote connection to LP slot 1 established
Press CTRL-X or type 'exit' to disconnect it
LP-1>en
LP-1#sh ip network 0.0.0.0
D:Dynamic  P:Permanent  F:Forward  U:Us  C:Connected Network
W:Wait ARP  I:ICMP Deny  K:Drop  R:Fragment  S:Snap Encap N:CamInvalid
      IP Address         Next Hop        MAC              Type  Port  Vlan  Pri
      0.0.0.0/0          10.0.0.1*    0012.f293.ad02   PF    15/1*  1     0

      OutgoingIf  ArpIndex PPCR_ID   CamLevel   Parent  DontAge Index Is_trunk
      eth 15/1    4        1:1       31              0               0 0

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000  0x00000001   0     0x0005ffff (L3, right)  2

        CAM Entry Flag: 00000001H
        PPCR : 1:1 CIDX: 0x0005ffff (L3, right) (IP_NETWORK: 0x56000)

        pram_index_programmed: ppcr[0] 0x0000014c
use_index: 0
IP-nh-Pram 0: 0x2ebeec10, ref_count 1
n_paths = 2, type = ECMP_PHY_VE, is_default  = 1, vrf_index = 0
  path[0]: FORWARD, out_intf eth 15/1, nh 10.0.0.1, out_port 15/1, is_trunk 0
  path[1]: FORWARD, out_intf eth 16/1, nh 10.0.0.5, out_port 16/1, is_trunk 0
Pram info: alloc_count 2 use_count 2
  pram[0]: idx 0, pram_idx[0] 0x0000014c
  pram[1]: idx 1, pram_idx[0] 0x0000014d

The top half still shows a single port, but down it shows this:

n_paths = 2, type = ECMP_PHY_VE, is_default  = 1, vrf_index = 0
  path[0]: FORWARD, out_intf eth 15/1, nh 10.0.0.1, out_port 15/1, is_trunk 0
  path[1]: FORWARD, out_intf eth 16/1, nh 10.0.0.5, out_port 16/1, is_trunk 0

n paths is the number of paths. The router is also doing ECMP. It then shows which ports outbound it’ll send traffic.

On a route with only a single hop the bit above are shown as so:

n_paths = 1, type = NON_ECMP, is_default  = 0, vrf_index = 0
  path[0]: FORWARD, out_intf eth 1/20, nh 10.0.0.8, out_port 1/20, is_trunk 0

Restricting users to only view parts of the SNMP tree – Junos

This is a similar post to this one over here where I described how to do it in IOS: http://mellowd.co.uk/ccie/?p=2332

I recently had a project where I had to give certain customers full read-access to a subinterface on an Juniper SRX. I wanted them to see the system via SNMP, but only see their subinterface and subinterface stats.

The first part of this is getting the SNMP ifindex value. This you can get very easily:

darrenolocal> show interfaces ge-0/0/0.0 | match SNMP
  Logical interface ge-0/0/0.0 (Index 73) (SNMP ifIndex 656)

For this subinterface I need to reference 656

Now we create the view:

darrenolocal> show configuration snmp
view CUSTOMER1 {
    oid ifName.656;
    oid ifDescr.656;
    oid ifInErrors.656;
    oid ifOutErrors.656;
    oid ifOperStatus.656;
    oid ifInOctets.656;
    oid ifOutOctets.656;
    oid sysDescr.0;
    oid sysUpTime.0;
    oid sysContact.0;
    oid sysName.0;
    oid sysLocation.0;
    oid ifNumber.656;
    oid ifHCInOctets.656;
    oid ifHCOutOctets.656;
    oid ifIndex.656;
    oid ifNumber.*;
}

Here I’ve given them a number of oids to view that interface index value. I’ve also allowed them to see the system name, uptime, and various other things.

I now bind this view to a community and allow only the customer to view it:

darrenolocal> show configuration snmp
community CUSTOMER1 {
    view CUSTOMER1;
    clients {
        192.168.100.1/32;
    }
}

As a test, let’s do an snmpwalk from the monitoring station using this community now:

C:\snmpwalk>snmpwalk -v 1 -c CUSTOMER1 192.168.31.252
iso.3.6.1.2.1.1.1.0 = STRING: "Juniper Networks, Inc. srx210h internet router, kernel JUNOS 12.1X45-D10 #0: 2013-07
-04 06:05:04 UTC     [email protected]:[email protected]/k
ernels/JSRXNLE/kernel Build date: 2013-07-04 07:32:04 U"
iso.3.6.1.2.1.1.3.0 = Timeticks: (7437040) 20:39:30.40
iso.3.6.1.2.1.1.4.0 = ""
iso.3.6.1.2.1.1.5.0 = ""
iso.3.6.1.2.1.1.6.0 = ""
iso.3.6.1.2.1.2.1.0 = INTEGER: 48
iso.3.6.1.2.1.2.2.1.1.656 = INTEGER: 656
iso.3.6.1.2.1.2.2.1.2.656 = STRING: "ge-0/0/0.0"
iso.3.6.1.2.1.2.2.1.8.656 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.10.656 = Counter32: 19240538
iso.3.6.1.2.1.2.2.1.14.656 = Counter32: 0
iso.3.6.1.2.1.2.2.1.16.656 = Counter32: 19226558
iso.3.6.1.2.1.2.2.1.20.656 = Counter32: 0
iso.3.6.1.2.1.31.1.1.1.1.656 = STRING: "ge-0/0/0.0"
iso.3.6.1.2.1.31.1.1.1.6.656 = Counter64: 19240538
iso.3.6.1.2.1.31.1.1.1.10.656 = Counter64: 19226558
End of MIB

A nice short walk giving them just what you want and nothing more.