OSPF as the PE-CE routing protocols deep dive – Part 3 of 3 – Loop Prevention

Read part 1
Read part 2
Read part 3

 
When customer sites are single-homed, there is no possibility of a loop forming, unless of course your customer decides to set up a bunch of GRE tunnels and run OSPF over that, but I digress. If a site is multi-homed, or two sites have a back-door between them, it’s essential that route from BGP going into OSPF, do not go back into BGP.

Let’s create a slightly different diagram for this one. R3 is now also a PE router:

The loop prevention used ultimately depends on whether a prefix comes in as internal or external. If a sham-link is configured and all OSPF routes are intra-area, no loop prevention is needed. Standard SPF is run everything is fine. This is because everything is seen in area 0, and SPF can run with full knowledge of the entire area.

As soon as type3s and type5s are used, OSPF becomes a little more distance vector like. ABRs/ASBRs originate new LSAs and other OSPF router believe what is told to them. This makes is possible for loops to appear when multual redistribution is occuring.

The down bit

Let’s go back to RFC 4577, specifically section 4.2.5.1

When a type 3 LSA is sent from a PE router to a CE router, the DN bit [OSPF-DN] in the LSA Options field MUST be set. This is used to ensure that if any CE router sends this type 3 LSA to a PE router, the PE router will not redistribute it further.

When a PE router needs to distribute to a CE router a route that comes from a site outside the latter’s OSPF domain, the PE router presents itself as an ASBR (Autonomous System Border Router), and distributes the route in a type 5 LSA. The DN bit [OSPF-DN] MUST be set in these LSAs to ensure that they will be ignored by any other PE routers that receive them.

There are deployed implementations that do not set the DN bit, but instead use OSPF route tagging to ensure that a type 5 LSA generated by a PE router will be ignored by any other PE router that may receive it. A special OSPF route tag, which we will call the VPN Route Tag (see Section 4.2.5.2), is used for this purpose. To ensure backward compatibility, all implementations adhering to this specification MUST by default support the VPN Route Tag procedures specified in Sections 4.2.5.2, 4.2.8.1, and 4.2.8.2. When it is no longer necessary to use the VPN Route Tag in a particular deployment, its use (both sending and receiving) may be disabled by configuration.

Essentially, if an LSA arrives at a PE with the down bit set, that will never be redistributed into BGP. This prevents the route from leaking in from one PE back into another PE.

Down Bit – IOS

R7 is advertising it’s loopback address. No sham-links are used and so R4 will originate a type3 LSA to R6:

R6#show ip ospf database summary 7.7.7.7  adv-router 4.4.4.4

            OSPF Router with ID (6.6.6.6) (Process ID 1)

                Summary Net Link States (Area 0)

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 441
  Options: (No TOS-capability, DC, Downward)
  LS Type: Summary Links(Network)
  Link State ID: 7.7.7.7 (summary Network Number)
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000003
  Checksum: 0x5636
  Length: 28
  Network Mask: /32
        MTID: 0         Metric: 2

Options state ‘Downward’ – This LSA is flooded to R6 -> R5 -> R3. R3, another PE, will have the LSA (all databases need to match remember) but it will not use the LSA. The routing bit will not be set, and it will not redistribute that into BGP either:

R3#  show ip ospf database summary 7.7.7.7  adv-router 4.4.4.4

            OSPF Router with ID (10.0.35.3) (Process ID 1)

                Summary Net Link States (Area 0)

  LS age: 597
  Options: (No TOS-capability, DC, Downward)
  LS Type: Summary Links(Network)
  Link State ID: 7.7.7.7 (summary Network Number)
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000003
  Checksum: 0x5636
  Length: 28
  Network Mask: /32
        MTID: 0         Metric: 2

The same happens vice-versa. Any LSA originated by R3 to R5, will be received but not used by R4.

Down Bit – IOS-XR

No change in IOS-XR behaviour. You need to be sure your domain-ids match to get a type3 between IOS and IOS-XE:

R6#sh ip ospf database summary 7.7.7.7 adv-router 4.4.4.4

            OSPF Router with ID (6.6.6.6) (Process ID 1)

                Summary Net Link States (Area 0)

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 20
  Options: (No TOS-capability, DC, Downward)
  LS Type: Summary Links(Network)
  Link State ID: 7.7.7.7 (summary Network Number)
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0x5A34
  Length: 28
  Network Mask: /32
        MTID: 0         Metric: 2

Down bit set on the type3.

Route tags – IOS

Let’s go back to the RFC to see what this is all about. Section 4.2.5.2

If a particular VRF in a PE is associated with an instance of OSPF, then by default it MUST be configured with a special OSPF route tag value, which we call the VPN Route Tag. By default, this route tag MUST be included in the Type 5 LSAs that the PE originates (as the result of receiving a BGP-distributed VPN-IPv4 route, see Section 4.2.8) and sends to any of the attached CEs.

The configuration and inclusion of the VPN Route Tag is required for backward compatibility with deployed implementations that do not set the DN bit in type 5 LSAs. The inclusion of the VPN Route Tag may be disabled by configuration if it has been determined that it is no longer needed for backward compatibility.

The value of the VPN Route Tag is arbitrary but must be distinct from any OSPF Route Tag being used within the OSPF domain. Its value MUST therefore be configurable. If the Autonomous System number of the VPN backbone is two bytes long, the default value SHOULD be an automatically computed tag based on that Autonomous System number

If the Autonomous System number is four bytes long, then a Route Tag value MUST be configured, and it MUST be distinct from any Route Tag used within the VPN itself.

If a PE router needs to use OSPF to distribute to a CE router a route that comes from a site outside the CE router’s OSPF domain, the PE router SHOULD present itself to the CE router as an Autonomous System Border Router (ASBR) and SHOULD report such routes as AS-external routes. That is, these PE routers originate Type 5 LSAs reporting the extra-domain routes as AS-external routes. Each such Type 5 LSA MUST contain an OSPF route tag whose value is that of the VPN Route Tag. This tag identifies the route as having come from a PE router. The VPN Route Tag MUST be used to ensure that a Type 5 LSA originated by a PE router is not redistributed through the OSPF area to another PE router.

Note that it says the OSPF should set a route-tag when the implementation doesn’t support setting the down bit in type5 LSAs. Also note in the previous RFC quote that it did note an implementation could set the down bit in type5s if desired. At this point I’ve stopped advertising R7’s loopback directly into OSPF and simply redistributed the loopback. This ensures that the LSA is external.

Usually when an ASBR originates a type5, that type5 remains unchanged in the domain. i.e. the originating router is the same. However according to the quote above, the PE need to originate a new type5 to the attached CE. This we see on R6:

R6#show ip ospf database external 7.7.7.7  adv-router 4.4.4.4

            OSPF Router with ID (6.6.6.6) (Process ID 1)

                Type-5 AS External Link States

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 38
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 7.7.7.7 (External Network Number )
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0x77C7
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 20
        Forward Address: 0.0.0.0
        External Route Tag: 3489661028

Notice no down bit. Also note the originator of this type5 is R4 itself. Finally the route has an external route tag of 3489661028

Much like the down bit, if a PE router receives an external LSA with a domain tag that matches it’s own, that LSA will not be used or redistributed

R3#show ip ospf 1 database external 7.7.7.7 adv-router 4.4.4.4

            OSPF Router with ID (10.0.35.3) (Process ID 1)

                Type-5 AS External Link States

  LS age: 744
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 7.7.7.7 (External Network Number )
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0x77C7
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 20
        Forward Address: 0.0.0.0
        External Route Tag: 3489661028

No routing bit set, no redistribution happening.

Route tags – IOS-XR

R6#sh ip ospf database external 7.7.7.7 adv-router 4.4.4.4

            OSPF Router with ID (6.6.6.6) (Process ID 1)

                Type-5 AS External Link States

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 11
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 7.7.7.7 (External Network Number )
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0xEFCE
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 20
        Forward Address: 0.0.0.0
        External Route Tag: 3489661028

IOS-XR and IOS have the same behaviour.

IOS – 32bit AS number – Route-tag

The RFC states that when using 16bit AS numbers, the domain tag is automatically derived. When using a 32bit AS number, it should be manually configured. You are able to manually set this even when using a 16bit number with the domain-tag command. You can see above that when using a 16bit number it was automatic. Let’s move to a 32bit number and see what we see.
A quick change of the BGP sessions:

R4#sh run | sec router bgp
router bgp 4294967295
 no bgp default ipv4-unicast
 bgp log-neighbor-changes
 neighbor 2.2.2.2 remote-as 4294967295
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 3.3.3.3 remote-as 4294967295
 neighbor 3.3.3.3 update-source Loopback0

Take a look at the type5 on R6. The domain-tag matches the 32bit AS number directly. This is not 100% confirming to the RFC which states it should be manually set:

R6#sh ip ospf database external 7.7.7.7 adv-router 4.4.4.4

            OSPF Router with ID (6.6.6.6) (Process ID 1)

                Type-5 AS External Link States

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 76
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 7.7.7.7 (External Network Number )
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0x2C48
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 20
        Forward Address: 0.0.0.0
        External Route Tag: 4294967295

Of course, R3 will not use that LSA as it’s domain-tag matches.

Considering the domain-tag matches, it stands to reason that any inter-AS VPN using OSPF would be susceptible to routing loops as each SP will have a different domain-tag. One of them could manually set it to match the other.

32bit AS number – Route-tag – IOS-XR

IOS-XR’s 32bit external behaviour is identical to IOS:

R6#sh ip ospf database external 7.7.7.7 adv-router 4.4.4.4

            OSPF Router with ID (6.6.6.6) (Process ID 1)

                Type-5 AS External Link States

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 76
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 7.7.7.7 (External Network Number )
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0xA44F
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 20
        Forward Address: 0.0.0.0
        External Route Tag: 4294967295

Once again, IOS and IOS-XR have the same behaviour.

Notes

  • Unlike parts 1 and 2 of this blog, IOS and IOS-XR finally show identical behaviour when it comes to loop prevention.

Embedded traffic capture on Junos and IOS-XE

IOS-XE and Junos both give you the ability to sniff packets directly on the device itself. This is pretty handy for troubleshooting without having to send an engineer to site with a laptop, potentialy with downtime.

Both are very flexible, so I won’t go over every single option possible on both. Rather I’ll just go over a basic capture and view on both platforms. For this post I’ll use a simple topology with an LACP interface between them to show how to get around a limitation or two:

I’ve enabled OSPF over the aggregated interface.

IOS-XE Setup

I want to view the OSPF hello packets over the port-channel. IOS-XE will not allow you to specify a port-channel interface, but you can specify a range. I’ll simply use a range of interfaces currently in the port-channel. Note that this is done in privileged exec mode and not in configuration mode:

C3850#monitor capture NEW_CAP interface range gi1/0/1 , gi2/0/1 both
C3850#monitor capture NEW_CAP match any
C3850#monitor capture NEW_CAP file location flash:CAP1.pcap

You are able to push the capture through an ACL to match all kinds of particular things. There are a load of options to change if needed. On a 3850 stack, the output needs to go to the current active switches’ flash or USB.

Without configuring any other options, take a look at the defaults used:

C3850#show monitor capture NEW_CAP

Status Information for Capture NEW_CAP
  Target Type:
   Interface: GigabitEthernet1/0/1, Direction: both
   Interface: GigabitEthernet2/0/1, Direction: both
   Status : Inactive
  Filter Details:
    Capture all packets
  Buffer Details:
   Buffer Type: LINEAR (default)
  File Details:
   Associated file name: flash:CAP1.pcap
  Limit Details:
   Number of Packets to capture: 0 (no limit)
   Packet Capture duration: 0 (no limit)
   Packet Size to capture: 0 (no limit)
   Packets per second: 0 (no limit)
   Packet sampling rate: 0 (no sampling)

There are no limits imposed anywhere. If you leave a capture on running to the flash, it could very easily fill the flash. I’ll impose a limit of 60 seconds on this capture to ensure we don’t fill up the flash:

C3850#monitor capture NEW_CAP limit duration 60

IOS-XE Capture

Let’s start the capture:

C3850#monitor capture NEW_CAP start
*Feb 26 08:23:48.854 GMT: %BUFCAP-6-ENABLE: Capture Point NEW_CAP enabled.

I can either run this for 60 seconds, or stop it manually:

C3850#monitor capture NEW_CAP stop
*Feb 26 08:24:19.584 GMT: %BUFCAP-6-DISABLE: Capture Point NEW_CAP disabled.

Very simple.

IOS-XE – View Captures

IOS-XE has a terse output:

C3850#show monitor capture file flash:CAP1.pcap
  1   0.000000     10.0.0.2 -> 224.0.0.5    OSPF Hello Packet
  2   7.868018     10.0.0.2 -> 224.0.0.5    OSPF Hello Packet
  3  15.429030     10.0.0.2 -> 224.0.0.5    OSPF Hello Packet
  4  23.035002     10.0.0.2 -> 224.0.0.5    OSPF Hello Packet

You can also see the entire detail:

C3850#show monitor capture file flash:CAP1.pcap  detailed
Frame 1: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Feb 26, 2014 08:23:53.939938000 UTC
    Epoch Time: 1393403033.939938000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ip:ospf]
Ethernet II, Src: 3c:61:04:d9:73:80 (3c:61:04:d9:73:80), Dst: 01:00:5e:00:00:05 (01:00:5e:00:00:05)
    Destination: 01:00:5e:00:00:05 (01:00:5e:00:00:05)
        Address: 01:00:5e:00:00:05 (01:00:5e:00:00:05)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 3c:61:04:d9:73:80 (3c:61:04:d9:73:80)
        Address: 3c:61:04:d9:73:80 (3c:61:04:d9:73:80)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 10.0.0.2 (10.0.0.2), Dst: 224.0.0.5 (224.0.0.5)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
        .... ..0. = ECN-Capable Transport (ECT): 0

etc....
etc....

It’s also possible to capture directly to the screen:

C3850#monitor capture NEW_CAP start display detailed
A file by the same capture file name already exists, overwrite?[confirm]
Frame 1: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Feb 26, 2014 08:31:20.753958000 UTC
    Epoch Time: 1393403480.753958000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ip:ospf]
Ethernet II, Src: 3c:61:04:d9:73:80 (3c:61:04:d9:73:80), Dst: 01:00:5e:00:00:05

IOS-XE Notes

  • The capture configuration is not saved in global config, but the config is still there. In order to remove your monitor session you need to explicitly delete it from privileged exec mode:
C3850#no monitor capture NEW_CAP
  • Embedded wireshark can capture data-plane traffic, as well as control-place traffic

Junos Capture

Start up a shell:

darreno@SRX110> start shell user root
Password:

Junos has tcpdump built-in. For this part I’ll write a file to the tmp folder which we can the view later:

root@SRX110% tcpdump -i ae0.0 -w /tmp/CAP2.pcap
Address resolution is ON. Use  to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on ae0.0, capture size 96 bytes

Junos – view captures

We can use tcpdump to view the files we created:

root@SRX110% tcpdump -qn -r /tmp/CAP2.pcap
23:51:58.856149 Out IP truncated-ip - 20 bytes missing! 10.0.0.2 > 224.0.0.5: OSPFv2, Hello, length 60
23:51:58.991515  In IP 10.0.0.1 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:08.531670  In IP 10.0.0.1 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:08.744550 Out IP truncated-ip - 20 bytes missing! 10.0.0.2 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:17.460023 Out IP truncated-ip - 20 bytes missing! 10.0.0.2 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:17.640020  In IP 10.0.0.1 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:25.978974 Out IP truncated-ip - 20 bytes missing! 10.0.0.2 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:26.888403  In IP 10.0.0.1 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:33.517479 Out IP truncated-ip - 20 bytes missing! 10.0.0.2 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:36.858979  In IP 10.0.0.1 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:42.147688 Out IP truncated-ip - 20 bytes missing! 10.0.0.2 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:46.407409  In IP 10.0.0.1 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:49.663809 Out IP truncated-ip - 20 bytes missing! 10.0.0.2 > 224.0.0.5: OSPFv2, Hello, length 60
23:52:55.448971  In IP 10.0.0.1 > 224.0.0.5: OSPFv2, Hello, length 60

If you wanted to quickly see traffic going over an interface without saving a file, you can do it directly from the cli:

darreno@SRX110> monitor traffic interface ae0.0 detail
Address resolution is ON. Use  to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on ae0.0, capture size 1514 bytes

Reverse lookup for 10.0.0.1 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use  to avoid reverse lookups on IP addresses.

23:56:24.203372  In IP (tos 0xc0, ttl   1, id 65445, offset 0, flags [none], proto: OSPF (89), length: 80) 10.0.0.1 > 224.0.0.5: OSPFv2, Hello, length 60 [len 48]
        Router-ID 192.168.255.100, Backbone Area, Authentication Type: none (0)
        Options [External, LLS]
          Hello Timer 10s, Dead Timer 40s, Mask 255.255.255.0, Priority 1
          Neighbor List:
            10.0.0.2
          LLS: checksum: 0xfff6, length: 3
            Extended Options (1), length: 4
              Options: 0x00000001 [LSDB resync]
23:56:24.527779 Out IP (tos 0xc0, ttl   1, id 62974, offset 0, flags [none], proto: OSPF (89), length: 80) 10.0.0.2 > 224.0.0.5: OSPFv2, Hello, length 60 [len 48]
        Router-ID 10.0.0.2, Backbone Area, Authentication Type: none (0)
        Options [External, LLS]
          Hello Timer 10s, Dead Timer 40s, Mask 255.255.255.0, Priority 128
          Neighbor List:
            192.168.255.100
          LLS: checksum: 0xfff6, length: 3
            Extended Options (1), length: 4
              Options: 0x00000001 [LSDB resync]

Junos notes

  • Traffic captured on Junos is control-plane traffic only. It cannot capture data-plane traffic

The System is using a non-recommended Boot mode – 3850

I recently upgraded a pair of 3850s running IOS-XE and when rebooting I noticed the following message appear on boot:

The System is using a non-recommended Boot mode.
Not all features may be available in this boot mode.
Please check the product configuration guide.

In the past I’ve always loaded the .bin file and just booted off that. The 3850 works like this, but it seems to prefer expanding the file and then loading the packages.conf – This show you how to change it quick

  • Clean the flash
  • Expand current image in RAM to flash
  • Configure switch to boot new package
  • Reload switch
C3850#software clean file flash:
Preparing clean operation ...
[2]: Cleaning up unnecessary package files
[2]: Preparing packages list to delete ...
[2]: Files that will be deleted:
    cat3k_caa-base.SPA.03.03.02SE.pkg
    cat3k_caa-drivers.SPA.03.03.02SE.pkg
    cat3k_caa-infra.SPA.03.03.02SE.pkg
    cat3k_caa-iosd-universalk9.SPA.150-1.EZ2.pkg
    cat3k_caa-platform.SPA.03.03.02SE.pkg
    cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin
    cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.conf
    cat3k_caa-wcm.SPA.10.1.121.0.pkg
    packages.conf

[2]: Do you want to proceed with the deletion? [yes/no]: yes
[2]: Clean up completed


C3850#software expand running to flash:
Preparing expand operation ...
[2]: Expanding the running bundle
[2]: Copying package files
[2]: Package files copied
[2]: Finished expanding the running bundle

C3850#conf t
Enter configuration commands, one per line.  End with CNTL/Z.

C3850(config)#boot system flash:packages.conf
C3850(config)#end
C3850#wr me
Building configuration...

C3850#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]

Full BGP in dynamips

After having trouble with IOS-XRv yesterday, I wondered if an emulated 7200 could take a full table.

In particular I’m running:

Cisco IOS Software, 7200 Software (C7200-ADVIPSERVICESK9-M), Version 12.2(33)SRE7

The ram itself isn’t even high:

Cisco 7206VXR (NPE400) processor (revision A) with 491520K/32768K bytes of memory.

The emulated 7200 has no issue taking the full table:

C7200# sh ip bgp sum
BGP router identifier 1.1.31.253, local AS number 65500
BGP table version is 482914, main routing table version 482914
479027 network entries using 61315456 bytes of memory
479027 path entries using 24909404 bytes of memory
76560/76531 BGP path/bestpath attribute entries using 9493440 bytes of memory
68802 BGP AS-PATH entries using 2856590 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 98574890 total bytes of memory
BGP activity 479427/400 prefixes, 479482/455 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.31.252      4        65550   82140      11   482914    0    0 00:07:06   479027

Memory utilisation on the virtual machine I’m running this on is a paltry 160MB – Which includes the OS itself.

Performance isn’t too bad either. I wouldn’t push transit traffic through it, but it would make a good looking glass to be able to see BGP routes and trace through it

FULL BGP Table on IOS-XRv – Seems broken

I’ve attempted to put a full table into IOS-XRv, but I’m getting all kinds of nasty messages.

First the session came up:

RP/0/0/CPU0:XR1#show bgp ipv4 un sum
Thu Feb 20 17:27:07.990 UTC
BGP router identifier 1.1.224.62, local AS number 65500
BGP generic scan interval 60 secs
BGP table state: Active
Table ID: 0xe0000000   RD version: 2
BGP main routing table version 2
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker               2          1          2          0           1           2

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
1.1.31.252     0 39326   78356       3        0    0    0 00:01:52     479556

Soon after I was getting all manner of errors:

RP/0/0/CPU0:XR1#RP/0/0/CPU0:Feb 20 17:27:29.308 : shmwin_svr[68]: %OS-SHMWIN_SVR-3-NOVMEM : Exhausted virtual memory, top 1 consumer, ifc-ipv4: 50305 KBytes
RP/0/0/CPU0:Feb 20 17:27:29.308 : shmwin_svr[68]: %OS-SHMWIN_SVR-3-NOVMEM : Exhausted virtual memory, top 2 consumer, statsd_db_g: 5103 KBytes
RP/0/0/CPU0:Feb 20 17:27:29.308 : shmwin_svr[68]: %OS-SHMWIN_SVR-3-NOVMEM : Exhausted virtual memory, top 3 consumer, aib: 2159 KBytes
RP/0/0/CPU0:Feb 20 17:27:29.308 : shmwin_svr[68]: %OS-SHMWIN_SVR-3-NOVMEM : Exhausted virtual memory, top 4 consumer, statsd_db_l: 1131 KBytes
RP/0/0/CPU0:Feb 20 17:27:29.308 : shmwin_svr[68]: %OS-SHMWIN_SVR-3-NOVMEM : Exhausted virtual memory, top 5 consumer, im_rd: 1131 KBytes
RP/0/0/CPU0:Feb 20 17:27:29.328 : fib_mgr[207]: %OS-SHMWIN-3-ALLOC_ARENA_FAILED : SHMWIN: Failed to allocate new arena from the server : 'SHMWIN_SVR' detected the 'fatal' condition 'VM is exhausted or totally fragmented'
RP/0/0/CPU0:Feb 20 17:27:29.328 : fib_mgr[207]: %ROUTING-FIB-3-PD_FAIL : FIB platform error: fib_leaf_insert 5014 Cannot insert in switching leaf 97.73.48.0/21 [0xafff21a8] type 5 flags 10000001 refcnt 1 prot ipv4 table default tableid e0000000 vrfid 60000000 ---LDI: [0xacbf25a4] type 6 flags 90141 refcnt 1 type 3 depth 1 num_slots 1 num_buckets 1 LDI pl 0xacbd306c ---PL [0xacbd306c] type 7 flags 8020 refcnt 160238 path_cnt 1 max_depth 2 ldi 0xacbf25a4  ---: 0x16 Invalid argument  : pkg/bin/fib_mgr : (PID=417911) :  -Traceback= (TRUNCATED)
RP/0/0/CPU0:Feb 20 17:27:29.328 : fib_mgr[207]: %ROUTING-FIB-3-INTERNAL_ERR : FIB internal error: (!retry)  : pkg/bin/fib_mgr : (PID=417911) :  -Traceback= 98bf26a 42497c4 4241b80 424be26 422e1b2 42f41ee 42fa798 4214150 4211fe4 421427e 4218e75 421923a 8226a4b 8224bdc 4200bd5 421081c
RP/0/0/CPU0:Feb 20 17:27:29.418 : bgp[1048]: %OS-SHMWIN-3-ALLOC_ARENA_FAILED : SHMWIN: Failed to allocate new arena from the server : 'SHMWIN_SVR' detected the 'fatal' condition 'VM is exhausted or totally fragmented'
RP/0/0/CPU0:Feb 20 17:27:29.518 : fib_mgr[207]: %ROUTING-FIB-3-GTRIE : [3] : FIB error in generic trie operation: leaf alloc: Not enough memory

RP/0/0/CPU0:XR1#
RP/0/0/CPU0:XR1#show bgp ipv4 un nexthops RP/0/0/CPU0:Feb 20 17:27:39.337 : fib_mgr[207]: %OS-SHMWIN-3-ALLOC_ARENA_FAILED : SHMWIN: Failed to allocate new arena from the server : 'SHMWIN_SVR' detected the 'fatal' condition 'VM is exhausted or totally fragmented'
RP/0/0/CPU0:Feb 20 17:27:39.527 : fib_mgr[207]: %ROUTING-FIB-3-GTRIE : [3] : FIB error in generic trie operation: leaf alloc: Not enough memory
RP/0/0/CPU0:Feb 20 17:27:40.717 : bgp[1048]: %OS-SHMWIN-3-ALLOC_ARENA_FAILED : SHMWIN: Failed to allocate new arena from the server : 'SHMWIN_SVR' detected the 'fatal' condition 'VM is exhausted or totally fragmented'

The actual resources on the VM itself isn’t too bad:

After that, the BGP session dies, restarts, and the above happens over and over again.

The VM itself has 3Gb of RAM which is more than enough for the full table in both RIB and FIB.

Anyone else managed to get this working?