Tag Archives: bgp

VPLS on Junos signalled via LDP or BGP

Continuing on from the L2VPN on Junos post, let’s switch focus to VPLS. CCC is a point to point technology and so out of the question. That leaves both LDP and BGP to do our VC label signalling. As always, you can use either LDP or RSVP for your transport label signalling.

Slightly different topology this time, as I’m using to test different ways for the CE to attach to the VPLS. For now we’ll simply focus on T1, C2, and T2:
VPLS1 VPLS on Junos signalled via LDP or BGP

All three CE WAN interfaces are in the same subnet running OSPF. The goal is for them to be able to reach each other’s loopbacks. As far as the CE devices are concerned, they are simply plugged into a ‘big switch’

LDP

I’ll concentrate on the PE R3 for this example. We first need to let the router know that the interface pointing towards T1 will in fact be a VPLS interface:

darreno@M7i> show configuration interfaces fe-0/0/1
encapsulation ethernet-vpls;
unit 0;

Our regular RSVP MPLS config, nothing special. Note that LDP is configured for the loopback interface:

darreno@M7i> show configuration protocols
rsvp {
    interface all;
}
mpls {
    label-switched-path TO-R6 {
        to 6.6.6.6;
        no-cspf;
    }
    label-switched-path TO-R7 {
        to 7.7.7.7;
        no-cspf;
    }
    interface all;
}
ospf {
    traffic-engineering;
    area 0.0.0.0 {
        interface all;
    }
}
ldp {
    interface lo0.3;
}

Finally the LDP VPLS config itself. As there is no auto-discovery you need to let Junos know what other PE routers are participating in this VPLS:

darreno@M7i> show configuration routing-instances
VPLS1 {
    instance-type vpls;
    interface fe-0/0/1.0;
    protocols {
        vpls {
            vpls-id 1;
            neighbor 6.6.6.6;
            neighbor 7.7.7.7;
        }
    }
}

I’ve matched the above configs on R6 and R7. Let’s take a look at the network from T1′s perspective:

USERT1@M7i:T1> show ospf neighbor
Address          Interface              State     ID               Pri  Dead
192.168.0.2      fe-0/1/0.0             Full      12.12.12.12      128    37
192.168.0.3      fe-0/1/0.0             Full      14.14.14.14      128    36

USERT1@M7i:T1> show route protocol ospf

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

12.12.12.12/32     *[OSPF/10] 00:05:43, metric 1
                    > to 192.168.0.2 via fe-0/1/0.0
14.14.14.14/32     *[OSPF/10] 00:27:15, metric 1
                    > to 192.168.0.3 via fe-0/1/0.0
224.0.0.5/32       *[OSPF/10] 2d 06:44:41, metric 1
                      MultiRecv

USERT1@M7i:T1> ping 12.12.12.12 rapid count 30
PING 12.12.12.12 (12.12.12.12): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 12.12.12.12 ping statistics ---
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.085/1.338/6.357/0.935 ms

USERT1@M7i:T1> ping 14.14.14.14 rapid count 30
PING 14.14.14.14 (14.14.14.14): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 14.14.14.14 ping statistics ---
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.081/1.496/11.077/1.784 ms

T1 considers T2 and C2 to be directly connected via L2. There is an OSPF neighbourship between all three and routes are learned. The data plane is also functioning correctly.

BGP

Let’s turn our attention now to BGP. There are a number of advantages to using BGP, especially if you already run BGP in the SP network. There is another address family which will not only advertise VC labels between PE routers, it will also allow PE routers to auto-discover any other PE configured in the same VPLS.

I’ll keep the interface config the same as above. You may notice that there is more configuration for the BGP version, but in the long run there is less config as that same BGP session is good for all your VPLS instances on the PE.

Let’s start with our BGP config:

darreno@M7i> show configuration routing-options autonomous-system
100;

darreno@M7i> show configuration protocols bgp
group iBGP {
    local-address 3.3.3.3;
    family l2vpn {
        signaling;
    }
    peer-as 100;
    neighbor 6.6.6.6;
    neighbor 7.7.7.7;
}

The BGP VPLS config is slightly different. We now have site-identifiers, but no manual neighbour config. As with our L3VPN set up, we need both RD and RTs configured.

darreno@M7i> show configuration routing-instances
VPLS1 {
    instance-type vpls;
    interface fe-0/0/1.0;
    route-distinguisher 100:200;
    vrf-target target:100:200;
    protocols {
        vpls {
            site T1 {
                site-identifier 1;
            }
        }
    }
}

We test from our CE once again:

USERT1@M7i:T1> show ospf neighbor
Address          Interface              State     ID               Pri  Dead
192.168.0.2      fe-0/1/0.0             Full      12.12.12.12      128    34
192.168.0.3      fe-0/1/0.0             Full      14.14.14.14      128    36

USERT1@M7i:T1> show route protocol ospf

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

12.12.12.12/32     *[OSPF/10] 00:03:34, metric 1
                    > to 192.168.0.2 via fe-0/1/0.0
14.14.14.14/32     *[OSPF/10] 00:04:26, metric 1
                    > to 192.168.0.3 via fe-0/1/0.0
224.0.0.5/32       *[OSPF/10] 2d 07:00:30, metric 1
                      MultiRecv

USERT1@M7i:T1> ping 12.12.12.12 rapid count 30
PING 12.12.12.12 (12.12.12.12): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 12.12.12.12 ping statistics ---
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.061/1.480/10.779/1.728 ms

USERT1@M7i:T1> ping 14.14.14.14 rapid count 30
PING 14.14.14.14 (14.14.14.14): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 14.14.14.14 ping statistics ---
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.079/1.183/1.394/0.088 ms

LDP & BGP

There is another way to get this to work. You can use BGP for auto-discovery, while using LDP to advertise the VC labels. This is the same way Brocade Netiron boxes do this, and inter-op is the only reason I would do it this way. If you have BGP running already, why not just let it do both discovery and VC assignment?

The configuration on PE R3 has been changed as follows:

darreno@M7i> show configuration protocols bgp
group iBGP {
    local-address 3.3.3.3;
    family l2vpn {
        auto-discovery-only;
    }
    peer-as 100;
    neighbor 6.6.6.6;
    neighbor 7.7.7.7;
}

darreno@M7i> show configuration routing-instances
VPLS1 {
    instance-type vpls;
    interface fe-0/0/1.0;
    route-distinguisher 100:200;
    l2vpn-id l2vpn-id:100:200;
    vrf-target target:100:200;
    protocols {
        vpls;
    }
}

CE-CE connectivity has been tested as above with no issues at all.

L2VPN on Junos using CCC/Martini/Kompella

There are three main ways to provide a point to point L2 link over MPLS on Junos. Below I’ll give a brief description and show how to configure all three.

For the below descriptions I’ll be using this simple topology. All these devices are running as logical-systems on a single MX5.
L2vpn L2VPN on Junos using CCC/Martini/Kompella

CE Configs

My two CE devices will be configured the same for all three types below:
CE T1:

root@MX5:T1> show configuration interfaces ge-1/1/0
unit 0 {
    family inet {
        address 2.2.2.1/24;
    }
}

CE BB:

root@MX5:BB> show configuration interfaces ge-1/1/3
unit 0 {
    family inet {
        address 2.2.2.2/24;
    }
}

CCC

Circuit Cross-Connect is a legacy type of L2 point to point link found in Junos. CCC over MPLS requires a unique RSVP LSP per circuit per direction. It needs a unique LSP as there is no VC (inner) label. Any frames going over an LSP belongs to a specific circuit.

L2vpnCCC L2VPN on Junos using CCC/Martini/Kompella

Customers frames in either direction is encapsulated into another L2 frame with an RSVP label. This label determines which circuit the frame belongs to on the other side via the configuration.
CCC Header L2VPN on Junos using CCC/Martini/Kompella

The core network has already been set up with OSPF/RSVP/MPLS so I won’t go over that. I’ll concentrate on the PE boxes themselves. I need to create a regular RSVP LSP each way. The CCC config sits under the protocols connections stanza
R3

interfaces {
    ge-1/1/1 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R6 {
            to 6.6.6.6;
            no-cspf;
        }
        interface all;
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
    connections {
        remote-interface-switch R6 {
            interface ge-1/1/1.0;
            transmit-lsp TO-R6;
            receive-lsp TO-R3;
        }
    }
}

R6 has a similar config back to R3. To confirm on the PE we can view via show connections:

root@MX5:R3> show connections | find R6
R6                                rmt-if      Up      Aug 26 16:41:00           2
  ge-1/1/1.0                        intf  Up
  TO-R6                             tlsp  Up
  TO-R3                             rlsp  Up

Finally we can test from the CE device:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.464/0.490/1.041/0.081 ms

Martini

Martini allows you to use the same LSP for multiple circuits. The circuits can be multiplexed as there is a second MPLS label in the stack. The bottom label is the VC label and it tells the PE what L2VPN the frame belongs to. Martini uses LDP to signal the VC label. Note that transport labels can use either LDP or RSVP. If using RSVP, you need to enable LDP on the loopback interfaces of both PE routers. Of course you still need at least two LSPs are LSPs are unidirectional.

The frame header will now contain an RSVP/LDP transport label as well as an LDP-signalled VC label. This allows the PEs to multiplex multiple circuits over the same LSP.
Martini Header L2VPN on Junos using CCC/Martini/Kompella

Martini is also configured under the protocols stanza.
This is R6′s config:

interfaces {
    ge-1/1/2 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R3 {
            to 3.3.3.3;
            no-cspf;
        }
        interface all;
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
    ldp {
        interface lo0.6;
    }
    l2circuit {
        neighbor 3.3.3.3 {
            interface ge-1/1/2.0 {
                virtual-circuit-id 1;
            }
        }
    }
}

R3 and R6 should have a targeted LDP session and the circuit should be up:

root@MX5:R6> show ldp neighbor
Address            Interface          Label space ID         Hold time
3.3.3.3            lo0.6              3.3.3.3:0                39

root@MX5:R6> show l2circuit connections | find Neigh
Neighbor: 3.3.3.3
    Interface                 Type  St     Time last up          # Up trans
    ge-1/1/2.0(vc 1)          rmt   Up     Aug 26 17:28:31 2013           1
      Remote PE: 3.3.3.3, Negotiated control-word: Yes (Null)
      Incoming label: 299840, Outgoing label: 299904
      Negotiated PW status TLV: No
      Local interface: ge-1/1/2.0, Status: Up, Encapsulation: ETHERNET

All looks good. Check from CE to CE again:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.468/0.498/0.720/0.074 ms

Kompella

Kompella also uses a two label stack as Martini. The VC label is signaled via BGP. Once again your transport underneath can be either RSVP or LDP. Kompella has a lot more configuration than the above two, however as it uses BGP it means you can just add the l2vpn address family to your existing BGP deployment.

Kompella Header L2VPN on Junos using CCC/Martini/Kompella

I’ll show R3′s config here:

interfaces {
    ge-1/1/1 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R6 {
            to 6.6.6.6;
            no-cspf;
        }
        interface all;
    }
    bgp {
        group iBGP {
            local-address 3.3.3.3;
            family l2vpn {
                signaling;
            }
            peer-as 100;
            neighbor 6.6.6.6;
        }
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
}
routing-instances {
    CUS1 {
        instance-type l2vpn;
        interface ge-1/1/1.0;
        route-distinguisher 100:100;
        vrf-target target:100:1;
        protocols {
            l2vpn {
                encapsulation-type ethernet;
                interface ge-1/1/1.0;
                site T1 {
                    site-identifier 1;
                    interface ge-1/1/1.0;
                }
            }
        }
    }
}
routing-options {
    autonomous-system 100;
}

Kompella uses a routing-instance exactly like a regular L3VPN in Junos. This includes the RD and RT. We can check the BGP session:

root@MX5:R3> show bgp summary
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l2vpn.0
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
6.6.6.6                 100         35         35       0       0       13:52 Establ
  bgp.l2vpn.0: 1/1/1/0
  CUS1.l2vpn.0: 1/1/1/0

Final confirmation is to ping from CE to CE:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.469/0.497/0.714/0.072 ms

Conclusion

  • CCC cannot multiplex multiple circuits over the same LSP, hence once you start ramping up your circuits it’s not at all scalable.
  • CCC however is supported on Juniper’s EX3200 range which gives you a cheap option to offer L2VPN over MPLS
  • Martini and Kompella both multiplex over the same LSP by signalling VC labels
  • Martini is simple to configure and makes sense if you’re a small ISP offering a couple of links here and there, especially if you’re already running LDP
  • Kompella is a lot more complicated to configure, however it runs off the back of BGP. This means that you can use your existing BGP topology to run L3VPN/L2VPN/MGVPN/etc with a single signalling protocol.
  • BGP also scales incredibly well so its the best option for larger deployments.

Carrier Supporting Carrier

CSC, or Carrier Supporting Carrier, takes inter-AS L3 VPN to the next level. Let’s say that you are an ISP and you are offering L3VPN MPLS services to your customers in England. You take over another ISP located in say, Australia, and two of your UK customers are also located in Australia. They would like their offices in both locations connected over the MPLS cloud.

It would be very expensive to run a new line between those locations. You do, however, still want to provide L3VPN services to your customers. The is where CSC comes in. CSC allows another ISP to connect both sides of your ISP network together. It also ensures the the core ISP doesn’t learn about any of your customer prefixes, as it doesn’t need to.

Let’s take the following diagram into consideration. (click the image for full size)
Screen Shot 2013 05 04 at 21.01.11 1024x214 Carrier Supporting Carrier
Routers 1, 7, 14, 6, 9, and 8 are all part of ISP2. There are three routers located in each geographical location. Routers 2, 3, 15, and 5 are part of the core carrier. This network stretches to both locations. The rest are customer routers injecting their loopback addresses into OSPF to test.

The core carrier is running IS-IS and LDP. ISP100 is running OSPF and LDP. I won’t go into the regular IGP+LDP config as is pretty straightforward.

With CSC, there are a few new terms to deal with. R14 and R8 are going to be regular PE routers for our ISP. R1 and R6 are going to be called CSC-CE (Customer Supporting Carrier – Customer Edge) routers. R2 and R5 are going to be CSC-PE (Customer Supporting Carrier – Provider Edge) routers. All other ISP routers are simply core routers.

The terminology above assumes that you are speaking in regards to being the core carrier (ISP 500 in this case). That is, ISP500′s edge routers are ‘PE’ and R1 and R6 is the customer’s (ISP) PE routers (Called CE in this case)

It’s a little confusing, depending on which view you take, but it’s really not that difficult.

Initial R14/R8 config – regular PE

The first thing we can start off with is to ensure R14′s PE config is correct. I am running OSPF with the CE routers and learning routes from them:

vrf definition CUS1
 rd 14.14.14.14:1
 route-target export 100:1
 route-target import 100:1
 !
 address-family ipv4
 exit-address-family
!
vrf definition CUS2
 rd 14.14.14.14:2
 route-target export 100:2
 route-target import 100:2
 !
 address-family ipv4
 exit-address-family
!
interface FastEthernet1/1
 vrf forwarding CUS1
 ip address 10.14.16.14 255.255.255.0
 ip ospf network point-to-point
 ip ospf 3 area 0
!
interface FastEthernet2/0
 vrf forwarding CUS2
 ip address 10.14.15.14 255.255.255.0
 ip ospf network point-to-point
 ip ospf 2 area 0

R8 has a similar config on the other side so I won’t put it here.

CSC-PE config

R2 and R5 are going to act as PE routers for the core carrier. They will have their AS200 facing interfaces in a VRF. R2 and R5 will be running regular VPNv4 BGP with each other.
csc pe Carrier Supporting Carrier
R2:

vrf definition CSC_AS100
 rd 2.2.2.2:500
 route-target export 500:100
 route-target import 500:100
 !
 address-family ipv4
 exit-address-family
!
interface FastEthernet1/0
 vrf forwarding CSC_AS100
 ip address 10.1.2.2 255.255.255.0
!
router bgp 500
 no bgp default ipv4-unicast
 neighbor 5.5.5.5 remote-as 500
 neighbor 5.5.5.5 update-source Loopback0
 !
 address-family vpnv4
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 send-community extended
 exit-address-family

Once again, R5 has a similar config on the other side so I’m not putting it here.

CSC-CE

Eventually we need R14 and R8 to peer with each other via VPNv4.
csc pe customer Carrier Supporting CarrierThis is much like option C in the inter-AS config in which two PE routers peer with each other even though they are not directly connected in the same AS. In order to do so we need each of them to have a valid route to each other. As R1 and R6 each have routes to their local PE device, they need to learn routes from the other side through the core carrier. To do this I’ll be running BGP to the core carrier. I also need to ensure that I’m sending and receiving labeled BGP routes and the end LSP has to be end to end. No part of the path can be unlabelled. I need to advertise the PE’s (R14 and R8) loopback are advertised over to the core carrier:

interface FastEthernet1/0
 ip address 10.1.2.1 255.255.255.0
 mpls bgp forwarding
!
router bgp 100
 bgp log-neighbor-changes
 neighbor 10.1.2.2 remote-as 500
 !
 address-family ipv4
  network 14.14.14.14 mask 255.255.255.255
  neighbor 10.1.2.2 activate
  neighbor 10.1.2.2 allowas-in 1
  neighbor 10.1.2.2 send-label
  no auto-summary
 exit-address-family

I’ll need to allowas-in 1 as both sides are running the same AS number. Without it, the CSC-CE routers would reject the BGP update.

I then need to ensure the CSC-CE routers are redistributing those learned prefixes into the IGP:

router ospf 1
 redistribute bgp 100 subnets

R6 again has a similar config.

The end result of it so far is that R8 and R14 should now be able to ping each other from their respective loopbacks:

R8#ping 14.14.14.14 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 14.14.14.14, timeout is 2 seconds:
Packet sent with a source address of 8.8.8.8
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/45/56 ms
R8#traceroute 14.14.14.14 so lo0

Type escape sequence to abort.
Tracing the route to 14.14.14.14

  1 10.8.9.9 [MPLS: Label 24 Exp 0] 56 msec 44 msec 32 msec
  2 10.6.9.6 [MPLS: Label 24 Exp 0] 60 msec 24 msec 60 msec
  3 10.5.6.5 [MPLS: Label 16 Exp 0] 44 msec 44 msec 48 msec
  4 10.5.13.13 [MPLS: Labels 20/34 Exp 0] 48 msec 44 msec 44 msec
  5 10.3.13.3 [MPLS: Labels 19/34 Exp 0] 44 msec 44 msec 40 msec
  6 10.1.2.2 [MPLS: Label 34 Exp 0] 40 msec 44 msec 44 msec
  7 10.1.2.1 [MPLS: Label 19 Exp 0] 44 msec 44 msec 28 msec
  8 10.1.7.7 [MPLS: Label 18 Exp 0] 56 msec 32 msec 12 msec
  9 10.7.14.14 52 msec *  72 msec

Which they can.

PE config – continued

Now that the PE routers have connectivity to each other, we can set up the VPNv4 BGP session:

router bgp 100
 no bgp default ipv4-unicast
 neighbor 8.8.8.8 remote-as 100
 neighbor 8.8.8.8 update-source Loopback0
 !
 address-family vpnv4
  neighbor 8.8.8.8 activate
  neighbor 8.8.8.8 send-community extended
 exit-address-family

Is the session up?

R14#show bgp vpnv4 unicast all summary
BGP router identifier 14.14.14.14, local AS number 100
BGP table version is 1, main routing table version 1

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
8.8.8.8         4          100       4       4        1    0    0 00:00:50        0

Yes, but no prefixes learnt. We still need to redistribute between our vrf aware OSPF processes and BGP:

router ospf 3 vrf CUS1
 redistribute bgp 100 subnets
!
router ospf 2 vrf CUS2
 redistribute bgp 100 subnets
!
router bgp 100
 !
address-family ipv4 vrf CUS1
  redistribute ospf 3 vrf CUS1
 exit-address-family
!
address-family ipv4 vrf CUS2
  redistribute ospf 2 vrf CUS2
 exit-address-family

Verification

As always with these types of configs, we need to ensure both the control and data planes are working correctly. First let’s see the control plane update of R16′s loopback over to R11. The PE router of R14 should be learning this as a vrf prefix:

R14#show ip route vrf CUS1 16.16.16.16

Routing Table: CUS1
Routing entry for 16.16.16.16/32
  Known via "ospf 3", distance 110, metric 2, type intra area
  Redistributing via bgp 100
  Advertised by bgp 100
  Last update from 10.14.16.16 on FastEthernet1/1, 00:18:35 ago
  Routing Descriptor Blocks:
  * 10.14.16.16, from 16.16.16.16, 00:18:35 ago, via FastEthernet1/1
      Route metric is 2, traffic share count is 1

This prefix is converted into a VPNv4 prefix and advertised over to R8:

R8#show bgp vpnv4 un rd 14.14.14.14:1 16.16.16.16
BGP routing table entry for 14.14.14.14:1:16.16.16.16/32, version 3
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    14.14.14.14 (metric 1) from 14.14.14.14 (14.14.14.14)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:1 OSPF DOMAIN ID:0x0005:0x000000030200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.14.16.14:0
      mpls labels in/out nolabel/27

This should end up in the correct vrf table:

R8#show ip route vrf CUS1 16.16.16.16

Routing Table: CUS1
Routing entry for 16.16.16.16/32
  Known via "bgp 100", distance 200, metric 2, type internal
  Redistributing via ospf 2
  Advertised by ospf 2 subnets
  Last update from 14.14.14.14 00:07:03 ago
  Routing Descriptor Blocks:
  * 14.14.14.14 (default), from 14.14.14.14, 00:07:03 ago
      Route metric is 2, traffic share count is 1
      AS Hops 0
      MPLS label: 27
      MPLS Flags: MPLS Required

Finally R11 should receive that as on OSPF route:

R11#show ip route 16.16.16.16
Routing entry for 16.16.16.16/32
  Known via "ospf 1", distance 110, metric 2
  Tag Complete, Path Length == 1, AS 100, , type extern 2, forward metric 1
  Last update from 10.8.11.8 on FastEthernet1/0, 00:06:38 ago
  Routing Descriptor Blocks:
  * 10.8.11.8, from 10.8.11.8, 00:06:38 ago, via FastEthernet1/0
      Route metric is 2, traffic share count is 1
      Route tag 3489661028

So our control plane is all good so far. Let’s check our data plane forwarding:

R11#ping 16.16.16.16 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 16.16.16.16, timeout is 2 seconds:
Packet sent with a source address of 11.11.11.11
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/57/76 ms


R11#traceroute 16.16.16.16 so lo0

Type escape sequence to abort.
Tracing the route to 16.16.16.16

  1 10.8.11.8 8 msec 12 msec 12 msec
  2 10.8.9.9 [MPLS: Labels 24/27 Exp 0] 52 msec 56 msec 48 msec
  3 10.6.9.6 [MPLS: Labels 24/27 Exp 0] 56 msec 48 msec 48 msec
  4 10.5.6.5 [MPLS: Labels 16/27 Exp 0] 56 msec 40 msec 68 msec
  5 10.5.13.13 [MPLS: Labels 20/34/27 Exp 0] 52 msec 44 msec 52 msec
  6 10.3.13.3 [MPLS: Labels 19/34/27 Exp 0] 56 msec 48 msec 40 msec
  7 10.2.3.2 [MPLS: Labels 34/27 Exp 0] 52 msec 52 msec 40 msec
  8 10.1.2.1 [MPLS: Labels 19/27 Exp 0] 44 msec 44 msec 48 msec
  9 10.1.7.7 [MPLS: Labels 18/27 Exp 0] 52 msec 40 msec 60 msec
 10 10.14.16.14 [MPLS: Label 27 Exp 0] 40 msec 44 msec 36 msec
 11 10.14.16.16 44 msec *  52 msec

No problems there :)

Carrier Supporing Carrier Supporting Carrier

So let’s be silly and go a step further. What if our final customer is actually another ISP offering L3VPN to it’s customers? Let’s change our topology slightly(click the image for full size)
Screen Shot 2013 05 06 at 19.14.59 1024x208 Carrier Supporting Carrier

I’m not going to show all the config here as it’s simply too much to fit into a blog post. However the config itself is pretty much like so. You’ll need to click the image for the larger version:
cscsc 1024x327 Carrier Supporting Carrier

R10 and R15 are our final CE routers. Once all is configured does it all actually work?

R10#traceroute 15.15.15.15 so lo0

Type escape sequence to abort.
Tracing the route to 15.15.15.15

  1 10.10.11.11 12 msec 12 msec 8 msec
  2 10.8.11.8 [MPLS: Labels 16/17 Exp 0] 72 msec 48 msec 68 msec
  3 10.8.9.9 [MPLS: Labels 22/24/17 Exp 0] 36 msec 80 msec 44 msec
  4 10.6.9.6 [MPLS: Labels 20/24/17 Exp 0] 52 msec 56 msec 52 msec
  5 10.5.6.5 [MPLS: Labels 23/24/17 Exp 0] 64 msec 52 msec 52 msec
  6 10.5.13.13 [MPLS: Labels 20/23/24/17 Exp 0] 48 msec 60 msec 52 msec
  7 10.3.13.3 [MPLS: Labels 19/23/24/17 Exp 0] 56 msec 56 msec 48 msec
  8 10.2.3.2 [MPLS: Labels 23/24/17 Exp 0] 56 msec 44 msec 56 msec
  9 10.1.2.1 [MPLS: Labels 19/24/17 Exp 0] 48 msec 48 msec 64 msec
 10 10.1.7.7 [MPLS: Labels 16/24/17 Exp 0] 56 msec 52 msec 60 msec
 11 10.7.14.14 [MPLS: Labels 24/17 Exp 0] 44 msec 56 msec 56 msec
 12 10.15.16.16 [MPLS: Label 17 Exp 0] 56 msec 56 msec 44 msec
 13 10.15.16.15 56 msec *  76 msec

It does indeed. At this point we are up to a four label stack in AS500. If we were running RSVP-TE and FRR we would have even more labels sitting on top.

There is a much easier way to do this of course. The original Customer Carrier could just buy some some or virtual leased line or VPLS from AS500 and they would be directly connected over the same subnet. They could then run MPLS over that link and as far as anyone cares R1 and R6 would be directly connected to each other.

But of course this is a topic on the CCIE SP after all…

MPLS L3VPN – Route Distinguisher vs Route Target vs VPN label

A lot of people confuse the above 3 items. I’ll explain exactly what each of the 3 above items do, how you can see them, and how the routers use them to provide a L3VPN service.

Let’s take the following topology for this post:

RTRDVPN3 MPLS L3VPN   Route Distinguisher vs Route Target vs VPN label

Here we have 2 L3VPN customers running over our MPLS core. R5 is advertising 5.5.5.5/32. R8 is also advertising 5.5.5.5/32

Route Distinguisher:

The route distinguisher’s sole job is to keep a route unique while the PE routers advertise NLRI (Network Layer Reachability Information) to each other. If R5 and R8 both advertise 5.5.5.5/32 to R3, how will R3 advertise both of those routes to R4 while keeping them unique. The VPNV4 family itself doesn’t run in a VRF. It runs in the global routing instance and hence it needs something to distinguish a route.

Let’s take a quick look at the vrf RD config for both customers and then the vpnv4 route for 6.6.6.6/32 in the BGP table on R3:

R3#sh run | include ip vrf | rd
ip vrf CUS1
 rd 3.3.3.3:100
ip vrf CUS2
 rd 3.3.3.3:200

R3#sh bgp vpnv4 unicast rd 3.3.3.3:200 6.6.6.6
BGP routing table entry for 3.3.3.3:200:6.6.6.6/32, version 106
Paths: (1 available, best #1, table CUS2)
  Not advertised to any peer
  Local, imported path from 4.4.4.4:200:6.6.6.6/32
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:200 OSPF DOMAIN ID:0x0005:0x000000030200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.47.4:0
      mpls labels in/out nolabel/21

R3#sh bgp vpnv4 unicast rd 3.3.3.3:100 6.6.6.6
BGP routing table entry for 3.3.3.3:100:6.6.6.6/32, version 108
Paths: (1 available, best #1, table CUS1)
  Not advertised to any peer
  Local, imported path from 4.4.4.4:100:6.6.6.6/32
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:100 OSPF DOMAIN ID:0x0005:0x000000020200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.46.4:0
      mpls labels in/out nolabel/23

You can see that R3 has 2 vpnv4 routes for 6.6.6.6/32 – 3.3.3.3:200:6.6.6.6/32 and 3.3.3.3:100:6.6.6.6/32. They are unique as one contains :100: and the other contains :200: – Note that R4 does not have to match this RD in any way. It simple needs to be able to accept 2 unique routes. This is especially important when using route reflectors as RR’s will normally only advertise the best route to it’s clients. If they were not unique, the RR would only be advertising one of these routes. The RD in no way determines what VPN a route actually belongs to.

That’s all the route distinguisher does. No more.

Route Target:

The route target’s job is to tell the PE routers what VPN a route actually belongs to. Let’s take a look at the target config on R3:

R3#sh run | inc ip vrf|target
ip vrf CUS1
 route-target export 100:100
 route-target import 100:100
ip vrf CUS2
 route-target export 100:200
 route-target import 100:200

When R3 receives an advertisement from R5, not only does it change the route into a vpnv4 route with the RD to make it unique, it also adds a community value to that advertisement. This is an RT value. Once this NLRI gets to R4, R4 will ensure that only routes that have a certain RT, will be placed in their respective VRF. As an example let’s have a look at the advertisements of 5.5.5.5 from R3 to R4:

R3#sh bgp vpnv4 unicast rd 3.3.3.3:100 5.5.5.5
BGP routing table entry for 3.3.3.3:100:5.5.5.5/32, version 37
Paths: (1 available, best #1, table CUS1)
  Advertised to update-groups:
     9
  Local
    10.0.35.5 from 0.0.0.0 (3.3.3.3)
      Origin incomplete, metric 2, localpref 100, weight 32768, valid, sourced, best
      Extended Community: RT:100:100 OSPF DOMAIN ID:0x0005:0x000000020200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.35.3:0
      mpls labels in/out 24/nolabel

We can see the extended community of 100:100 is encoded into this NLRI on R3. This is advertised to R4:

R4#sh bgp vpnv4 unicast rd 3.3.3.3:100 5.5.5.5
BGP routing table entry for 3.3.3.3:100:5.5.5.5/32, version 178
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    3.3.3.3 (metric 4) from 3.3.3.3 (3.3.3.3)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:100 OSPF DOMAIN ID:0x0005:0x000000020200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.35.3:0
      mpls labels in/out nolabel/24

R4#sh run | include ip vrf | 100:100
ip vrf CUS1
 route-target export 100:100
 route-target import 100:100

R4 has an import 100:100 configuration under it’s VRF, and hence matching the community of 100:100 on the received NLRI, the PE router knows that the advertisement is meant for vrf CUS1. Note that the RD has nothing to do with this.

VPN Label:

The VPN label is to determine what VPN a packet belongs to. But hang on, surely that’s what the RT is for? No. The RT is for the control plane, while the VPN label is for the data plane. Let’s expand on that idea a bit. When R3 advertises NLRI to R4, the RT is used to determine where a route actually belongs. When it comes to R5 actually sending a packet to R6, the VPN label is used. Why? Because when a packet is sent, there is no field in the packet that the route-target is stored. Only the route advertisement contains the route-target as a community value. When R5 sends a ping to R6 from it’s loopback, it’s simply a packet with a destination address of 6.6.6.6 and a source address of 5.5.5.5.

So with L3VPNs we have two labels. The top label is the transport label and the bottom label is the VPN label. PHP will pop the transport label off the second to last router, but the VPN label will only be popped by the actual PE in question. When that frame comes in with the VPN label, R6 knows which VRF that packet belongs to.

VPN labels are advertised in the NLRI along with the RT. Let’s take a look at the 2 VPN labels that R4 is advertising to R3:

R3#sh bgp vpnv4 unicast rd 4.4.4.4:100 6.6.6.6
BGP routing table entry for 4.4.4.4:100:6.6.6.6/32, version 6
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:100 OSPF DOMAIN ID:0x0005:0x000000020200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.46.4:0
      mpls labels in/out nolabel/21
R3#sh bgp vpnv4 unicast rd 4.4.4.4:200 6.6.6.6
BGP routing table entry for 4.4.4.4:200:6.6.6.6/32, version 8
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:200 OSPF DOMAIN ID:0x0005:0x000000030200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.47.4:0
      mpls labels in/out nolabel/23

We can see that R3 will use a VPN label of 21 when sending traffic to the CUS1 VRF, while it’ll use VPN label 23 when sending to CUS2′s VRF.

Let’s run a traceroute from R5 and R8 to confirm this.
CUS1:

R5#traceroute 6.6.6.6

Type escape sequence to abort.
Tracing the route to 6.6.6.6

  1 10.0.35.3 36 msec *  52 msec
  2 10.0.13.1 [MPLS: Labels 20/21 Exp 0] 120 msec 120 msec 132 msec
  3 10.0.12.2 [MPLS: Labels 18/21 Exp 0] 92 msec 148 msec 104 msec
  4 10.0.46.4 [MPLS: Label 21 Exp 0] 104 msec 100 msec 68 msec
  5 10.0.46.6 172 msec *  140 msec

CUS2:

R8#traceroute 6.6.6.6

Type escape sequence to abort.
Tracing the route to 6.6.6.6

  1 10.0.38.3 44 msec 64 msec 40 msec
  2 10.0.13.1 [MPLS: Labels 20/23 Exp 0] 132 msec 132 msec 88 msec
  3 10.0.12.2 [MPLS: Labels 18/23 Exp 0] 124 msec 156 msec 104 msec
  4 10.0.47.4 [MPLS: Label 23 Exp 0] 192 msec 96 msec 76 msec
  5 10.0.47.7 156 msec *  116 msec

The first hop for CUS1 shows a label stack of 20/21 – 20 being transport and 21 being the VPN. CUS2 uses 20/23 – Notice that as the egress PE is the same, the same transport label is used.

On the 3rd hop on both, R2 is popping off the transport label. Both frames now get to R4. One has a VPN label of 21 and the second a label of 23. R4 knows which VRF both packets belong to and sends them on their way to the correct routers.

Hopefully this helps clear this up for some of you..

Hurricane Sandy’s affect on the core BGP table

A couple of interesting graphs. You can see in the following diagrams how Sandy affected the BGP table. These images I got from the CIDR report

450 AS numbers disappeared as well as roughly 3000 prefixes. Which of course gradually returned

I don’t have figures for RTT to various sites as my guess is that a fair amount of trans-Atlantic traffic would’ve been quite higher.

Update: RIPE has a good RTT graph over here

plot2 Hurricane Sandys affect on the core BGP table
plot Hurricane Sandys affect on the core BGP table