Tag Archives: bgp

OSPF as the PE-CE routing protocols deep dive – Part 1 of 3 – Redistribution

Read part 1
Read part 2
Read part 3

 
When doing L3VPN, using OSPF is actually one of the more complicated options. Vector-based protocols like RIP, EIGRP, and BGP are comparatively simple.

RFC4577 is a great RFC that goes over how OSPF and BGP should operate when it comes to using OSPF as the PE-CE routing protocol.

I wanted to go into detail some of what is noted on the RFC to see just how both IOS and IOS-XR interpret the RFC. Also it makes it a bit fun by purposely trying to break the RFC and seeing what happens.

First, a quick refresh of how PE-CE protocols work when not using BGP as the PE-CE routing protocol. I’m going to brush very lightly over this.

Consider the following network. R2, R3, and R4 are ISP routers in which R2 and R4 are PE routers. R7, R5, and R6 belong to the customer. R7 and R5 are both connected to the same PE while R6 is connected to another PE.

The CE routers are running OSPF with the PE routers. The PE routers redistribute these OSPF routes into BGP and then converts them to VPNv4 NLRI. These VPNv4 NLRIare advetised to other PE routers via BGP. The PE also converts these VPNv4 routes back into OSPF and then off to the CE router:

LSA Translation

Taking the above image as an example. R7 is running OSPF with R2. R2 is also running OSPF with R5 and so any LSA updates are sent to R5 from R7 as per standard OSPF rules. When R2 needs to advertise the route over to R4, that LSA needs to be converted to a VPNv4 route. R4 will then convert that VPNv4 route back to an OSPF route on the other side. So how does the RFC state this LSA must be translated?

Section 4.2.6 of the RFC states:

For every address prefix that was installed in the VRF by one of its associated OSPF instances, the PE must create a VPN-IPv4 route in BGP. Each such route will have some of the
following Extended Communities attributes:

- The OSPF Domain Identifier Extended Communities attribute. If the OSPF instance that installed the route has a non-NULL primary Domain Identifier, this MUST be present; if that OSPF instance has only a NULL Domain Identifier, it MAY be omitted. This attribute is encoded with a two-byte type field, and its type is 0005, 0105, or 0205. For backward compatibility, the type 8005 MAY be used as well and is treated as if it were 0005. If the OSPF instance has a NULL Domain Identifier, and the OSPF Domain Identifier Extended Communities attribute is present, then the attribute’s value field must be all zeroes, and its type field may be any of 0005, 0105, 0205, or 8005.

- OSPF Route Type Extended Communities Attribute. This attribute MUST be present. It is encoded with a two-byte type field, and its type is 0306. To ensure backward compatibility, the type 8000 SHOULD be accepted as well and treated as if it were type 0306. The remaining six bytes of the Attribute are encoded as follows:

Area Number – Route Type – Options

In the test network I have already configured mutual redistribution between OSPF and BGP on both PE routers. Let’s see if the VPNv4 routes match what we expect from the RFC. R7 is advertising it’s loopback into OSPF. R2 converts this to a VPNv4 route. Let’s dig into the VPNv4 route itself:

R2#show bgp vpnv4 un all  7.7.7.7
BGP routing table entry for 2.2.2.2:1:7.7.7.7/32, version 28
Paths: (1 available, best #1, table A)
  Advertised to update-groups:
     1
  Local
    10.0.27.7 from 0.0.0.0 (2.2.2.2)
      Origin incomplete, metric 2, localpref 100, weight 32768, valid, sourced, best
      Extended Community: RT:1:1 OSPF DOMAIN ID:0x0005:0x000000010200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:2.2.2.2:0
      mpls labels in/out 26/nolabel

The route has a number of extended communities. The first one we’ll look at is the domain id value of

OSPF DOMAIN ID:0x0005:0x000000010200

IOS has encoded a type 005 domain ID with a value of 000000010200. This is interesting as I have not hard-coded a domain ID. Section 4.2.4 of the RFC states:

Each OSPF instance MUST be associated with one or more Domain Identifiers. This MUST be configurable, and the default value (if none is configured) SHOULD be NULL.

I have not configured one yet there is one. This means IOS is configuring one automatically even though it SHOULD be null.

The second community we’ll look at is the Route Type Extended Communities Attribute:

OSPF RT:0.0.0.0:2:0

The RFC states that the RT is broken up as follows:

  1. 32-bit Area number
  2. Route-type
  3. Options

From our value above we can see that the original OSPF LSA is from area 0. Our RT says that this route comes from a type-2 LSA, but that’s incorrect as 7.7.7.7 is coming in via a type-1 LSA so that is a bit odd (as we shall see in a bit, it doesn’t actually matter whether this value is 1, 2, or 3 at the end of day). The final byte is the Options byte which is currently zero.

This VPNv4 update is now sent over to R4, who needs to take that information and create a new OSPF LSA and advertise it to R6. What does the RFC say about how the PE needs to do this?

VPNv4 routes received via BGP

Sescion 4.2.8.1 of the RFC states:

With respect to a particular OSPF instance associated with a VRF, a VPN-IPv4 route that is installed in the VRF and then selected as the preferred route is treated as an External Route if one of the following conditions holds:

- The route type field of the OSPF Route Type Extended Community has an OSPF route type of “external”

- The route is from a different domain from the domain of the OSPF instance

What this means is that if a route comes into a PE as an External or NSSA-External , it will always be so. It can never change. If a route comes in with a type of 1, 2, or 3; and the domain-id matches – then the local PE will originate a new type-3 LSA. i.e. the route will appear inter-area on the other customer sites.
If a route comes in with a type of 1, 2, or 3; and the domain-id does not match, then it becomes an external route.
All my routers are currently running IOS and OSPF process ID 100. This means currently all the domain-ids match. This means that R4 should be originating a new type-3 LSA. We can verify this on R6:

R6#sh ip ospf database summary 7.7.7.7

            OSPF Router with ID (6.6.6.6) (Process ID 1)

                Summary Net Link States (Area 0)

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 638
  Options: (No TOS-capability, DC, Downward)
  LS Type: Summary Links(Network)
  Link State ID: 7.7.7.7 (summary Network Number)
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0x1EDF
  Length: 28
  Network Mask: /32
        MTID: 0         Metric: 2

We see the 7.7.7.7/32 LSA coming from 4.4.4.4. This means the OSPF route should be inter area:

R6#sh ip route 7.7.7.7
Routing entry for 7.7.7.7/32
  Known via "ospf 1", distance 110, metric 3, type inter area
  Last update from 10.0.46.4 on FastEthernet1/0, 00:11:25 ago
  Routing Descriptor Blocks:
  * 10.0.46.4, from 4.4.4.4, 00:11:25 ago, via FastEthernet1/0
      Route metric is 3, traffic share count is 1

Let’s change the domain-id on R4 to Null to see if this will change the route-type:

R4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R4(config)#router ospf 1
R4(config-router)#domain-id Null
R4(config-router)#end

Verify:

R6#sh ip route 7.7.7.7
Routing entry for 7.7.7.7/32
  Known via "ospf 1", distance 110, metric 2
  Tag Complete, Path Length == 1, AS 100, , type extern 2, forward metric 1
  Last update from 10.0.46.4 on FastEthernet1/0, 00:00:08 ago
  Routing Descriptor Blocks:
  * 10.0.46.4, from 4.4.4.4, 00:00:08 ago, via FastEthernet1/0
      Route metric is 2, traffic share count is 1
      Route tag 3489661028

R6#sh ip ospf database external 7.7.7.7

            OSPF Router with ID (6.6.6.6) (Process ID 1)

                Type-5 AS External Link States

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 69
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 7.7.7.7 (External Network Number )
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0x863A
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 2
        Forward Address: 0.0.0.0
        External Route Tag: 3489661028

As expected, the route is now external.

IOS-XR

I’ve swapped out R4 with an IOS-XR box and configured it the same. How has R6′s loopback been converted into a VPNv4 route?

R2#show bgp vpnv4 un all 6.6.6.6
BGP routing table entry for 1:1:6.6.6.6/32, version 4
Paths: (1 available, best #1, table A)
  Not advertised to any peer
  Local
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:1:1 OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:4.4.4.4:0
      mpls labels in/out nolabel/16012

What’s interesting here is that IOS-XR follows the RFC a little bit more closely in that there is no implicit default Domain-ID. This means a L3VPN where some of your routers are IOS and some are IOS-XR, their Domain-IDs are not going to match unless you change the defaults. This should also mean on R6 I should be seeing external routes from R7 and R5:

RP/0/3/CPU0:R6#sho route ipv4 7.7.7.7
Thu Jan  2 17:05:59.526 UTC

Routing entry for 7.7.7.7/32
  Known via "ospf 1", distance 110, metric 2
  Tag 3489661028, type extern 2
  Installed Jan  2 17:01:21.626 for 00:04:38
  Routing Descriptor Blocks
    10.19.20.19, from 4.4.4.4, via POS0/7/0/0
      Route metric is 2
  No advertising protos.
RP/0/3/CPU0:R6#sho route ipv4 5.5.5.5
Thu Jan  2 17:06:05.378 UTC

Routing entry for 5.5.5.5/32
  Known via "ospf 1", distance 110, metric 2
  Tag 3489661028, type extern 2
  Installed Jan  2 17:01:21.625 for 00:04:43
  Routing Descriptor Blocks
    10.19.20.19, from 4.4.4.4, via POS0/7/0/0
      Route metric is 2
  No advertising protos.

Let’s hard-code the Domain-ID on R4 to ensure they now match:

RP/0/0/CPU0:R4#conf
Thu Jan  2 17:06:40.703 UTC
RP/0/0/CPU0:R4(config)#router ospf 100 vrf A domain-id type 0005 value 0000006$
RP/0/0/CPU0:R4(config)#end
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:yes
RP/0/3/CPU0:R6#sho route ipv4 5.5.5.5
Thu Jan  2 17:08:01.164 UTC

Routing entry for 5.5.5.5/32
  Known via "ospf 1", distance 110, metric 3, type inter area
  Installed Jan  2 17:07:33.737 for 00:00:27
  Routing Descriptor Blocks
    10.19.20.19, from 4.4.4.4, via POS0/7/0/0
      Route metric is 3
  No advertising protos.

Knowing the implicit defaults on both platforms can certainly save you from headaches.

Multiple Domain-IDs

IOS gives you the option to have secondary domain-IDs. The configuration guide doesn’t give all that information on what exactly it does, so it’s time to break out Wireshark. First I’ll configure multiple secondary domain-ids on R2:

R2#sh run | sec router ospf
router ospf 1 vrf A
 domain-id type 0005 value 000000010200
 domain-id type 0005 value 000000020200 secondary
 domain-id type 0005 value 000000030200 secondary
 domain-id type 0005 value 000000040200 secondary
 log-adjacency-changes
 redistribute bgp 100 subnets

Will this make R2 generate VPNv3 update with multiple extended OSPF communities? I’m capturing BGP traffic on R4′s core interface and done a route refresh:

No. The VPNv4 update still only has a single domain-id. Secondary domain-ids are for a receiving PE to look at. If it receives OSPF updates from multiple different domain-id’s, if the ID matches any of the local secondary IDs, then it is considered a match. In order for this to work, all sides will need to match multiple IDs to consider everything internal as each PE can only originate a single ID outbound.

VPLS on Junos signalled via LDP or BGP

Continuing on from the L2VPN on Junos post, let’s switch focus to VPLS. CCC is a point to point technology and so out of the question. That leaves both LDP and BGP to do our VC label signalling. As always, you can use either LDP or RSVP for your transport label signalling.

Slightly different topology this time, as I’m using to test different ways for the CE to attach to the VPLS. For now we’ll simply focus on T1, C2, and T2:

All three CE WAN interfaces are in the same subnet running OSPF. The goal is for them to be able to reach each other’s loopbacks. As far as the CE devices are concerned, they are simply plugged into a ‘big switch’

LDP

I’ll concentrate on the PE R3 for this example. We first need to let the router know that the interface pointing towards T1 will in fact be a VPLS interface:

darreno@M7i> show configuration interfaces fe-0/0/1
encapsulation ethernet-vpls;
unit 0;

Our regular RSVP MPLS config, nothing special. Note that LDP is configured for the loopback interface:

darreno@M7i> show configuration protocols
rsvp {
    interface all;
}
mpls {
    label-switched-path TO-R6 {
        to 6.6.6.6;
        no-cspf;
    }
    label-switched-path TO-R7 {
        to 7.7.7.7;
        no-cspf;
    }
    interface all;
}
ospf {
    traffic-engineering;
    area 0.0.0.0 {
        interface all;
    }
}
ldp {
    interface lo0.3;
}

Finally the LDP VPLS config itself. As there is no auto-discovery you need to let Junos know what other PE routers are participating in this VPLS:

darreno@M7i> show configuration routing-instances
VPLS1 {
    instance-type vpls;
    interface fe-0/0/1.0;
    protocols {
        vpls {
            vpls-id 1;
            neighbor 6.6.6.6;
            neighbor 7.7.7.7;
        }
    }
}

I’ve matched the above configs on R6 and R7. Let’s take a look at the network from T1′s perspective:

USERT1@M7i:T1> show ospf neighbor
Address          Interface              State     ID               Pri  Dead
192.168.0.2      fe-0/1/0.0             Full      12.12.12.12      128    37
192.168.0.3      fe-0/1/0.0             Full      14.14.14.14      128    36

USERT1@M7i:T1> show route protocol ospf

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

12.12.12.12/32     *[OSPF/10] 00:05:43, metric 1
                    > to 192.168.0.2 via fe-0/1/0.0
14.14.14.14/32     *[OSPF/10] 00:27:15, metric 1
                    > to 192.168.0.3 via fe-0/1/0.0
224.0.0.5/32       *[OSPF/10] 2d 06:44:41, metric 1
                      MultiRecv

USERT1@M7i:T1> ping 12.12.12.12 rapid count 30
PING 12.12.12.12 (12.12.12.12): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 12.12.12.12 ping statistics ---
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.085/1.338/6.357/0.935 ms

USERT1@M7i:T1> ping 14.14.14.14 rapid count 30
PING 14.14.14.14 (14.14.14.14): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 14.14.14.14 ping statistics ---
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.081/1.496/11.077/1.784 ms

T1 considers T2 and C2 to be directly connected via L2. There is an OSPF neighbourship between all three and routes are learned. The data plane is also functioning correctly.

BGP

Let’s turn our attention now to BGP. There are a number of advantages to using BGP, especially if you already run BGP in the SP network. There is another address family which will not only advertise VC labels between PE routers, it will also allow PE routers to auto-discover any other PE configured in the same VPLS.

I’ll keep the interface config the same as above. You may notice that there is more configuration for the BGP version, but in the long run there is less config as that same BGP session is good for all your VPLS instances on the PE.

Let’s start with our BGP config:

darreno@M7i> show configuration routing-options autonomous-system
100;

darreno@M7i> show configuration protocols bgp
group iBGP {
    local-address 3.3.3.3;
    family l2vpn {
        signaling;
    }
    peer-as 100;
    neighbor 6.6.6.6;
    neighbor 7.7.7.7;
}

The BGP VPLS config is slightly different. We now have site-identifiers, but no manual neighbour config. As with our L3VPN set up, we need both RD and RTs configured.

darreno@M7i> show configuration routing-instances
VPLS1 {
    instance-type vpls;
    interface fe-0/0/1.0;
    route-distinguisher 100:200;
    vrf-target target:100:200;
    protocols {
        vpls {
            site T1 {
                site-identifier 1;
            }
        }
    }
}

We test from our CE once again:

USERT1@M7i:T1> show ospf neighbor
Address          Interface              State     ID               Pri  Dead
192.168.0.2      fe-0/1/0.0             Full      12.12.12.12      128    34
192.168.0.3      fe-0/1/0.0             Full      14.14.14.14      128    36

USERT1@M7i:T1> show route protocol ospf

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

12.12.12.12/32     *[OSPF/10] 00:03:34, metric 1
                    > to 192.168.0.2 via fe-0/1/0.0
14.14.14.14/32     *[OSPF/10] 00:04:26, metric 1
                    > to 192.168.0.3 via fe-0/1/0.0
224.0.0.5/32       *[OSPF/10] 2d 07:00:30, metric 1
                      MultiRecv

USERT1@M7i:T1> ping 12.12.12.12 rapid count 30
PING 12.12.12.12 (12.12.12.12): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 12.12.12.12 ping statistics ---
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.061/1.480/10.779/1.728 ms

USERT1@M7i:T1> ping 14.14.14.14 rapid count 30
PING 14.14.14.14 (14.14.14.14): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 14.14.14.14 ping statistics ---
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.079/1.183/1.394/0.088 ms

LDP & BGP

There is another way to get this to work. You can use BGP for auto-discovery, while using LDP to advertise the VC labels. This is the same way Brocade Netiron boxes do this, and inter-op is the only reason I would do it this way. If you have BGP running already, why not just let it do both discovery and VC assignment?

The configuration on PE R3 has been changed as follows:

darreno@M7i> show configuration protocols bgp
group iBGP {
    local-address 3.3.3.3;
    family l2vpn {
        auto-discovery-only;
    }
    peer-as 100;
    neighbor 6.6.6.6;
    neighbor 7.7.7.7;
}

darreno@M7i> show configuration routing-instances
VPLS1 {
    instance-type vpls;
    interface fe-0/0/1.0;
    route-distinguisher 100:200;
    l2vpn-id l2vpn-id:100:200;
    vrf-target target:100:200;
    protocols {
        vpls;
    }
}

CE-CE connectivity has been tested as above with no issues at all.

L2VPN on Junos using CCC/Martini/Kompella

There are three main ways to provide a point to point L2 link over MPLS on Junos. Below I’ll give a brief description and show how to configure all three.

For the below descriptions I’ll be using this simple topology. All these devices are running as logical-systems on a single MX5.

CE Configs

My two CE devices will be configured the same for all three types below:
CE T1:

root@MX5:T1> show configuration interfaces ge-1/1/0
unit 0 {
    family inet {
        address 2.2.2.1/24;
    }
}

CE BB:

root@MX5:BB> show configuration interfaces ge-1/1/3
unit 0 {
    family inet {
        address 2.2.2.2/24;
    }
}

CCC

Circuit Cross-Connect is a legacy type of L2 point to point link found in Junos. CCC over MPLS requires a unique RSVP LSP per circuit per direction. It needs a unique LSP as there is no VC (inner) label. Any frames going over an LSP belongs to a specific circuit.

Customers frames in either direction is encapsulated into another L2 frame with an RSVP label. This label determines which circuit the frame belongs to on the other side via the configuration.

The core network has already been set up with OSPF/RSVP/MPLS so I won’t go over that. I’ll concentrate on the PE boxes themselves. I need to create a regular RSVP LSP each way. The CCC config sits under the protocols connections stanza
R3

interfaces {
    ge-1/1/1 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R6 {
            to 6.6.6.6;
            no-cspf;
        }
        interface all;
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
    connections {
        remote-interface-switch R6 {
            interface ge-1/1/1.0;
            transmit-lsp TO-R6;
            receive-lsp TO-R3;
        }
    }
}

R6 has a similar config back to R3. To confirm on the PE we can view via show connections:

root@MX5:R3> show connections | find R6
R6                                rmt-if      Up      Aug 26 16:41:00           2
  ge-1/1/1.0                        intf  Up
  TO-R6                             tlsp  Up
  TO-R3                             rlsp  Up

Finally we can test from the CE device:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.464/0.490/1.041/0.081 ms

Martini

Martini allows you to use the same LSP for multiple circuits. The circuits can be multiplexed as there is a second MPLS label in the stack. The bottom label is the VC label and it tells the PE what L2VPN the frame belongs to. Martini uses LDP to signal the VC label. Note that transport labels can use either LDP or RSVP. If using RSVP, you need to enable LDP on the loopback interfaces of both PE routers. Of course you still need at least two LSPs are LSPs are unidirectional.

The frame header will now contain an RSVP/LDP transport label as well as an LDP-signalled VC label. This allows the PEs to multiplex multiple circuits over the same LSP.

Martini is also configured under the protocols stanza.
This is R6′s config:

interfaces {
    ge-1/1/2 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R3 {
            to 3.3.3.3;
            no-cspf;
        }
        interface all;
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
    ldp {
        interface lo0.6;
    }
    l2circuit {
        neighbor 3.3.3.3 {
            interface ge-1/1/2.0 {
                virtual-circuit-id 1;
            }
        }
    }
}

R3 and R6 should have a targeted LDP session and the circuit should be up:

root@MX5:R6> show ldp neighbor
Address            Interface          Label space ID         Hold time
3.3.3.3            lo0.6              3.3.3.3:0                39

root@MX5:R6> show l2circuit connections | find Neigh
Neighbor: 3.3.3.3
    Interface                 Type  St     Time last up          # Up trans
    ge-1/1/2.0(vc 1)          rmt   Up     Aug 26 17:28:31 2013           1
      Remote PE: 3.3.3.3, Negotiated control-word: Yes (Null)
      Incoming label: 299840, Outgoing label: 299904
      Negotiated PW status TLV: No
      Local interface: ge-1/1/2.0, Status: Up, Encapsulation: ETHERNET

All looks good. Check from CE to CE again:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.468/0.498/0.720/0.074 ms

Kompella

Kompella also uses a two label stack as Martini. The VC label is signaled via BGP. Once again your transport underneath can be either RSVP or LDP. Kompella has a lot more configuration than the above two, however as it uses BGP it means you can just add the l2vpn address family to your existing BGP deployment.

I’ll show R3′s config here:

interfaces {
    ge-1/1/1 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R6 {
            to 6.6.6.6;
            no-cspf;
        }
        interface all;
    }
    bgp {
        group iBGP {
            local-address 3.3.3.3;
            family l2vpn {
                signaling;
            }
            peer-as 100;
            neighbor 6.6.6.6;
        }
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
}
routing-instances {
    CUS1 {
        instance-type l2vpn;
        interface ge-1/1/1.0;
        route-distinguisher 100:100;
        vrf-target target:100:1;
        protocols {
            l2vpn {
                encapsulation-type ethernet;
                interface ge-1/1/1.0;
                site T1 {
                    site-identifier 1;
                    interface ge-1/1/1.0;
                }
            }
        }
    }
}
routing-options {
    autonomous-system 100;
}

Kompella uses a routing-instance exactly like a regular L3VPN in Junos. This includes the RD and RT. We can check the BGP session:

root@MX5:R3> show bgp summary
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l2vpn.0
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
6.6.6.6                 100         35         35       0       0       13:52 Establ
  bgp.l2vpn.0: 1/1/1/0
  CUS1.l2vpn.0: 1/1/1/0

Final confirmation is to ping from CE to CE:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.469/0.497/0.714/0.072 ms

Conclusion

  • CCC cannot multiplex multiple circuits over the same LSP, hence once you start ramping up your circuits it’s not at all scalable.
  • CCC however is supported on Juniper’s EX3200 range which gives you a cheap option to offer L2VPN over MPLS
  • Martini and Kompella both multiplex over the same LSP by signalling VC labels
  • Martini is simple to configure and makes sense if you’re a small ISP offering a couple of links here and there, especially if you’re already running LDP
  • Kompella is a lot more complicated to configure, however it runs off the back of BGP. This means that you can use your existing BGP topology to run L3VPN/L2VPN/MGVPN/etc with a single signalling protocol.
  • BGP also scales incredibly well so its the best option for larger deployments.

Carrier Supporting Carrier

CSC, or Carrier Supporting Carrier, takes inter-AS L3 VPN to the next level. Let’s say that you are an ISP and you are offering L3VPN MPLS services to your customers in England. You take over another ISP located in say, Australia, and two of your UK customers are also located in Australia. They would like their offices in both locations connected over the MPLS cloud.

It would be very expensive to run a new line between those locations. You do, however, still want to provide L3VPN services to your customers. The is where CSC comes in. CSC allows another ISP to connect both sides of your ISP network together. It also ensures the the core ISP doesn’t learn about any of your customer prefixes, as it doesn’t need to.

Let’s take the following diagram into consideration. (click the image for full size)

Routers 1, 7, 14, 6, 9, and 8 are all part of ISP2. There are three routers located in each geographical location. Routers 2, 3, 15, and 5 are part of the core carrier. This network stretches to both locations. The rest are customer routers injecting their loopback addresses into OSPF to test.

The core carrier is running IS-IS and LDP. ISP100 is running OSPF and LDP. I won’t go into the regular IGP+LDP config as is pretty straightforward.

With CSC, there are a few new terms to deal with. R14 and R8 are going to be regular PE routers for our ISP. R1 and R6 are going to be called CSC-CE (Customer Supporting Carrier – Customer Edge) routers. R2 and R5 are going to be CSC-PE (Customer Supporting Carrier – Provider Edge) routers. All other ISP routers are simply core routers.

The terminology above assumes that you are speaking in regards to being the core carrier (ISP 500 in this case). That is, ISP500′s edge routers are ‘PE’ and R1 and R6 is the customer’s (ISP) PE routers (Called CE in this case)

It’s a little confusing, depending on which view you take, but it’s really not that difficult.

Initial R14/R8 config – regular PE

The first thing we can start off with is to ensure R14′s PE config is correct. I am running OSPF with the CE routers and learning routes from them:

vrf definition CUS1
 rd 14.14.14.14:1
 route-target export 100:1
 route-target import 100:1
 !
 address-family ipv4
 exit-address-family
!
vrf definition CUS2
 rd 14.14.14.14:2
 route-target export 100:2
 route-target import 100:2
 !
 address-family ipv4
 exit-address-family
!
interface FastEthernet1/1
 vrf forwarding CUS1
 ip address 10.14.16.14 255.255.255.0
 ip ospf network point-to-point
 ip ospf 3 area 0
!
interface FastEthernet2/0
 vrf forwarding CUS2
 ip address 10.14.15.14 255.255.255.0
 ip ospf network point-to-point
 ip ospf 2 area 0

R8 has a similar config on the other side so I won’t put it here.

CSC-PE config

R2 and R5 are going to act as PE routers for the core carrier. They will have their AS200 facing interfaces in a VRF. R2 and R5 will be running regular VPNv4 BGP with each other.

R2:

vrf definition CSC_AS100
 rd 2.2.2.2:500
 route-target export 500:100
 route-target import 500:100
 !
 address-family ipv4
 exit-address-family
!
interface FastEthernet1/0
 vrf forwarding CSC_AS100
 ip address 10.1.2.2 255.255.255.0
!
router bgp 500
 no bgp default ipv4-unicast
 neighbor 5.5.5.5 remote-as 500
 neighbor 5.5.5.5 update-source Loopback0
 !
 address-family vpnv4
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 send-community extended
 exit-address-family

Once again, R5 has a similar config on the other side so I’m not putting it here.

CSC-CE

Eventually we need R14 and R8 to peer with each other via VPNv4.
This is much like option C in the inter-AS config in which two PE routers peer with each other even though they are not directly connected in the same AS. In order to do so we need each of them to have a valid route to each other. As R1 and R6 each have routes to their local PE device, they need to learn routes from the other side through the core carrier. To do this I’ll be running BGP to the core carrier. I also need to ensure that I’m sending and receiving labeled BGP routes and the end LSP has to be end to end. No part of the path can be unlabelled. I need to advertise the PE’s (R14 and R8) loopback are advertised over to the core carrier:

interface FastEthernet1/0
 ip address 10.1.2.1 255.255.255.0
 mpls bgp forwarding
!
router bgp 100
 bgp log-neighbor-changes
 neighbor 10.1.2.2 remote-as 500
 !
 address-family ipv4
  network 14.14.14.14 mask 255.255.255.255
  neighbor 10.1.2.2 activate
  neighbor 10.1.2.2 allowas-in 1
  neighbor 10.1.2.2 send-label
  no auto-summary
 exit-address-family

I’ll need to allowas-in 1 as both sides are running the same AS number. Without it, the CSC-CE routers would reject the BGP update.

I then need to ensure the CSC-CE routers are redistributing those learned prefixes into the IGP:

router ospf 1
 redistribute bgp 100 subnets

R6 again has a similar config.

The end result of it so far is that R8 and R14 should now be able to ping each other from their respective loopbacks:

R8#ping 14.14.14.14 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 14.14.14.14, timeout is 2 seconds:
Packet sent with a source address of 8.8.8.8
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/45/56 ms
R8#traceroute 14.14.14.14 so lo0

Type escape sequence to abort.
Tracing the route to 14.14.14.14

  1 10.8.9.9 [MPLS: Label 24 Exp 0] 56 msec 44 msec 32 msec
  2 10.6.9.6 [MPLS: Label 24 Exp 0] 60 msec 24 msec 60 msec
  3 10.5.6.5 [MPLS: Label 16 Exp 0] 44 msec 44 msec 48 msec
  4 10.5.13.13 [MPLS: Labels 20/34 Exp 0] 48 msec 44 msec 44 msec
  5 10.3.13.3 [MPLS: Labels 19/34 Exp 0] 44 msec 44 msec 40 msec
  6 10.1.2.2 [MPLS: Label 34 Exp 0] 40 msec 44 msec 44 msec
  7 10.1.2.1 [MPLS: Label 19 Exp 0] 44 msec 44 msec 28 msec
  8 10.1.7.7 [MPLS: Label 18 Exp 0] 56 msec 32 msec 12 msec
  9 10.7.14.14 52 msec *  72 msec

Which they can.

PE config – continued

Now that the PE routers have connectivity to each other, we can set up the VPNv4 BGP session:

router bgp 100
 no bgp default ipv4-unicast
 neighbor 8.8.8.8 remote-as 100
 neighbor 8.8.8.8 update-source Loopback0
 !
 address-family vpnv4
  neighbor 8.8.8.8 activate
  neighbor 8.8.8.8 send-community extended
 exit-address-family

Is the session up?

R14#show bgp vpnv4 unicast all summary
BGP router identifier 14.14.14.14, local AS number 100
BGP table version is 1, main routing table version 1

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
8.8.8.8         4          100       4       4        1    0    0 00:00:50        0

Yes, but no prefixes learnt. We still need to redistribute between our vrf aware OSPF processes and BGP:

router ospf 3 vrf CUS1
 redistribute bgp 100 subnets
!
router ospf 2 vrf CUS2
 redistribute bgp 100 subnets
!
router bgp 100
 !
address-family ipv4 vrf CUS1
  redistribute ospf 3 vrf CUS1
 exit-address-family
!
address-family ipv4 vrf CUS2
  redistribute ospf 2 vrf CUS2
 exit-address-family

Verification

As always with these types of configs, we need to ensure both the control and data planes are working correctly. First let’s see the control plane update of R16′s loopback over to R11. The PE router of R14 should be learning this as a vrf prefix:

R14#show ip route vrf CUS1 16.16.16.16

Routing Table: CUS1
Routing entry for 16.16.16.16/32
  Known via "ospf 3", distance 110, metric 2, type intra area
  Redistributing via bgp 100
  Advertised by bgp 100
  Last update from 10.14.16.16 on FastEthernet1/1, 00:18:35 ago
  Routing Descriptor Blocks:
  * 10.14.16.16, from 16.16.16.16, 00:18:35 ago, via FastEthernet1/1
      Route metric is 2, traffic share count is 1

This prefix is converted into a VPNv4 prefix and advertised over to R8:

R8#show bgp vpnv4 un rd 14.14.14.14:1 16.16.16.16
BGP routing table entry for 14.14.14.14:1:16.16.16.16/32, version 3
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    14.14.14.14 (metric 1) from 14.14.14.14 (14.14.14.14)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:1 OSPF DOMAIN ID:0x0005:0x000000030200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.14.16.14:0
      mpls labels in/out nolabel/27

This should end up in the correct vrf table:

R8#show ip route vrf CUS1 16.16.16.16

Routing Table: CUS1
Routing entry for 16.16.16.16/32
  Known via "bgp 100", distance 200, metric 2, type internal
  Redistributing via ospf 2
  Advertised by ospf 2 subnets
  Last update from 14.14.14.14 00:07:03 ago
  Routing Descriptor Blocks:
  * 14.14.14.14 (default), from 14.14.14.14, 00:07:03 ago
      Route metric is 2, traffic share count is 1
      AS Hops 0
      MPLS label: 27
      MPLS Flags: MPLS Required

Finally R11 should receive that as on OSPF route:

R11#show ip route 16.16.16.16
Routing entry for 16.16.16.16/32
  Known via "ospf 1", distance 110, metric 2
  Tag Complete, Path Length == 1, AS 100, , type extern 2, forward metric 1
  Last update from 10.8.11.8 on FastEthernet1/0, 00:06:38 ago
  Routing Descriptor Blocks:
  * 10.8.11.8, from 10.8.11.8, 00:06:38 ago, via FastEthernet1/0
      Route metric is 2, traffic share count is 1
      Route tag 3489661028

So our control plane is all good so far. Let’s check our data plane forwarding:

R11#ping 16.16.16.16 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 16.16.16.16, timeout is 2 seconds:
Packet sent with a source address of 11.11.11.11
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/57/76 ms


R11#traceroute 16.16.16.16 so lo0

Type escape sequence to abort.
Tracing the route to 16.16.16.16

  1 10.8.11.8 8 msec 12 msec 12 msec
  2 10.8.9.9 [MPLS: Labels 24/27 Exp 0] 52 msec 56 msec 48 msec
  3 10.6.9.6 [MPLS: Labels 24/27 Exp 0] 56 msec 48 msec 48 msec
  4 10.5.6.5 [MPLS: Labels 16/27 Exp 0] 56 msec 40 msec 68 msec
  5 10.5.13.13 [MPLS: Labels 20/34/27 Exp 0] 52 msec 44 msec 52 msec
  6 10.3.13.3 [MPLS: Labels 19/34/27 Exp 0] 56 msec 48 msec 40 msec
  7 10.2.3.2 [MPLS: Labels 34/27 Exp 0] 52 msec 52 msec 40 msec
  8 10.1.2.1 [MPLS: Labels 19/27 Exp 0] 44 msec 44 msec 48 msec
  9 10.1.7.7 [MPLS: Labels 18/27 Exp 0] 52 msec 40 msec 60 msec
 10 10.14.16.14 [MPLS: Label 27 Exp 0] 40 msec 44 msec 36 msec
 11 10.14.16.16 44 msec *  52 msec

No problems there :)

Carrier Supporing Carrier Supporting Carrier

So let’s be silly and go a step further. What if our final customer is actually another ISP offering L3VPN to it’s customers? Let’s change our topology slightly(click the image for full size)

I’m not going to show all the config here as it’s simply too much to fit into a blog post. However the config itself is pretty much like so. You’ll need to click the image for the larger version:

R10 and R15 are our final CE routers. Once all is configured does it all actually work?

R10#traceroute 15.15.15.15 so lo0

Type escape sequence to abort.
Tracing the route to 15.15.15.15

  1 10.10.11.11 12 msec 12 msec 8 msec
  2 10.8.11.8 [MPLS: Labels 16/17 Exp 0] 72 msec 48 msec 68 msec
  3 10.8.9.9 [MPLS: Labels 22/24/17 Exp 0] 36 msec 80 msec 44 msec
  4 10.6.9.6 [MPLS: Labels 20/24/17 Exp 0] 52 msec 56 msec 52 msec
  5 10.5.6.5 [MPLS: Labels 23/24/17 Exp 0] 64 msec 52 msec 52 msec
  6 10.5.13.13 [MPLS: Labels 20/23/24/17 Exp 0] 48 msec 60 msec 52 msec
  7 10.3.13.3 [MPLS: Labels 19/23/24/17 Exp 0] 56 msec 56 msec 48 msec
  8 10.2.3.2 [MPLS: Labels 23/24/17 Exp 0] 56 msec 44 msec 56 msec
  9 10.1.2.1 [MPLS: Labels 19/24/17 Exp 0] 48 msec 48 msec 64 msec
 10 10.1.7.7 [MPLS: Labels 16/24/17 Exp 0] 56 msec 52 msec 60 msec
 11 10.7.14.14 [MPLS: Labels 24/17 Exp 0] 44 msec 56 msec 56 msec
 12 10.15.16.16 [MPLS: Label 17 Exp 0] 56 msec 56 msec 44 msec
 13 10.15.16.15 56 msec *  76 msec

It does indeed. At this point we are up to a four label stack in AS500. If we were running RSVP-TE and FRR we would have even more labels sitting on top.

There is a much easier way to do this of course. The original Customer Carrier could just buy some some or virtual leased line or VPLS from AS500 and they would be directly connected over the same subnet. They could then run MPLS over that link and as far as anyone cares R1 and R6 would be directly connected to each other.

But of course this is a topic on the CCIE SP after all…

MPLS L3VPN – Route Distinguisher vs Route Target vs VPN label

A lot of people confuse the above 3 items. I’ll explain exactly what each of the 3 above items do, how you can see them, and how the routers use them to provide a L3VPN service.

Let’s take the following topology for this post:

Here we have 2 L3VPN customers running over our MPLS core. R5 is advertising 5.5.5.5/32. R8 is also advertising 5.5.5.5/32

Route Distinguisher:

The route distinguisher’s sole job is to keep a route unique while the PE routers advertise NLRI (Network Layer Reachability Information) to each other. If R5 and R8 both advertise 5.5.5.5/32 to R3, how will R3 advertise both of those routes to R4 while keeping them unique. The VPNV4 family itself doesn’t run in a VRF. It runs in the global routing instance and hence it needs something to distinguish a route.

Let’s take a quick look at the vrf RD config for both customers and then the vpnv4 route for 6.6.6.6/32 in the BGP table on R3:

R3#sh run | include ip vrf | rd
ip vrf CUS1
 rd 3.3.3.3:100
ip vrf CUS2
 rd 3.3.3.3:200

R3#sh bgp vpnv4 unicast rd 3.3.3.3:200 6.6.6.6
BGP routing table entry for 3.3.3.3:200:6.6.6.6/32, version 106
Paths: (1 available, best #1, table CUS2)
  Not advertised to any peer
  Local, imported path from 4.4.4.4:200:6.6.6.6/32
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:200 OSPF DOMAIN ID:0x0005:0x000000030200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.47.4:0
      mpls labels in/out nolabel/21

R3#sh bgp vpnv4 unicast rd 3.3.3.3:100 6.6.6.6
BGP routing table entry for 3.3.3.3:100:6.6.6.6/32, version 108
Paths: (1 available, best #1, table CUS1)
  Not advertised to any peer
  Local, imported path from 4.4.4.4:100:6.6.6.6/32
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:100 OSPF DOMAIN ID:0x0005:0x000000020200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.46.4:0
      mpls labels in/out nolabel/23

You can see that R3 has 2 vpnv4 routes for 6.6.6.6/32 – 3.3.3.3:200:6.6.6.6/32 and 3.3.3.3:100:6.6.6.6/32. They are unique as one contains :100: and the other contains :200: – Note that R4 does not have to match this RD in any way. It simple needs to be able to accept 2 unique routes. This is especially important when using route reflectors as RR’s will normally only advertise the best route to it’s clients. If they were not unique, the RR would only be advertising one of these routes. The RD in no way determines what VPN a route actually belongs to.

That’s all the route distinguisher does. No more.

Route Target:

The route target’s job is to tell the PE routers what VPN a route actually belongs to. Let’s take a look at the target config on R3:

R3#sh run | inc ip vrf|target
ip vrf CUS1
 route-target export 100:100
 route-target import 100:100
ip vrf CUS2
 route-target export 100:200
 route-target import 100:200

When R3 receives an advertisement from R5, not only does it change the route into a vpnv4 route with the RD to make it unique, it also adds a community value to that advertisement. This is an RT value. Once this NLRI gets to R4, R4 will ensure that only routes that have a certain RT, will be placed in their respective VRF. As an example let’s have a look at the advertisements of 5.5.5.5 from R3 to R4:

R3#sh bgp vpnv4 unicast rd 3.3.3.3:100 5.5.5.5
BGP routing table entry for 3.3.3.3:100:5.5.5.5/32, version 37
Paths: (1 available, best #1, table CUS1)
  Advertised to update-groups:
     9
  Local
    10.0.35.5 from 0.0.0.0 (3.3.3.3)
      Origin incomplete, metric 2, localpref 100, weight 32768, valid, sourced, best
      Extended Community: RT:100:100 OSPF DOMAIN ID:0x0005:0x000000020200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.35.3:0
      mpls labels in/out 24/nolabel

We can see the extended community of 100:100 is encoded into this NLRI on R3. This is advertised to R4:

R4#sh bgp vpnv4 unicast rd 3.3.3.3:100 5.5.5.5
BGP routing table entry for 3.3.3.3:100:5.5.5.5/32, version 178
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    3.3.3.3 (metric 4) from 3.3.3.3 (3.3.3.3)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:100 OSPF DOMAIN ID:0x0005:0x000000020200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.35.3:0
      mpls labels in/out nolabel/24

R4#sh run | include ip vrf | 100:100
ip vrf CUS1
 route-target export 100:100
 route-target import 100:100

R4 has an import 100:100 configuration under it’s VRF, and hence matching the community of 100:100 on the received NLRI, the PE router knows that the advertisement is meant for vrf CUS1. Note that the RD has nothing to do with this.

VPN Label:

The VPN label is to determine what VPN a packet belongs to. But hang on, surely that’s what the RT is for? No. The RT is for the control plane, while the VPN label is for the data plane. Let’s expand on that idea a bit. When R3 advertises NLRI to R4, the RT is used to determine where a route actually belongs. When it comes to R5 actually sending a packet to R6, the VPN label is used. Why? Because when a packet is sent, there is no field in the packet that the route-target is stored. Only the route advertisement contains the route-target as a community value. When R5 sends a ping to R6 from it’s loopback, it’s simply a packet with a destination address of 6.6.6.6 and a source address of 5.5.5.5.

So with L3VPNs we have two labels. The top label is the transport label and the bottom label is the VPN label. PHP will pop the transport label off the second to last router, but the VPN label will only be popped by the actual PE in question. When that frame comes in with the VPN label, R6 knows which VRF that packet belongs to.

VPN labels are advertised in the NLRI along with the RT. Let’s take a look at the 2 VPN labels that R4 is advertising to R3:

R3#sh bgp vpnv4 unicast rd 4.4.4.4:100 6.6.6.6
BGP routing table entry for 4.4.4.4:100:6.6.6.6/32, version 6
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:100 OSPF DOMAIN ID:0x0005:0x000000020200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.46.4:0
      mpls labels in/out nolabel/21
R3#sh bgp vpnv4 unicast rd 4.4.4.4:200 6.6.6.6
BGP routing table entry for 4.4.4.4:200:6.6.6.6/32, version 8
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    4.4.4.4 (metric 4) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:200 OSPF DOMAIN ID:0x0005:0x000000030200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.0.47.4:0
      mpls labels in/out nolabel/23

We can see that R3 will use a VPN label of 21 when sending traffic to the CUS1 VRF, while it’ll use VPN label 23 when sending to CUS2′s VRF.

Let’s run a traceroute from R5 and R8 to confirm this.
CUS1:

R5#traceroute 6.6.6.6

Type escape sequence to abort.
Tracing the route to 6.6.6.6

  1 10.0.35.3 36 msec *  52 msec
  2 10.0.13.1 [MPLS: Labels 20/21 Exp 0] 120 msec 120 msec 132 msec
  3 10.0.12.2 [MPLS: Labels 18/21 Exp 0] 92 msec 148 msec 104 msec
  4 10.0.46.4 [MPLS: Label 21 Exp 0] 104 msec 100 msec 68 msec
  5 10.0.46.6 172 msec *  140 msec

CUS2:

R8#traceroute 6.6.6.6

Type escape sequence to abort.
Tracing the route to 6.6.6.6

  1 10.0.38.3 44 msec 64 msec 40 msec
  2 10.0.13.1 [MPLS: Labels 20/23 Exp 0] 132 msec 132 msec 88 msec
  3 10.0.12.2 [MPLS: Labels 18/23 Exp 0] 124 msec 156 msec 104 msec
  4 10.0.47.4 [MPLS: Label 23 Exp 0] 192 msec 96 msec 76 msec
  5 10.0.47.7 156 msec *  116 msec

The first hop for CUS1 shows a label stack of 20/21 – 20 being transport and 21 being the VPN. CUS2 uses 20/23 – Notice that as the egress PE is the same, the same transport label is used.

On the 3rd hop on both, R2 is popping off the transport label. Both frames now get to R4. One has a VPN label of 21 and the second a label of 23. R4 knows which VRF both packets belong to and sends them on their way to the correct routers.

Hopefully this helps clear this up for some of you..