Carrier Supporting Carrier

CSC, or Carrier Supporting Carrier, takes inter-AS L3 VPN to the next level. Let’s say that you are an ISP and you are offering L3VPN MPLS services to your customers in England. You take over another ISP located in say, Australia, and two of your UK customers are also located in Australia. They would like their offices in both locations connected over the MPLS cloud.

It would be very expensive to run a new line between those locations. You do, however, still want to provide L3VPN services to your customers. The is where CSC comes in. CSC allows another ISP to connect both sides of your ISP network together. It also ensures the the core ISP doesn’t learn about any of your customer prefixes, as it doesn’t need to.

Let’s take the following diagram into consideration. (click the image for full size)

Routers 1, 7, 14, 6, 9, and 8 are all part of ISP2. There are three routers located in each geographical location. Routers 2, 3, 15, and 5 are part of the core carrier. This network stretches to both locations. The rest are customer routers injecting their loopback addresses into OSPF to test.

The core carrier is running IS-IS and LDP. ISP100 is running OSPF and LDP. I won’t go into the regular IGP+LDP config as is pretty straightforward.

With CSC, there are a few new terms to deal with. R14 and R8 are going to be regular PE routers for our ISP. R1 and R6 are going to be called CSC-CE (Customer Supporting Carrier – Customer Edge) routers. R2 and R5 are going to be CSC-PE (Customer Supporting Carrier – Provider Edge) routers. All other ISP routers are simply core routers.

The terminology above assumes that you are speaking in regards to being the core carrier (ISP 500 in this case). That is, ISP500’s edge routers are ‘PE’ and R1 and R6 is the customer’s (ISP) PE routers (Called CE in this case)

It’s a little confusing, depending on which view you take, but it’s really not that difficult.

Initial R14/R8 config – regular PE

The first thing we can start off with is to ensure R14’s PE config is correct. I am running OSPF with the CE routers and learning routes from them:

vrf definition CUS1
 rd 14.14.14.14:1
 route-target export 100:1
 route-target import 100:1
 !
 address-family ipv4
 exit-address-family
!
vrf definition CUS2
 rd 14.14.14.14:2
 route-target export 100:2
 route-target import 100:2
 !
 address-family ipv4
 exit-address-family
!
interface FastEthernet1/1
 vrf forwarding CUS1
 ip address 10.14.16.14 255.255.255.0
 ip ospf network point-to-point
 ip ospf 3 area 0
!
interface FastEthernet2/0
 vrf forwarding CUS2
 ip address 10.14.15.14 255.255.255.0
 ip ospf network point-to-point
 ip ospf 2 area 0

R8 has a similar config on the other side so I won’t put it here.

CSC-PE config

R2 and R5 are going to act as PE routers for the core carrier. They will have their AS200 facing interfaces in a VRF. R2 and R5 will be running regular VPNv4 BGP with each other.

R2:

vrf definition CSC_AS100
 rd 2.2.2.2:500
 route-target export 500:100
 route-target import 500:100
 !
 address-family ipv4
 exit-address-family
!
interface FastEthernet1/0
 vrf forwarding CSC_AS100
 ip address 10.1.2.2 255.255.255.0
!
router bgp 500
 no bgp default ipv4-unicast
 neighbor 5.5.5.5 remote-as 500
 neighbor 5.5.5.5 update-source Loopback0
 !
 address-family vpnv4
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 send-community extended
 exit-address-family

Once again, R5 has a similar config on the other side so I’m not putting it here.

CSC-CE

Eventually we need R14 and R8 to peer with each other via VPNv4.
This is much like option C in the inter-AS config in which two PE routers peer with each other even though they are not directly connected in the same AS. In order to do so we need each of them to have a valid route to each other. As R1 and R6 each have routes to their local PE device, they need to learn routes from the other side through the core carrier. To do this I’ll be running BGP to the core carrier. I also need to ensure that I’m sending and receiving labeled BGP routes and the end LSP has to be end to end. No part of the path can be unlabelled. I need to advertise the PE’s (R14 and R8) loopback are advertised over to the core carrier:

interface FastEthernet1/0
 ip address 10.1.2.1 255.255.255.0
 mpls bgp forwarding
!
router bgp 100
 bgp log-neighbor-changes
 neighbor 10.1.2.2 remote-as 500
 !
 address-family ipv4
  network 14.14.14.14 mask 255.255.255.255
  neighbor 10.1.2.2 activate
  neighbor 10.1.2.2 allowas-in 1
  neighbor 10.1.2.2 send-label
  no auto-summary
 exit-address-family

I’ll need to allowas-in 1 as both sides are running the same AS number. Without it, the CSC-CE routers would reject the BGP update.

I then need to ensure the CSC-CE routers are redistributing those learned prefixes into the IGP:

router ospf 1
 redistribute bgp 100 subnets

R6 again has a similar config.

The end result of it so far is that R8 and R14 should now be able to ping each other from their respective loopbacks:

R8#ping 14.14.14.14 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 14.14.14.14, timeout is 2 seconds:
Packet sent with a source address of 8.8.8.8
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/45/56 ms
R8#traceroute 14.14.14.14 so lo0

Type escape sequence to abort.
Tracing the route to 14.14.14.14

  1 10.8.9.9 [MPLS: Label 24 Exp 0] 56 msec 44 msec 32 msec
  2 10.6.9.6 [MPLS: Label 24 Exp 0] 60 msec 24 msec 60 msec
  3 10.5.6.5 [MPLS: Label 16 Exp 0] 44 msec 44 msec 48 msec
  4 10.5.13.13 [MPLS: Labels 20/34 Exp 0] 48 msec 44 msec 44 msec
  5 10.3.13.3 [MPLS: Labels 19/34 Exp 0] 44 msec 44 msec 40 msec
  6 10.1.2.2 [MPLS: Label 34 Exp 0] 40 msec 44 msec 44 msec
  7 10.1.2.1 [MPLS: Label 19 Exp 0] 44 msec 44 msec 28 msec
  8 10.1.7.7 [MPLS: Label 18 Exp 0] 56 msec 32 msec 12 msec
  9 10.7.14.14 52 msec *  72 msec

Which they can.

PE config – continued

Now that the PE routers have connectivity to each other, we can set up the VPNv4 BGP session:

router bgp 100
 no bgp default ipv4-unicast
 neighbor 8.8.8.8 remote-as 100
 neighbor 8.8.8.8 update-source Loopback0
 !
 address-family vpnv4
  neighbor 8.8.8.8 activate
  neighbor 8.8.8.8 send-community extended
 exit-address-family

Is the session up?

R14#show bgp vpnv4 unicast all summary
BGP router identifier 14.14.14.14, local AS number 100
BGP table version is 1, main routing table version 1

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
8.8.8.8         4          100       4       4        1    0    0 00:00:50        0

Yes, but no prefixes learnt. We still need to redistribute between our vrf aware OSPF processes and BGP:

router ospf 3 vrf CUS1
 redistribute bgp 100 subnets
!
router ospf 2 vrf CUS2
 redistribute bgp 100 subnets
!
router bgp 100
 !
address-family ipv4 vrf CUS1
  redistribute ospf 3 vrf CUS1
 exit-address-family
!
address-family ipv4 vrf CUS2
  redistribute ospf 2 vrf CUS2
 exit-address-family

Verification

As always with these types of configs, we need to ensure both the control and data planes are working correctly. First let’s see the control plane update of R16’s loopback over to R11. The PE router of R14 should be learning this as a vrf prefix:

R14#show ip route vrf CUS1 16.16.16.16

Routing Table: CUS1
Routing entry for 16.16.16.16/32
  Known via "ospf 3", distance 110, metric 2, type intra area
  Redistributing via bgp 100
  Advertised by bgp 100
  Last update from 10.14.16.16 on FastEthernet1/1, 00:18:35 ago
  Routing Descriptor Blocks:
  * 10.14.16.16, from 16.16.16.16, 00:18:35 ago, via FastEthernet1/1
      Route metric is 2, traffic share count is 1

This prefix is converted into a VPNv4 prefix and advertised over to R8:

R8#show bgp vpnv4 un rd 14.14.14.14:1 16.16.16.16
BGP routing table entry for 14.14.14.14:1:16.16.16.16/32, version 3
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    14.14.14.14 (metric 1) from 14.14.14.14 (14.14.14.14)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:1 OSPF DOMAIN ID:0x0005:0x000000030200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.14.16.14:0
      mpls labels in/out nolabel/27

This should end up in the correct vrf table:

R8#show ip route vrf CUS1 16.16.16.16

Routing Table: CUS1
Routing entry for 16.16.16.16/32
  Known via "bgp 100", distance 200, metric 2, type internal
  Redistributing via ospf 2
  Advertised by ospf 2 subnets
  Last update from 14.14.14.14 00:07:03 ago
  Routing Descriptor Blocks:
  * 14.14.14.14 (default), from 14.14.14.14, 00:07:03 ago
      Route metric is 2, traffic share count is 1
      AS Hops 0
      MPLS label: 27
      MPLS Flags: MPLS Required

Finally R11 should receive that as on OSPF route:

R11#show ip route 16.16.16.16
Routing entry for 16.16.16.16/32
  Known via "ospf 1", distance 110, metric 2
  Tag Complete, Path Length == 1, AS 100, , type extern 2, forward metric 1
  Last update from 10.8.11.8 on FastEthernet1/0, 00:06:38 ago
  Routing Descriptor Blocks:
  * 10.8.11.8, from 10.8.11.8, 00:06:38 ago, via FastEthernet1/0
      Route metric is 2, traffic share count is 1
      Route tag 3489661028

So our control plane is all good so far. Let’s check our data plane forwarding:

R11#ping 16.16.16.16 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 16.16.16.16, timeout is 2 seconds:
Packet sent with a source address of 11.11.11.11
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/57/76 ms


R11#traceroute 16.16.16.16 so lo0

Type escape sequence to abort.
Tracing the route to 16.16.16.16

  1 10.8.11.8 8 msec 12 msec 12 msec
  2 10.8.9.9 [MPLS: Labels 24/27 Exp 0] 52 msec 56 msec 48 msec
  3 10.6.9.6 [MPLS: Labels 24/27 Exp 0] 56 msec 48 msec 48 msec
  4 10.5.6.5 [MPLS: Labels 16/27 Exp 0] 56 msec 40 msec 68 msec
  5 10.5.13.13 [MPLS: Labels 20/34/27 Exp 0] 52 msec 44 msec 52 msec
  6 10.3.13.3 [MPLS: Labels 19/34/27 Exp 0] 56 msec 48 msec 40 msec
  7 10.2.3.2 [MPLS: Labels 34/27 Exp 0] 52 msec 52 msec 40 msec
  8 10.1.2.1 [MPLS: Labels 19/27 Exp 0] 44 msec 44 msec 48 msec
  9 10.1.7.7 [MPLS: Labels 18/27 Exp 0] 52 msec 40 msec 60 msec
 10 10.14.16.14 [MPLS: Label 27 Exp 0] 40 msec 44 msec 36 msec
 11 10.14.16.16 44 msec *  52 msec

No problems there :)

Carrier Supporing Carrier Supporting Carrier

So let’s be silly and go a step further. What if our final customer is actually another ISP offering L3VPN to it’s customers? Let’s change our topology slightly(click the image for full size)

I’m not going to show all the config here as it’s simply too much to fit into a blog post. However the config itself is pretty much like so. You’ll need to click the image for the larger version:

R10 and R15 are our final CE routers. Once all is configured does it all actually work?

R10#traceroute 15.15.15.15 so lo0

Type escape sequence to abort.
Tracing the route to 15.15.15.15

  1 10.10.11.11 12 msec 12 msec 8 msec
  2 10.8.11.8 [MPLS: Labels 16/17 Exp 0] 72 msec 48 msec 68 msec
  3 10.8.9.9 [MPLS: Labels 22/24/17 Exp 0] 36 msec 80 msec 44 msec
  4 10.6.9.6 [MPLS: Labels 20/24/17 Exp 0] 52 msec 56 msec 52 msec
  5 10.5.6.5 [MPLS: Labels 23/24/17 Exp 0] 64 msec 52 msec 52 msec
  6 10.5.13.13 [MPLS: Labels 20/23/24/17 Exp 0] 48 msec 60 msec 52 msec
  7 10.3.13.3 [MPLS: Labels 19/23/24/17 Exp 0] 56 msec 56 msec 48 msec
  8 10.2.3.2 [MPLS: Labels 23/24/17 Exp 0] 56 msec 44 msec 56 msec
  9 10.1.2.1 [MPLS: Labels 19/24/17 Exp 0] 48 msec 48 msec 64 msec
 10 10.1.7.7 [MPLS: Labels 16/24/17 Exp 0] 56 msec 52 msec 60 msec
 11 10.7.14.14 [MPLS: Labels 24/17 Exp 0] 44 msec 56 msec 56 msec
 12 10.15.16.16 [MPLS: Label 17 Exp 0] 56 msec 56 msec 44 msec
 13 10.15.16.15 56 msec *  76 msec

It does indeed. At this point we are up to a four label stack in AS500. If we were running RSVP-TE and FRR we would have even more labels sitting on top.

There is a much easier way to do this of course. The original Customer Carrier could just buy some some or virtual leased line or VPLS from AS500 and they would be directly connected over the same subnet. They could then run MPLS over that link and as far as anyone cares R1 and R6 would be directly connected to each other.

But of course this is a topic on the CCIE SP after all…

© 2009-2019 Darren O'Connor All Rights Reserved -- Copyright notice by Blog Copyright