Tag Archives: mpls

When a vlan is not a vlan

What is a vlan? What is a vlan-id? Are they the same thing?

Generally yes, but in the ISP world a vlan-id can also be a circuit identifier. While your view of a vlan might be a single broadcast domain, you’ll soon see that multiple vlan IDs can share the same single broadcast domain, or the same vlan-id could be in a completely different broadcast domain.

The Problem

I’ve written about this before. Carriers, at least in the UK, are offering more and more aggregated links to Service Providers. Each circuit to customer sites is aggregated over a single high-bandwidth link to your PE router. This cuts down on ports, cables, and man hours to plug them in.

Old way:

carrier old When a vlan is not a vlan

New way:

carrier new When a vlan is not a vlan
How are the p2p circuits aggregated over the core high-bandwidth link? Each p2p link is separated by a vlan tag on the PoP side. So we could say that any packet coming out of the core PE with vlan 2000 goes to site 1, while packets with vlan 3000 go to site 2. What happens if site 1 and site 2 are going to the same customer? What if you are providing a VPLS service to them? It’s essential to note that the vlan tag imposed by the carrier is used simply to determine what packet goes to which circuit. As we control the MPLS core, it’s ultimately up to us to decide which packet belongs in which broadcast domain, and that is regardless of the vlan id used by the carrier.

Relevant Initial Core Config

I’ll use the following topology:
vlans core When a vlan is not a vlan

R1, R2, and R3 are the core of the network. R1 is a Brocade Netiron running MPLS. R2 is a Cisco me3600x running MPLS. R2 is an me3600x running bridge-groups with no MPLS.

CE1, CE2, and CE3 are all customer routers.

R1 – Brocade XMR

interface ethernet 2/4
 port-name TO-R2
 enable
 route-only
 ip ospf area 0
 ip ospf network point-to-point
 ip address 10.10.10.10/24
!
router mpls
 policy
  traffic-eng ospf area 0

  mpls-interface e2/4

 lsp R1-R2
  to 192.168.224.4
  adaptive
  enable

R2 – Cisco me3600x running MPLS

mpls traffic-eng tunnels
!
router ospf 1
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng area 0
!
interface GigabitEthernet0/1
 description TO-R1
 no switchport
 ip address 10.10.10.11 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls traffic-eng tunnels
!
interface Tunnel0
 ip unnumbered Loopback0
 tunnel mode mpls traffic-eng
 tunnel destination 192.168.224.61
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng path-option 5 dynamic
 tunnel mpls traffic-eng record-route

There is no IP and MPLS configuration on R3 as it’s not running MPLS. I’ll show how the bridge-group is configured when I get to that part.

CPE Config

I’ll be using vlan 3000 to get to CE1, vlan 2000 to get to CE2, and double-tag vlan 3500,2500 to get to CE3. Each CE has their WAN interface in the same subnet as each other running OSPF. I’ll also enable OSPF on their loopbacks and WAN links.

CE1

This is a Juniper EX3200:

root@CE1> show configuration interfaces ge-0/0/0
vlan-tagging;
unit 3000 {
    vlan-id 3000;
    family inet {
        address 1.1.1.1/24;
    }
}

root@CE1> show configuration interfaces lo0.0
family inet {
    address 10.10.10.10/32;
}

root@CE1> show configuration protocols ospf
area 0.0.0.0 {
    interface ge-0/0/0.3000;
    interface lo0.0;
}

CE2

This is a Cisco 3750G:

interface Loopback0
 ip address 20.20.20.20 255.255.255.255
 ip ospf 1 area 0
!
interface Vlan2000
 ip address 1.1.1.2 255.255.255.0
 ip ospf 1 area 0
!
interface GigabitEthernet1/0/1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2000
 switchport mode trunk

CE3

This is a Cisco 1841:

interface Loopback0
 ip address 30.30.30.30 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0.32
 encapsulation dot1Q 3500 second-dot1q 2500
 ip address 1.1.1.3 255.255.255.0
 ip ospf 1 area 0

VPLS Config

As you can see, each CPE will be using a different vlan tag. One site is even sending a double-tagged frame. They all need to be in the same broadcast domain. No problem as we are simply going to use the vlan tag to determine the service.

R2

Gi0/2 will create a LDP-signalled VPLS VC to R1 (aka manual set up). Interface gi0/2 vlan 2000 will be part of VPLS id 501:

ethernet evc TEST-EVC
 uni count 20
!
l2vpn vfi context TEST-VPLS
 vpn id 501
 member 192.168.224.61 encapsulation mpls
!
interface GigabitEthernet0/2
 switchport trunk allowed vlan none
 switchport mode trunk
 mtu 9800
 service instance 1 ethernet TEST-EVC
  encapsulation dot1q 2000
  rewrite ingress tag pop 1 symmetric
  bridge-domain 501
 !
interface Vlan501
 no ip address
 member vfi TEST-VPLS

What’s important to note here is that the me3600x still uses bridge-groups for VPLS, but it’s not exactly the same as just using bridge-groups by itself. You’ll see this soon enough when we configure R3.

R1

R1 will create a VPLS to R2. Vlan 3000 on interface 2/5 will be part of the same VPLS:

router mpls
 vpls TEST-VPLS 501
  vpls-peer 192.168.224.4
  vpls-mtu 1500
  vlan 3000
   tagged ethe 2/5

At this point R1 and R2 have the VPLS set up between them. Each CE is using different vlans on their WAN, but they are in fact on the same broadcast domain:

CE2#sh ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1         128   FULL/DR         00:00:39    1.1.1.1         Vlan2000

CE2#ping 1.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/17 ms

CE2#ping 10.10.10.10 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
Packet sent with a source address of 20.20.20.20
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms

The vlan-id used on the CPE, was merely used to push the frame into the correct VPLS. The VPLS itself is the broadcast domain, the vlan tag is irrelevant as its stripped on inbound into the PE router. You CAN however, ensure that the PE router does NOT strip the vlan tag. This has interesting use cases when you purposely want to separate on vlan id with in the VPLS. I wrote more on this over here so please give it a read. Both the Brocade and Cisco default to VC mode 5 when setting up a VPLS.

Bridge Group Config

I’m going to set up R3 so that it only uses bridge-groups. No routing or MPLS involved. Bridge-Groups work very similar to VPLS, though it’s on a single box. Traffic can be pushed from a bridge-group into a VPLS if needed. The bridge-group determines the broadcast domain. I can have multiple different vlans in the same bridge group.

For R3, gi0/2 is the interface pointing towards the core, while gi0/1 is pointing towards the customer. I’ll use different vlan ids on each, but they will be in the same bridge-group:

ethernet evc TEST
!
vlan 501
 name TEST-CE
!
interface GigabitEthernet0/1
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 1 ethernet TEST
  encapsulation dot1q 501
  rewrite ingress tag pop 1 symmetric
  bridge-domain 501
 !
interface GigabitEthernet0/2
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 1 ethernet TEST
  encapsulation dot1q 3500 second-dot1q 2500
  rewrite ingress tag pop 2 symmetric
  bridge-domain 501

I’m not going into detail, but I will cover the basics. When gi0/2 receives a double-tagged frame that matches 3500,2500 inbound, the me3600x will pop both tags off and the resulting frame will be part of bridge-group 501. Symmetric means that when a frame leaves gi0/2, it will re-add vlans 3500,2500 on top of the frame. As gi0/1 is also in bridge-group 501, the customer frame will be forwarded out that port, and it will have a single vlan tag of 501 popped on top.

At this point gi0/1 is connected to R1 eth2/3. For this customer I would be expecting a single tag of 501 coming inbound, and so I’ll place that vlan id into the VPLS from above:

 vpls TEST-VPLS 501
  vlan 501
   tagged ethe 2/3

Now all three CE routers should be fully adjacent:

CE3#sh ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1         128   FULL/DR         00:00:35    1.1.1.1         FastEthernet0/0.32
1.1.1.2           1   FULL/DROTHER    00:00:37    1.1.1.2         FastEthernet0/0.32

CE3#ping 10.10.10.10 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
Packet sent with a source address of 30.30.30.30
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
CE3#ping 20.20.20.20 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 20.20.20.20, timeout is 2 seconds:
Packet sent with a source address of 30.30.30.30
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/12 ms

Conclusions:

vlan tags have multiple uses. In most networks it informs the switches which vlan, and therefore broadcast domain, a frame is part of. They can also be circuit identifiers showing which VPLS/Circuit the frame belongs to. They can also be both at the same time, depending on the VPLS VC type you’re using.

For the above network it’s extremely simplified. Care must be taken when forwarding certain layer2 control frames. Most are sent untagged out tagged interfaces. Cisco’s RSTP+ and STP tag each vlan BPDU with a the same vlan-id. If you’re using vlan 2000 on one side and vlan 3000 on the other, and the BPDU gets through, one side will shut down their WAN link due to receiving a BPDU with a vlan tag that doesn’t match the BPDU data inside the frame.

My book – MPLS for Enterprise Engineers is now available from multiple channels

I put together a beginners MPLS book for Juniper. I’ve noticed, when interviewing candidates, that often they have a good knowledge of routing protocols, but lack in MPLS. This is to be expected unless they’ve worked at an ISP. The book is targeted towards those users.

J-net

https://www.juniper.net/us/en/community/junos/training-certification/day-one/networking-technologies-series/mpls-enterprise-engineers/

Amazon

http://www.amazon.com/dp/B00IU1KCJ0
http://www.amazon.co.uk/dp/B00IU1KCJ0

iTunes

https://itunes.apple.com/us/book/day-one-mpls-for-enterprise/id836201741?mt=11

Vervante (Print version)

http://store.vervante.com/c/v/V4081705490.html

L2VPN on Junos using CCC/Martini/Kompella

There are three main ways to provide a point to point L2 link over MPLS on Junos. Below I’ll give a brief description and show how to configure all three.

For the below descriptions I’ll be using this simple topology. All these devices are running as logical-systems on a single MX5.
L2vpn L2VPN on Junos using CCC/Martini/Kompella

CE Configs

My two CE devices will be configured the same for all three types below:
CE T1:

root@MX5:T1> show configuration interfaces ge-1/1/0
unit 0 {
    family inet {
        address 2.2.2.1/24;
    }
}

CE BB:

root@MX5:BB> show configuration interfaces ge-1/1/3
unit 0 {
    family inet {
        address 2.2.2.2/24;
    }
}

CCC

Circuit Cross-Connect is a legacy type of L2 point to point link found in Junos. CCC over MPLS requires a unique RSVP LSP per circuit per direction. It needs a unique LSP as there is no VC (inner) label. Any frames going over an LSP belongs to a specific circuit.

L2vpnCCC L2VPN on Junos using CCC/Martini/Kompella

Customers frames in either direction is encapsulated into another L2 frame with an RSVP label. This label determines which circuit the frame belongs to on the other side via the configuration.
CCC Header L2VPN on Junos using CCC/Martini/Kompella

The core network has already been set up with OSPF/RSVP/MPLS so I won’t go over that. I’ll concentrate on the PE boxes themselves. I need to create a regular RSVP LSP each way. The CCC config sits under the protocols connections stanza
R3

interfaces {
    ge-1/1/1 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R6 {
            to 6.6.6.6;
            no-cspf;
        }
        interface all;
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
    connections {
        remote-interface-switch R6 {
            interface ge-1/1/1.0;
            transmit-lsp TO-R6;
            receive-lsp TO-R3;
        }
    }
}

R6 has a similar config back to R3. To confirm on the PE we can view via show connections:

root@MX5:R3> show connections | find R6
R6                                rmt-if      Up      Aug 26 16:41:00           2
  ge-1/1/1.0                        intf  Up
  TO-R6                             tlsp  Up
  TO-R3                             rlsp  Up

Finally we can test from the CE device:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.464/0.490/1.041/0.081 ms

Martini

Martini allows you to use the same LSP for multiple circuits. The circuits can be multiplexed as there is a second MPLS label in the stack. The bottom label is the VC label and it tells the PE what L2VPN the frame belongs to. Martini uses LDP to signal the VC label. Note that transport labels can use either LDP or RSVP. If using RSVP, you need to enable LDP on the loopback interfaces of both PE routers. Of course you still need at least two LSPs are LSPs are unidirectional.

The frame header will now contain an RSVP/LDP transport label as well as an LDP-signalled VC label. This allows the PEs to multiplex multiple circuits over the same LSP.
Martini Header L2VPN on Junos using CCC/Martini/Kompella

Martini is also configured under the protocols stanza.
This is R6′s config:

interfaces {
    ge-1/1/2 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R3 {
            to 3.3.3.3;
            no-cspf;
        }
        interface all;
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
    ldp {
        interface lo0.6;
    }
    l2circuit {
        neighbor 3.3.3.3 {
            interface ge-1/1/2.0 {
                virtual-circuit-id 1;
            }
        }
    }
}

R3 and R6 should have a targeted LDP session and the circuit should be up:

root@MX5:R6> show ldp neighbor
Address            Interface          Label space ID         Hold time
3.3.3.3            lo0.6              3.3.3.3:0                39

root@MX5:R6> show l2circuit connections | find Neigh
Neighbor: 3.3.3.3
    Interface                 Type  St     Time last up          # Up trans
    ge-1/1/2.0(vc 1)          rmt   Up     Aug 26 17:28:31 2013           1
      Remote PE: 3.3.3.3, Negotiated control-word: Yes (Null)
      Incoming label: 299840, Outgoing label: 299904
      Negotiated PW status TLV: No
      Local interface: ge-1/1/2.0, Status: Up, Encapsulation: ETHERNET

All looks good. Check from CE to CE again:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.468/0.498/0.720/0.074 ms

Kompella

Kompella also uses a two label stack as Martini. The VC label is signaled via BGP. Once again your transport underneath can be either RSVP or LDP. Kompella has a lot more configuration than the above two, however as it uses BGP it means you can just add the l2vpn address family to your existing BGP deployment.

Kompella Header L2VPN on Junos using CCC/Martini/Kompella

I’ll show R3′s config here:

interfaces {
    ge-1/1/1 {
        encapsulation ethernet-ccc;
        unit 0 {
            family ccc;
        }
    }
}
protocols {
    rsvp {
        interface all;
    }
    mpls {
        label-switched-path TO-R6 {
            to 6.6.6.6;
            no-cspf;
        }
        interface all;
    }
    bgp {
        group iBGP {
            local-address 3.3.3.3;
            family l2vpn {
                signaling;
            }
            peer-as 100;
            neighbor 6.6.6.6;
        }
    }
    ospf {
        area 0.0.0.0 {
            interface all;
        }
    }
}
routing-instances {
    CUS1 {
        instance-type l2vpn;
        interface ge-1/1/1.0;
        route-distinguisher 100:100;
        vrf-target target:100:1;
        protocols {
            l2vpn {
                encapsulation-type ethernet;
                interface ge-1/1/1.0;
                site T1 {
                    site-identifier 1;
                    interface ge-1/1/1.0;
                }
            }
        }
    }
}
routing-options {
    autonomous-system 100;
}

Kompella uses a routing-instance exactly like a regular L3VPN in Junos. This includes the RD and RT. We can check the BGP session:

root@MX5:R3> show bgp summary
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l2vpn.0
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
6.6.6.6                 100         35         35       0       0       13:52 Establ
  bgp.l2vpn.0: 1/1/1/0
  CUS1.l2vpn.0: 1/1/1/0

Final confirmation is to ping from CE to CE:

root@MX5:T1> ping 2.2.2.2 rapid count 10
PING 2.2.2.2 (2.2.2.2): 56 data bytes
!!!!!!!!!!
--- 2.2.2.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.469/0.497/0.714/0.072 ms

Conclusion

  • CCC cannot multiplex multiple circuits over the same LSP, hence once you start ramping up your circuits it’s not at all scalable.
  • CCC however is supported on Juniper’s EX3200 range which gives you a cheap option to offer L2VPN over MPLS
  • Martini and Kompella both multiplex over the same LSP by signalling VC labels
  • Martini is simple to configure and makes sense if you’re a small ISP offering a couple of links here and there, especially if you’re already running LDP
  • Kompella is a lot more complicated to configure, however it runs off the back of BGP. This means that you can use your existing BGP topology to run L3VPN/L2VPN/MGVPN/etc with a single signalling protocol.
  • BGP also scales incredibly well so its the best option for larger deployments.

ME3600X Bridge Group into a Brocade VPLS for H-QoS

The Cisco ME3600X and ME3800X series support both VPLS and H-VPLS. MPLS features require an expensive license. If you are currenlty joining a few leased lines together into a VPLS you don’t always need the more expensive MPLS license.

Any carrier circuit terminating on the Cisco ME can be placed into a bridge-group based on the dot1q tag. This can then be passed along or re-written to a new vlan tag into a Brocade XMR port which stick that frame into a VPLS. Why not just terminate the carrier circuit directly onto the XMR? I’ve recently had to do this because the XMR line cards we have do not support H-QoS. More and more carriers are aggregating many B end circuits into single high-bandwidth A end gig links. I need to be able to shape traffic outbound on a per-vlan basis, and within that queue give priority to certain queues. The ME3600X can do H-QoS and its exactly the reason I’m using it.

Let’s show a quick diagram so we know what we are talking about. Click the image for the full image:
Carrier Ports 2 300x195 ME3600X Bridge Group into a Brocade VPLS for H QoS

On the right I have two SRX210s and an 1841 sending tagged traffic into the carrier network. That carrier network is multiplexing all three circuits into a single link on the A end. That goes into the ME3600X. From the ME3600X it goes off to a Brocade XMR. That Brocade is connected to another Brocade over the MPLS core and finally to another 1841 on the other side.

On the ME3600X I could easily span each vlan over to the XMR and have the XMR VPLS them back. The issue with that is that if hosts behind SRX is sending a ton of traffic to a host at SRX2, why waste bandwidth hairpinning traffic over to the XMR when it could be done at the ME3600X?

In other words, instead of doing this:
VPLS PIN1 ME3600X Bridge Group into a Brocade VPLS for H QoS

We do something like this:
ME3400 PIN ME3600X Bridge Group into a Brocade VPLS for H QoS

Now you may be asking, why not just get this carrier to stick them in the VPLS? That could work if all these links were coming from the same carrier, but often they aren’t.

ME3600X EVC config

ethernet evc TESTLAB
!
interface GigabitEthernet0/1
 description Link to Carrier
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 1 ethernet TESTLAB
  description SRX1
  encapsulation dot1q 2000
  rewrite ingress tag pop 1 symmetric
  bridge-domain 150
 !
 service instance 2 ethernet TESTLAB
  description SRX2
  encapsulation dot1q 2001
  rewrite ingress tag pop 1 symmetric
  bridge-domain 150
 !
 service instance 3 ethernet TESTLAB
  description 1841
  encapsulation dot1q 2002 second-dot1q 100
  rewrite ingress tag pop 2 symmetric
  bridge-domain 150
 !
interface GigabitEthernet0/24
 description Link to XMR
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 150 ethernet
  description VPLS Core
  encapsulation dot1q 150
  rewrite ingress tag pop 1 symmetric
  bridge-domain 150
 !
end

I have three service instances configured on gi0/1 – each matching the vlan id that the carrier is using for transport. Notice that I can match on both an outer and inner tag that I’m originating from the 1841. Gi0/24 is the port connected to the XMR and that’s in the same bridge-group with a vlan tag of 150.

On the XMR side I’m simply matching on vlan 150 and placing those frames in the VPLS.

Brocade Config

vpls DARREN-TESTING 3200
  vpls-peer 172.10.10.1
  vlan 150
   tagged ethe 2/20

Connectivity verification

All my CPEs are running OSPF on their WAN links. I’ve also hard-coded their MAC addresses so it’ll be easy to see in this post.

darreno@JR1> show ospf neighbor
Address          Interface              State     ID               Pri  Dead
10.0.0.4         ge-0/0/1.2000          Full      4.4.4.4            1    39
10.0.0.3         ge-0/0/1.2000          2Way      3.3.3.3            1    34
10.0.0.2         ge-0/0/1.2000          Full      2.2.2.2          128    32

SRX1 has three neighbours. Fully adjacent with the DR and BDR. This means I get get to the remote VPLS 1841. Let’s take a look at the mac address table for the VPLS on the PE connected to the ME3600x:

SSH@pe2#sh mac vpls 3200

Total MAC entries for VPLS 3200: 5 (Local: 3, Remote: 2)

VPLS       MAC Address    L/R Port  Vlan(In-Tag)/Peer ISID      Age
====       ===========    === ====  ================= ====      ===
3200       0000.1111.0000 L   2/20  150               NA        0
3200       0000.2222.0000 L   2/20  150               NA        0
3200       0000.3333.0000 L   2/20  150               NA        0
3200       0000.4444.0000 R   1/10  172.10.10.1       NA        0

The MAC’s for SRX1, SRX2, and the first 1841 are all via tag 150 out interface 2/20. The remote 1841′s MAC is learned via the remote PE router.

We should see all four MACs on thee ME3600X out their respective tagged ports:

ME3600X#show mac address-table bridge-domain 150 | begin DYNAMIC
 150    0000.1111.0000    DYNAMIC     Gi0/1+Efp1
 150    0000.2222.0000    DYNAMIC     Gi0/1+Efp2
 150    0000.3333.0000    DYNAMIC     Gi0/1+Efp3
 150    0000.4444.0000    DYNAMIC     Gi0/24+Efp150

The ME3600X is also telling us which service instance under the physical port it’s learning the MAC address from.

QoS Config

Let’s assume the the circuit tagged with vlan 2000 has only got 5Mb, lan 2001 has 10Mb, and vlan 2001 has 15Mb. I want to shape each EVC to their respective speed, and then give priority to DSCP EF packets. I also want to police that queue to 50% to ensure that priority queue cannot hog the link.

class-map match-all EF
 match dscp ef
!
policy-map QoS
 class EF
  priority
  police cir percent 50
   conform-action transmit
   exceed-action drop
 class class-default
  queue-limit percent 100
!
policy-map VLAN2000
 class class-default
  shape average 5000000
   service-policy QoS
policy-map VLAN2001
 class class-default
  shape average 10000000
   service-policy QoS
policy-map VLAN2002
 class class-default
  shape average 15000000
   service-policy QoS

Each policy is then attached to the service instance itself. I’ll use service instance 1 as an example here:

interface GigabitEthernet0/1
 service instance 1 ethernet TESTLAB
  description SRX1
  encapsulation dot1q 2000
  rewrite ingress tag pop 1 symmetric
  service-policy output VLAN2000
  bridge-domain 150

Each service instance can have an individual policy. So we have broken up the physical port into many virtual circuits. Each VC has their own shaper and priority queue.

QoS Verification

If you apply the policy as above, you can’t use show policy-map interface anymore. Instead you need to use show ethernet service instance policy-map

ME3600X#show ethernet service instance policy-map
  GigabitEthernet0/1: EFP 1

  Service-policy output: VLAN2000

    Class-map: class-default (match-any)
      8 packets, 644 bytes
      5 minute offered rate 0000 bps, drop rate 0000 bps
      Match: any
  Traffic Shaping
    Average Rate Traffic Shaping
    Shape 5000 (kbps)
      Output Queue:
        Default Queue-limit 49152 bytes
        Tail Packets Drop: 0
        Tail Bytes Drop: 0

      Service-policy : QoS

        Class-map: EF (match-all)
          0 packets, 0 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match:  dscp ef (46)
          Strict Priority
          police:
            cir percent 50 % bc 250 ms
            cir 2500000 bps, bc 78000 bytes
            conform-action transmit
            exceed-action drop
          conform: 0 (packets) 0 (bytes)
          exceed: 0 (packets) 0 (bytes)
          conform: 0 bps, exceed: 0 bps
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

        Class-map: class-default (match-any)
          8 packets, 644 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match: any
          Queue-limit 100 percent
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

OSPF packets are going through and that’s being matched by class-default. We can force some EF traffic by pinging with a TOS value:

darreno@JR2> ping 10.0.0.1 rapid tos 184
PING 10.0.0.1 (10.0.0.1): 56 data bytes
!!!!!
--- 10.0.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 3.766/4.635/7.131/1.269 ms
ME3600X#show ethernet service instance policy-map
  GigabitEthernet0/1: EFP 1

  Service-policy output: VLAN2000

    Class-map: class-default (match-any)
      328 packets, 26028 bytes
      5 minute offered rate 2000 bps, drop rate 0000 bps
      Match: any
  Traffic Shaping
    Average Rate Traffic Shaping
    Shape 5000 (kbps)
      Output Queue:
        Default Queue-limit 49152 bytes
        Tail Packets Drop: 0
        Tail Bytes Drop: 0

      Service-policy : QoS

        Class-map: EF (match-all)
          5 packets, 530 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match:  dscp ef (46)
          Strict Priority
          police:
            cir percent 50 % bc 250 ms
            cir 2500000 bps, bc 78000 bytes
            conform-action transmit
            exceed-action drop
          conform: 5 (packets) 510 (bytes)
          exceed: 0 (packets) 0 (bytes)
          conform: 0 bps, exceed: 0 bps
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

        Class-map: class-default (match-any)
          323 packets, 25498 bytes
          5 minute offered rate 2000 bps, drop rate 0000 bps
          Match: any
          Queue-limit 100 percent
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

There we see the five EF packets.

So there you have it. Not too difficult at all to get the basics working.

Carrier Supporting Carrier

CSC, or Carrier Supporting Carrier, takes inter-AS L3 VPN to the next level. Let’s say that you are an ISP and you are offering L3VPN MPLS services to your customers in England. You take over another ISP located in say, Australia, and two of your UK customers are also located in Australia. They would like their offices in both locations connected over the MPLS cloud.

It would be very expensive to run a new line between those locations. You do, however, still want to provide L3VPN services to your customers. The is where CSC comes in. CSC allows another ISP to connect both sides of your ISP network together. It also ensures the the core ISP doesn’t learn about any of your customer prefixes, as it doesn’t need to.

Let’s take the following diagram into consideration. (click the image for full size)
Screen Shot 2013 05 04 at 21.01.11 1024x214 Carrier Supporting Carrier
Routers 1, 7, 14, 6, 9, and 8 are all part of ISP2. There are three routers located in each geographical location. Routers 2, 3, 15, and 5 are part of the core carrier. This network stretches to both locations. The rest are customer routers injecting their loopback addresses into OSPF to test.

The core carrier is running IS-IS and LDP. ISP100 is running OSPF and LDP. I won’t go into the regular IGP+LDP config as is pretty straightforward.

With CSC, there are a few new terms to deal with. R14 and R8 are going to be regular PE routers for our ISP. R1 and R6 are going to be called CSC-CE (Customer Supporting Carrier – Customer Edge) routers. R2 and R5 are going to be CSC-PE (Customer Supporting Carrier – Provider Edge) routers. All other ISP routers are simply core routers.

The terminology above assumes that you are speaking in regards to being the core carrier (ISP 500 in this case). That is, ISP500′s edge routers are ‘PE’ and R1 and R6 is the customer’s (ISP) PE routers (Called CE in this case)

It’s a little confusing, depending on which view you take, but it’s really not that difficult.

Initial R14/R8 config – regular PE

The first thing we can start off with is to ensure R14′s PE config is correct. I am running OSPF with the CE routers and learning routes from them:

vrf definition CUS1
 rd 14.14.14.14:1
 route-target export 100:1
 route-target import 100:1
 !
 address-family ipv4
 exit-address-family
!
vrf definition CUS2
 rd 14.14.14.14:2
 route-target export 100:2
 route-target import 100:2
 !
 address-family ipv4
 exit-address-family
!
interface FastEthernet1/1
 vrf forwarding CUS1
 ip address 10.14.16.14 255.255.255.0
 ip ospf network point-to-point
 ip ospf 3 area 0
!
interface FastEthernet2/0
 vrf forwarding CUS2
 ip address 10.14.15.14 255.255.255.0
 ip ospf network point-to-point
 ip ospf 2 area 0

R8 has a similar config on the other side so I won’t put it here.

CSC-PE config

R2 and R5 are going to act as PE routers for the core carrier. They will have their AS200 facing interfaces in a VRF. R2 and R5 will be running regular VPNv4 BGP with each other.
csc pe Carrier Supporting Carrier
R2:

vrf definition CSC_AS100
 rd 2.2.2.2:500
 route-target export 500:100
 route-target import 500:100
 !
 address-family ipv4
 exit-address-family
!
interface FastEthernet1/0
 vrf forwarding CSC_AS100
 ip address 10.1.2.2 255.255.255.0
!
router bgp 500
 no bgp default ipv4-unicast
 neighbor 5.5.5.5 remote-as 500
 neighbor 5.5.5.5 update-source Loopback0
 !
 address-family vpnv4
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 send-community extended
 exit-address-family

Once again, R5 has a similar config on the other side so I’m not putting it here.

CSC-CE

Eventually we need R14 and R8 to peer with each other via VPNv4.
csc pe customer Carrier Supporting CarrierThis is much like option C in the inter-AS config in which two PE routers peer with each other even though they are not directly connected in the same AS. In order to do so we need each of them to have a valid route to each other. As R1 and R6 each have routes to their local PE device, they need to learn routes from the other side through the core carrier. To do this I’ll be running BGP to the core carrier. I also need to ensure that I’m sending and receiving labeled BGP routes and the end LSP has to be end to end. No part of the path can be unlabelled. I need to advertise the PE’s (R14 and R8) loopback are advertised over to the core carrier:

interface FastEthernet1/0
 ip address 10.1.2.1 255.255.255.0
 mpls bgp forwarding
!
router bgp 100
 bgp log-neighbor-changes
 neighbor 10.1.2.2 remote-as 500
 !
 address-family ipv4
  network 14.14.14.14 mask 255.255.255.255
  neighbor 10.1.2.2 activate
  neighbor 10.1.2.2 allowas-in 1
  neighbor 10.1.2.2 send-label
  no auto-summary
 exit-address-family

I’ll need to allowas-in 1 as both sides are running the same AS number. Without it, the CSC-CE routers would reject the BGP update.

I then need to ensure the CSC-CE routers are redistributing those learned prefixes into the IGP:

router ospf 1
 redistribute bgp 100 subnets

R6 again has a similar config.

The end result of it so far is that R8 and R14 should now be able to ping each other from their respective loopbacks:

R8#ping 14.14.14.14 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 14.14.14.14, timeout is 2 seconds:
Packet sent with a source address of 8.8.8.8
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/45/56 ms
R8#traceroute 14.14.14.14 so lo0

Type escape sequence to abort.
Tracing the route to 14.14.14.14

  1 10.8.9.9 [MPLS: Label 24 Exp 0] 56 msec 44 msec 32 msec
  2 10.6.9.6 [MPLS: Label 24 Exp 0] 60 msec 24 msec 60 msec
  3 10.5.6.5 [MPLS: Label 16 Exp 0] 44 msec 44 msec 48 msec
  4 10.5.13.13 [MPLS: Labels 20/34 Exp 0] 48 msec 44 msec 44 msec
  5 10.3.13.3 [MPLS: Labels 19/34 Exp 0] 44 msec 44 msec 40 msec
  6 10.1.2.2 [MPLS: Label 34 Exp 0] 40 msec 44 msec 44 msec
  7 10.1.2.1 [MPLS: Label 19 Exp 0] 44 msec 44 msec 28 msec
  8 10.1.7.7 [MPLS: Label 18 Exp 0] 56 msec 32 msec 12 msec
  9 10.7.14.14 52 msec *  72 msec

Which they can.

PE config – continued

Now that the PE routers have connectivity to each other, we can set up the VPNv4 BGP session:

router bgp 100
 no bgp default ipv4-unicast
 neighbor 8.8.8.8 remote-as 100
 neighbor 8.8.8.8 update-source Loopback0
 !
 address-family vpnv4
  neighbor 8.8.8.8 activate
  neighbor 8.8.8.8 send-community extended
 exit-address-family

Is the session up?

R14#show bgp vpnv4 unicast all summary
BGP router identifier 14.14.14.14, local AS number 100
BGP table version is 1, main routing table version 1

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
8.8.8.8         4          100       4       4        1    0    0 00:00:50        0

Yes, but no prefixes learnt. We still need to redistribute between our vrf aware OSPF processes and BGP:

router ospf 3 vrf CUS1
 redistribute bgp 100 subnets
!
router ospf 2 vrf CUS2
 redistribute bgp 100 subnets
!
router bgp 100
 !
address-family ipv4 vrf CUS1
  redistribute ospf 3 vrf CUS1
 exit-address-family
!
address-family ipv4 vrf CUS2
  redistribute ospf 2 vrf CUS2
 exit-address-family

Verification

As always with these types of configs, we need to ensure both the control and data planes are working correctly. First let’s see the control plane update of R16′s loopback over to R11. The PE router of R14 should be learning this as a vrf prefix:

R14#show ip route vrf CUS1 16.16.16.16

Routing Table: CUS1
Routing entry for 16.16.16.16/32
  Known via "ospf 3", distance 110, metric 2, type intra area
  Redistributing via bgp 100
  Advertised by bgp 100
  Last update from 10.14.16.16 on FastEthernet1/1, 00:18:35 ago
  Routing Descriptor Blocks:
  * 10.14.16.16, from 16.16.16.16, 00:18:35 ago, via FastEthernet1/1
      Route metric is 2, traffic share count is 1

This prefix is converted into a VPNv4 prefix and advertised over to R8:

R8#show bgp vpnv4 un rd 14.14.14.14:1 16.16.16.16
BGP routing table entry for 14.14.14.14:1:16.16.16.16/32, version 3
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Local
    14.14.14.14 (metric 1) from 14.14.14.14 (14.14.14.14)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:100:1 OSPF DOMAIN ID:0x0005:0x000000030200
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.14.16.14:0
      mpls labels in/out nolabel/27

This should end up in the correct vrf table:

R8#show ip route vrf CUS1 16.16.16.16

Routing Table: CUS1
Routing entry for 16.16.16.16/32
  Known via "bgp 100", distance 200, metric 2, type internal
  Redistributing via ospf 2
  Advertised by ospf 2 subnets
  Last update from 14.14.14.14 00:07:03 ago
  Routing Descriptor Blocks:
  * 14.14.14.14 (default), from 14.14.14.14, 00:07:03 ago
      Route metric is 2, traffic share count is 1
      AS Hops 0
      MPLS label: 27
      MPLS Flags: MPLS Required

Finally R11 should receive that as on OSPF route:

R11#show ip route 16.16.16.16
Routing entry for 16.16.16.16/32
  Known via "ospf 1", distance 110, metric 2
  Tag Complete, Path Length == 1, AS 100, , type extern 2, forward metric 1
  Last update from 10.8.11.8 on FastEthernet1/0, 00:06:38 ago
  Routing Descriptor Blocks:
  * 10.8.11.8, from 10.8.11.8, 00:06:38 ago, via FastEthernet1/0
      Route metric is 2, traffic share count is 1
      Route tag 3489661028

So our control plane is all good so far. Let’s check our data plane forwarding:

R11#ping 16.16.16.16 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 16.16.16.16, timeout is 2 seconds:
Packet sent with a source address of 11.11.11.11
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/57/76 ms


R11#traceroute 16.16.16.16 so lo0

Type escape sequence to abort.
Tracing the route to 16.16.16.16

  1 10.8.11.8 8 msec 12 msec 12 msec
  2 10.8.9.9 [MPLS: Labels 24/27 Exp 0] 52 msec 56 msec 48 msec
  3 10.6.9.6 [MPLS: Labels 24/27 Exp 0] 56 msec 48 msec 48 msec
  4 10.5.6.5 [MPLS: Labels 16/27 Exp 0] 56 msec 40 msec 68 msec
  5 10.5.13.13 [MPLS: Labels 20/34/27 Exp 0] 52 msec 44 msec 52 msec
  6 10.3.13.3 [MPLS: Labels 19/34/27 Exp 0] 56 msec 48 msec 40 msec
  7 10.2.3.2 [MPLS: Labels 34/27 Exp 0] 52 msec 52 msec 40 msec
  8 10.1.2.1 [MPLS: Labels 19/27 Exp 0] 44 msec 44 msec 48 msec
  9 10.1.7.7 [MPLS: Labels 18/27 Exp 0] 52 msec 40 msec 60 msec
 10 10.14.16.14 [MPLS: Label 27 Exp 0] 40 msec 44 msec 36 msec
 11 10.14.16.16 44 msec *  52 msec

No problems there :)

Carrier Supporing Carrier Supporting Carrier

So let’s be silly and go a step further. What if our final customer is actually another ISP offering L3VPN to it’s customers? Let’s change our topology slightly(click the image for full size)
Screen Shot 2013 05 06 at 19.14.59 1024x208 Carrier Supporting Carrier

I’m not going to show all the config here as it’s simply too much to fit into a blog post. However the config itself is pretty much like so. You’ll need to click the image for the larger version:
cscsc 1024x327 Carrier Supporting Carrier

R10 and R15 are our final CE routers. Once all is configured does it all actually work?

R10#traceroute 15.15.15.15 so lo0

Type escape sequence to abort.
Tracing the route to 15.15.15.15

  1 10.10.11.11 12 msec 12 msec 8 msec
  2 10.8.11.8 [MPLS: Labels 16/17 Exp 0] 72 msec 48 msec 68 msec
  3 10.8.9.9 [MPLS: Labels 22/24/17 Exp 0] 36 msec 80 msec 44 msec
  4 10.6.9.6 [MPLS: Labels 20/24/17 Exp 0] 52 msec 56 msec 52 msec
  5 10.5.6.5 [MPLS: Labels 23/24/17 Exp 0] 64 msec 52 msec 52 msec
  6 10.5.13.13 [MPLS: Labels 20/23/24/17 Exp 0] 48 msec 60 msec 52 msec
  7 10.3.13.3 [MPLS: Labels 19/23/24/17 Exp 0] 56 msec 56 msec 48 msec
  8 10.2.3.2 [MPLS: Labels 23/24/17 Exp 0] 56 msec 44 msec 56 msec
  9 10.1.2.1 [MPLS: Labels 19/24/17 Exp 0] 48 msec 48 msec 64 msec
 10 10.1.7.7 [MPLS: Labels 16/24/17 Exp 0] 56 msec 52 msec 60 msec
 11 10.7.14.14 [MPLS: Labels 24/17 Exp 0] 44 msec 56 msec 56 msec
 12 10.15.16.16 [MPLS: Label 17 Exp 0] 56 msec 56 msec 44 msec
 13 10.15.16.15 56 msec *  76 msec

It does indeed. At this point we are up to a four label stack in AS500. If we were running RSVP-TE and FRR we would have even more labels sitting on top.

There is a much easier way to do this of course. The original Customer Carrier could just buy some some or virtual leased line or VPLS from AS500 and they would be directly connected over the same subnet. They could then run MPLS over that link and as far as anyone cares R1 and R6 would be directly connected to each other.

But of course this is a topic on the CCIE SP after all…