Tag Archives: carrier

ME3600X Bridge Group into a Brocade VPLS for H-QoS

The Cisco ME3600X and ME3800X series support both VPLS and H-VPLS. MPLS features require an expensive license. If you are currenlty joining a few leased lines together into a VPLS you don’t always need the more expensive MPLS license.

Any carrier circuit terminating on the Cisco ME can be placed into a bridge-group based on the dot1q tag. This can then be passed along or re-written to a new vlan tag into a Brocade XMR port which stick that frame into a VPLS. Why not just terminate the carrier circuit directly onto the XMR? I’ve recently had to do this because the XMR line cards we have do not support H-QoS. More and more carriers are aggregating many B end circuits into single high-bandwidth A end gig links. I need to be able to shape traffic outbound on a per-vlan basis, and within that queue give priority to certain queues. The ME3600X can do H-QoS and its exactly the reason I’m using it.

Let’s show a quick diagram so we know what we are talking about. Click the image for the full image:
Carrier Ports 2 300x195 ME3600X Bridge Group into a Brocade VPLS for H QoS

On the right I have two SRX210s and an 1841 sending tagged traffic into the carrier network. That carrier network is multiplexing all three circuits into a single link on the A end. That goes into the ME3600X. From the ME3600X it goes off to a Brocade XMR. That Brocade is connected to another Brocade over the MPLS core and finally to another 1841 on the other side.

On the ME3600X I could easily span each vlan over to the XMR and have the XMR VPLS them back. The issue with that is that if hosts behind SRX is sending a ton of traffic to a host at SRX2, why waste bandwidth hairpinning traffic over to the XMR when it could be done at the ME3600X?

In other words, instead of doing this:
VPLS PIN1 ME3600X Bridge Group into a Brocade VPLS for H QoS

We do something like this:
ME3400 PIN ME3600X Bridge Group into a Brocade VPLS for H QoS

Now you may be asking, why not just get this carrier to stick them in the VPLS? That could work if all these links were coming from the same carrier, but often they aren’t.

ME3600X EVC config

ethernet evc TESTLAB
!
interface GigabitEthernet0/1
 description Link to Carrier
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 1 ethernet TESTLAB
  description SRX1
  encapsulation dot1q 2000
  rewrite ingress tag pop 1 symmetric
  bridge-domain 150
 !
 service instance 2 ethernet TESTLAB
  description SRX2
  encapsulation dot1q 2001
  rewrite ingress tag pop 1 symmetric
  bridge-domain 150
 !
 service instance 3 ethernet TESTLAB
  description 1841
  encapsulation dot1q 2002 second-dot1q 100
  rewrite ingress tag pop 2 symmetric
  bridge-domain 150
 !
interface GigabitEthernet0/24
 description Link to XMR
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 150 ethernet
  description VPLS Core
  encapsulation dot1q 150
  rewrite ingress tag pop 1 symmetric
  bridge-domain 150
 !
end

I have three service instances configured on gi0/1 – each matching the vlan id that the carrier is using for transport. Notice that I can match on both an outer and inner tag that I’m originating from the 1841. Gi0/24 is the port connected to the XMR and that’s in the same bridge-group with a vlan tag of 150.

On the XMR side I’m simply matching on vlan 150 and placing those frames in the VPLS.

Brocade Config

vpls DARREN-TESTING 3200
  vpls-peer 172.10.10.1
  vlan 150
   tagged ethe 2/20

Connectivity verification

All my CPEs are running OSPF on their WAN links. I’ve also hard-coded their MAC addresses so it’ll be easy to see in this post.

darreno@JR1> show ospf neighbor
Address          Interface              State     ID               Pri  Dead
10.0.0.4         ge-0/0/1.2000          Full      4.4.4.4            1    39
10.0.0.3         ge-0/0/1.2000          2Way      3.3.3.3            1    34
10.0.0.2         ge-0/0/1.2000          Full      2.2.2.2          128    32

SRX1 has three neighbours. Fully adjacent with the DR and BDR. This means I get get to the remote VPLS 1841. Let’s take a look at the mac address table for the VPLS on the PE connected to the ME3600x:

SSH@pe2#sh mac vpls 3200

Total MAC entries for VPLS 3200: 5 (Local: 3, Remote: 2)

VPLS       MAC Address    L/R Port  Vlan(In-Tag)/Peer ISID      Age
====       ===========    === ====  ================= ====      ===
3200       0000.1111.0000 L   2/20  150               NA        0
3200       0000.2222.0000 L   2/20  150               NA        0
3200       0000.3333.0000 L   2/20  150               NA        0
3200       0000.4444.0000 R   1/10  172.10.10.1       NA        0

The MAC’s for SRX1, SRX2, and the first 1841 are all via tag 150 out interface 2/20. The remote 1841′s MAC is learned via the remote PE router.

We should see all four MACs on thee ME3600X out their respective tagged ports:

ME3600X#show mac address-table bridge-domain 150 | begin DYNAMIC
 150    0000.1111.0000    DYNAMIC     Gi0/1+Efp1
 150    0000.2222.0000    DYNAMIC     Gi0/1+Efp2
 150    0000.3333.0000    DYNAMIC     Gi0/1+Efp3
 150    0000.4444.0000    DYNAMIC     Gi0/24+Efp150

The ME3600X is also telling us which service instance under the physical port it’s learning the MAC address from.

QoS Config

Let’s assume the the circuit tagged with vlan 2000 has only got 5Mb, lan 2001 has 10Mb, and vlan 2001 has 15Mb. I want to shape each EVC to their respective speed, and then give priority to DSCP EF packets. I also want to police that queue to 50% to ensure that priority queue cannot hog the link.

class-map match-all EF
 match dscp ef
!
policy-map QoS
 class EF
  priority
  police cir percent 50
   conform-action transmit
   exceed-action drop
 class class-default
  queue-limit percent 100
!
policy-map VLAN2000
 class class-default
  shape average 5000000
   service-policy QoS
policy-map VLAN2001
 class class-default
  shape average 10000000
   service-policy QoS
policy-map VLAN2002
 class class-default
  shape average 15000000
   service-policy QoS

Each policy is then attached to the service instance itself. I’ll use service instance 1 as an example here:

interface GigabitEthernet0/1
 service instance 1 ethernet TESTLAB
  description SRX1
  encapsulation dot1q 2000
  rewrite ingress tag pop 1 symmetric
  service-policy output VLAN2000
  bridge-domain 150

Each service instance can have an individual policy. So we have broken up the physical port into many virtual circuits. Each VC has their own shaper and priority queue.

QoS Verification

If you apply the policy as above, you can’t use show policy-map interface anymore. Instead you need to use show ethernet service instance policy-map

ME3600X#show ethernet service instance policy-map
  GigabitEthernet0/1: EFP 1

  Service-policy output: VLAN2000

    Class-map: class-default (match-any)
      8 packets, 644 bytes
      5 minute offered rate 0000 bps, drop rate 0000 bps
      Match: any
  Traffic Shaping
    Average Rate Traffic Shaping
    Shape 5000 (kbps)
      Output Queue:
        Default Queue-limit 49152 bytes
        Tail Packets Drop: 0
        Tail Bytes Drop: 0

      Service-policy : QoS

        Class-map: EF (match-all)
          0 packets, 0 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match:  dscp ef (46)
          Strict Priority
          police:
            cir percent 50 % bc 250 ms
            cir 2500000 bps, bc 78000 bytes
            conform-action transmit
            exceed-action drop
          conform: 0 (packets) 0 (bytes)
          exceed: 0 (packets) 0 (bytes)
          conform: 0 bps, exceed: 0 bps
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

        Class-map: class-default (match-any)
          8 packets, 644 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match: any
          Queue-limit 100 percent
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

OSPF packets are going through and that’s being matched by class-default. We can force some EF traffic by pinging with a TOS value:

darreno@JR2> ping 10.0.0.1 rapid tos 184
PING 10.0.0.1 (10.0.0.1): 56 data bytes
!!!!!
--- 10.0.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 3.766/4.635/7.131/1.269 ms
ME3600X#show ethernet service instance policy-map
  GigabitEthernet0/1: EFP 1

  Service-policy output: VLAN2000

    Class-map: class-default (match-any)
      328 packets, 26028 bytes
      5 minute offered rate 2000 bps, drop rate 0000 bps
      Match: any
  Traffic Shaping
    Average Rate Traffic Shaping
    Shape 5000 (kbps)
      Output Queue:
        Default Queue-limit 49152 bytes
        Tail Packets Drop: 0
        Tail Bytes Drop: 0

      Service-policy : QoS

        Class-map: EF (match-all)
          5 packets, 530 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match:  dscp ef (46)
          Strict Priority
          police:
            cir percent 50 % bc 250 ms
            cir 2500000 bps, bc 78000 bytes
            conform-action transmit
            exceed-action drop
          conform: 5 (packets) 510 (bytes)
          exceed: 0 (packets) 0 (bytes)
          conform: 0 bps, exceed: 0 bps
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

        Class-map: class-default (match-any)
          323 packets, 25498 bytes
          5 minute offered rate 2000 bps, drop rate 0000 bps
          Match: any
          Queue-limit 100 percent
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

There we see the five EF packets.

So there you have it. Not too difficult at all to get the basics working.

Fun with QinQ

Not all service providers have blanket coverage of an entire country. When a service provider gives service to a customer, more than likely the customer will be using a line from an external carrier of the ISP’s choosing.

Metro Ethernet is becoming more and more common these days and it does give us as designers a lot of flexibility. However each time a customer purchases a leased line, that requires another port in your core. That’s fine if the circuit is a nice gig, but quite often a lot of office will have anything from 2Mb to 1Gb. Do you really want to waste your core gig ports for a 2Mb circuit? Not really.

A lot of carriers are now offering aggregated ethernet links. Essentially this means that each customer site has a separate port (of course) but these are all aggregated when the carrier hands off to us. We get a single link carrying a bunch of customer circuits. Now you can sell hundreds of 2Mb circuits and only use a single port.
QinQ VPLS 1 Fun with QinQ

But how do we separate traffic then? Well you’ll come to an agreement with the carrier and each circuit ordered will have a vlan tag on the core side. This means customer 1 site 1′s traffic will arrive on vlan 1000. customer2 site 1′s traffic will arrive on vlan 1001. Now you just need to stick each tag into an MPLS solution and all is good.
QinQ VPLS 2 Fun with QinQ

But now what happens when you need to run multiple virtual circuits to a single customer site over their leased line? The vlan tag is already used by the carrier. What if I need to run 2 vrf’s for the customer and I need a WAN interface in each vrf?

We could do QinQ, but let’s think about this. In order to do QinQ I need another device at the customer site to pop another vlan tag on. I then need to get that tag into the core, pop off the tag and stick it back into the core. This could get messy.
Another problem with regular Cisco switches is that I can only do dot1q encapsulation on a port based. This means I need to use a port for every customer again, negating the advantage of the aggregated port to begin with. Or I can send traffic to another switch on a per-port basis, then get the second switch to aggregate the vlans back into the core. This will work, but what a nightmare to support.
Option1:
QinQ VPLS 3 Fun with QinQ

Option2:
QinQ VPLS 4 Fun with QinQ

Instead of a regular switch I could use a ME3400G and do selective QinQ. This allows me to specify that vlan 10 and 20 gets vlan 1000 popped onto it, and vlan 15 and 25, on the same port, gets vlan 1001 popped onto it.

It’s still another device that we could do without.

Let’s tackle the problem at the customer site first. Ideally I would like to do away with a separate device that does my QinQ. I can’t send double-tagged frames directly out my Cisco. Are we sure about this?

interface GigabitEthernet0/1.500
 encapsulation dot1Q 1000 second-dot1q 500
 ip address 172.16.255.1 255.255.255.0
end

The second-dot1q command allows you to both send and receive QinQ traffic directly from a router interface. The first tag is the metro tag, while the second tag is the inner tag. Let’s create another interface in another vrf.

interface GigabitEthernet0/1.600
 encapsulation dot1Q 1000 second-dot1q 600
 ip vrf forwarding 1
 ip address 192.168.1.2 255.255.255.0

Perfect. I can create as many sub-interfaces as I need using a single metro tag.

What about my core? In my core I’m using Brocade Netirons. Let’s see if I can terminate a double-tagged frame:

vpls Test_Customer 500
  vpls-peer 192.168.1.1
   vlan 1000 inner-vlan 500
   tagged ethe 15/1
!
 vpls Test_Customer 600
  vpls-peer 192.168.1.50
   vlan 1000 inner-vlan 600
   tagged ethe 15/1

No problems there. I can create 2 completely separate VPLS instances using the same single metro tag.

So far so good…

Let’s say for whatever reason I now need to run a point-to-point link directly from my core to the CPE at the customer site. I don’t want to terminate this point-to-point link into a VLL/VPLS. This could be for management, to provide voice or internet access, whatever. Unfortunately my Brocade device cannot terminate a double-tagged frame both in the local table as well as in a VPLS. Now we’re close to going back to our earlier switch examples to pop off that pesky outer tag.

But never fear, there is always a way. I did note above that we can’t terminate a double-tagged frame on both. How about sending some traffic as double-tagged and certain traffic as single? Can we do this?

Let’s start again with the CPE and show the full relevant config:

interface GigabitEthernet0/1.1000
 encapsulation dot1Q 1000
 ip address 10.0.0.1 255.255.255.0
!
interface GigabitEthernet0/1.600
 encapsulation dot1Q 1000 second-dot1q 600
 ip vrf forwarding 1
 ip address 192.168.1.2 255.255.255.0
!
interface GigabitEthernet0/1.500
 encapsulation dot1Q 1000 second-dot1q 500
 ip address 172.16.255.1 255.255.255.0

Works just fine on the Cisco. What about my Brocade?

vlan 1000 name Test
 tagged ethe 15/1
 router-interface ve 1000
!
interface ve 1000
 ip address 10.0.0.2/24
!
router mpls
!
vpls Test_Customer 500
  vpls-peer 192.168.1.1
   vlan 1000 inner-vlan 500
   tagged ethe 15/1
!
 vpls Test_Customer 600
  vpls-peer 192.168.1.50
   vlan 1000 inner-vlan 600
   tagged ethe 15/1

Everything tested and everything works perfectly. We’ve managed to remove all kinds of kit and also managed to simply the solution.