Tag Archives: isp

When a vlan is not a vlan

What is a vlan? What is a vlan-id? Are they the same thing?

Generally yes, but in the ISP world a vlan-id can also be a circuit identifier. While your view of a vlan might be a single broadcast domain, you’ll soon see that multiple vlan IDs can share the same single broadcast domain, or the same vlan-id could be in a completely different broadcast domain.

The Problem

I’ve written about this before. Carriers, at least in the UK, are offering more and more aggregated links to Service Providers. Each circuit to customer sites is aggregated over a single high-bandwidth link to your PE router. This cuts down on ports, cables, and man hours to plug them in.

Old way:

carrier old When a vlan is not a vlan

New way:

carrier new When a vlan is not a vlan
How are the p2p circuits aggregated over the core high-bandwidth link? Each p2p link is separated by a vlan tag on the PoP side. So we could say that any packet coming out of the core PE with vlan 2000 goes to site 1, while packets with vlan 3000 go to site 2. What happens if site 1 and site 2 are going to the same customer? What if you are providing a VPLS service to them? It’s essential to note that the vlan tag imposed by the carrier is used simply to determine what packet goes to which circuit. As we control the MPLS core, it’s ultimately up to us to decide which packet belongs in which broadcast domain, and that is regardless of the vlan id used by the carrier.

Relevant Initial Core Config

I’ll use the following topology:
vlans core When a vlan is not a vlan

R1, R2, and R3 are the core of the network. R1 is a Brocade Netiron running MPLS. R2 is a Cisco me3600x running MPLS. R2 is an me3600x running bridge-groups with no MPLS.

CE1, CE2, and CE3 are all customer routers.

R1 – Brocade XMR

interface ethernet 2/4
 port-name TO-R2
 enable
 route-only
 ip ospf area 0
 ip ospf network point-to-point
 ip address 10.10.10.10/24
!
router mpls
 policy
  traffic-eng ospf area 0

  mpls-interface e2/4

 lsp R1-R2
  to 192.168.224.4
  adaptive
  enable

R2 – Cisco me3600x running MPLS

mpls traffic-eng tunnels
!
router ospf 1
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng area 0
!
interface GigabitEthernet0/1
 description TO-R1
 no switchport
 ip address 10.10.10.11 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls traffic-eng tunnels
!
interface Tunnel0
 ip unnumbered Loopback0
 tunnel mode mpls traffic-eng
 tunnel destination 192.168.224.61
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng path-option 5 dynamic
 tunnel mpls traffic-eng record-route

There is no IP and MPLS configuration on R3 as it’s not running MPLS. I’ll show how the bridge-group is configured when I get to that part.

CPE Config

I’ll be using vlan 3000 to get to CE1, vlan 2000 to get to CE2, and double-tag vlan 3500,2500 to get to CE3. Each CE has their WAN interface in the same subnet as each other running OSPF. I’ll also enable OSPF on their loopbacks and WAN links.

CE1

This is a Juniper EX3200:

root@CE1> show configuration interfaces ge-0/0/0
vlan-tagging;
unit 3000 {
    vlan-id 3000;
    family inet {
        address 1.1.1.1/24;
    }
}

root@CE1> show configuration interfaces lo0.0
family inet {
    address 10.10.10.10/32;
}

root@CE1> show configuration protocols ospf
area 0.0.0.0 {
    interface ge-0/0/0.3000;
    interface lo0.0;
}

CE2

This is a Cisco 3750G:

interface Loopback0
 ip address 20.20.20.20 255.255.255.255
 ip ospf 1 area 0
!
interface Vlan2000
 ip address 1.1.1.2 255.255.255.0
 ip ospf 1 area 0
!
interface GigabitEthernet1/0/1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2000
 switchport mode trunk

CE3

This is a Cisco 1841:

interface Loopback0
 ip address 30.30.30.30 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0.32
 encapsulation dot1Q 3500 second-dot1q 2500
 ip address 1.1.1.3 255.255.255.0
 ip ospf 1 area 0

VPLS Config

As you can see, each CPE will be using a different vlan tag. One site is even sending a double-tagged frame. They all need to be in the same broadcast domain. No problem as we are simply going to use the vlan tag to determine the service.

R2

Gi0/2 will create a LDP-signalled VPLS VC to R1 (aka manual set up). Interface gi0/2 vlan 2000 will be part of VPLS id 501:

ethernet evc TEST-EVC
 uni count 20
!
l2vpn vfi context TEST-VPLS
 vpn id 501
 member 192.168.224.61 encapsulation mpls
!
interface GigabitEthernet0/2
 switchport trunk allowed vlan none
 switchport mode trunk
 mtu 9800
 service instance 1 ethernet TEST-EVC
  encapsulation dot1q 2000
  rewrite ingress tag pop 1 symmetric
  bridge-domain 501
 !
interface Vlan501
 no ip address
 member vfi TEST-VPLS

What’s important to note here is that the me3600x still uses bridge-groups for VPLS, but it’s not exactly the same as just using bridge-groups by itself. You’ll see this soon enough when we configure R3.

R1

R1 will create a VPLS to R2. Vlan 3000 on interface 2/5 will be part of the same VPLS:

router mpls
 vpls TEST-VPLS 501
  vpls-peer 192.168.224.4
  vpls-mtu 1500
  vlan 3000
   tagged ethe 2/5

At this point R1 and R2 have the VPLS set up between them. Each CE is using different vlans on their WAN, but they are in fact on the same broadcast domain:

CE2#sh ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1         128   FULL/DR         00:00:39    1.1.1.1         Vlan2000

CE2#ping 1.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/17 ms

CE2#ping 10.10.10.10 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
Packet sent with a source address of 20.20.20.20
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms

The vlan-id used on the CPE, was merely used to push the frame into the correct VPLS. The VPLS itself is the broadcast domain, the vlan tag is irrelevant as its stripped on inbound into the PE router. You CAN however, ensure that the PE router does NOT strip the vlan tag. This has interesting use cases when you purposely want to separate on vlan id with in the VPLS. I wrote more on this over here so please give it a read. Both the Brocade and Cisco default to VC mode 5 when setting up a VPLS.

Bridge Group Config

I’m going to set up R3 so that it only uses bridge-groups. No routing or MPLS involved. Bridge-Groups work very similar to VPLS, though it’s on a single box. Traffic can be pushed from a bridge-group into a VPLS if needed. The bridge-group determines the broadcast domain. I can have multiple different vlans in the same bridge group.

For R3, gi0/2 is the interface pointing towards the core, while gi0/1 is pointing towards the customer. I’ll use different vlan ids on each, but they will be in the same bridge-group:

ethernet evc TEST
!
vlan 501
 name TEST-CE
!
interface GigabitEthernet0/1
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 1 ethernet TEST
  encapsulation dot1q 501
  rewrite ingress tag pop 1 symmetric
  bridge-domain 501
 !
interface GigabitEthernet0/2
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 1 ethernet TEST
  encapsulation dot1q 3500 second-dot1q 2500
  rewrite ingress tag pop 2 symmetric
  bridge-domain 501

I’m not going into detail, but I will cover the basics. When gi0/2 receives a double-tagged frame that matches 3500,2500 inbound, the me3600x will pop both tags off and the resulting frame will be part of bridge-group 501. Symmetric means that when a frame leaves gi0/2, it will re-add vlans 3500,2500 on top of the frame. As gi0/1 is also in bridge-group 501, the customer frame will be forwarded out that port, and it will have a single vlan tag of 501 popped on top.

At this point gi0/1 is connected to R1 eth2/3. For this customer I would be expecting a single tag of 501 coming inbound, and so I’ll place that vlan id into the VPLS from above:

 vpls TEST-VPLS 501
  vlan 501
   tagged ethe 2/3

Now all three CE routers should be fully adjacent:

CE3#sh ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1         128   FULL/DR         00:00:35    1.1.1.1         FastEthernet0/0.32
1.1.1.2           1   FULL/DROTHER    00:00:37    1.1.1.2         FastEthernet0/0.32

CE3#ping 10.10.10.10 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
Packet sent with a source address of 30.30.30.30
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
CE3#ping 20.20.20.20 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 20.20.20.20, timeout is 2 seconds:
Packet sent with a source address of 30.30.30.30
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/12 ms

Conclusions:

vlan tags have multiple uses. In most networks it informs the switches which vlan, and therefore broadcast domain, a frame is part of. They can also be circuit identifiers showing which VPLS/Circuit the frame belongs to. They can also be both at the same time, depending on the VPLS VC type you’re using.

For the above network it’s extremely simplified. Care must be taken when forwarding certain layer2 control frames. Most are sent untagged out tagged interfaces. Cisco’s RSTP+ and STP tag each vlan BPDU with a the same vlan-id. If you’re using vlan 2000 on one side and vlan 3000 on the other, and the BPDU gets through, one side will shut down their WAN link due to receiving a BPDU with a vlan tag that doesn’t match the BPDU data inside the frame.