Juniper EX – Private vlans

I’ve gone over pvlans before on IOS, so I’m going to cover Juniper’s implementation today. This post will be based on the following topology:
PVLANS-1
There are five hosts and a single router. Host1 and Host3 are in the same community vlan, while Host2, Host4, and Host5 are in isolated vlans. R1 is the default gateway for all hosts.
The vlan plan is laid out as follows:

PVLAN Table
Host Switch Vlan
1 SW1 Community 501
2 SW1 Isolated 502
3 SW2 Community 501
4 SW2 Isolated 502
5 SW2 Isolated 502
R1 SW3 Primary 500

Community vlan – SW1

The community vlan has a vlan-id and primary vlan specified. You enable the interface under tha vlan config:

set vlans HOST_COMM vlan-id 501
set vlans HOST_COMM primary-vlan HOST_PRIMARY
set vlans HOST_COMM interface ge-0/0/4.0

Isolated and Primary Van – SW1

Isolated vlans are configured directly under the primary vlan. You also specify the interfaces in this vlan under the vlans hierarchy. Finally, as this pvlans span multiple switches, you need to ensure the trunk interfaces are pvlans aware:

set vlans HOST_PRIMARY vlan-id 500
set vlans HOST_PRIMARY interface ge-0/0/1.0 pvlan-trunk
set vlans HOST_PRIMARY interface ge-0/0/3.0 pvlan-trunk
set vlans HOST_PRIMARY interface ge-0/0/5.0
set vlans HOST_PRIMARY no-local-switching
set vlans HOST_PRIMARY isolation-id 502

Personally I really don’t like the Junos way of doing isolated vlans. Interface ge-0/0/5.0 is an isolated port as its untagged and no-local-switching is configured. Configuring the promiscuous port to R1 from SW3 is configured like so:

set vlans HOST_PRIMARY vlan-id 500
set vlans HOST_PRIMARY interface ge-0/0/2.0 pvlan-trunk
set vlans HOST_PRIMARY interface ge-0/0/1.0 pvlan-trunk
set vlans HOST_PRIMARY interface ge-0/0/0.0
set vlans HOST_PRIMARY no-local-switching
set vlans HOST_PRIMARY isolation-id 502

There is no difference in the vlan configuration between an actual isolated port and a promiscuous port. What makes the difference is the interface config itself on both switches:

[email protected]> show configuration interfaces ge-0/0/5
unit 0 {
    family ethernet-switching {
        port-mode access;
    }
}

While SW3’s port to R1 is tagged:

[email protected]> show configuration interfaces ge-0/0/0
unit 0 {
    family ethernet-switching {
        port-mode trunk;
    }
}

If I wanted SW3’s like to R1 to be untagged it would change it to an isolated port. If I needed a host to send tagged traffic into an isolated vlan (like an ESX server), Junos makes that a promiscuous port. This is a lack of flexibility that I don’t like. The switches should be able to put devices in isolated or promiscuous mode by config separate to the fact that the host-facing port has a dot1q tag or not.

Verification

Show vlans extensive shows the pvlan information. It would be nice if Junos had a separate show pvlans command:

[email protected]> show vlans extensive
VLAN: HOST_COMM, Created at: Wed Mar 19 04:00:58 2014
802.1Q Tag: 501, Internal index: 14, Admin State: Enabled, Origin: Static
Private VLAN Mode: Community, Primary VLAN: HOST_PRIMARY
Protocol: Port Mode, Mac aging time: 300 seconds
Number of interfaces: Tagged 2 (Active = 2), Untagged  1 (Active = 1)
      ge-0/0/1.0*, tagged, trunk, pvlan-trunk
      ge-0/0/3.0*, tagged, trunk, pvlan-trunk
      ge-0/0/4.0*, untagged, access

Here we see ge-0/0/4.0 is an access port in vlan HOST_COMM. ge-0/0/1.0 and ge-0/0/3.0 are pvlan trunks as expected.

LAN: HOST_PRIMARY, Created at: Wed Mar 19 04:00:58 2014
802.1Q Tag: 500, Internal index: 16, Admin State: Enabled, Origin: Static
Private VLAN Mode: Primary
Protocol: Port Mode, Mac aging time: 300 seconds
Number of interfaces: Tagged 2 (Active = 2), Untagged  2 (Active = 2)
      ge-0/0/1.0*, tagged, trunk, pvlan-trunk
      ge-0/0/3.0*, tagged, trunk, pvlan-trunk
      ge-0/0/4.0*, untagged, access
      ge-0/0/5.0*, untagged, access
Secondary VLANs: Isolated 1, Community  1, Inter-switch-isolated  1
  Isolated VLANs :
      __pvlan_HOST_PRIMARY_ge-0/0/5.0__
  Community VLANs :
      HOST_COMM
  Inter-switch-isolated VLAN :
      __pvlan_HOST_PRIMARY_isiv__

VLAN: __pvlan_HOST_PRIMARY_ge-0/0/5.0__, Created at: Wed Mar 19 04:26:16 2014
Internal index: 17, Admin State: Enabled, Origin: Static
Private VLAN Mode: Isolated, Primary VLAN: HOST_PRIMARY
Protocol: Port Mode, Mac aging time: 300 seconds
Number of interfaces: Tagged 2 (Active = 2), Untagged  1 (Active = 1)
      ge-0/0/1.0*, tagged, trunk, pvlan-trunk
      ge-0/0/3.0*, tagged, trunk, pvlan-trunk
      ge-0/0/5.0*, untagged, access

VLAN: __pvlan_HOST_PRIMARY_isiv__, Created at: Wed Mar 19 04:26:16 2014
802.1Q Tag: 502, Internal index: 18, Admin State: Enabled, Origin: Static
Private VLAN Mode: Inter-switch-isolated, Primary VLAN: HOST_PRIMARY
Protocol: Port Mode, Mac aging time: 300 seconds
Number of interfaces: Tagged 2 (Active = 2), Untagged  0 (Active = 0)
      ge-0/0/1.0*, tagged, trunk, pvlan-trunk
      ge-0/0/3.0*, tagged, trunk, pvlan-trunk

A lot of information above, buit it does show which ports are connected to the primary vlan and which are isolated. It also shows which community and isolated vlans are connected to the primary vlan.

Ultimately the end result is that Host1 should be able to ping Host3 and Router1, but nothing else:

[email protected]_HOST> ping routing-instance HOST1 10.0.0.2 rapid
PING 10.0.0.2 (10.0.0.2): 56 data bytes
.....
--- 10.0.0.2 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

{master:0}
[email protected]_HOST> ping routing-instance HOST1 10.0.0.3 rapid
PING 10.0.0.3 (10.0.0.3): 56 data bytes
!!!!!
--- 10.0.0.3 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.056/1.175/1.415/0.136 ms

{master:0}
[email protected]_HOST> ping routing-instance HOST1 10.0.0.4 rapid
PING 10.0.0.4 (10.0.0.4): 56 data bytes
.....
--- 10.0.0.4 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

{master:0}
[email protected]_HOST> ping routing-instance HOST1 10.0.0.5 rapid
PING 10.0.0.5 (10.0.0.5): 56 data bytes
.....
--- 10.0.0.5 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

{master:0}
[email protected]_HOST> ping routing-instance HOST1 10.0.0.254 rapid
PING 10.0.0.254 (10.0.0.254): 56 data bytes
!!!!!
--- 10.0.0.254 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.007/1.189/1.638/0.229 ms

I won’t go every single possible combination a there will simply be too much text, but I’ll go over Host4 and Router1.

Host4 should only be able to ping Router1 and nothing else:

[email protected]_HOST> ping routing-instance HOST4 10.0.0.1 rapid
PING 10.0.0.1 (10.0.0.1): 56 data bytes
.....
--- 10.0.0.1 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

{master:0}
[email protected]_HOST> ping routing-instance HOST4 10.0.0.2 rapid
PING 10.0.0.2 (10.0.0.2): 56 data bytes
.....
--- 10.0.0.2 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

{master:0}
[email protected]_HOST> ping routing-instance HOST4 10.0.0.3 rapid
PING 10.0.0.3 (10.0.0.3): 56 data bytes
.....
--- 10.0.0.3 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

{master:0}
[email protected]_HOST> ping routing-instance HOST4 10.0.0.5 rapid
PING 10.0.0.5 (10.0.0.5): 56 data bytes
.....
--- 10.0.0.5 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

{master:0}
[email protected]_HOST> ping routing-instance HOST4 10.0.0.254 rapid
PING 10.0.0.254 (10.0.0.254): 56 data bytes
!!!!!
--- 10.0.0.254 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.997/1.763/3.471/0.927 ms

Finally, Router1 should be able to ping all hosts:

[email protected]_HOST> ping routing-instance ROUTER1 10.0.0.1 rapid
PING 10.0.0.1 (10.0.0.1): 56 data bytes
!!!!!
--- 10.0.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.011/1.168/1.301/0.122 ms

{master:0}
[email protected]_HOST> ping routing-instance ROUTER1 10.0.0.2 rapid
PING 10.0.0.2 (10.0.0.2): 56 data bytes
!!!!!
--- 10.0.0.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.963/1.194/1.432/0.180 ms

{master:0}
[email protected]_HOST> ping routing-instance ROUTER1 10.0.0.3 rapid
PING 10.0.0.3 (10.0.0.3): 56 data bytes
!!!!!
--- 10.0.0.3 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.988/1.165/1.438/0.161 ms

{master:0}
[email protected]_HOST> ping routing-instance ROUTER1 10.0.0.4 rapid
PING 10.0.0.4 (10.0.0.4): 56 data bytes
!!!!!
--- 10.0.0.4 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.997/1.194/1.444/0.200 ms

{master:0}
[email protected]_HOST> ping routing-instance ROUTER1 10.0.0.5 rapid
PING 10.0.0.5 (10.0.0.5): 56 data bytes
!!!!!
--- 10.0.0.5 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.015/1.174/1.324/0.123 ms

Private Vlans – Control Plane/Data Plane

Private vlans are actually very easy to configure. But what’s actually happening at the data plane level? Knowing this helps when we need to span our pvlans through switches that do not support private vlans.

 

Let’s use the following topology for these tests. The routers are just acting as hosts. I also have a laptop connected to port fa0/48 on SW3 to capture some frames. All routers are in the 10.0.0.x/24 range.

 

SW1 is a 3750 and SW2 is a 3560. Both support pvlans. SW3 is a 3550 which has no concept of private vlans.

The first thing I’m going to do is create the primary private vlan with an id of 50. I’ll then create a secondary community vlan with a vlaue of 24.  I’ll then put R2’s and R4’s port into the pvlan.

R1:

vlan 24
  private-vlan community
!
vlan 50
  private-vlan primary
  private-vlan association 24
!
interface FastEthernet1/0/4
 switchport private-vlan host-association 50 24
 switchport mode private-vlan host

At this point if I ping R4 from R2 the ping fails:

R2#ping 10.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.4, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

If two community ports are trying to speak to each other, SW1 and SW2 will use the community vlan value id for traffic in both ways. The community vlan is 24, so let’s try creating a regular vlan 24 on SW3:

SW3(config)#vlan 24
SW3(config-vlan)#end

Can we ping now?

R2#ping 10.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/6/20 ms

It doesn’t matter that SW3 doesn’t support pvlans at this point. SW1 and SW2 simply use a generic vlan tag of 24. This can be shown in the wireshark capture for both the ping request and response:


SW3 doesn’t know this is a private-vlan. It simply sees traffic for vlan 24 passing through it’s trunk ports. Does this mean that we can put R3’s port into vlan 24 and it’ll be able to ping both R2 and R4? Well it should, but let’s find out.
SW3:

interface FastEthernet0/1
 switchport access vlan 24
 switchport mode access

R2:

R2#ping 10.0.0.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/7/16 ms

No problems there. Wireshark shows the same regular vlan 24 being used between the switches.

Now let’s add a promiscuous port. I’ll make port fa1/0/1 on SW1 a promiscuous port which will represent the gateway on this subnet.
SW1:

interface FastEthernet1/0/1
 switchport private-vlan mapping 50 24
 switchport mode private-vlan promiscuous

Can R4 and R2 ping R1?

R4#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/6/16 ms
R2#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R4 can ping, R2 can’t. Why it can’t will become clear very shortly. Let’s add vlan 50 on SW3 and try again on R2.

SW3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW3(config)#vlan 50
SW3(config-vlan)#end
R2#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/8/20 ms

Now R2 can ping. To figure out why, let’s have a look at what wireshark is telling us.


It’s quite clear from the captures above. When traffic needs to get from a community port to an promiscuous port, SW1 and SW2 will use vlan 24 unidirectionally to get to the promiscuous port. Traffic going from the promiscuous port back to the community ports will use the primary vlan, which is id 50. (Isolated ports are the same in this regard)
As soon as vlan 50 is allowed through SW3, R2 can speak to the segment gateway. Once again SW3 has no idea that vlan 50 is any sort of private vlan.

However, this does cause a problem if R3 tries to get to the default gateway. R3 belongs to vlan 24 only, and so will never receive any reply from R1:

R3#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R3 is sending out an ARP request which R1 does  receive. This can be proved with a debug arp on R1:

R1#debug arp
ARP packet debugging is on
R1#
*Dec  9 13:04:55.339: IP ARP: rcvd req src 10.0.0.3 ca02.0b28.0008, dst 10.0.0.1 FastEthernet0/0
*Dec  9 13:04:55.339: IP ARP: sent rep src 10.0.0.1 ca00.0b28.0008,
                 dst 10.0.0.3 ca02.0b28.0008 FastEthernet0/0

The problem is that the reply is getting sent over vlan 50, of which R3 is not a part of. What happens if we make R3 part of vlan 50 on SW3?

SW3(config)#int fa0/1
SW3(config-if)#swit acc vlan 50
SW3(config-if)#end
R3#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/7/16 ms

That worked. Surely SW1 would assume that incoming traffic on vlan 50 can’t be correct as traffic from the associated ports should come in on the secondary vlan? Well there is a reason this works which I’ll cover in a bit. Wireshark proves that both the request and replys were send using vlan 50. Of course R3 no longer has access to R2 or R4:

R3#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R3#ping 10.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.4, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

What if we wanted to create an SVI on all 3 of our switches so that each had a layer3 interface in the vlan? Let’s try to create int vlan 24 on SW1:

SW1(config)#int vlan 24
SW1(config-if)#
*Mar  1 02:16:48.839: %PV-6-PV_SVI_DOWN: Vlan 24's interface remains down because this vlan is a secondary vlan.
*Mar  1 02:16:48.848: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan24, changed state to down

This doesn’t work. SVI’s will only work when you use the primary vlan id:

interface Vlan50
 ip address 10.0.0.11 255.255.255.0

This SVI is not part of the community though. This means it should be able to ping R1, R3, but not R2 or R4:

SW1#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/208/1015 ms
SW1#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
SW1#ping 10.0.0.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/206/1007 ms
SW1#ping 10.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.4, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Which is what we see. In fact if I ping from SW2 to R1, the vlans in both directions should be vlan 50. Are they?

Yes they are. This is why the ping from R3 worked earlier. When the frame entered SW1’s trunk interface encapsulated with vlan 50, it has no idea this isn’t coming from a pvlan switch. It simply sees a source of vlan 50, which is valid. Yes community and isolated ports will be coming inbound with their respective secondary vlans, but a frame coming in tagged with the primary vlan is still a valid frame.

What happens if I don’t want to use R1 as the gateway. Rather I would like to use SW2’s SVI interface. At the moment none of the community ports can actually ping the SVIs as they are not promiscuous. We can’t specifically make them promiscuous, but we can map vlans to an SVI interface:
SW2:

interface Vlan50
 ip address 10.0.0.12 255.255.255.0
 private-vlan mapping 24
R2#ping 10.0.0.12

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.12, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
R4#ping 10.0.0.12

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.12, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/7/20 ms

For now, this means that SW3’s SVI should be able to ping SW1, SW2, R1, but neither R2 nor R4:

SW3#ping 10.0.0.11

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/8 ms
SW3#ping 10.0.0.12

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.12, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms
SW3#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/202/1000 ms
SW3#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
SW3#ping 10.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.4, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

There are a couple of interesting things to note. If I remove the associations in my vlan config, then any interface using pvlans associated with the wrongly configured pvlan will go down:

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#vlan 50
SW1(config-vlan)#no private-vlan association 24
SW1(config-vlan)#end
SW1#
*Mar  1 03:07:08.604: %PV-6-PV_MSG: Purged a private vlan mapping, Primary 50, Secondary 24
*Mar  1 03:07:08.621: %SYS-5-CONFIG_I: Configured from console by console
*Mar  1 03:07:09.619: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/1, changed state to down
*Mar  1 03:07:09.619: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/4, changed state to down

This can be a bit tricky to troubleshoot, as there is no reason given for the port being down:

SW1#sh int fa1/0/1
FastEthernet1/0/1 is up, line protocol is down (notconnect)

The only way to figure that out (at least I can find) is to have a look at the vlans configured, or check the switchport command:

SW1#sh int fa1/0/1 switchport | include Administrative|Operational
Administrative Mode: private-vlan promiscuous
Operational Mode: down
Administrative Trunking Encapsulation: negotiate
Administrative Native VLAN tagging: enabled
Administrative private-vlan host-association: none
Administrative private-vlan mapping: 50 (VLAN0050) 24 (VLAN0024)
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none

The administrative mode shows private-vlan, but the operational mode shows none. Not that easy to find.

Conclusions

  • It’s not essential to have a switch that suports pvlan as a transport switch between 2 pvlan switches
  • While you could add a host to the transport switch, it’s either going to be able to speak to the hosts, or the gateway. Not both at the same time
  • Just keep hosts or the routing SVI off the transport switch. Keep these limited to the pvlan switches
  • In the data-plane, the only distinguishing feature of the packet is the dot1q header. This is why you can use non pvlan switches in the middle. Your transport switches are simply forwarding based on the MAC address in the vlans configured over the trunk links
  • SVIs can only be created in the primary vlan id. A Secondary SVI id will not come up
  • SVIs are not promiscuous by default. You need to map secondary vlans to an SVI port
  • If you need to monitor pvlan traffic on a non-pvlan switch, you’ll need to monitor both the primary and secondary vlan id.