Juniper EX Virtual-Chassis notes

I’ve been deploying some EX VCs recently so this post will go over some configuration and verification commands. To start with I have two EX4200s in my lab connected via the built-in VC ports. I’m running code version 12.3R6.6

VC-1

VC Ports

When booting this type of configuration, the switches will automatically attempt to create a virtual-chassis. i.e. without configuring anything they are already in a virtual-chassis:

[email protected]> show virtual-chassis

Virtual Chassis ID: f365.a9c6.1714
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BN0208105351 ex4200-24p 128  Backup       N  1  vcp-0
                                                                 1  vcp-1
1 (FPC 1)  Prsnt    BN0208084463 ex4200-24p 128  Master*      N  0  vcp-0
                                                                 0  vcp-1

Member ID for next new member: 2 (FPC 2)

Here we see two members, both of which are present. Both have a default priority of 128. vcp-0/1 are the purpose-made VC ports. We can drill a bit deeper to see the bandwidth offered by these ports:

[email protected]> show virtual-chassis vc-port
fpc0:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           2    Up           32000        1   vcp-1
vcp-1       Dedicated           1    Up           32000        1   vcp-0

fpc1:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           2    Up           32000        0   vcp-1
vcp-1       Dedicated           1    Up           32000        0   vcp-0

VCEP Ports

VCP cables are limited in length. Juniper allows you to use the uplink module on the ex4200 to create a VC. These are called VCEP ports and requires manual configuration to do so. You can mix and match both VCP and VCEP ports at the same time. I’ve added the following switch to my topology:
VC-2
Turning these ports into VCEP ports is done in operational mode!

[email protected]> request virtual-chassis vc-port set pic-slot 1 port 0

Once this is done on both, I can verify that the vcep ports are created and the switched has joined the VC:

[email protected]> show virtual-chassis vc-port
fpc0:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           2    Up           32000        1   vcp-1
vcp-1       Dedicated           1    Up           32000        1   vcp-0
1/0         Configured         -1    Up           1000         2   vcp-255/1/0

fpc1:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           2    Up           32000        0   vcp-1
vcp-1       Dedicated           1    Up           32000        0   vcp-0

fpc2:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           1    Down         32000
vcp-1       Dedicated           2    Down         32000
1/0         Configured         -1    Up           1000         0   vcp-255/1/0

The advantage of using VCEP ports is that my ethernet cables can run as long as standard ethernet. The disadvantages include losing some front ports as well as much lower bandwidth. In the above the VCP ports give 32Gb while the VCEP port only gives 1Gb. Of course I could use a 10Gb module for the same duty.

Deterministic Master

Juniper has a great page showing how a master is elected right here. It basically goes like this:

  • Choose the member with the highest user-configured mastership priority (255 is the highest possible value). A switch with a mastership priority of 0 will always stay in the linecard role.
  • Choose the member that was master the last time the Virtual Chassis configuration booted.
  • Choose the member that has been included in the Virtual Chassis configuration for the longest period of time. (For this to be a deciding factor, there has to be a minimum time lapse of 1 minute between the power-ons of the individual interconnected member switches.)
  • Choose the member with the lowest MAC address.

I’d like to ensure SW1 is the master when possible:

{master:1}[edit]
[email protected]# set virtual-chassis member 0 mastership-priority 255

I can verify that it’s now 255:

[email protected]> show virtual-chassis

Virtual Chassis ID: f365.a9c6.1714
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BN0208105351 ex4200-24p 255  Master*      N  1  vcp-0
                                                                 2  vcp-255/1/0
                                                                 1  vcp-1
1 (FPC 1)  Prsnt    BN0208084463 ex4200-24p 128  Backup       N  0  vcp-0
                                                                 0  vcp-1
2 (FPC 2)  Prsnt    BP0213260138 ex4200-48t 128  Linecard     N  0  vcp-255/1/0

Member ID for next new member: 3 (FPC 3)

Note that this mastership is pre-emptive and you cannot change that behaviour. This can be disruptive and the current best practise is to actually ensure all devices are configured with the same priority. – Virtual Chassis Technology Best Practises

Management port

Each ex4200 has an onboard ethernet management port referred to as me0.0 in the configuration. In a VC you con configure a vme port. This is a management address that moves to whichever switch is the master. Note that packets coming into ANY members me port will get directed to the master switch via the VCP ports:

[email protected]> show configuration interfaces vme
unit 0 {
    family inet {
        address 192.168.0.0/31;
    }
}

LCD Menu

The LCD panel of switches in a VC will inform you of their current role. RE for master, BK for backup, LC for linecard.

VCCP

Juniper uses VCCP as the VC protocol. This is actually customised IS-IS and you cannot configure it. You are able to extract some information though:

[email protected]> show virtual-chassis protocol interface
fpc0:
--------------------------------------------------------------------------
IS-IS interface database:
Interface             State         Metric
internal-0/27         Up             7
internal-1/24         Up             7
vcp-0.32768           Up             7
vcp-1.32768           Up             7
vcp-255/1/0.32768     Up             240

fpc1:
--------------------------------------------------------------------------
IS-IS interface database:
Interface             State         Metric
internal-0/27         Up             7
internal-1/24         Up             7
vcp-0.32768           Up             7
vcp-1.32768           Up             7

fpc2:
--------------------------------------------------------------------------
IS-IS interface database:
Interface             State         Metric
internal-0/24         Up             7
internal-1/25         Up             7
internal-2/24         Up             7
internal-2/27         Up             7
vcp-0.32768           Down           7
vcp-1.32768           Down           7
vcp-255/1/0.32768     Up             240

Upgrading

When upgrading, all member switches will be upgraded:

[email protected]> request system software add /tmp/usb/jinstall-ex-4200-13.2X50-D19.2-domestic-signed.tgz

[Mar 13 10:55:09]: Retrieving software images. This process can take several minutes. Please be patient..

[Mar 13 10:56:19]: Retrieving version and model information from /tmp/usb/jinstall-ex-4200-13.2X50-D19.2-domestic-signed.tgz

[Mar 13 10:57:37]: Checking pending install on fpc0

[Mar 13 10:57:37]: Checking pending install on fpc2

[Mar 13 10:57:38]: Checking pending install on fpc1
[Mar 13 10:58:04]: Pushing bundle to fpc0
[Mar 13 10:58:30]: Pushing bundle to fpc2

[Mar 13 10:58:59]: Validating on fpc0

[Mar 13 10:59:13]: Validating on fpc2

[Mar 13 10:59:14]: Validating on fpc1
[Mar 13 10:59:14]: Done with validate on all virtual chassis members

fpc0:
Verify the signature of the new package
Verified jinstall-ex-4200-13.2X50-D19.2-export.tgz signed by PackageProduction_13_2_0
WARNING: A reboot is required to install the software
WARNING:     Use the 'request system reboot' command immediately

fpc2:
Verify the signature of the new package
Verified jinstall-ex-4200-13.2X50-D19.2-export.tgz signed by PackageProduction_13_2_0
WARNING: A reboot is required to install the software
WARNING:     Use the 'request system reboot' command immediately

fpc1:
Verify the signature of the new package
Verified jinstall-ex-4200-13.2X50-D19.2-export.tgz signed by PackageProduction_13_2_0
WARNING: A reboot is required to install the software
WARNING:     Use the 'request system reboot' command immediately

You can reboot individual switches, but a member in the wrong version will not join the VC stack:

[email protected]> request system reboot member 2
Reboot the system ? [yes,no] (no) yes


Rebooting fpc2
[email protected]> show virtual-chassis

Virtual Chassis ID: f365.a9c6.1714
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BN0208105351 ex4200-24p 255  Backup       N  1  vcp-0
                                                                 2  vcp-255/1/0
                                                                 1  vcp-1
1 (FPC 1)  Prsnt    BN0208084463 ex4200-24p 255  Master*      N  0  vcp-0
                                                                 0  vcp-1
2 (FPC 2)  Inactive BP0213260138 ex4200-48t 255  Linecard     N  0  vcp-255/1/0

The last switches remains inactive until all versions match. You can reboot all switches by issuing a request system reboot

Note, it’s possible to do NSSU on EX switches. Capabilities and versions matter. Juniper has a long document here showing how it works so I won’t repeat the information here.

NSR/NSB/GRES

A VC stack gives you multiple routing-engines of which one is active and a second is backup. Graceful routing engine switchover and non-stop bridging and routing are supported, but not enabled by default. It’s a simple matter to enable:

{master:1}[edit]
[email protected]# set chassis redundancy graceful-switchover
{master:1}[edit]
[email protected]# set ethernet-switching-options nonstop-bridging
{master:1}[edit]
[email protected]# set routing-options nonstop-routing

In order to verify you need to issue show system switchover on the backup RE:

[email protected]> show system switchover
fpc0:
--------------------------------------------------------------------------
Graceful switchover: On
Configuration database: Ready
Kernel database: Ready
Peer state: Steady State

GRE Tunneling

Juniper added support for GRE on EX in 12.1 onwards. Juniper requires you to convert a hardware port into a tunnel interface to do GRE encapsulation. The issue with this is that the gre interface is configured on the tunnel interface that was created. As this is tied to a single physical port, the loss of that port means the GRE tunnel goes down. Essentially this means if you lose the switch that is doing the tunnelling you lose the GRE tunnel. The only way around this is to create multiple tunnel interfaces with multiple GRE tunnels to systems that require it.
Configuring Generic Routing Encapsulation Tunneling

MPLS L3VPN without an MPLS core – MPLSoGRE

A lot of people seem to get terms wrong and assume that MPLS means L3VPN MPLS. But L3VPN is just a type of MPLS application. As is EoMPLS/ATOM/VLL/VPLS. Regular MPLS is simply MPLS. No MP-BGP involved.

In fact it’s perfectly possible to run L3VPN over a non-MPLS core. You can just run the PE to PE links over GRE. The PE routers still run MPLS, but the P routers don’t. Let’s have a look at this using the following topology:

For whatever reason R1 and R2 do not run MPLS. No LDP, RSVP, nothing. All they are running is OSPF. This is the extent of R1’s config:

interface Loopback0
 ip address 1.1.1.1 255.255.255.255
 ip ospf 1 area 0
!
interface GigabitEthernet1/0
 ip address 10.0.13.1 255.255.255.0
 ip ospf 1 area 0
!
interface GigabitEthernet2/0
 ip address 10.0.12.1 255.255.255.0
 ip ospf 1 area 0

R5 and R6 are our CPEs both running OSPF. Each has a loopback that they are advertising into OSPF. Nothing special so I’m not even going to show the config.

As always with a lot of MPLS application work, 95% of the work is done by the PE kit. Let’s start with R3:

ip vrf CUS1
 rd 3.3.3.3:100
 route-target export 200:100
 route-target import 200:100
!
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
 ip ospf 1 area 0
!
interface GigabitEthernet1/0
 ip vrf forwarding CUS1
 ip address 10.0.35.3 255.255.255.0
 ip ospf 2 area 0
!
interface GigabitEthernet2/0
 ip address 10.0.13.3 255.255.255.0
 ip ospf 1 area 0
!
router ospf 2 vrf CUS1
 redistribute bgp 200 subnets
!
router bgp 200
 no bgp default ipv4-unicast
 neighbor 4.4.4.4 remote-as 200
 neighbor 4.4.4.4 update-source Loopback0
 !
 address-family vpnv4
  neighbor 4.4.4.4 activate
  neighbor 4.4.4.4 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CUS1
  no synchronization
  redistribute ospf 2 vrf CUS1
 exit-address-family

You’ll notice that there is no LDP or RSVP running on this box either. However at this point even though route-advertisements are working, traffic from R5 to R6 will not actually work. When the traffic hit’s R1 it’ll just drop it.

What we need is a GRE tunnel from R3 to R4. We then run LDP through this tunnel and ensure that the routers use the tunnel as a next-hop to get to each others loopback. Note that we aren’t using LDP to advertise labels here. We are simply enabling it as the tunnel interface will be receiving the bottom of stack VPN label and will error if it receives this on a non-MPLs interface

interface Tunnel34
 ip unnumbered Loopback0
 mpls ip
 tunnel source GigabitEthernet2/0
 tunnel destination 10.0.24.4
!
ip route 4.4.4.4 255.255.255.255 Tunnel34

So, does traffic flow?

R5#ping 6.6.6.6 source 5.5.5.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 5.5.5.5
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 108/121/144 ms

Let’s take a look at traceroute:

R5#traceroute 6.6.6.6

Type escape sequence to abort.
Tracing the route to 6.6.6.6

  1 10.0.35.3 72 msec 84 msec 44 msec
  2 10.0.46.4 [MPLS: Label 17 Exp 0] 104 msec 152 msec 108 msec
  3 10.0.46.6 120 msec *  144 msec

You do see an MPLS hop, but that’s simple the VPN label. There is no transport label used and MPLS is not enabled anywhere in the core.

Let’s take a deeper dive into each hop to see exactly what packets are sent out each interface.

R5 sends ping to 6.6.6.6

1st link:

Here we see a regular ICMP packet. Source address 5.5.5.5 destination address 6.6.6.6

2nd link:

R3 has imposed both an MPLS VPN label on, as well as a GRE header. Notice I now have 2 IP headers, the GRE header, and the MPLS VPN label. The MPLS label of 17 will remain unchanged end to end, as it’s just a VPN label, not a transport label.

3rd Link:

Very similar to the last capture. 2 IP headers, GRE header, and the unchanged MPLS VPN label

4th Link:

Again exactly the same. Notice that if I was using MPLS as the transport, the outer label would’ve been popped now for PHP. But as mentioned before, this is a VPN label so goes through unchanged.

5th Link:

Finally the last hop. The GRE, outer IP label, as well as VPN label have all been removed and the packet is sent on it’s way to R6.

So to simplify, this is how the packets gets through this network: