Category Archives: Brocade

VC mode 4/5 – Brocade VLL/VPLS

When doing L2VPN, there are two different ways in which packets are encapsulated over the MPLS link. I’ll take the following quote from RFC4488 directly to spare myself from explaining it all:

4.1. Ethernet Tagged Mode

The Ethernet frame will be encapsulated according to the procedures
defined later in this document for tagged mode. It should be noted
that if the VLAN identifier is modified by the egress PE, the
Ethernet spanning tree protocol might fail to work properly. If this
issue is of significance, the VLAN identifier MUST be selected in
such a way that it matches on the attachment circuits at both ends of
the PW.

If the PE detects a failure on the Ethernet physical port, or the
port is administratively disabled, it MUST send a PW status
notification message for all PWs associated with the port.

This mode uses service-delimiting tags to map input Ethernet frames
to respective PWs and corresponds to PW type 0×0004 “Ethernet Tagged
Mode” [IANA].

4.2. Ethernet Raw Mode

The Ethernet frame will be encapsulated according to the procedures
defined later in this document for raw mode. If the PE detects a
failure on the Ethernet input port, or the port is administratively
disabled, the PE MUST send an appropriate PW status notification
message to the corresponding remote PE.

In this mode, all Ethernet frames received on the attachment circuit
of PE1 will be transmitted to PE2 on a single PW. This service
corresponds to PW type 0×0005 “Ethernet” [IANA].

4.4.1. Raw Mode vs. Tagged Mode

When the PE receives an Ethernet frame, and the frame has a VLAN tag,
we can distinguish two cases:

1. The tag is service-delimiting. This means that the tag was
placed on the frame by some piece of service provider-operated
equipment, and the tag is used by the service provider to
distinguish the traffic. For example, LANs from different
customers might be attached to the same service provider
switch, which applies VLAN tags to distinguish one customer’s
traffic from another’s, and then forwards the frames to the PE.

2. The tag is not service-delimiting. This means that the tag was
placed in the frame by a piece of customer equipment, and is
not meaningful to the PE.

Whether or not the tag is service-delimiting is determined by local
configuration on the PE.

If an Ethernet PW is operating in raw mode, service-delimiting tags
are NEVER sent over the PW. If a service-delimiting tag is present
when the frame is received from the attachment circuit by the PE, it
MUST be stripped (by the NSP) from the frame before the frame is sent
to the PW.

If an Ethernet PW is operating in tagged mode, every frame sent on
the PW MUST have a service-delimiting VLAN tag. If the frame as
received by the PE from the attachment circuit does not have a
service-delimiting VLAN tag, the PE must prepend the frame with a
dummy VLAN tag before sending the frame on the PW. This is the
default operating mode. This is the only REQUIRED mode.

In both modes, non-service-delimiting tags are passed transparently
across the PW as part of the payload. It should be noted that a
single Ethernet packet may contain more than one tag. At most, one
of these tags may be service-delimiting. In any case, the NSP
function may only inspect the outermost tag for the purpose of
adapting the Ethernet frame to the pseudowire.

In both modes, the service-delimiting tag values have only local
significance, i.e., are meaningful only at a particular PE-CE
interface. When tagged mode is used, the PE that receives a frame
from the PW may rewrite the tag value, or may strip the tag entirely,
or may leave the tag unchanged, depending on its configuration. When
raw mode is used, the PE that receives a frame may or may not need to
add a service-delimiting tag before transmitting the frame on the
attachment circuit; however, it MUST not rewrite or remove any tags
that are already present.

Quite a mouthful. In my testing of both modes on a Brocade Netiron XMR I’m seeing slightly different results. Let’s take a look. For this post I’ll use the following topology:
VC 4 5 VC mode 4/5    Brocade VLL/VPLS
I have three sites and on those sites the customer has a router. We also have a switch on site which will be taking the customer’s tagged frames and impose another vlan tag on top for QinQ. That QinQ tag goes over the attachment circuit to the PE router.

On the PE router we’ll be putting it all in the same VPLS. I have mirrored a core port and will show exactly what the packet format is when traffic goes between these three routers. One thing to note is that vc-modes have to match. Your VPLS peers will not come up if they don’t. I am using LDP-signhalled VPLS over RSVP for this example.

VPLS VC-MODE 4

I’ll be showing the config of PE3. The others are near-idential. Only the outer vlan tag is set to the local service-deliminating vlan id.

router mpls
 vpls VC_TEST 9000
  vc-mode tagged
  vpls-peer 10.11.224.60 10.11.224.61
  vlan 500
   tagged ethe 1/13

We can verify the session is both up and operating at the right mode:

[email protected]# sh mpls vpls id 9000
VPLS VC_TEST, Id 9000, Max mac entries: 8192
 Total vlans: 1, Tagged ports: 1 (1 Up), Untagged ports 0 (0 Up)
 IFL-ID: n/a
  Vlan 500
   L2 Protocol: NONE
   Tagged: ethe 1/13
 VC-Mode: Tagged
 Total VPLS peers: 2 (2 Operational)
 Peer address: 10.11.224.60, State: Operational, Uptime: 3 hr 28 min
  Tnnl in use: tnl0(1097)[RSVP]    Peer Index:0
  Local VC lbl: 983040, Remote VC lbl: 983060
  Local VC MTU: 8974, Remote VC MTU: 8974
  Local VC-Type: Ethernet Tagged(0x04), Remote VC-Type: Ethernet Tagged(0x04)
 Peer address: 10.11.224.61, State: Operational, Uptime: 3 hr 28 min
  Tnnl in use: tnl1(1104)[RSVP]    Peer Index:1
  Local VC lbl: 983041, Remote VC lbl: 983100
  Local VC MTU: 8974, Remote VC MTU: 8974
  Local VC-Type: Ethernet Tagged(0x04), Remote VC-Type: Ethernet Tagged(0x04)
 CPU-Protection: OFF
 Local Switching: Enabled
 Extended Counter: ON
 Multicast Snooping: Disabled

A quick overview of how the QinQ SW3 is configured. Again S1 and S2 will be the same except for their vlan-id:

interface GigabitEthernet1/0/22
 switchport access vlan 500
 switchport trunk encapsulation dot1q
 switchport mode dot1q-tunnel
!
interface GigabitEthernet1/0/23
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 500
 switchport mode trunk

CPE3 has a subinterface configured sending tagged traffic:

interface FastEthernet0/0.10
 encapsulation dot1Q 10
 ip address 10.10.10.3 255.255.255.0
 ip ospf 1 area 0

Let’s see if R3 can ping R2′s subinterface:

R3#ping 10.10.10.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

Yes it can. Let’s refer back to the RFC and see if we can see what packet format should be going over the MPLS core, before we look at a packet itself.

If an Ethernet PW is operating in tagged mode, every frame sent on
the PW MUST have a service-delimiting VLAN tag. If the frame as
received by the PE from the attachment circuit does not have a
service-delimiting VLAN tag, the PE must prepend the frame with a
dummy VLAN tag before sending the frame on the PW. This is the
default operating mode. This is the only REQUIRED mode.

In both modes, the service-delimiting tag values have only local
significance, i.e., are meaningful only at a particular PE-CE
interface. When tagged mode is used, the PE that receives a frame
from the PW may rewrite the tag value, or may strip the tag entirely,
or may leave the tag unchanged, depending on its configuration.

On PE3 above, vlan 500 is the service-deliminating vlan tag. On PE2 it’s vlan 700. The second quote can very easily be understood differently by different vendors. The RFC itself is saying it could be one of three things!

Either way let’s ping from R3 to R2 and capture that in wireshark and see what a single packet looks like:
vc4 ping VC mode 4/5    Brocade VLL/VPLS
Both the service and customer vlan are sent across the link. Communication is still working so PE2 swaps the service-deliminating tag with it’s own service tag (vlan 600) outbound. The reverse is true when a packet comes in from R2 back to R3:
vc4 ping reply1 VC mode 4/5    Brocade VLL/VPLS

There is an important note here. The Brocade Netiron will not allow me to specify ‘any’ coming into a port to be put into a VPLS. i.e. If I want to carry cvlans across the core I need to pop another tag on top before it gets into a VPLS instance.

VPLS VC-MODE 5

I’ve changed all my PE routers to use vc mode 5 for this VPLS instance:

[email protected]#sh mpls vpls id 9000
VPLS VC_TEST, Id 9000, Max mac entries: 8192
 Total vlans: 1, Tagged ports: 1 (1 Up), Untagged ports 0 (0 Up)
 IFL-ID: n/a
  Vlan 500
   L2 Protocol: NONE
   Tagged: ethe 1/13
 VC-Mode: Raw
 Total VPLS peers: 2 (2 Operational)
 Peer address: 10.11.224.60, State: Operational, Uptime: 1 min
  Tnnl in use: tnl0(1097)[RSVP]    Peer Index:0
  Local VC lbl: 983040, Remote VC lbl: 983062
  Local VC MTU: 8974, Remote VC MTU: 8974
  Local VC-Type: Ethernet(0x05), Remote VC-Type: Ethernet(0x05)
 Peer address: 10.11.224.61, State: Operational, Uptime: 1 min
  Tnnl in use: tnl1(1104)[RSVP]    Peer Index:1
  Local VC lbl: 983041, Remote VC lbl: 983106
  Local VC MTU: 8974, Remote VC MTU: 8974
  Local VC-Type: Ethernet(0x05), Remote VC-Type: Ethernet(0x05)
 CPU-Protection: OFF
 Local Switching: Enabled
 Extended Counter: ON
 Multicast Snooping: Disabled

I’ll ping R2 from R3 again and see what we see this time:
vc5 ping1 VC mode 4/5    Brocade VLL/VPLS
This time the service-deliminating tag has been stripped. Only the inner-vlan is being sent across the MPLS LSP.

VLL VC-MODE 4

VLLs are different in that they are point-to-point. The Netiron will allow me to tag any vlan without having to push another vlan tag on top first.

I’ve changed the switches to not do QinQ anymore. They are merely passing the tagged vlans from the CPE directly to the core interfaces.

I’ve set up vc-mode 4 and ping across. What do we expect? As there is no service-deliminating vlan tag, the RFC states the device should add a dummy vlan tag. Let’s see if it does:
vc4 ping vll VC mode 4/5    Brocade VLL/VPLS

The Brocade has added vlan 1 on top of vlan 10. This is what we expect.

VLL VC-MODE 5

I’ve now changed it to vc-mode 5. This should mean that no dummy vlan is sent, but we should still see our vlan 10:
vc5 ping vll VC mode 4/5    Brocade VLL/VPLS

Which is exactly what we see.

Why is this important?

Knowing how this works is mainly due to interop. I’ll be doing the same tests above on both Cisco and juniper kit to see the differences. The main thing to remember is that VC mode 4 and VC mode 5 peers will NOT come up.

Annoyingly, on the Brocade the VPLS defaults to vc-type 5 while the VLL defaults to vc-type 4.

Even if they do match, we can see from the RFC above that it’s not at all easy to determine exactly what the router is supposed to do. This can cause you to run into all manner of interop issues.

I’ve already read out there that vc-type 5 simply means untagged frames. This is not true. Both vc-mode 4 and 5 can send cvlan tags across the MPLS cloud.

Odd behaviour of down interfaces and OSPF on Brocade Netiron

I ran across an odd problem on a Brocade XMR recently. I had created a static route redistributed into OSPF but ended up with a traffic loop. I managed to figure out why this was happening, but the behaviour I see should not be happening. Note I’ve changed the first octect to 10. in all the below addresses.

Consider this network:
OSPF issue Brocade Odd behaviour of down interfaces and OSPF on Brocade Netiron
R1, R2, and R3 are on OSPF area 0. R4 has the prefix 10.46.204.0/29 sitting behind it. R3 has a static route to this prefix via R4 and that route is getting redistributed into OSPF. R4 has a static default to R3.

My PC is connected to the core network, so let’s do a traceroute:

H:\>tracert -d 10.46.204.1

Tracing route to 10.46.204.1 over a maximum of 30 hops

  1    <1 ms    <1 ms     5 ms  10.196.226.126
  2    <1 ms    <1 ms    <1 ms  10.255.1.1
  3     1 ms     1 ms     1 ms  10.255.0.20
  4     1 ms     1 ms     1 ms  10.71.16.198
  5     1 ms     1 ms     1 ms  10.248.31.29
  6     1 ms     1 ms     1 ms  10.248.31.10
  7     1 ms     1 ms     1 ms  10.248.31.2
  8     8 ms     1 ms     1 ms  10.248.31.61
  9     2 ms     1 ms     1 ms  10.248.31.17
 10     2 ms     1 ms     1 ms  10.248.31.16
 11     2 ms     2 ms     2 ms  10.248.31.17
 12     2 ms    12 ms     2 ms  10.248.31.16
 13     2 ms     2 ms     2 ms  10.248.31.17
 14     2 ms    15 ms     4 ms  10.248.31.16
 15     3 ms     3 ms    14 ms  10.248.31.17
 16    14 ms     4 ms     3 ms  10.248.31.16
 17     3 ms    14 ms     3 ms  10.248.31.17
 18     4 ms     3 ms    14 ms  10.248.31.16
 19    14 ms     4 ms     3 ms  10.248.31.17
 20     4 ms    13 ms     5 ms  10.248.31.16
 21     5 ms     4 ms    14 ms  10.248.31.17
 22    14 ms     5 ms     4 ms  10.248.31.16
 23    15 ms     5 ms     4 ms  10.248.31.17
 24     4 ms    14 ms     6 ms  10.248.31.16
 25     6 ms    17 ms     5 ms  10.248.31.17
 26     6 ms    16 ms     6 ms  10.248.31.16
 27     6 ms    16 ms     6 ms  10.248.31.17
 28     6 ms    16 ms     6 ms  10.248.31.16
 29     7 ms    17 ms     6 ms  10.248.31.17
 30     6 ms    16 ms     7 ms  10.248.31.16

Trace complete.

Clearly we have a problem.

Troubleshooting

Let's look at all three router's view of that route.

R1:

SSH@R1#sh ip route 10.46.204.1
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.46.204.0/29     10.248.31.17    ve 62         110/10        O2   34m19s -

R2:

SSH@R2#sh ip route 10.46.204.1
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       0.0.0.0/0          10.248.31.16    ve 62         110/210       O1   1h7m   -

R3:

SSH@R3#sh ip route 10.46.204.1
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.46.204.0/29     10.248.31.22    eth 2/1       1/1           S    1d2h   -

R2 is clearly wrong. It doesn't have the route in it's table and therefore is following the default route back to R1.

Let's check the OSPF LSA for this prefix:

SSH@R2#sh ip ospf database external-link-state link-state-id 10.46.204.0
Ospf ext link-state by link-state ID 10.46.204.0 are in the following:

Type-5 AS External Link States

Index Age  LS ID           Router          Netmask  Metric   Flag Fwd Address   SyncState
444   1387 10.46.204.0     10.196.224.61  fffffff8 0000000a 0000 10.248.31.22   Done
  LSA Header:  age: 1387, options: 0x02, seq-nbr: 0x80000067, length: 36
  NetworkMask: 255.255.255.248
  TOS 0:  metric_type: 2, metric: 10
          forwarding_address: 10.248.31.22
          external_route_tag: 0

I don't see anything wrong here. The metric is valid. The forwarding address is also reachable. Let's check the route to the forwarding address on R2:

SSH@R2#sh ip route 10.248.31.22
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.248.31.22/31    10.248.31.21    ve 5          110/200       O    1h59m  -

It's reachable over the directly connected interface which is valid. So what's going on?

After taking a look around I saw that 10.248.31.22 was configured on a local interface on R2. However that interface was down. If an interface is down it should not consider that to be an active route. In fact the show ip route command above the router knows that 10.2148.31.22 is a remote address.

This is the local interface on R2 in question:

SSH@R2#sh run int eth 2/1
interface ethernet 2/1
 port-name R3 eth2/1
 enable
 route-only
 ip ospf area 0
 ip ospf network point-to-point
 ip address 10.248.31.22/31

SSH@R2#sh int eth 2/1 | include protocol
GigabitEthernet2/1 is down, line protocol is down

I tested the exact same config above in both Junos and IOS and both systems do not have the same problem. It doesn't matter if the forwrding address is a locally configured address as long as that address is not active

OSPF forwarding address

Why does OSPF set the forwarding address anyway? Most type5 LSAs wil have a forwarding address of 0.0.0.0 which effectively means to forward packets to the destination to the ASBR that originated the Type5. There are some very specific circumstances as to when an ASBR sets the forwarding address to a non-0.0.0.0 value which can be found on this site http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009405a.shtml?referring_site=bodynav

At least on IOS, this is what Cisco is saying:

The forwarding address is set to 0.0.0.0 if the ASBR redistributes routes and OSPF is not enabled on the next hop interface for those routes. This is true in the figure if Router 1 does not have OSPF enabled on the Ethernet interface.

These conditions set the forwarding address field to a non-zero address:

OSPF is enabled on the ASBR's next hop interface AND

ASBR's next hop interface is non-passive under OSPF AND

ASBR's next hop interface is not point-to-point AND

ASBR's next hop interface is not point-to-multipoint AND

ASBR's next hop interface address falls under the network range specified in the router ospf command.

Any other conditions besides these set the forwarding address to 0.0.0.0.

R3's link to R4 in my topology does indeed have ospf enabled on the link, but I am running passive. Brocade doesn't seem to have thorough documentation on this so it's behaviour is slightly different to Cisco. Let's remove OSPF from the interface though and check the type 5 LSA again:

SSH@R3#conf t
SSH@R3(config)#int eth 2/1
SSH@R3(config-if-e1000-2/1)#no ip ospf area 0
SSH@R2#sh ip ospf database external-link-state link-state-id 10.46.204.0
Ospf ext link-state by link-state ID 10.46.204.0 are in the following:

Type-5 AS External Link States

Index Age  LS ID           Router          Netmask  Metric   Flag Fwd Address   SyncState
444   15   10.46.204.0     10.196.224.61  fffffff8 0000000a 0000 0.0.0.0        Done
  LSA Header:  age: 15, options: 0x02, seq-nbr: 0x80000069, length: 36
  NetworkMask: 255.255.255.248
  TOS 0:  metric_type: 2, metric: 10
          forwarding_address: 0.0.0.0
          external_route_tag: 0

The forward address has changed, which should mean the route is installed:

SSH@R2#sh ip route 10.46.204.0
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.46.204.0/29     10.248.31.21    ve 5          110/10        O2   0m48s  -

Conclusion

What matters is the running state of a device. Whether a forwarding address is configured on an interface or not is irrelevant if the interface is not active. The interface configured with an identical address is not up. I even hard shut the interface but that made no difference.

Either way, it's an interesting quirk to know.

Always check the forwarding table – IOS, Junos, Netiron

Most bigger routers these days use a distributed system. One of the bigger differences is the separation on the control and forwarding plane. When troubleshooting or verifying it’s essential to view both. Too many engineers simply show the control plane output. While these should match, they don’t always. Note that the forwarding table doesn’t have to be distributed to different hardware.

For the examples below I’ll simply be viewing a default route learned through OSPF. The router in question will always have two equal costs out of the network so you would expect to see two routes.

IOS

First we check the routing table:

R1#sh ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
  Known via "ospf 1", distance 110, metric 1, candidate default path
  Tag 1, type extern 2, forward metric 2
  Last update from 10.0.12.2 on GigabitEthernet2/0, 00:00:33 ago
  Routing Descriptor Blocks:
  * 10.0.13.3, from 10.0.24.4, 00:00:33 ago, via GigabitEthernet1/0
      Route metric is 1, traffic share count is 1
      Route tag 1
    10.0.12.2, from 10.0.24.4, 00:00:33 ago, via GigabitEthernet2/0
      Route metric is 1, traffic share count is 1
      Route tag 1

Two ways to get to 0.0.0.0 – What does the forwarding table show? For this I’ll choose an IP that would follow the default route:

R1#sh ip cef 4.2.2.1
0.0.0.0/0
  nexthop 10.0.12.2 GigabitEthernet2/0
  nexthop 10.0.13.3 GigabitEthernet1/0

Both control plane and data plane agree.

Netiron

Routing table:

[email protected]#sh ip route 0.0.0.0
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       0.0.0.0/0          10.0.0.1        eth 15/1      110/110       O1   1h22m  -
        0.0.0.0/0          10.0.0.2        eth 16/1      110/110       O1   1h22m  -

In order to show the forwarding table you use show route x.x.x.x detail. Note that I’m executing this command on an XMR16 and I will get the forwarding entry for every single module. I’m going to only show the output for the first module:

[email protected]#sh ip route 4.2.2.1 detail
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       0.0.0.0/0          10.0.0.1        eth 15/1      110/110       O1   1h24m  -
        0.0.0.0/0          10.0.0.1        eth 16/1      110/110       O1   1h24m  -
        Nexthop Entry ID:65540, Paths: 2, Ref_Count:707/712

D:Dynamic  P:Permanent  F:Forward  U:Us  C:Connected Network E: ESI VLAN
W:Wait ARP  I:ICMP Deny  K:Drop  R:Fragment  S:Snap Encap N:CamInvalid

Module S1:
      IP Address         Next Hop        MAC              Type  Port  Vlan  Pri
      0.0.0.0/0          10.0.0.1       0012.f293.a802   PF    16/1   1     0

      OutgoingIf  ArpIndex PPCR_ID   CamLevel   Parent  DontAge Index Is_trunk
      eth 16/1    5        1:1       31              0               0 0

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000               0     0x0005ffff (L3, right)  2

        CAM Entry Flag: 00000001H
        PPCR : 1:1 CIDX: 0x0005ffff (L3, right) (IP_NETWORK: 0x56000)

        pram_index_programmed: ppcr[0] 0x0000014c

The output is a little cryptic so I’ll highlight the important bits. First the paths show as two:

Nexthop Entry ID:65540, Paths: 2, Ref_Count:707/712

But the actual next-hop is only showing a single:

     0.0.0.0/0          10.0.0.1       0012.f293.a802   PF    16/1   1     0

This is a cosmetic error. The most important bit is here:

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000               0     0x0005ffff (L3, right)  2

The hardware path count is two, which is what we expect.

Junos

Finally Junos. First up we look at the route table:

lab@Vega_SRX6> show route 0.0.0.0

inet.0: 32 destinations, 32 routes (29 active, 3 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[OSPF/150] 00:00:12, metric 0, tag 0
                      to 172.30.0.17 via ge-0/0/4.126
                    > to 172.30.0.89 via ge-0/0/4.146

Two routes, our forwarding table should match?

lab@Vega_SRX6> show route forwarding-table destination 4.2.2.1
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            user     1 0:c:29:86:21:55    ucst   584    13 ge-0/0/4.146
default            perm     0                    rjct    36     5

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   534     1

Well no, it doesn’t. While the route table shows two routes, only one is being used by the forwarding table. Junos will not install multiple next-hops into the forwarding-table unless you tell it to:

lab@Vega_SRX6> show configuration policy-options policy-statement BALANCE
then {
    load-balance per-packet;
}
lab@Vega_SRX6> show configuration routing-options forwarding-table
export BALANCE;

Let’s check again:

lab@Vega_SRX6> show route forwarding-table destination 4.2.2.1
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            user     1                    ulst 262142     7
                              0:c:29:25:21:57    ucst   612    11 ge-0/0/4.126
                              0:c:29:86:21:55    ucst   584     9 ge-0/0/4.146
default            perm     0                    rjct    36     5

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   534     1

This time we have both in the forwarding table. Note that while the policy states load-blance per-packet, it’s actually doing per-flow load-sharing.

Conclusion

I have seen routers disagree as to what they think they are doing compared to what they are doing. You need to check both tables above to note what both are doing. This could help immensely when a router is dropping packets it’s supposed to be forwarding, due to your FIB having no entry. I might write a bit on this as I’ve seen it happen more than once.

EDIT – 04/11/13

I’ve since found another way to verify this on the Brocades. If you rconsole onto the line card itself you can see a bit more:

SSH@XMR16#rconsole 1
Remote connection to LP slot 1 established
Press CTRL-X or type 'exit' to disconnect it
LP-1>en
LP-1#sh ip network 0.0.0.0
D:Dynamic  P:Permanent  F:Forward  U:Us  C:Connected Network
W:Wait ARP  I:ICMP Deny  K:Drop  R:Fragment  S:Snap Encap N:CamInvalid
      IP Address         Next Hop        MAC              Type  Port  Vlan  Pri
      0.0.0.0/0          10.0.0.1*    0012.f293.ad02   PF    15/1*  1     0

      OutgoingIf  ArpIndex PPCR_ID   CamLevel   Parent  DontAge Index Is_trunk
      eth 15/1    4        1:1       31              0               0 0

      U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
      0000e000  0x00000001   0     0x0005ffff (L3, right)  2

        CAM Entry Flag: 00000001H
        PPCR : 1:1 CIDX: 0x0005ffff (L3, right) (IP_NETWORK: 0x56000)

        pram_index_programmed: ppcr[0] 0x0000014c
use_index: 0
IP-nh-Pram 0: 0x2ebeec10, ref_count 1
n_paths = 2, type = ECMP_PHY_VE, is_default  = 1, vrf_index = 0
  path[0]: FORWARD, out_intf eth 15/1, nh 10.0.0.1, out_port 15/1, is_trunk 0
  path[1]: FORWARD, out_intf eth 16/1, nh 10.0.0.5, out_port 16/1, is_trunk 0
Pram info: alloc_count 2 use_count 2
  pram[0]: idx 0, pram_idx[0] 0x0000014c
  pram[1]: idx 1, pram_idx[0] 0x0000014d

The top half still shows a single port, but down it shows this:

n_paths = 2, type = ECMP_PHY_VE, is_default  = 1, vrf_index = 0
  path[0]: FORWARD, out_intf eth 15/1, nh 10.0.0.1, out_port 15/1, is_trunk 0
  path[1]: FORWARD, out_intf eth 16/1, nh 10.0.0.5, out_port 16/1, is_trunk 0

n paths is the number of paths. The router is also doing ECMP. It then shows which ports outbound it’ll send traffic.

On a route with only a single hop the bit above are shown as so:

n_paths = 1, type = NON_ECMP, is_default  = 0, vrf_index = 0
  path[0]: FORWARD, out_intf eth 1/20, nh 10.0.0.8, out_port 1/20, is_trunk 0

Moving routes between a VRF and the global (default) RIB – Part 2 – Brocade Netiron

Part 1 – Cisco IOS
Part 2 – Brocade Netiron
Part 3 – Juniper Junos

Part 2 of this series will focus on Brocade’s Netiron implementation. I have a few XMRs here to test with.

Slightly different set-up here for this lab:
Drawing9 31 Moving routes between a VRF and the global (default) RIB – Part 2 – Brocade Netiron

I have firewalls acting as my CE kit this time. FW2 has a loopback address of 22.22.22.22/32 that needs to be able to speak to 172.24.0.97, FW1′s IP address that sites in the global IGP table.

Copying global routes to VRF

Brocade simply calls this feature Inter-VRF routing. Though it does allow you to leak routes between a customer vrf and the ‘default’ vrf. The default vrf is in fact the global RIB. The nice thing about Brocade’s implementation is that routes do not need to be BGP routes. They can be static, IGP, or BGP learned. This gives you a lot more flexibility when it comes to doing this sort of thing. To test I am going to run OSPF as the CE-PE routing protocol between R6 and FW2.

Let’s first check the VRF table on R6 first to ensure we don’t see 172.24.0.97:

SSH@R6#sh ip route vrf CUS1
Total number of IP routes: 2
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.0.222.0/24      DIRECT          ve 22         0/0           D    1m23s  -
2       22.22.22.22/32     10.0.222.22     ve 22         110/2         O    1m3s   -

Only the directly connected link and FW’s loopback are in the table as expected. I’ll create a prefix list, route-map, and then import that into the vrf:

ip prefix-list ISP_SERVER seq 5 permit 172.24.0.0/24
!
route-map MAP_ISP_SERVER permit 10
 match ip address prefix-list ISP_SERVER
!
vrf CUS1
 address-family ipv4
   import routes vrf default-vrf route-map MAP_ISP_SERVER

One thing to note is that the global RIB is named default-vrf – all lowercase. Context-sensitive help doesn’t show this. You need to either know this, or get it out the configuration guide.

Is that route now in the vrf?

SSH@R6#sh ip route vrf CUS1 172.24.0.97
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       172.24.0.0/24      10.0.16.1       ve 16         110/4         O    1m56s  default-vrf

It is. It’s nice enough to tell us the source vrf in the same command. You’ll notice the source protocol is still OSPF. Even though the PE is running OSPF with the CE device, that route is not advertised. The route, even though it’s in the vrf table on the PE, is still from another process. You’ll need to redistribute from the global process into the customers vrf process:

router ospf vrf CUS1
 area 0
 redistribute ospf

Note that this redistribute command will only redistribute routes that are already leaked into the local vrf table on the PE device.

FW2 should now have that route as an external OSPF route:

FW2-> get route ip 172.24.0.97
 Dest for 172.24.0.97
--------------------------------------------------------------------------------------
trust-vr       : => 172.24.0.0/24 (id=14) via 10.0.222.2 (vr: trust-vr)
                    Interface ethernet1/1.1 , metric 10

Copying VRf routes to global

With Brocade, the reverse is just as easy. One of the main differences is that instead of exporting routes out of the vrf, we simply import directly into the global RIB:

ip prefix-list CUS1_RANGE seq 5 permit 22.22.22.22/32
!
route-map MAP_CUS1_RANGE permit 10
 match ip address prefix-list CUS1_RANGE
!
ip import routes vrf CUS1 route-map MAP_CUS1_RANGE

If we now check the global table for FW2′s loopback:

SSH@R6#show ip route 22.22.22.22
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       22.22.22.22/32     10.0.222.22     ve 22         110/2         O    1m7s   CUS1

There it is. Once again it’s an OSPF route and it also shows us the source vrf. We also need to redistribute between processes in order for those vrf OSPF routes to get into the global OSPF table:

router ospf
 area 0
 redistribute ospf

Unfortunately the Netiron doesn’t allow me to specify which process to redistribute. The above command would redistribute ALL OSPF routes that are already leaked into the local table. 95% of the time that’s what I would want to do, but it’s not the most flexible solution. I could push the routes I wanted through a route-map if I wanted.

Verification

FW2 should now be able to ping 172.24.0.97, but only if it sources those pings from 22.22.22.22/32:

FW2-> ping 172.24.0.97
Type escape sequence to abort

Sending 5, 100-byte ICMP Echos to 172.24.0.97, timeout is 1 seconds
.....
Success Rate is 0 percent (0/5)
FW2-> ping 172.24.0.97 from  loopback.1
Type escape sequence to abort

Sending 5, 100-byte ICMP Echos to 172.24.0.97, timeout is 1 seconds from loopback.1
!!!!!
Success Rate is 100 percent (5/5), round-trip time min/avg/max=2/2/3 ms

Which is exactly what we see.

Final Notes

Brocade allows the use of both IPv4 and IPv6 with all the above commands. Simply substitute ip for ipv6 and you’re good to go.

VPLS Interop – Junos and Netiron – Part 1 of 2

VPLS is a LAN emulation service that can be run over an MPLS backbone. I do not have a ton of free devices on hand and so my acual lab will only consist of two CE devices even though it supports more than two. VPLS runs over an MPLS core constructed by RSVP-TE or LDP LSP tunnels. VPLS itself requires either LDP (RFC 4762) or BGP (RFC 4761) as the VC singnalling protocol. Note that when LDP is used, this is a targeted LDP session and has nothing to do with the protocol you use for the LSPs itself. To prove this I will be using RSVP-TE LSP tunnels with LDP and BGP on top.

Part one of this series will use LDP as the VC signalling protocol. Part two will use BGP.

I was originally going to include Cisco’s IOS as well, but you need a 6500 or 7600 and I don’t have a spare. The 7200 platform does not support VPLS.

This the the topology I’m going to use:
VPLS Juniper Brocade1 VPLS Interop   Junos and Netiron   Part 1 of 2
R2 and R3 are P routers. R1 is a Junos PE and R8 is a Netiron PE. R6 and R10 are both CPEs

The actual P router config is standard RSVP-TE which I have covered extensively on this site already. The CPE’s have both been configured to be in the same subnet (10.0.0.0/24)

PE Config

Junos

darreno> show configuration interfaces fe-0/0/2
encapsulation ethernet-vpls;
unit 0 {
    family vpls;
}

darreno> show configuration protocols ldp
interface lo0.1;

darreno> show configuration routing-instances
MELLOWD-VPLS {
    instance-type vpls;
    interface fe-0/0/2.0;
    protocols {
        vpls {
            vpls-id 150;
            neighbor 8.8.8.8;
        }
    }
}

On Junos you need to enable LDP on the loopback interface, even when running RSVP. You need to ensure VPLS encapsulation is on the physical interface. Finally you need to create the VPLS instance and tie this all together. You specify neighbours under the process. This actually creates the targeted LDP sessions (i.e. there is no need to specify a T-LDP session separately)

PE Config

Netiron

Brocade’s Netiron config is actually very simple compared to the above

router mpls
 mpls-interface ve2

  vpls MELLOWD-VPLS 150
  vpls-peer 1.1.1.1
  vpls-mtu 1500
  vlan 550
   tagged ethe 3/19

That’s it. When you enable rsvp on an interface, and then set up a VPLS with a neighbour, it automatically sets up a T-LDP session with it’s peer. Under the VLAN I’ve said tagged eth 3/19 as it’ll be receiving tagged frames from the CPE router.

Verification

Control Plane

Let’s check to see if the session is actually up:

darreno> show vpls connections
Layer-2 VPN connections:

Legend for connection status (St)
[deleted for brevity]

Legend for interface status
Up -- operational
Dn -- down

Instance: MELLOWD-VPLS
  VPLS-id: 150
    Neighbor                  Type  St     Time last up          # Up trans
    8.8.8.8(vpls-id 150)      rmt   Up     Mar  8 13:52:45 2013           1
      Remote PE: 8.8.8.8, Negotiated control-word: No
      Incoming label: 800000, Outgoing label: 983040
      Negotiated PW status TLV: No
      Local interface: vt-0/2/0.1048579, Status: Up, Encapsulation: ETHERNET
        Description: Intf - vpls MELLOWD-VPLS neighbor 8.8.8.8 vpls-id 150
SSH@XMR_R8# show mpls vpls id 150
VPLS MELLOWD-VPLS, Id 150, Max mac entries: 8192
 Routing Interface Id 150
 Total vlans: 1, Tagged ports: 1 (1 Up), Untagged ports 0 (0 Up)
 IFL-ID: n/a
  Vlan 550
   L2 Protocol: NONE
   Tagged: ethe 3/19
 VC-Mode: Raw
 Total VPLS peers: 1 (1 Operational)
 Peer address: 1.1.1.1, State: Operational, Uptime: 55 min
  Tnnl in use: tnl2(299984)[RSVP]    Peer Index:0
  Local VC lbl: 983040, Remote VC lbl: 800000
  Local VC MTU: 1500, Remote VC MTU: 1500
  Local VC-Type: Ethernet(0x05), Remote VC-Type: Ethernet(0x05)
 CPU-Protection: OFF
 Local Switching: Enabled
 Extended Counter: ON
 Multicast Snooping: Disabled

Both sessions are up. Both see the others VC labels. Can we show that LDP is actually used?

darreno> show ldp neighbor
Address            Interface          Label space ID         Hold time
8.8.8.8            lo0.1              8.8.8.8:0                36
SSH@XMR_R8#sh mpls ldp neighbor
 Number of link neighbors: 0
 Number of targeted neighbors: 1

Nbr Transport       Interface         Nbr LDP ID          Max Hold  Time Left
1.1.1.1             (targeted)        1.1.1.1:0           45        42

Verification

Data Plane

So can our CPE’s ping each others?

R10#ping 10.0.0.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/12 ms
USER6:R6> ping 10.0.0.10 rapid count 5
PING 10.0.0.10 (10.0.0.10): 56 data bytes
!!!!!
--- 10.0.0.10 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.368/1.460/1.755/0.148 ms

No problems there. It’s always handy to check MAC addresses learned over the VPLS. We can check like so:

darreno> show route forwarding-table family vpls
Routing table: MELLOWD-VPLS.vpls
VPLS:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct  1007     1
fe-0/0/2.0         user     0                    comp  1150     2
vt-0/2/0.1048579   user     0                    comp  1258     2
00:12:f2:93:a9:00/48 dynm     0                  indr 262142     5
                              10.0.4.14         Push 983040, Push 299920(top)  1233     2 ae1.13
00:13:19:22:8f:91/48 dynm     0                  ucst  1146     4 fe-0/0/2.0
00:90:69:a5:13:f1/48 dynm     0                  ucst  1146     4 fe-0/0/2.0
8c:b6:4f:63:46:b8/48 dynm     0                  indr 262142     5
                              10.0.4.14         Push 983040, Push 299920(top)  1233     2 ae1.13
SSH@XMR_R8#sh mac vpls 150

Total MAC entries for VPLS 150: 4 (Local: 1, Remote: 3)

VPLS       MAC Address    L/R/IB Port  Vlan(In-Tag)/Peer ISID      Age  Type
====       ===========    ====== ====  ================= ====      ===  ====
150        8cb6.4f63.46b8 L      3/19  550               NA        70   NA
150        0a0a.0009.0004 R      3/7   1.1.1.1           NA        250  NA
150        0013.1922.8f91 R      3/7   1.1.1.1           NA        0    NA
150        0090.69a5.13f1 R      3/7   1.1.1.1           NA        70   NA

Both outputs show local and remote locations of MAC addresses. Both also show the neighbour ID of who has the directly connected MAC address.

Management over the VPLS

A handy new feature on the Netiron is the ability to have a layer3 interface over the VPLS. This can be handy when you need to manage the CPE devices. While in the past you may need to have a ‘break-in’ interface also attached to the VPLS, you can now do it directly on the Netiron.

interface ve 150
 ip address 10.0.0.8/24
!
router mpls

 vpls MELLOWD-VPLS 150
  router-interface ve 150

This essentially works like in SVI interface on a vlan. Let’s check if we have communication:

SSH@XMR_R8#ping 10.0.0.10
Sending 1, 16-byte ICMP Echo to 10.0.0.10, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 10.0.0.10       : bytes=16 time=1ms TTL=255
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.
SSH@XMR_R8#ping 10.0.0.6
Sending 1, 16-byte ICMP Echo to 10.0.0.6, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 10.0.0.6        : bytes=16 time=1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.