So how does the JNCIE-SP compare to the CCIE R&S?

This is not an apples to apples comparison. The tracks are different so it’s difficult to give a thorough review.

Now that I have taken and passed both I can give a little info on how they compare. I’ll be blunt right from the start. The CCIE R&S was a LOT more difficult.

I work in an ISP for a living and so MPLS/BGP is a quite an easy subject. Requirements in exams can be quite ‘odd’ – but no more so than real life. I’ve been in this game too long to know that you some customer requirements are simply off the charts in craziness.

The CCIE R&S is a lot more time-intense. You have a 2 hour TS section followed by 6 hours config. In the JNCIE-SP it certainly felt like I had roughly the same amount of config work, but I had 8 hours to do so. Yes there was some TS in the config section itself, but it’s certainly not like the CCIE TS. It’s perhaps this reason I finished with well over 3 hours to spare and simply used that time to verify everything at least three times.

On the way to my CCIE I learned a lot of lessons which I pulled over into my JNCIE prep. One of the most important ones is learning how to verify. Config is easy. How do you prove that something works though? This is absolutely vital in both exams as well as real life. Another lesson was don’t believe what you read until you’ve labbed it up and seen for yourself. Not just how your devices operate, but what kind of packets do they send at each stage? Load up wireshark and see for yourself.

I much preferred the CCIE lab room over the JNCIE lab room. Cisco provide much bigger screens which is helpful on an 8 hour lab exam. Juniper provided only laptops which was upsetting considering we pay a lot of money for these exams.

Cisco provide you with a result generally within 3 days, but usually by the next day. I waited just over two weeks to get my JNCIE result. Sitting a lab exam is a bit of a rush and that rush dies out pretty quickly. By the time I got my JNCIE the lab rush was long gone and it certainly felt that way when I received my result. While result time is not ‘vital’ to get quick, it made a big difference to how I felt. Maybe that’s just me?

resource-wise there is a lot more training out there for CCIEs. I used the very excellent INE vol2 workbook for my CCIE prep. For the JNCIE-SP I used the InetZero SP workbook. There is less material in the InetZero book compared to the equivalent INE WB. For this reason I think it’s a bit too expensive for what it is, however it’s still a great resource to learn though. If you need go deep configuring MPLS and BGP policies in Junos it’s most certainly the book for you.

I also took one Proteus Networks SP mock exam which I felt was very close to the real exam in difficulty. I initially wanted to take two but ran out of time to get my second one in.

Overall I would have spent a lot more time in the lab if I had not done my CCIE first. Once your know OSPF you know it. The biggest thing is simply learning how a different vendor does it’s defaults, and how you configure what you need.

I’ve definitely have a lot of fun doing both. What next?

The tool is not important

/rant

Recently Russia took their Olympic torch up into space on a space walk. Lot’s of fan-fair was made about it. While that is a great achievement it blows over a very important detail:

The torch is not important.

I know full well that you cannot take a flame up to space, but that’s not the point. The point is that the glorification of tools over what those tools are used for is very prevelent these days. The Olympic flame is what’s important, not the tool used to carry it.

I see plenty of debate when people say, “this tool is crap”, “that tool blows” etc. What’s the point of even having these debates?

Why does it matter if you use Windows or MacOS at work? Does it matter if your packets are forwarded by a Juniper or Brocade forwarding engine? What matters is that the packet gets from A to B. What matters is that the tool itself is up to the job.

I understand that certain people have favourites. But just because you like vendor A’s router does not automatically mean vendor B’s router is rubbish. Is the device capable of providing the solution I require is all that matters ultimately.

To put it another way, think of a game franchise like Super Mario Bros. Does anyone care what PCs those coders used? Does anyone care what language they used? No-one cares. What people care about are the games themselves.

/end rant

Odd behaviour of down interfaces and OSPF on Brocade Netiron

I ran across an odd problem on a Brocade XMR recently. I had created a static route redistributed into OSPF but ended up with a traffic loop. I managed to figure out why this was happening, but the behaviour I see should not be happening. Note I’ve changed the first octect to 10. in all the below addresses.

Consider this network:

R1, R2, and R3 are on OSPF area 0. R4 has the prefix 10.46.204.0/29 sitting behind it. R3 has a static route to this prefix via R4 and that route is getting redistributed into OSPF. R4 has a static default to R3.

My PC is connected to the core network, so let’s do a traceroute:

H:\>tracert -d 10.46.204.1

Tracing route to 10.46.204.1 over a maximum of 30 hops

  1    <1 ms    <1 ms     5 ms  10.196.226.126
  2    <1 ms    <1 ms    <1 ms  10.255.1.1
  3     1 ms     1 ms     1 ms  10.255.0.20
  4     1 ms     1 ms     1 ms  10.71.16.198
  5     1 ms     1 ms     1 ms  10.248.31.29
  6     1 ms     1 ms     1 ms  10.248.31.10
  7     1 ms     1 ms     1 ms  10.248.31.2
  8     8 ms     1 ms     1 ms  10.248.31.61
  9     2 ms     1 ms     1 ms  10.248.31.17
 10     2 ms     1 ms     1 ms  10.248.31.16
 11     2 ms     2 ms     2 ms  10.248.31.17
 12     2 ms    12 ms     2 ms  10.248.31.16
 13     2 ms     2 ms     2 ms  10.248.31.17
 14     2 ms    15 ms     4 ms  10.248.31.16
 15     3 ms     3 ms    14 ms  10.248.31.17
 16    14 ms     4 ms     3 ms  10.248.31.16
 17     3 ms    14 ms     3 ms  10.248.31.17
 18     4 ms     3 ms    14 ms  10.248.31.16
 19    14 ms     4 ms     3 ms  10.248.31.17
 20     4 ms    13 ms     5 ms  10.248.31.16
 21     5 ms     4 ms    14 ms  10.248.31.17
 22    14 ms     5 ms     4 ms  10.248.31.16
 23    15 ms     5 ms     4 ms  10.248.31.17
 24     4 ms    14 ms     6 ms  10.248.31.16
 25     6 ms    17 ms     5 ms  10.248.31.17
 26     6 ms    16 ms     6 ms  10.248.31.16
 27     6 ms    16 ms     6 ms  10.248.31.17
 28     6 ms    16 ms     6 ms  10.248.31.16
 29     7 ms    17 ms     6 ms  10.248.31.17
 30     6 ms    16 ms     7 ms  10.248.31.16

Trace complete.

Clearly we have a problem.

Troubleshooting

Let's look at all three router's view of that route.

R1:

[email protected]#sh ip route 10.46.204.1
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.46.204.0/29     10.248.31.17    ve 62         110/10        O2   34m19s -

R2:

[email protected]#sh ip route 10.46.204.1
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       0.0.0.0/0          10.248.31.16    ve 62         110/210       O1   1h7m   -

R3:

[email protected]#sh ip route 10.46.204.1
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.46.204.0/29     10.248.31.22    eth 2/1       1/1           S    1d2h   -

R2 is clearly wrong. It doesn't have the route in it's table and therefore is following the default route back to R1.

Let's check the OSPF LSA for this prefix:

[email protected]#sh ip ospf database external-link-state link-state-id 10.46.204.0
Ospf ext link-state by link-state ID 10.46.204.0 are in the following:

Type-5 AS External Link States

Index Age  LS ID           Router          Netmask  Metric   Flag Fwd Address   SyncState
444   1387 10.46.204.0     10.196.224.61  fffffff8 0000000a 0000 10.248.31.22   Done
  LSA Header:  age: 1387, options: 0x02, seq-nbr: 0x80000067, length: 36
  NetworkMask: 255.255.255.248
  TOS 0:  metric_type: 2, metric: 10
          forwarding_address: 10.248.31.22
          external_route_tag: 0

I don't see anything wrong here. The metric is valid. The forwarding address is also reachable. Let's check the route to the forwarding address on R2:

[email protected]#sh ip route 10.248.31.22
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.248.31.22/31    10.248.31.21    ve 5          110/200       O    1h59m  -

It's reachable over the directly connected interface which is valid. So what's going on?

After taking a look around I saw that 10.248.31.22 was configured on a local interface on R2. However that interface was down. If an interface is down it should not consider that to be an active route. In fact the show ip route command above the router knows that 10.2148.31.22 is a remote address.

This is the local interface on R2 in question:

[email protected]#sh run int eth 2/1
interface ethernet 2/1
 port-name R3 eth2/1
 enable
 route-only
 ip ospf area 0
 ip ospf network point-to-point
 ip address 10.248.31.22/31

[email protected]#sh int eth 2/1 | include protocol
GigabitEthernet2/1 is down, line protocol is down

I tested the exact same config above in both Junos and IOS and both systems do not have the same problem. It doesn't matter if the forwrding address is a locally configured address as long as that address is not active

OSPF forwarding address

Why does OSPF set the forwarding address anyway? Most type5 LSAs wil have a forwarding address of 0.0.0.0 which effectively means to forward packets to the destination to the ASBR that originated the Type5. There are some very specific circumstances as to when an ASBR sets the forwarding address to a non-0.0.0.0 value which can be found on this site http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009405a.shtml?referring_site=bodynav

At least on IOS, this is what Cisco is saying:

The forwarding address is set to 0.0.0.0 if the ASBR redistributes routes and OSPF is not enabled on the next hop interface for those routes. This is true in the figure if Router 1 does not have OSPF enabled on the Ethernet interface.

These conditions set the forwarding address field to a non-zero address:

OSPF is enabled on the ASBR's next hop interface AND

ASBR's next hop interface is non-passive under OSPF AND

ASBR's next hop interface is not point-to-point AND

ASBR's next hop interface is not point-to-multipoint AND

ASBR's next hop interface address falls under the network range specified in the router ospf command.

Any other conditions besides these set the forwarding address to 0.0.0.0.

R3's link to R4 in my topology does indeed have ospf enabled on the link, but I am running passive. Brocade doesn't seem to have thorough documentation on this so it's behaviour is slightly different to Cisco. Let's remove OSPF from the interface though and check the type 5 LSA again:

[email protected]#conf t
[email protected](config)#int eth 2/1
[email protected](config-if-e1000-2/1)#no ip ospf area 0
[email protected]#sh ip ospf database external-link-state link-state-id 10.46.204.0
Ospf ext link-state by link-state ID 10.46.204.0 are in the following:

Type-5 AS External Link States

Index Age  LS ID           Router          Netmask  Metric   Flag Fwd Address   SyncState
444   15   10.46.204.0     10.196.224.61  fffffff8 0000000a 0000 0.0.0.0        Done
  LSA Header:  age: 15, options: 0x02, seq-nbr: 0x80000069, length: 36
  NetworkMask: 255.255.255.248
  TOS 0:  metric_type: 2, metric: 10
          forwarding_address: 0.0.0.0
          external_route_tag: 0

The forward address has changed, which should mean the route is installed:

[email protected]#sh ip route 10.46.204.0
Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
STATIC Codes - d:DHCPv6
        Destination        Gateway         Port          Cost          Type Uptime src-vrf
1       10.46.204.0/29     10.248.31.21    ve 5          110/10        O2   0m48s  -

Conclusion

What matters is the running state of a device. Whether a forwarding address is configured on an interface or not is irrelevant if the interface is not active. The interface configured with an identical address is not up. I even hard shut the interface but that made no difference.

Either way, it's an interesting quirk to know.

3750G QoS for EF and BE traffic

I was forced to use a 3750G as a router yesterday for a WAN link that was only 70Mb. The LAN interfaces were all gig. The customer wanted to ensure that 30% of the bandwidth was available for EF marked packets. Everything else was to get 70%

A lot of people have trouble with QoS on the 3750. This is mainly due to the tiny buffers, complexity, and the defaults it uses.

Let’s use the following network for this post:

The laptop on the left is connected on a gig port running iperf on linux. The laptop on the right is connected to a hard-coded 100Mb port. However the link itself needs to act like a 70Mb port as the carrier is policing it to 70Mb.

Before we turn any QoS on, let’s get a benchmark. I’m going to send 5 session from the iperf server with DSCP 0 and 5 session with DSCP EF:

Server

$ iperf -c 37.46.204.2 -w 128k -t 600 -i 5 --tos 0 -P 5
$ iperf -c 37.46.204.2 -w 128k -t 600 -i 5 --tos 184 -P 5

Both outputs show bandwidth used is 50/50:

DSCP 0
[SUM]  0.0- 5.0 sec  28.9 MBytes  48.5 Mbits/sec
DSCP EF
[SUM]  5.0-10.0 sec  29.1 MBytes  48.8 Mbits/sec

Also to note is the output drops on gi1/0/15. Remember we are going from a gig interface to a 100Mb interface. This is after 30 seconds:

QOS_TEST#sh int gi1/0/15 | include drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2145

MLS QoS on

I’ll now simply turn QoS on and nothing else:

QOS_TEST(config)#mls qos
QOS_TEST(config)#end

Running the same iperf commands above I see this:

DSCP 0
[SUM]  0.0- 5.0 sec  55.6 MBytes  93.3 Mbits/sec
DSCP EF
[SUM]  5.0-10.0 sec  2.55 MBytes  4.27 Mbits/sec

Voice packets are only getting 4% of the interface speed. Why is this? This is a default on the 3750 and you’ll need to do a little digging. First we need to see which queue EF packets will get into:

QOS_TEST#sh mls qos maps dscp-output-q
   Dscp-outputq-threshold map:
     d1 :d2    0     1     2     3     4     5     6     7     8     9
     ------------------------------------------------------------
      0 :    02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01
      1 :    02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01
      2 :    03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01
      3 :    03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
      4 :    01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01
      5 :    04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
      6 :    04-01 04-01 04-01 04-01

It’s a bit cryptic, but we can see that DSCP value 46 will map to queue 1, while DSCP 0 maps to queue 2.
Let’s now check the default queueing structure on our interface:

QOS_TEST#sh mls qos interface gi1/0/15 queueing
GigabitEthernet1/0/15
Egress Priority Queue : disabled
Shaped queue weights (absolute) :  25 0 0 0
Shared queue weights  :  25 25 25 25
The port bandwidth limit : 100  (Operational Bandwidth:100.0)
The port is mapped to qset : 1

Shaped queue weights shows 25 0 0 0. This actually means that queue 1 is used 1/25 of the interface speed. 100/25 = 4. This is why we are seeing 4Mb for EF traffic.

Ater 30 seconds I took a new reading of the drops and we see this:

QOS_TEST#sh int gi1/0/15 | include drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 8022

A lot worse.

The fix

The first thing we need to do is remove the shaping off the interface:

QOS_TEST(config-if)#srr-queue bandwidth shape 0 0 0 0

Now I want to give 30% to EF and 70% to BE. I don’t want these to be hard-policed so I use the share command:

srr-queue bandwidth share 30 70 1 1

The share command allows other queues to use the bandwidth if those queues are not full. These numbers are not 1/x like the shape command. Rather IOS will add all the values up (102 in our case) and then give 102/30’s worth of bandwidth to queue 1.

This is all great for 100Mb, but remember our link is getting policed to 70Mb. So we need to add this:

srr-queue bandwidth limit 70

Let’s verify:

QOS_TEST#sh mls qos interface gi1/0/15 queueing
GigabitEthernet1/0/15
Egress Priority Queue : disabled
Shaped queue weights (absolute) :  0 0 0 0
Shared queue weights  :  30 70 1 1
The port bandwidth limit : 70  (Operational Bandwidth:70.38)
The port is mapped to qset : 1

iperf again:

DSCP 0
[SUM] 10.0-15.0 sec  29.0 MBytes  48.7 Mbits/sec
DSCP EF
[SUM] 10.0-15.0 sec  11.2 MBytes  18.9 Mbits/sec

Note too that if I just send EF or BE packets, each can use up to 70Mb. It’s only if both are sending for a total over 70Mb do they get their shares as above.

One issue that still remains is that I’m getting these drops after 30 seconds:

QOS_TEST#sh int gi1/0/15 | include drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 5212

In order to properly tune buffers I thoroughly recommend a read through this document: https://supportforums.cisco.com/docs/DOC-8093

Of course in the real world you should be calculating what the maximum amount of voice traffic you are going to send. You would never have ‘more’ voice traffic than if every person in your company was on an external call.

If I change the above test so that the server is sending 15Mb of UDP traffic marked DSCP EF, then I can see that the TCP BE traffic drops while no drops are on the EF queue:

iperf -c 37.46.204.2 -u -b 15m -p 5002 -t 5

No packets dropped in the EF stream:

[  3]  0.0- 5.0 sec  7.76 MBytes  13.0 Mbits/sec  1.776 ms    0/ 5532 (0%)

Checking the port drop statistics on the 3750G:

QOS_TEST#sh platform port-asic stats drop gi1/0/15

  Interface Gi1/0/15 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 276
      Weight 1 Frames 0
      Weight 2 Frames 0

No voice packets dropped there either.

EDIT – 05/11/13

Be very careful when enabling priority-queue out. Regardless of bandwidth configured, if you enable this under your interface the switch will always service any priority packet before any other packet. IT does not take bandwidth into account at all.

Let’s test by adding priority queue on the switchport:

QOS_TEST(config)#int gi1/0/15
QOS_TEST(config-if)#priority-queue out

I’ve now started flooding DSCP 0 and DSCP EF traffic. EF traffic is getting 70Mb while I’m not getting ANY result on the DSCP 0 traffic. All those packets are getting dropped. It could be very easy to DDOS your service by just sending EF frames.

As soon as I disable priority-queue out, EF drops to 30% and non-EF goes up to 70%

Also to note is that the srr-queue bandwidth limit command only affects traffic going out the interface. Traffic coming into the interface is not affected by this command.