Tag Archives: lab

ESXi whitebox server

I usually have access to an ESX box at work where I can run multiple VMs and virtual routers for labbing and testing. I’ve also wanted one at home. It’s nice to be able to quickly spin up VMs when needed without always running them through my laptop.

While virtual routers don’t need lots of resources, I did want a beefy machine as there are a few servers I’d like to get running that need lots of CPU power.

Requirements

  • 32GB RAM capable with ECC
  • Fast CPU with at least 4 physical cores
  • Quiet
  • Small
  • OOB (ilo/IPMI/Etc)
  • Low power

Specifically these are the things I don’t need:

  • Optical drive
  • Hard drive
  • GPU

The point of the box is to sit in the corner with only power and network connected. If anything went wrong, I don’t want to have to connect a monitor to it. I’m also not running any tasks requiring video output on the server itself. All VMs will be logged in via SSH.

I already have a Synology DS411 which will provide an iSCSI connection to the ESX server. Hence no need for internal hard drives.

My initial build was going to be built around an Intel i7 4790. However the i7 doesn’t support ECC ram and it also has a built-in HD4000 GPU which I don’t need.

Parts list

I ended up going for the Intel Xeon E3-1230v3. 4 cores, 8 threads, all the virtulisation support I need. It has no built-in GPU. It supports up to 32GB ECC RAM. Intel have released a newer 1231v3, but I couldn’t find a good price for it in the UK and all it gives is an extra 100MHz which I’m not fussed about.

RAM is quite pricey at the moment. While I wanted 32GB, I’ll start with 16GB for now and add another 16GB when prices drop.

For the motherboard, I went with the SuperMicro X10SLL-F. It supports both the CPU and RAM and also has built-in IPMI. The board has two onboard Intel NICs, i217LM and i210AT. The board also has an on-board VGA card. I’m not going to use that, but it will be handy if I can’t log into both the server and IPMI. It also has an a-type USB slot on board which is quite handy as you’ll see later.

For the PSU, I wanted both silent and efficient. I don’t need a huge amount of wattage either as I have no GPU. I ended up with the Seasonic G-360. 80-Plus certified and very quiet.

Part of the problem with an ESXi whitebox is ensuring that VMWare recognises all your components. I did extensive research in order to ensure this was the case. There are a couple of things I hit upon, but they were easily fixed.

Final part list:

  • Intel Xeon E3-1230 V3
  • SuperMicro X10SLL-F
  • 2 X 8GB Crucial DDR3 PC3-12800 ECC Unbuffered
  • Seasonic G-360 PSU
  • Aerocool Dead Silence Gaming Cube Case
  • Old 1Gb USB flash drive

Building and installing

The SuperMicro board has a dedicated IPMI port and so I can do the entire install remotely. I’ll mount the ISO over the network, and all do all the config this way. This is the screen you see when first logging onto IPMI web interface:
Capture
I decided to install Vmware itself on a USB stick. What’s nice about this motherboard is that it has a USB port on the motherboard itself, meaning no external USB key required. This keeps it a bit neater.

The SuperMicro has two external NICs, one Intel 217 and an Intel 210. I’ve installed VMware 5.5 update 2 and the I210 Intel card is supported out of the box. No need to hack any drivers into the ISO. I’m more than happy with one NIC for now so I’ve no need to try and get the 217 working.

Once VMWare was installed, I created a 300GB iSCSI LUN from my Synology and attached VMWare to that. The install and set up really was painless.

Vmware shows my system as:
system

Virtual devices

I wanted to start a basic lab, so I have the following VMs running in my lab:
VMs

With all my VMs running, I see hardly any CPU and quite a bit of RAM usage as I expected:
resource
For now the RAM amount is fine. As I ramp up the lab and prices drop, I’ll add another 16GB to the system.

Power usage

As I wanted this to be low power, I’ve done full wattage readings on power usage.

  • Server off, IPMI on – 3.7 Watts
  • Server on, no VMs running – 23 Watts
  • Server on, all lab VMs running – 34 Watts

Not at all bad. In another post I’ll show the Synology power draw as well as the power draw if all VMs are using full CPU. I’ll also go over how I automate my VMs starting and shutting down.

HSRP Object tracking

HSRP can track interfaces, which is pretty handy if they are tracking the WAN interface to get out the network. There are times when tracking the interface itself is not enough. This is a great example:

Our customer is running HSRP between 2 routers connected to his local LAN. All PC’s connected to his switch are using 10.1.1.254 as their default gateway. R1 is the primary HSRP router and R2 is the backup.

HSRP allows you to track an interface and lower the priority if this happens. So let’s say R1′s WAN interface goes down, HSRP will notice that, and allow R2 to take over the HSRP group. This allows the customer to continue to get to the cloud. A basic config here:
R1:

interface FastEthernet0/1
 description LAN
 ip address 10.1.1.1 255.255.255.0
 standby version 2
 standby 1 ip 10.1.1.254
 standby 1 priority 110
 standby 1 preempt
 standby 1 track FastEthernet0/0 65

R2:

interface FastEthernet0/1
 description LAN
 ip address 10.1.1.2 255.255.255.0
 standby version 2
 standby 1 ip 10.1.1.254
 standby 1 preempt
 standby 1 track FastEthernet0/0 50

What happens if the interface stays up though? In the above diagram, R1′s WAN interface connects to a switch. Now this could be a local switch or a 3rd party NTE (which often happens on leased lines) – The circuit from the switch to the cloud could be down, but the port between the router and switch are still up.
As far as HSRP is concerned, that interface is up and healthy. All LAN PC’s will continue to send traffic to R1, but that traffic gets dropped at the switch. Another case is if you’ve got some sort of MPLS/VPLS solution with your provider. They could have a problem and all traffic is getting black-holed inside their network. But R1 still thinks the link is healthy and sends it on it’s way.

There is a better way of doing this.

IOS allows you to track object. An object could be an IP SLA instance. That IP SLA instance could very easily be an ICMP echo from another device in the cloud. See where I’m going here?

Assume R3 has a rock solid connection. We could assume is a big bad router with multiple power feeds on multiple phases with multiple WAN connections. 192.168.1.1 is the loopback of this device accessible from multiple connections. Basically we assume that 192.168.1.1 is ALWAYS available.

Let’s create an IP SLA instance that tests connectivity to 192.168.1.1. We then tell HSRP to track reachability to that instance. If it cannot get to the instance, we can assume that the link to it is dead, regardless of whether R1′s WAN interface is up or not. First up is the IP SLA instance:

ip sla monitor 10
 type echo protocol ipIcmpEcho 192.168.1.1
 frequency 5
ip sla monitor schedule 10 life forever start-time now

Here we’ve told the router to ping 192.168.1.1 every 5 seconds and never stop. If I get a reply, consider the SLA a success. If I get no response, consider it a failure.

Now we tell IOS that we want to create an object and track this IP SLA instance:

track 100 rtr 10

I create an object labelled 100 that is tracking instance 10 of IP SLA we created above.

We now amend R1′s HSRP config as follows:

interface FastEthernet0/1
 description LAN
 ip address 10.1.1.1 255.255.255.0
 no ip redirects
 standby version 2
 standby 1 ip 10.1.1.254
 standby 1 priority 110
 standby 1 preempt
 standby 1 track FastEthernet0/0 65
 standby 1 track 100 decrement 65

I’ve kept the interface tracking there as if it goes down, why wait for IP SLA to timeout?

But does it work? Let’s have a look and see. Let’s kill S1′s connection into the cloud. Once that’s done, let’s have a look at R1:

R13#sh standby
FastEthernet0/1 - Group 1 (version 2)
  State is Standby
    7 state changes, last state change 00:01:08
  Virtual IP address is 10.1.1.254
  Active virtual MAC address is 0000.0c9f.f001
    Local virtual MAC address is 0000.0c9f.f001 (v2 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.052 secs
  Preemption enabled
  Active router is 10.1.1.2, priority 100 (expires in 9.052 sec)
  Standby router is local
  Priority 45 (configured 110)
    Track interface FastEthernet0/0 state Up decrement 65
    Track object 100 state Down decrement 65
  IP redundancy name is "hsrp-Fa0/1-1" (default)

R1#sh int fa0/0
FastEthernet0/0 is up, line protocol is up

IOS is telling us that although the interface is up, it’s passed the HSRP group to R2. Now we don’t have to worry about traffic getting black-holed!

btw, if you need to ping an address and can’t guarantee 100% availability, you could just as easily track 2 objects. Weight it so that only if pings fail to both will the HSRP group failover.

MHSRP & OSPF design

I’m going to start blogging about various design stuff that I do these days at work. Maybe you can get some ideas from these types of posts.

I’m going to run this simple topology:

The client has 2 networks at the first site – 192.168.1.0/24 and 10.10.10.0/24. These are both connected to R1 and R2 (via a switch not shown) – R1 and R2 are running MHSRP, with R1 being the master of 192.168.1.254 and R2 being the master of 10.10.10.254. Both routers are the backup of each others group.

I want the link from R1 to be used for 192.168.1.0 and I want the link from R2 to be used for 10.10.10.0. I also want return traffic to take the same path. i.e. if R3 needs to send traffic to 10.10.10.0 – it needs to prefer the direct path to R2.

Another constraint is that I need convergence to be fast, therefore I’m not going to run HSRP on the WAN side of R1 and R2 and have to wait for the IGP to re-converge.

The final constraint is that I don’t want to have to use route-map’s on R3 itself. Let’s pretend there are going to be a lot more routers connected to OSPF area 0. I want to make it as simple as possible when newer routers come into the area. Therefore all this routing needs to be controlled by R1 and R2 themselves.

So far, pretty simple. So let’s get cracking!

Let’s do R1 first, this is the relevant config:

interface FastEthernet0/1
ip address 10.10.10.1 255.255.255.0 secondary
ip address 192.168.1.1 255.255.255.0
standby version 2
standby 1 ip 192.168.1.254
standby 1 priority 110
standby 1 preempt
standby 1 track FastEthernet0/0 50
standby 2 ip 10.10.10.254
standby 2 preempt
standby 2 track FastEthernet0/0 50

And now R2:

interface FastEthernet0/1
ip address 10.10.10.2 255.255.255.0
ip address 192.168.1.2 255.255.255.0 secondary
standby version 2
standby 1 ip 192.168.1.254
standby 1 preempt
standby 1 track FastEthernet0/0 50
standby 2 ip 10.10.10.254
standby 2 priority 110
standby 2 preempt
standby 2 track FastEthernet0/0 50

Now let’s set up OSPF. Remember that I want 192.168.1.0 traffic to go via R1 and 10.10.10.0 to go via R2. There are a number of ways of doing this.

First let’s try and set up regular OSPF with a metric:

router ospf 1
 R1(config-router)#network 10.10.10.0 0.0.0.255 area 0 ?
  cr>
 R1(config-router)#network 10.10.10.0 0.0.0.255 area 0

OSPF will not allow you to specify a metric under the network command!

There is another way. Run 2 OSPF processes and redistribute with a higher metric.

router ospf 1
 redistribute ospf 2 metric 50 metric-type 1 subnets
 network 1.1.1.0 0.0.0.255 area 0
 network 192.168.1.0 0.0.0.255 area 0
!
router ospf 2
 network 10.10.10.0 0.0.0.255 area 0

Let’s check R3′s routing table:

R3#sh ip route
     1.0.0.0/24 is subnetted, 1 subnets
C       1.1.1.0 is directly connected, FastEthernet0/0
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.50.0 is directly connected, Loopback0
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/51] via 1.1.1.1, 00:00:39, FastEthernet0/0
O    192.168.1.0/24 [110/2] via 1.1.1.1, 00:00:39, FastEthernet0/0

It works, but I think its messy. We could also use route-maps which would give us a lot finer control over newer subnets that may be added (without having to run more OSPF processes)

Let’s remove that OSPF config from R1 and do the following:

router ospf 1
 redistribute connected metric-type 1 subnets route-map 10_HIGH
 network 1.1.1.0 0.0.0.255 area 0
!
ip prefix-list 10NET seq 5 permit 10.10.10.0/24
!
route-map 10_HIGH permit 10
 match ip address prefix-list 10NET
 set metric +100
!
route-map 10_HIGH permit 20

Now we do this on R2:

router ospf 1
 redistribute connected metric-type 1 subnets route-map 192_HIGH
 network 1.1.1.0 0.0.0.255 area 0
!
ip prefix-list 192NET seq 5 permit 192.168.1.0/24
!
route-map 192_HIGH permit 10
 match ip address prefix-list 192NET
 set metric +100
!
route-map 192_HIGH permit 20

This will ensure R1 redistributes the 10.10.10.0 network with a higher metric to R3, and vice-versa for R2. R3 will have a possible route in it’s database, but won’t use it until the preferred link is down/

Let’s have a look at R3′s routing table:

R3#sh ip route ospf
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/21] via 1.1.1.2, 00:00:28, FastEthernet0/0
O E1 192.168.1.0/24 [110/21] via 1.1.1.1, 00:00:28, FastEthernet0/0

Excellent, just what we want. but do the backups work correctly? Let’s shut down R1′s interface to the customer LAN and see what happens:

R1(config)#int fa1/0
R1(config-if)#shut
R1(config-if)#
*Mar  1 00:34:20.711: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 1 state Active -> Init
*Mar  1 00:34:20.723: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 2 state Standby -> Init
*Mar  1 00:34:22.723: %LINK-5-CHANGED: Interface FastEthernet1/0, changed state to administratively down
*Mar  1 00:34:23.723: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0, changed state to down

What does R3 see?

R3#sh ip route ospf
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/21] via 1.1.1.2, 00:00:18, FastEthernet0/0
O E1 192.168.1.0/24 [110/101] via 1.1.1.2, 00:00:18, FastEthernet0/0

Both are now going via R2, which is exactly what we wanted. Let’s now reconnect R1 and disconnect R2′s WAN link:

R1(config-if)#int fa1/0
R1(config-if)#no shut
R1(config-if)#
*Mar  1 00:35:47.775: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 1 state Listen -> Active
*Mar  1 00:35:47.859: %LINK-3-UPDOWN: Interface FastEthernet1/0, changed state to up
*Mar  1 00:35:48.859: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0, changed state to up
R2(config)#int fa0/0
R2(config-if)#shut
R2(config-if)#
*Mar  1 00:36:16.179: %TRACKING-5-STATE: 1 interface Fa0/0 line-protocol Up->Down
*Mar  1 00:36:16.187: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.3 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
*Mar  1 00:36:16.187: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.1.1 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
*Mar  1 00:36:18.179: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Mar  1 00:36:18.231: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 2 state Active -> Speak
*Mar  1 00:36:19.179: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down

What does R3 see this time?

R3#sh ip route ospf
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/101] via 1.1.1.1, 00:00:03, FastEthernet0/0
O E1 192.168.1.0/24 [110/21] via 1.1.1.1, 00:00:03, FastEthernet0/0

Great. Let’s just make sure it all works properly again if all links are up:

R2(config-if)#int fa0/0
R2(config-if)#no shut
R2(config-if)#
*Mar  1 00:37:53.291: %TRACKING-5-STATE: 1 interface Fa0/0 line-protocol Down->Up
*Mar  1 00:37:54.235: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 2 state Standby -> Active
*Mar  1 00:37:55.287: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar  1 00:37:56.287: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
R3#sh ip route ospf
     10.0.0.0/24 is subnetted, 1 subnets
O E1    10.10.10.0 [110/21] via 1.1.1.2, 00:00:06, FastEthernet0/0
O E1 192.168.1.0/24 [110/21] via 1.1.1.1, 00:00:06, FastEthernet0/0

Let’s see what happens when a client is pinging a network directly connected to R3 over the 192.168.1.0 network. I’ll be using a router as a client, so it’ll be a ping flood.

R4#ping 172.16.50.1 repeat 1000

I’ll quickly shut R1′s LAN interface. Wait a bit, and then no shut it. What happens to R4′s pings?

Sending 1000, 100-byte ICMP Echos to 172.16.50.1, timeout is 2 seconds:
!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (998/1000), round-trip min/avg/max = 1/44/160 ms

I lost only 2 pings! Once when I shut the interface, and the second when it came back up again. Pretty good!

It wont be as fast if the WAN link goes down though. R2 will take over for sending packets out, but return traffic will still go to R1. Why? Well remember R3 still thinks R1 is there. Unless they were directly connected, or you wait for OSPF to realise the peer is down, it’ll continue to send traffic that way. You can speed this up by reducing your OSPF hello timers though. This will need to be balanced on how much OSPF traffic you want going across your WAN link.

Lab Solution – MPLS Lab #2

Sometimes its so hard to simply find the time to do what I promised. I hope this will spur up some conversation. I still stress that you should always try to do the lab without my help first. This will ensure you learn how to do it properly. Also remember that there are always multiple ways to do certain labs, so don’t take my solution as gospel.

This solution is for the lab I posted here: http://mellowd.co.uk/ccie/?p=527

  • CPE1 and CPE5 belong to Customer1
  • CPE2 and CPE6 belong to Customer2
  • Both customers are running OSPF as their IGP’s
  • The loopbacks as shown in the topology must be advertised into OSPF. Cutomer1 should be able to ping all loopbacks in their networks and Customer2 should be able to ping everything in theirs.
  • Both customers are now running a project together, and need 2 of their offices connected. CPE1 from Customer1 should be able to communicate with CPE6 from Customer2 and vice-versa
  • It’s essential that CPE2 and CPE5 are NOT able to get to all loopbacks. ONLY CPE1 and CPE6 should be able to communicate with each other. This new configuration should not break the previous VPN’s in place
  • Do this without using any ACL’s, Prefix-lists, Route-maps or the like

We start by doing a regular MPLS VPN config – The same for which we did for the first MPLS VPN lab. All the MPLS-specific config is here:

CPE1

interface Loopback0
 ip address 192.168.1.1 255.255.255.0
!
interface FastEthernet0/0
 ip address 10.1.1.1 255.255.255.0
 duplex auto
 speed auto
!
router ospf 1
 log-adjacency-changes
 network 10.1.1.0 0.0.0.255 area 0
 network 192.168.1.0 0.0.0.255 area 0

CPE2:

interface Loopback0
 ip address 172.16.1.1 255.255.255.0
!
interface FastEthernet0/0
 ip address 10.1.2.1 255.255.255.0
 duplex auto
 speed auto
!
router ospf 1
 log-adjacency-changes
 network 10.1.2.0 0.0.0.255 area 0
 network 172.16.1.0 0.0.0.255 area 0

CPE5:

interface Loopback0
 ip address 192.168.2.1 255.255.255.0
!
interface FastEthernet0/0
 ip address 10.1.3.1 255.255.255.0
 duplex auto
 speed auto
!
router ospf 1
 log-adjacency-changes
 network 10.1.3.0 0.0.0.255 area 0
 network 192.168.2.0 0.0.0.255 area 0

CPE6:

interface Loopback0
 ip address 172.16.2.1 255.255.255.0
!
interface FastEthernet0/0
 ip address 10.1.4.1 255.255.255.0
 duplex auto
 speed auto
!
router ospf 1
 log-adjacency-changes
 network 10.1.4.0 0.0.0.255 area 0
 network 172.16.2.0 0.0.0.255 area 0

Now for the 2 AR Routers:

ip cef
ip vrf CUS1
 rd 400:1
 route-target export 400:1
 route-target import 400:1

ip vrf CUS2
 rd 400:2
 route-target export 400:2
 route-target import 400:2

interface FastEthernet0/0
 ip vrf forwarding CUS1
 ip address 10.1.1.2 255.255.255.0

interface FastEthernet2/0
 ip vrf forwarding CUS2
 ip address 10.1.2.2 255.255.255.0

router ospf 2 vrf CUS1
redistribute bgp 400 metric 10 subnets
 network 10.1.1.0 0.0.0.255 area 0

router ospf 3 vrf CUS2
 redistribute bgp 400 metric 10 subnets
 network 10.1.2.0 0.0.0.255 area 0

router bgp 400
 bgp log-neighbor-changes
 neighbor 10.255.255.7 remote-as 400
 neighbor 10.255.255.7 update-source Loopback0

 address-family vpnv4
  neighbor 10.255.255.7 activate
  neighbor 10.255.255.7 send-community extended
 
 
 address-family ipv4 vrf CUS2
  redistribute ospf 3 vrf CUS2 metric 10
  no synchronization
 
 address-family ipv4 vrf CUS1
  redistribute ospf 2 vrf CUS1 metric 10
  no synchronization

A similar config is on AR3. (I’m not going to post it here otherwise this post will just get to big)

Let’s now concentrate on CPE1. The initial requirements were to allow CPE1 and CPE5 to speak to each other. Currently CPE1 has the following routing table:

CPE1#sh ip route

Gateway of last resort is not set

     10.0.0.0/24 is subnetted, 2 subnets
O IA    10.1.3.0 [110/11] via 10.1.1.2, 00:01:20, FastEthernet0/0
C       10.1.1.0 is directly connected, FastEthernet0/0
C    192.168.1.0/24 is directly connected, Loopback0
     192.168.2.0/32 is subnetted, 1 subnets
O IA    192.168.2.1 [110/11] via 10.1.1.2, 00:01:20, FastEthernet0/0

Can CPE1 ping the loopback subnet on CPE5? It sure can!

CPE1#ping 192.168.2.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 80/112/156 ms

Can CPE1 ping CPE6? No it can’t (as expected at this point)

CPE1#ping 172.16.2.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

We are now told that we need CPE1 and CPE6 to be able to speak to each other for a project. CPE2 and CPE5 need to be left out of this completely. We need to do this without using any ACL’s or the like.

There is a simple way of doing this. It’s called Extranet MPLS VPN. In the configuration above, each customer is given a route target. We can create a third route-target and have both CPE1 and CPE6 join that third route-target. We then simply don’t add CPE2 and CPE6 to that same route-target.

Let’s add it on AR1 and AR3:

AR1(config)#ip vrf CUS1
AR1(config-vrf)#route-target both 400:100
AR3(config)#ip vrf CUS2
AR3(config-vrf)#route-target both 400:100

If I now check the routing table on CPE1 I see the following:

CPE1#sh ip route

Gateway of last resort is not set

     172.16.0.0/32 is subnetted, 1 subnets
O E2    172.16.2.1 [110/10] via 10.1.1.2, 00:00:21, FastEthernet0/0
     10.0.0.0/24 is subnetted, 3 subnets
O IA    10.1.3.0 [110/11] via 10.1.1.2, 00:08:41, FastEthernet0/0
C       10.1.1.0 is directly connected, FastEthernet0/0
O E2    10.1.4.0 [110/10] via 10.1.1.2, 00:00:21, FastEthernet0/0
C    192.168.1.0/24 is directly connected, Loopback0
     192.168.2.0/32 is subnetted, 1 subnets
O IA    192.168.2.1 [110/11] via 10.1.1.2, 00:08:41, FastEthernet0/0

Can CPE1 now ping CPE6′s loopback subnet?

CPE1#ping 172.16.2.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/69/92 ms

It works :) – We now need to be sure that CPE2 and CPE5 still cannot see any of this.

CPE2#sh ip route

Gateway of last resort is not set

     172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C       172.16.1.0/24 is directly connected, Loopback0
O IA    172.16.2.1/32 [110/11] via 10.1.2.2, 22:04:42, FastEthernet0/0
     10.0.0.0/24 is subnetted, 2 subnets
C       10.1.2.0 is directly connected, FastEthernet0/0
O IA    10.1.4.0 [110/11] via 10.1.2.2, 22:05:56, FastEthernet0/0

As expected, it cannot ping anywhere in Customer1′s network:

CPE2#ping 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
CPE2#ping 192.168.2.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Job done. :D

Troubleshooting 101 – Can I get back?

Rule #1 – Just because Host 1 can get to Host 2, does NOT mean that Host 2 can get back to Host 1!

I can’t tell you how many times I’ve seen this out in the field. Here is a prime example.

This is quite a simply common topology. OfficeA and OfficeB have a WAN connection running between them. RouterA and RouterB are under control of the ISP and are running OSPF. RouterC is a customer-owned router, and so doesn’t run any OSPF with RouterA or RouterB.

OfficeB has an internet connection, and RouterB is injecting a default route to OfficeA via OSPF.

RouterB has a default route to RouterC. So let’s think about traffic flow now. A host connected to RouterA needs to send traffic to the internet. It will send it’s traffic to it’s default gateway, RouterA. RouterA has a default route injected into OSPF from RouterB and so sends traffic to RouterB.

RouterB has a default route and hence sends that traffic out to RouterC. Which then goes out to the internet.

Traffic then flows back from the internet to RouterC. The return address will be an IP that belongs in RouterA and RouterB’s routing table, however RouterC has no knowledge of that subnet (as it’s not participating in OSPF). RouterC will just use it’s default route and send that packet back out to the internet.  Eventually the TTL will kill that packet.

This can be fixed by putting a static route on RouterC to let it know that RouterA’s ip range needs to be sent off to RouterB instead.

A similar thing will happen if we add a server to SwitchA. That server’s default gateway will most likely be RouterC. If a host in OfficeA send a PING to that server, that server will then send traffic off to RouterC. If RouterC does not have the static route added above, it’ll send it out to the internet.

I’m well aware that the design in the picture is pretty bad, but I used it to illustrate a point. That point being that just because router’s know how to get from A to B, it does NOT mean they know how to route that traffic back. Make sure you understand this!