Protocol fundamentals – dot1q

I’ve noticed that a lot of people seem to get confused with what exactly dot1q is doing most of the time. It’s actually incredibly simply.

Tagging traffic, or Trunking in Cisco-talk, is a very straightforward process. I will not be discussing ISL here as not only do I not use it, but Cisco is phasing it out on their stuff anyway.

The dot1q tag is simply inserted into the layer2 header when a packet leaves a switchport over a trunk. If a frame leaves a switchport that is not a trunk, there is no dot1a tag inserted into it, regardless of what vlan the frame came from or is going to. This means that the following is a perfectly valid topology:
dot1qex1 Protocol fundamentals   dot1q
Let’s configure the above quickly.

3560TOP#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
3560TOP(config)#int range fa0/1, fa0/8
3560TOP(config-if-range)#switchport mode access
3560TOP(config-if-range)#switchport access vlan 10
% Access VLAN does not exist. Creating vlan 10
C3550#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
C3550(config)#int range fa0/1, fa0/8
C3550(config-if-range)#switchport mode access
C3550(config-if-range)#switchport access vlan 20
% Access VLAN does not exist. Creating vlan 20

Can the PC’s ping each other?

PC2#ping 10.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/10/28 ms
PC1#ping 10.1.1.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms

They both can ping each other. But aren’t the devices in different vlans? They sure are, yet they can still communicate. Why is this?

The issue is that the switch interlink are both access ports. An access port will not send or accept tagged traffic. Hence when SW1 sends PC1′s traffic over the link, the tag is removed. When that packet comes into SW2′s fa0/8 interface, that interface is part of vlan 20. SW2 will allow that frame to flow to PC2. The same happens vice-versa.

Let’s change the topology so that there is a trunk instead between the 2 switches. I also want to force the use of dot1q.

C3550#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
C3550(config)#default interface fa0/8
Interface FastEthernet0/8 set to default configuration
3560TOP#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
3560TOP(config)#default interface fa0/8
Interface FastEthernet0/8 set to default configuration
3560TOP(config-if)#switchport trunk encapsulation dot1q

Now can the PC’s ping each other?

PC2#ping 10.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

They cannot. It now works as expected SW1 now sends PC1′s frame over the link with a dot1q tag. In that tag is the vlan id of 10. When it gets to SW2, it’ll look at that tag and ensure the frame is not sent out any access port that is not in vlan 10.

Another important part of dot1q is the notion of a native vlan. The native vlan does not get tagged over a trunk unless you configure otherwise. Vlan1 is the default native vlan.

So let’s try something here. Let’s configure SW1 to think that vlan 10 is the native vlan, while on SW2 let’s change it to vlan20. Will my PC’s be able to ping each other?

3560TOP(config)#int fa0/8
3560TOP(config-if)#switchport trunk native vlan 10
*Mar  1 00:13:42.385: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 1 on FastEthernet0/8 VLAN10.
*Mar  1 00:13:42.385: %SPANTREE-2-BLOCK_PVID_PEER: Blocking FastEthernet0/8 on VLAN0001. Inconsistent peer vlan.
*Mar  1 00:13:42.385: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking FastEthernet0/8 on VLAN0010. Inconsistent local vlan.
*Mar  1 00:14:13.389: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking FastEthernet0/8 on VLAN0001. Port consistency restored.
*Mar  1 00:14:19.270: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/8 (10), with C3550 FastEthernet0/8 (20).

I’ve done the same on SW2, but for vlan 20. You can see that the switch is giving me all kinds of error messages.

However I cannot ping between my PCs:

PC2#ping 10.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

I tried disabling CDP, hard coding the trunk between the 2 switches and also disabling negotiation between the 2 links but no communication at all. Even though it should work theoretically, the switches just do not like the native vlan not matching on either side of the link

I was not happy with the outcome above, so I dug a little deeper. It seems STP is blocking communication. The BPDU’s still carry vlan information with them. What I’ve done on both switches is disable STP on vlans 10 and 20 on both switches:

3560TOP(config)#no spanning-tree vlan 10,20

Can I now ping?

PC2#ping 10.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms

Indeed I can. So even though there is a trunk between the 2 switches and both devices are in separate vlans, they can still communicate as SW1 and SW2 both think that the native vlan matches on each side. They have no idea that PC1 is in vlan 10 and PC2 is in vlan 20.

I don’t much like leaving the native vlan untagged. Thankfully we have an option to tag all frames:

3560TOP(config)#vlan dot1q tag native
C3550(config)#vlan dot1q tag native

What this does is essentially ignore the native vlan setting. All traffic, regardless of vlan, will be tagged over the link. This is proven by the fact I can now no longer ping again:

PC2#ping 10.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

It’s important to note that an untagged frame is identical to a frame that goes in and out of an access port. There is no difference. To prove this I’m going to put both PC1 and PC2 into vlan 10, and them create a trunk link to PC1. PC1 does not understand trunk links and will continue to send regular traffic. I’ll configure SW1 to untag the native vlan again and make vlan 10 the native vlan. Will this work?

3560TOP(config)#int fa0/8
3560TOP(config-if)#no switchport trunk native vlan 10
3560TOP(config-if)#exit
3560TOP(config)#no vlan dot1q tag native
3560TOP(config)#int fa0/1
3560TOP(config-if)#switchport trunk encap dot
3560TOP(config-if)#switch mode trunk
3560TOP(config-if)#switchport trunk native vlan 10

Can PC1 ping PC2?

PC1#ping 10.1.1.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/6/16 ms

It sure can. PC1 is simply sending untagged traffic off to SW1. SW1 assumes that untagged traffic is part of vlan10 and forwards those frames throughout vlan 10. This goes over the trunk to SW2, and sends those frames over the vlan10 access ports where it gets to PC2.

Of course routers and some servers can also send tagged traffic. Let’s change PC1 so that it sends out tagged traffic. Let’s change SW1 back to the regular native vlan on port fa0/1

3560TOP(config)#int fa0/1
3560TOP(config-if)#no switchport trunk native vlan 10
PC1(config)#int fa0/0
PC1(config-if)#no ip address
PC1(config-if)#int fa0/0.1
PC1(config-subif)#encapsulation dot1Q 10
PC1(config-subif)#ip address 10.1.1.1 255.255.255.0

Can I ping?

PC1#ping 10.1.1.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/8 ms

As expected I sure can.

Just like the switches, I can also make any single vlan I choose to be the native vlan. This frame will simply be sent untagged. Let’s change SW1′s fa0/1 interface to an access port again, and change PC1 to continue to use dot1q, but ensure that vlan 10 traffic is untagged:

3560TOP(config)#default interface fa0/1
Interface FastEthernet0/1 set to default configuration
3560TOP(config)#int fa0/1
3560TOP(config-if)#switch mode access
3560TOP(config-if)#switch access vlan 10
PC1(config)#int fa0/0.1
PC1(config-subif)#encapsulation dot1Q 10 native

Ping should still work, does it?

PC1#ping 10.1.1.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms

As you can see, it’s pretty simple stuff at the end of the day.

Building my topology

So now it’s time to actually build my topology. There are a number of issues I’d like to get my head around. It helps to know what the planned set up is going to be.

I plan to use my laptop for my studies. From my laptop I’ll be connecting to my dynamips box. My dynamips box is connected to 4 switches. I need to be able to console into all 4 switches remotely, but I don’t want to buy a terminal server. I also want to be able to telnet into all the routers running on the dynamips box.

This is how it’ll look:
dynsetup Building my topology

This is all possible of course, and I’ll be showing how I did it.

I needed 12 ‘breakout’ ports on the system. Essentially dynamips can map emulated router ports to real ports, allowing you to connect your emulated routers to real switches. I went and bought 3 of these on ebay (Sun Quad Fast PCI Ethernet Card 501-4366):
 Building my topology

However, first issue. I can only fit 2 NIC’s in my box. The cards are long, and the 3rd simply down not fit in the box:P1010975 Building my topologyYou can see that there is a heatsinked chip in the way as these cards are very long.

So I went and bought 4 of these:
NT0014 bg Building my topologyBut these things are pretty awful. They are more bulky than they look, and Ubuntu just doesn’t like to see more than one of them. It’s also messy.

I then decided to find a smaller 4 port NIC that would work. I looked around and found the Dlink DFE-580TX.
CategoryImages DFE580TXl Building my topology

Will this fit? It does indeed!
P1010979 Building my topology

I’ve downloaded and installed the latest version of Ubuntu 64bit server (At this time, 10.10)

Does Ubuntu see all my NICs?

darreno@ubuntu10:~$ lspci | grep Ethernet
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
04:04.0 Ethernet controller: D-Link System Inc DL10050 Sundance Ethernet (rev 15)
04:05.0 Ethernet controller: D-Link System Inc DL10050 Sundance Ethernet (rev 15)
04:06.0 Ethernet controller: D-Link System Inc DL10050 Sundance Ethernet (rev 15)
04:07.0 Ethernet controller: D-Link System Inc DL10050 Sundance Ethernet (rev 15)
05:00.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
05:01.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
05:02.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
05:03.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
06:00.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
06:01.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
06:02.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
06:03.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)

How has Ubuntu numbered those interfaces? You can find out like this:

darreno@ubuntu10:~$ sudo vi /etc/udev/rules.d/70-persistent-net.rules
# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:21:de:ed:1e",
 ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x108e:0x1001 (hme)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:20:8d:49:19",
 ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth6"

# PCI device 0x1186:0x1002 (sundance)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0d:88:cd:4f:7a",
 ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

You can see that eth0 is my onboard NIC. eth6 is one of the Sun’s ports and eth4 is one of the D-Link’s ports.

I need to set up my networking interfaces file so ifconfig knows that they are there. I’m going to be putting them in manual mode, and use a script to start them up when I need to run my topologies.

darreno@ubuntu10:~$ sudo vi /etc/network/interfaces
# The primary network interface
auto eth0
iface eth0 inet static
address 10.20.30.12
netmask 255.255.255.0
gateway 10.20.30.254
#
# Sun
auto eth1
iface eth1 inet manual

# D-Link
auto eth2
iface eth2 inet manual
#
auto eth3
iface eth3 inet manual
#
auto eth4
iface eth4 inet manual
#
auto eth5
iface eth5 inet manual

# Sun
auto eth6
iface eth6 inet manual
#
auto eth7
iface eth7 inet manual
etc....

I have no terminal server, so I’ve just done this: Why buy a terminal server when an old PC will do

Right. So now I have my 12 NIC ports. I have 4 serial cables connected for my switches. The only thing left now is the topology itself. Dynamips allows you to breakout your emulated routers to real switches. This is simple to do in the .net file.

Usually in a .net file, you specify that a particular router port is connected to another router port like so:

[[Router CR1]]
  model = 3725
  console = 2001
  slot1 = NM-4T
  slot2 = NM-1FE-TX
  s1/0 = AR1 s1/0
  s1/2 = AR3 s1/2
  Fa0/0 = CR3 Fa0/0
  Fa2/0 = CR2 Fa2/0

In the above configuration, I’m telling dynamips that R1′s S1/0 interface is connected to AR1′s S1/0 interface. R1′s Fa0/0 interface is connected to CR3′s Fa0/0 interface and so on.

Instead of doing it that way, you could do it this way:

[[Router CR1]]
  model = 3725
  console = 2001
  slot1 = NM-4T
  slot2 = NM-1FE-TX
  s1/0 = AR1 s1/0
  s1/2 = AR3 s1/2
  Fa0/0 = NIO_linux_eth:eth3
  Fa0/1 = NIO_linux_eth:eth2

In the above I’m telling dynamips to map R1′s Fa0/0 interface to eth3, and Fa0/1 to eth2. This then allows me to run a cat5 cable from eth3 on the server to a real switch. It’s also important to note that you can mix and match both modes so you can get very complex topologies.

Let’s cook up a quick bash script that will bring up my interfaces and start the dynamips process.

darreno@ubuntu10:~$ sudo vim /etc/ccie.sh
#!/bin/bash
#Bring interfaces up
ifconfig eth1 up
ifconfig eth2 up
ifconfig eth3 up
ifconfig eth4 up
ifconfig eth5 up
ifconfig eth6 up
ifconfig eth7 up
ifconfig eth8 up
ifconfig eth9 up
ifconfig eth10 up
ifconfig eth11 up
ifconfig eth12 up

#Start Hypervisor
dynamips -H 7200 &
sudo chmod +x /etc/ccie.sh
darreno@ubuntu10:~$ sudo /etc/ccie.sh
darreno@ubuntu10:~$ Cisco Router Simulation Platform (version 0.2.8-RC2-amd64)
Copyright (c) 2005-2007 Christophe Fillot.
Build date: May  9 2009 18:06:28

ILT: loaded table "mips64j" from cache.
ILT: loaded table "mips64e" from cache.
ILT: loaded table "ppc32j" from cache.
ILT: loaded table "ppc32e" from cache.

A quick look via ifconfig shows all 12 interfaces up.

Let’s test all of this with a simple topology. 3 routers to break out to a single 3560 switch:

autostart = False
[10.20.30.12:7200]
    workingdir = /data/dynamips/working
    [[3725]]
        image = /data/dynamips/ios/3725/c3725-adventerprisek9-mz.124-15.T14.UNCOMPRESSED.bin
        ram = 142
        idlepc = 0x6026be14
        ghostios = True
    [[ROUTER R5]]
        model = 3725
        console = 2005
        f0/0 = nio_linux_eth:eth4
        f0/1 = nio_linux_eth:eth5
    [[ROUTER R1]]
        model = 3725
        console = 2001
        f0/0 = nio_linux_eth:eth1
    [[ROUTER R3]]
        model = 3725
        console = 2003
        f0/0 = nio_linux_eth:eth2
        f0/1 = nio_linux_eth:eth3

I’ve started the topology up and connected the correct eth ports to the 3560. Let’s have a look at CDP:

R3#sh cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
3560TOP          Fas 0/1            158          S I      WS-C3560- Fas 0/6
3560TOP          Fas 0/0            129          S I      WS-C3560- Fas 0/4

What about on the switch itself?

darreno@ubuntu10:~$ telnet localhost 3000
3560TOP#sh cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
R3               Fas 0/6           152             R S I  3725      Fas 0/1
R3               Fas 0/4           151             R S I  3725      Fas 0/0
R1               Fas 0/5           150             R S I  3725      Fas 0/0
R5               Fas 0/7           149             R S I  3725      Fas 0/0

Finally, let’s do a layer 3 test between 2 routers going through the 3560:
R5

interface FastEthernet0/0
 ip address 10.1.1.5 255.255.255.0

R1

interface FastEthernet0/0
 ip address 10.1.1.1 255.255.255.0

R1#ping 10.1.1.5 repeat 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.1.1.5, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/4/24 ms

Perfect. Everything works just as expected :)

Lab almost complete

I’ve been busy with work and also getting my lab together so I can finally start my lab studies.

I just need a couple more pieces and configuration and then I’m good to go. I’ll be using my dynamips box for the 9 routers (3725s) and 4 switches ( 2 x 3550s & 2 x 3560s)

I’ll be sure to have a blog entry detailing how I’ve connected and configured everything to fit together. The dynamips box will also be my terminal server to connect to the 4 switches.