Passing an argument into Python

On April 16, 2014, in Python, by Darren

Up until now, I’ve been reading a file of a list of devices to log into and do some work on. That’s ideal when I have a load of routers to check, but sometimes I’d like to check one router quickly directly.

Using my OSPF Checker as an example,I currently iterate through a list of lines in devices.txt that the app then logs into. I can instead make it so that if an argument is passed in, that becomes the list and the file is ignored. If no argument is passed, I can read in the original list of devices.

To demonstrate, I’ll set up a really simple script:

#!/usr/bin/python3
import sys
print("You have passed in",sys.argv)

If I run this now form the cli I get the following:

darreno@Jumpbox:~/git/ospf_checker$ ./arg.py
You have passed in ['./arg.py']

There is only a single argument, as the script itself counts itself as arg location 0. If I pass in an argument:

darreno@Jumpbox:~/git/ospf_checker$ ./arg.py test
You have passed in ['./arg.py', 'test']

Most of the time I want to do some work on what’s been added after the script name. We can just be sure to look at that:

#!/usr/bin/python3
import sys
print("You have passed in",sys.argv[1])

Now when run:

darreno@Jumpbox:~/git/ospf_checker$ ./arg.py test
You have passed in test

If I run the script without any arguments, I’ll get a traceback as I’m referencing an index value that doesn’t exist:

darreno@Jumpbox:~/git/ospf_checker$ ./arg.py
Traceback (most recent call last):
  File "./arg.py", line 3, in 
    print("You have passed in",sys.argv[1])
IndexError: list index out of range

I can use this indexerror to do something else if no argument exists:

#!/usr/bin/python3
import sys
try:
    print("You have passed in",sys.argv[1])
except IndexError:
    print("No argument was passed, I'll do something else")
darreno@Jumpbox:~/git/ospf_checker$ ./arg.py test
You have passed in test

darreno@Jumpbox:~/git/ospf_checker$ ./arg.py
No argument was passed, I'll do something else

Very handy.

I’ve just scratched the surface of arguments so I’ll do a more in depth post sometime in the future

Tagged with:  

Defined Functions – Python

On April 14, 2014, in Python, by Darren

Currently my apps pretty much have a single main body in which all the data is collected, processed, and displayed/saved. A more Pythonic way of doing this is to get functions to do the work in a separate part of the application. In the main body I can then ask these functions to crunch some data I pass to it, and then receive a result back. At first this makes my code look a bit longer, but it should make it easier to debug in future as each defined function can be troubleshooted separately.

Let’s take a look at a very simple defined function:

>>> def square(x):
	x*=x
	return x

A function is defined named square. It accepts one argument. Whatever argument is passed to the function is squared, and the result passed back. I can now call square from the main body of the app and pass it my argument and I should get the square of that argument back:

>>> square(10)
100

If I have a variable with a value, I can pass the variable:

>>> a = 50
>>> square(a)
2500

It’s hard to see the benefits when I have a small app like above, but it makes a lot of sense in a much bigger app. I’ve now updated my OSPF Checker app to use definitions instead. The following is a few examples of the code:

def getip(i):
    ip = re.findall(r'Internet Address (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})',i)
    return ip

def getarea(i):
    area = re.findall(r'Area ([\s]{0,3}[0-9]{1,5})',i)
    return area

for i in ospf:
    ip = getip(i)
    area = getarea(i)

Each definition can now be changed as a separate ‘app’ – A great example is when I ran my script on an LNS box with virtual-access interfaces. The getip function didn’t take into account the slightly different output:

Virtual-Access111 is up, line protocol is up
  Interface is unnumbered. Using address of Loopback56146 (10.9.0.250), Area 0, Attached via Network Statement

I can go in and just change the getip function:

def getip(i):
    ip = re.findall(r'Internet Address (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})',i)
    if not ip:
        ip = re.findall(r'Interface is unnumbered. Using address of Loopback[0-9]{0,5}',i)
    return ip

App now picks up those sorts of interfaces just fine:

Int:	Virtual-Access266 
IP:	Interface is unnumbered. Using address of Loopback78437
Area:	0
Type:	POINT_TO_POINT
Cost:	1
Hello:	10
Dead:	40
Tagged with:  

Regex – Lookahead Assertions – Python

On April 14, 2014, in Python, by Darren

While trying to scrape information out of a router output, I realised that sometimes I just think about things in the wrong way. Basically I end up making my life difficult when there is usually an easier way to go.

Ultimately I’m not even happy with my result at the moment. I’m trying to work our better ways to get the info I need, but it’s a learning process so I’m happy with that.

Let’s start with this block of text:

cr1.123456#sh ip ospf interface
Loopback0 is up, line protocol is up
  Internet Address 10.90.3.38/32, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type LOOPBACK, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Loopback interface is treated as a stub Host
GigabitEthernet0/1 is up, line protocol is up
  Internet Address 172.16.13.150/29, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type BROADCAST, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Transmit Delay is 1 sec, State DR, Priority 1
  Designated Router (ID) 10.90.3.38, Interface address 172.16.13.150
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    No Hellos (Passive interface)
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 3/3, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 0, maximum is 0
  Last flood scan time is 0 msec, maximum is 0 msec
  Neighbor Count is 0, Adjacent neighbor count is 0
  Suppress hello for 0 neighbor(s)
  Message digest authentication enabled
      No key configured, using default key id 0
GigabitEthernet0/0.2561 is up, line protocol is up
  Internet Address 10.22.0.117/30, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type POINT_TO_POINT, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Transmit Delay is 1 sec, State POINT_TO_POINT
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:05
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 1/1, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 0 msec, maximum is 0 msec
  Neighbor Count is 1, Adjacent neighbor count is 1
    Adjacent with neighbor 217.196.224.51
  Suppress hello for 0 neighbor(s)
  Message digest authentication enabled
    Youngest key id is 1

I would like to extract information from the above text. I want my app to list each OSPF enabled interface, plus its properties.

Initially I had my script look for various expressions, and then iterate through to give me what I wanted. However there was a serious issue with this:

 i = re.findall('(?:GigabitEthernet|FastEthernet|Serial|Tunnel|Loopback|Dialer|BVI)[0-9]{1,4}/?[0-9]{0,4}.?[0-9]{0,4}/?[0-9]{0,3}/?[0-9]{0,3}/?[0-9]{0,3}:?[0-9]{0,3}',output)
ip = re.findall(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})',output)
A = re.findall(r'Area[\s]{0,3}[0-9]{1,5}',output)
n = re.findall(r'Network Type[\s]{0,3}[a-zA-Z_]{0,20}',output)
C = re.findall(r'Cost:[\s]{0,3}[0-9]{1,5}',output)
h = re.findall(r'Hello[\s][0-9]{1,3}',output)
D = re.findall(r'Dead[\s][0-9]{1,3}',output)
for a,b,c,d,e,f,g in zip(i,ip,A,n,C,h,D):
    print("\n"+a)
    print(b)
    print(c)
    print(d)
    print(e)
    print(f)
    print(g)

The problem with the above code is that Loopback interface don’t have hello and dead timers. This meant the output scraped ended up showing the first capture of a Hello: being matched with the Loopback0 interface. It also meant the last few interfaces were not shown. Basically the output was completely wrong:

Darrens-MacBook-Pro:Desktop darrenoconnor$ python3 1.py

Loopback0
10.90.3.38
Area 30395
Network Type LOOPBACK
Cost: 1
Hello 10
Dead 40

GigabitEthernet0/1
10.90.3.38
Area 30395
Network Type BROADCAST
Cost: 1
Hello 10
Dead 40

My first thought was to try and split the output into three separate strings. Each interface would be the start of the string and I could then search in the smaller string. Essentially like this:

cr1.123456#sh ip ospf interface

======== Begin String One ==========
Loopback0 is up, line protocol is up
  Internet Address 10.90.3.38/32, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type LOOPBACK, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Loopback interface is treated as a stub Host
========= End String One ===========

======== Begin String Two ==========
GigabitEthernet0/1 is up, line protocol is up
  Internet Address 172.16.13.150/29, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type BROADCAST, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
========= End String Two ===========

======== Begin String ETC ==========

At first this turned out to be a lot hardher than I thought it would be. Initially I tried to count the lines in which the interface name appeared. I would then split from 0 to the first point, then from the first point to the second point. this turned out to be a lot more difficult than I thought. Counting the lines where the interface names appeared was easy:

int = re.compile(r'GigabitEthernet|FastEthernet|Serial|Tunnel|Loopback|Dialer|BVI')
for (i,line) in enumerate(output):
   if int.match(line):
      for match in int.findall(line):
         print (i,line)

This gives me:

Darrens-MacBook-Pro:Desktop darrenoconnor$ python3 1.py
1 Loopback0 is up, line protocol is up

7 GigabitEthernet0/1 is up, line protocol is up

29 GigabitEthernet0/0.2561 is up, line protocol is up

But thinking about this, this simply cannot be very efficient. Get some output, convert to string, convert to list, convert back to string based on line numbers? Also Python doesn’t seem to make it very easy to create strings based on lines numbers of a list. At least I haven’t found.

So after trying for a few hours to get the above to work, I though there simply had to be a better way. Why not read the router output into a string, and then split that string based on the first occurence of the interface name. That should give me separte strings for each interface.

output_split = re.split(r'GigabitEthernet|FastEthernet|Serial|Tunnel|Loopback|Dialer|BVI', output)
print(len(output_split))
for i in output_split:
    print(i)

However this creates a big issue. When splitting on a matching regex group, the part matched will be removed from the string. I did get the right amount of splits at least. I would expect a list length of 4 as there are three interfaces as well as the first line of show ip ospf interfaces:

Darrens-MacBook-Pro:Desktop darrenoconnor$ python3 1.py
4

However look at my actual output:

 is up, line protocol is up
  Internet Address 10.90.3.38/32, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type LOOPBACK, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base

0/1 is up, line protocol is up
  Internet Address 172.16.13.150/29, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type BROADCAST, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name

The interface names matched are now gone!

Python’s re module allows you to do a lookahead assertion. Basically this means it can look forward and match a regex, but the matched regex is NOT removed when doing the split. This is exactly what we want.

Now I’ll match on a newline followed immidiately by my interface name. I’ll split on the newline, but do a lookahead assertion on the interface name itself. To do a lookahead assertion you match the lookahead with (?=) I still want to remote the newline so that will come before the lookahead. For now I’ll also add a couple of * and newlines so you can see where the split occurs when printed out.

output_split = re.split(r'\n(?=GigabitEthernet|FastEthernet|Serial|Tunnel|Loopback|Dialer|BVI)', output)
print(len(output_split))
for i in output_split:
    print("\n******\n"+i)

I should now have a list length of 4, and my interface names intact:

Darrens-MacBook-Pro:Desktop darrenoconnor$ python3 1.py
4

******
cr1.123456#sh ip ospf interface

******
Loopback0 is up, line protocol is up
  Internet Address 10.90.3.38/32, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type LOOPBACK, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Loopback interface is treated as a stub Host

******
GigabitEthernet0/1 is up, line protocol is up
  Internet Address 172.16.13.150/29, Area 303953, Attached via Network Statement
  Process ID 1, Router ID 10.90.3.38, Network Type BROADCAST, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base

The rest of the app is now easy. I can iterate through my split list. If no hello/dead timer is found in the loopback split, it doesn’t add it to the loopback section. My interfaces will now all be shown with the correct values.

Ultimately after cleaning up some outputs my output now gives me:

Darrens-MacBook-Pro:Desktop darrenoconnor$ python3 1.py

Int:	Loopback0
IP:	10.90.3.38
Area:	30395
Type:	LOOPBACK
Cost	1

Int:	GigabitEthernet0/1
IP:	172.16.13.150
Area:	30395
Type:	BROADCAST
Cost	1
Hello:	10
Dead:	40

Int:	GigabitEthernet0/0.2561
IP:	10.22.0.117
Area:	30395
Type:	POINT_TO_POINT
Cost	1
Hello:	10
Dead:	40

I’ve moved the above script now to github

There are still a large amount of changes I’d like to make. The next step is to move the regular expressions to some definitions. For now it’s doing what I want it to do.

Tagged with:  

If doing a show inventory on a Cisco device, the output looks as follows (hostnames and serials changed to protect the innocent)

foo.bar1#sh inv
NAME: "Chassis", DESCR: "Cisco 7204VXR, 4-slot chassis"
PID: CISCO7204VXR      , VID:    , SN: 35246813

NAME: "NPE-G2 0", DESCR: "Cisco 7200 Series Network Processing Engine NPE-G2"
PID: NPE-G2            , VID: V03 , SN: FOO16810684

NAME: "disk2", DESCR: "256MB Compact Flash Disk for NPE-G2"
PID: MEM-NPE-G2-FLD256 , VID:    , SN:

NAME: "Power Supply 1", DESCR: "Cisco 7200 AC Power Supply"
PID: PWR-7200-AC       , VID:    , SN:

I can read the output and via regular expressions, extract the needed information. Let’s say I want to grab the PID references as well as the serial numbers associated with it. Note that not every single PID has a serial number.

When creating a re.findall regular expression in Python, the results are fed back via a list. As an example:

>>> h = re.findall(r'pid:[\s\t]{0,3}[a-z0-9\-]{0,12}', text, re.IGNORECASE)
>>> h
['PID: CISCO7204VXR', 'PID: NPE-G2', 'PID: MEM-NPE-G2-F', 'PID: PWR-7200-AC']
>>> s = re.findall(r'sn:[\s\t]{0,3}[a-z0-9]{0,20}[?!\n\t\s]', text, re.IGNORECASE)
>>> s
['SN: 35246813\n', 'SN: FOO16810684\n', 'SN:\n\n']

I would like to print out the hardware, but followed immediately by the serial number field, even if a serial doesn’t exist. Essentially I need to iterate through both the hardware and serial lists at the same time. If I try to one at a time, I’ll just get all the PIDs lumped together while all the serial numbers lumped together.

Python makes this very easy to go through both at the same time:

h = re.findall(r'pid:[\s\t]{0,3}[a-z0-9\-]{0,12}', text, re.IGNORECASE)
s = re.findall(r'sn:[\s\t]{0,3}[a-z0-9]{0,20}[?!\n\t\s]', text, re.IGNORECASE)
>>> for i,j in zip(h,s):
	print(i+"\n"+j)

	
PID: CISCO7204VXR
SN: 35246813

PID: NPE-G2
SN: FOO16810684

PID: MEM-NPE-G2-F
SN:

Short and Sweet

Tagged with:  

Practising with regular expressions

On April 9, 2014, in Python, by Darren

My last post stated I needed a bit of practise with regular expressions. Python has a live interpreter which is extremely handy for testing out regular expression matches. I’ve created a text file and dumped a whole load of code version, serial, device type, IP, etc outputs in. I can then use regular expression matches to see if I can catch everything I want.

This is my random text with some values I want to capture:

Version 15.3(1)S 5B:9C:CD:5E:D9:65 So if he into shot half many long. SN: FOC1716X2LW
PID: ME-3600X-24TS-MSN: FCZ4836X2LW
PID: PWR-ME3KX-AC Attention he extremity unwilling on otherwise. Conviction up partiality as
China fully 5B:12:EA:4F:E0:57him every fat was world grave. Version 12.2(25)SEB
Version 15.2(4)S2-Version 12.2(50)SE3,abcd.1234.AB14
Version 12.4(4)XD5, SN: LIT171218BU
SN: FOC1716X2LW 192.168.10.5, 10.1.1.4/32; 8.8.8.8Version 15.2(4)S2
Version 12.4(15)T10 Yet e9:f4:43:3a:53:91jennings resolved disposed off.Version 12.2(55)SE1
Version 12.2(55)SE5 1.1.50.1, SN: ABC5256X2LW,,PID: ME-3600X-24TS-M

Regex Basics

Whole books have been written on regular expressions, so I’m not going to re-invent the wheel. However let’s go over some basics that I have found handy in the re module in Python.
If we wanted to match any occurrences of a – z at a position we can use:

[a-z]

This only matches lower-case a – z. If you wanted to match uppercase as well there are two ways to do so:

[a-zA-Z]

or

[a-z], text, re.IGNORECASE

If I wanted to match 2 numbers, I could do it two ways as well:

[0-9][0-9]

or

[0-9]{2}

What if I wanted to match anything from 1 – 3 numbers in succession? :

[0-9]{1,3}

Special characters can also be matched. Whitespace and Tab is as follows:

[\s\t]

Certain special characters need an escape character. Brackets is an example:

[\(\)]

Match IOS Version

First lets read our jumbled file and get it into a string:

>>> f = open('string.txt')
>>> text = f.read()
>>> type(text)
<class 'str'>

IOS seems to always use uppercase for Version, but lets not assume anything. I need to match Version, space, two numbers, dot, one or more numbers, open brackets, one or more numbers, close brackets, one or more letters or numbers:

v = re.findall(r'version[\s\t]{0,3}[0-9]{2}.[0-9]{0,3}\([0-9]{0,3}\)[a-z0-9]{0,3}', text, re.IGNORECASE)

When I echo back just v, I get a list of items it’s matched:

>>> v
['Version 15.3(1)S', 'Version 12.2(25)SEB', 'Version 15.2(4)S2', 'Version 12.2(50)SE3', 'Version 12.4(4)XD5', 'Version 15.2(4)S2',
'Version 12.4(15)T10', 'Version 12.2(55)SE1', 'Version 12.2(55)SE5']

To get a slightly cleaner output I could use an iterator and print one item in the string at a time:

>>> for i in v:
	print(i)

	
Version 15.3(1)S
Version 12.2(25)SEB
Version 15.2(4)S2
Version 12.2(50)SE3
Version 12.4(4)XD5
Version 15.2(4)S2
Version 12.4(15)T10
Version 12.2(55)SE1
Version 12.2(55)SE5

Match Hardware

A ‘show inventory’ on IOS lists all hardware prefaced with PID: – So let’s start with that:

>>> h = re.findall(r'pid:[\s\t]{0,3}[a-z0-9\-]{0,12}', text, re.IGNORECASE)
>>> for i in h:
	print(i)

	
PID: ME-3600X-24T
PID: PWR-ME3KX-AC
PID: ME-3600X-24T

The regex matches pid: (ignoring case), zero to three tabs or whitespace, followed by zero to twelve letters or numbers, including hyphens.

Match Serial Number

This is slightly easier. Show inventory on IOS prefaces the serial number with SN:

>>> s = re.findall(r'sn:[\s\t]{0,3}[a-z0-9\-]{0,20}', text, re.IGNORECASE)
>>> for i in s:
	print(i)

	
SN: FOC1716X2LW
SN: FCZ4836X2LW
SN: LIT171218BU
SN: FOC1716X2LW
SN: ABC5256X2LW

Match IP addresses

Note that this particular expression will also match invalid IPs. i.e. it will also match 999.999.999.999:

>>> ip = re.findall(r'((?:[0-9]{1,3}\.){3}[0-9]{1,3})', text)
>>> for i in ip:
	print(i)

	
192.168.10.5
10.1.1.4
8.8.8.8
1.1.50.1

Match MAC addresses

MAC address format can sometimes be quite different. Sometimes it looks like ff:ff:ff:ff:ff:ff, sometimes FF:FF:FF:FF:FF:FF, sometimes ffff.ffff.ffff. Let’s first concentrate on the first two:

>>> m = re.findall(r'((?:[0-9a-f]{2}:){5}[0-9a-f]{2})', text, re.IGNORECASE)
>>> for i in m:
	print(i)

	
5B:9C:CD:5E:D9:65
5B:12:EA:4F:E0:57
e9:f4:43:3a:53:91

To find the other type of MAC:

>>> m = re.findall(r'((?:[0-9a-f]{4}\.){2}[0-9a-f]{4})', text, re.IGNORECASE)
>>> for i in m:
	print(i)

	
abcd.1234.AB14

Conclusions

Regular expressions are awfully handy. I’m slowly getting more used to them. Once you figure out patterns it becomes a bit easier. The next step for me is to aggregate both MAC address patterns as well as search for all manner of IPv6 addresses!

Tagged with:  

© 2009-2014 Darren O'Connor All Rights Reserved