Basic Python Multithreading

The first ‘proper’ Python app I made logged onto a list of devices and pulled out OSPF state. This worked perfectly fine. The app correctly works out whether it can log into a device or not, and waits a few seconds to ensure a device actually responds.

The issue is that if I have a list of say 1000 devices, and 500 of them don’t respond, the amount of time you need to wait rapidly increases as it looks at each one in turn. Would it not be better for the app to be able to log into multiple devices at the same time in parallel? This would drastically reduce the runtime.

Basic Threading

Consider the following code:

#!/usr/bin/python

import threading

def thread_test():
    print "I am a thread"
    return

threads = []
for i in range(4):
    t = threading.Thread(target = thread_test)
    threads.append(t)
    t.start()

A module is defined called thread_test. I then spawn four threads, each of which run the module. I should therefore see four lines printed:

$ ./thread.py
I am a thread
I am a thread
I am a thread
I am a thread

Of course getting them all to do exactly the same thing is a bit boring. I may have a list of items I want to print. Let’s pass each item as an argument and print them out:

#!/usr/bin/python

import threading

list_of_items = ["cat", "banana", "house", "phone"]

def thread_test(item):
    print "I am a " + item
    return

threads = []
for word in list_of_items:
    t = threading.Thread(target = thread_test, args = (word,))
    threads.append(t)
    t.start()
$ ./thread.py
I am a cat
I am a banana
I am a house
I am a phone

If you’ve run this code, you may notice that sometimes your output gets a bit garbled:

$ ./thread.py
I am a catI am a banana

I am a house
I am a phone

$ ./thread.py
I am a cat
 I am a banana
I am a house
I am a phone

All four threads are trying to write to the screen at the same time. If outputting to the screen, or writing to a file, this output can look rather messy. Especially as the device and thread count goes up.

Locks

I can use locks to prevent this. Each thread can go do it’s business, but if I need to write to the screen or write to a file, I ensure only a single thread can do this at a time. As an example I’ll iterate through a list of 100. All those threads will create their data in memory at pretty much the same time, but I’ll ensure only one at a time can print and write to a file. I’ll also ensure that the application closes the file only after all threads are completed.

#!/usr/bin/python

import threading

lock = threading.Lock()

def thread_test(num):
    phrase = "I am number " + str(num)
    lock.acquire()
    print phrase
    f.write(phrase + "\n")
    lock.release()

threads = []
f = open("text.txt", 'w')
for i in range (100):
    t = threading.Thread(target = thread_test, args = (i,))
    threads.append(t)
    t.start()

while threading.activeCount() > 1:
    pass
else:
    f.close()

Any code between the locks acquiring and releasing can only be done one at a time. The example above doesn’t show a great example, but the action of getting data and waiting from a remote device can take a few seconds. If that can all be done at the same time, then results written once at a time to a file, it would speed things up immensely.

Update – 15/12/14

Ben Cardy below mentioned a great shortcut that most viewers might miss if not reading all the comments. For that reason I’ll put it up here. My code above acquires and releases a lock when needed. There is a simpler way to do this. If you code with lock, any code indented after will essentially be wrapped in lock codes. This is nice as you don’t have to remember to release the lock. Another benefit is that the with code will release the lock even if the thread throws an exception.

The last code above could be rewritten like so:

#!/usr/bin/python
 
import threading
 
lock = threading.Lock()
 
def thread_test(num):
    phrase = "I am number " + str(num)
    with lock:
        print phrase
        f.write(phrase + "\n")

 
threads = []
f = open("text.txt", 'w')
for i in range (100):
    t = threading.Thread(target = thread_test, args = (i,))
    threads.append(t)
    t.start()
 
while threading.activeCount() > 1:
    pass
else:
    f.close()

© 2009-2019 Darren O'Connor All Rights Reserved -- Copyright notice by Blog Copyright