Check if a python thread threw an exception
Question:
I have a set of tasks to do in parallel, but at the end of them, I need to know if any of the threads threw an exception.
I don’t need to handle the exception directly, I just need to know if one of the threads failed with an exception, so I can cleanly terminate the script
Here is a simple example:
#!/usr/bin/python
from time import sleep
from threading import Thread
def func(a):
for i in range(0,5):
print a
sleep(1)
def func_ex():
sleep(2)
raise Exception("Blah")
x = [Thread(target=func, args=("T1",)), Thread(target=func, args=("T2",)), Thread(target=func_ex, args=())]
print "Starting"
for t in x:
t.start()
print "Joining"
for t in x:
t.join()
print "End"
Before “End”, I want to iterate through the threads, see if any failed, and then decide if I can continue with the script, or if I need to exit at this point.
I don’t need to intercept the exception or stop the other threads, I just need to know at the end if any failed.
Answers:
By the time the join()
call on a thread returns the thread’s stack has been unwound and all information about exceptions has been lost. Thus, unfortunately, you’ll need to provide your own mechanism for registering exceptions; some techniques are discussed here.
A simple technique for situations where you do not need to handle the exception is to use a global list and append to it pertinent information. Your code would become something like:
#!/usr/bin/python
from time import sleep
from threading import Thread, current_thread #needed to get thread name or whatever identifying info you need
threadErrors = [] #global list
def func(a):
for i in range(0,5):
print a
sleep(1)
def func_ex():
global threadErrors #if you intend to change a global variable from within a different scope it has to be declared
try:
sleep(2)
raise Exception("Blah")
except Exception, e:
threadErrors.append([repr(e), current_thread.name]) #append a list of info
raise #re-raise the exception or use sys.exit(1) to let the thread die and free resources
x = [Thread(target=func, args=("T1",)), Thread(target=func, args=("T2",)), Thread(target=func_ex, args=())]
print "Starting"
for t in x:
t.start()
print "Joining"
for t in x:
t.join()
if len(threadErrors) > 0: #check if there are any errors
for e in threadErrors:
print(threadErrors[e][0]+' occurred in thread: '+threadErrors[e][1])
#do whatever with each error info
else:
#there are no errors so do normal clean-up stuff
#do clean-up that should happen in either case here
print "End"
Note: global variables are generally regarded as poor technique, but they are a simple mechanism for communicating between threads. You just have to remember that if one thread is sending info by this route the other thread has to be looking for it.
If you are going to do this in test, I would suggest using pytest-reraise
With it you can do something like this:
def test_assert(reraise):
def run():
with reraise:
assert False
reraise() # This will not raise anything yet
t = Thread(target=run)
t.start()
t.join()
reraise() # This will raise the assertion error
You can use the simple threading.Event class to know when some event has happened in another thread.
from threading import Event, Thread
def func(a, did_raise_exception):
try:
...
except:
did_raise_exception.set()
raise
did_raise_exception = Event()
x = [
Thread(target=func, args=("T1", did_raise_exception)),
Thread(target=func, args=("T2", did_raise_exception)),
...
]
print("Starting")
for t in x:
t.start()
print("Joining")
for t in x:
t.join()
if did_raise_exception.is_set():
# One of the child threads raised an exception.
# Terminate main thread.
...
I have a set of tasks to do in parallel, but at the end of them, I need to know if any of the threads threw an exception.
I don’t need to handle the exception directly, I just need to know if one of the threads failed with an exception, so I can cleanly terminate the script
Here is a simple example:
#!/usr/bin/python
from time import sleep
from threading import Thread
def func(a):
for i in range(0,5):
print a
sleep(1)
def func_ex():
sleep(2)
raise Exception("Blah")
x = [Thread(target=func, args=("T1",)), Thread(target=func, args=("T2",)), Thread(target=func_ex, args=())]
print "Starting"
for t in x:
t.start()
print "Joining"
for t in x:
t.join()
print "End"
Before “End”, I want to iterate through the threads, see if any failed, and then decide if I can continue with the script, or if I need to exit at this point.
I don’t need to intercept the exception or stop the other threads, I just need to know at the end if any failed.
By the time the join()
call on a thread returns the thread’s stack has been unwound and all information about exceptions has been lost. Thus, unfortunately, you’ll need to provide your own mechanism for registering exceptions; some techniques are discussed here.
A simple technique for situations where you do not need to handle the exception is to use a global list and append to it pertinent information. Your code would become something like:
#!/usr/bin/python
from time import sleep
from threading import Thread, current_thread #needed to get thread name or whatever identifying info you need
threadErrors = [] #global list
def func(a):
for i in range(0,5):
print a
sleep(1)
def func_ex():
global threadErrors #if you intend to change a global variable from within a different scope it has to be declared
try:
sleep(2)
raise Exception("Blah")
except Exception, e:
threadErrors.append([repr(e), current_thread.name]) #append a list of info
raise #re-raise the exception or use sys.exit(1) to let the thread die and free resources
x = [Thread(target=func, args=("T1",)), Thread(target=func, args=("T2",)), Thread(target=func_ex, args=())]
print "Starting"
for t in x:
t.start()
print "Joining"
for t in x:
t.join()
if len(threadErrors) > 0: #check if there are any errors
for e in threadErrors:
print(threadErrors[e][0]+' occurred in thread: '+threadErrors[e][1])
#do whatever with each error info
else:
#there are no errors so do normal clean-up stuff
#do clean-up that should happen in either case here
print "End"
Note: global variables are generally regarded as poor technique, but they are a simple mechanism for communicating between threads. You just have to remember that if one thread is sending info by this route the other thread has to be looking for it.
If you are going to do this in test, I would suggest using pytest-reraise
With it you can do something like this:
def test_assert(reraise):
def run():
with reraise:
assert False
reraise() # This will not raise anything yet
t = Thread(target=run)
t.start()
t.join()
reraise() # This will raise the assertion error
You can use the simple threading.Event class to know when some event has happened in another thread.
from threading import Event, Thread
def func(a, did_raise_exception):
try:
...
except:
did_raise_exception.set()
raise
did_raise_exception = Event()
x = [
Thread(target=func, args=("T1", did_raise_exception)),
Thread(target=func, args=("T2", did_raise_exception)),
...
]
print("Starting")
for t in x:
t.start()
print("Joining")
for t in x:
t.join()
if did_raise_exception.is_set():
# One of the child threads raised an exception.
# Terminate main thread.
...