Are Python built-in containers thread-safe?
Question:
I would like to know if the Python built-in containers (list, vector, set…) are thread-safe? Or do I need to implement a locking/unlocking environment for my shared variable?
Answers:
They are thread-safe as long as you don’t disable the GIL in C code for the thread.
You need to implement your own locking for all shared variables that will be modified in Python. You don’t have to worry about reading from the variables that won’t be modified (ie, concurrent reads are ok), so immutable types (frozenset, tuple, str) are probably safe, but it wouldn’t hurt. For things you’re going to be changing – list, set, dict, and most other objects, you should have your own locking mechanism (while in-place operations are ok on most of these, threads can lead to super-nasty bugs – you might as well implement locking, it’s pretty easy).
By the way, I don’t know if you know this, but locking is very easy in Python – create a threading.lock object, and then you can acquire/release it like this:
import threading
list1Lock = threading.Lock()
with list1Lock:
# change or read from the list here
# continue doing other stuff (the lock is released when you leave the with block)
In Python 2.5, do from __future__ import with_statement; Python 2.4 and before don’t have this, so you’ll want to put the acquire()/release() calls in try:...finally: blocks:
import threading
list1Lock = threading.Lock()
try:
list1Lock.acquire()
# change or read from the list here
finally:
list1Lock.release()
# continue doing other stuff (the lock is released when you leave the with block)
Some very good information about thread synchronization in Python.
Yes, but you still need to be careful of course
For example:
If two threads are racing to pop() from a list with only one item, One thread will get the item successfully and the other will get an IndexError
Code like this is not thread-safe
if L:
item=L.pop() # L might be empty by the time this line gets executed
You should write it like this
try:
item=L.pop()
except IndexError:
# No items left
The queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics.
I would like to know if the Python built-in containers (list, vector, set…) are thread-safe? Or do I need to implement a locking/unlocking environment for my shared variable?
They are thread-safe as long as you don’t disable the GIL in C code for the thread.
You need to implement your own locking for all shared variables that will be modified in Python. You don’t have to worry about reading from the variables that won’t be modified (ie, concurrent reads are ok), so immutable types (frozenset, tuple, str) are probably safe, but it wouldn’t hurt. For things you’re going to be changing – list, set, dict, and most other objects, you should have your own locking mechanism (while in-place operations are ok on most of these, threads can lead to super-nasty bugs – you might as well implement locking, it’s pretty easy).
By the way, I don’t know if you know this, but locking is very easy in Python – create a threading.lock object, and then you can acquire/release it like this:
import threading
list1Lock = threading.Lock()
with list1Lock:
# change or read from the list here
# continue doing other stuff (the lock is released when you leave the with block)
In Python 2.5, do from __future__ import with_statement; Python 2.4 and before don’t have this, so you’ll want to put the acquire()/release() calls in try:...finally: blocks:
import threading
list1Lock = threading.Lock()
try:
list1Lock.acquire()
# change or read from the list here
finally:
list1Lock.release()
# continue doing other stuff (the lock is released when you leave the with block)
Some very good information about thread synchronization in Python.
Yes, but you still need to be careful of course
For example:
If two threads are racing to pop() from a list with only one item, One thread will get the item successfully and the other will get an IndexError
Code like this is not thread-safe
if L:
item=L.pop() # L might be empty by the time this line gets executed
You should write it like this
try:
item=L.pop()
except IndexError:
# No items left
The queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics.