I notice that it is often suggested to use queues with multiple threads, instead of lists and
.pop(). Is this because lists are not thread-safe, or for some other reason?
Lists themselves are thread-safe. In CPython the GIL protects against concurrent accesses to them, and other implementations take care to use a fine-grained lock or a synchronized datatype for their list implementations. However, while lists themselves can’t go corrupt by attempts to concurrently access, the lists’s data is not protected. For example:
L += 1
is not guaranteed to actually increase L by one if another thread does the same thing, because
+= is not an atomic operation. (Very, very few operations in Python are actually atomic, because most of them can cause arbitrary Python code to be called.) You should use Queues because if you just use an unprotected list, you may get or delete the wrong item because of race conditions.
To clarify a point in Thomas’ excellent answer, it should be mentioned that
append() is thread safe.
This is because there is no concern that data being read will be in the same place once we go to write to it. The
append() operation does not read data, it only writes data to the list.
Here’s a comprehensive yet non-exhaustive list of examples of
list operations and whether or not they are thread safe.
Hoping to get an answer regarding the
obj in a_list language construct here.
I recently had this case where I needed to append to a list continuously in one thread, loop through the items and check if the item was ready, it was an AsyncResult in my case and remove it from the list only if it was ready.
I could not find any examples that demonstrated my problem clearly
Here is an example demonstrating adding to list in one thread continuously and removing from the same list in another thread continuously
The flawed version runs easily on smaller numbers but keep the numbers big enough and run a few times and you will see the error
The FLAWED version
import threading import time # Change this number as you please, bigger numbers will get the error quickly count = 1000 l =  def add(): for i in range(count): l.append(i) time.sleep(0.0001) def remove(): for i in range(count): l.remove(i) time.sleep(0.0001) t1 = threading.Thread(target=add) t2 = threading.Thread(target=remove) t1.start() t2.start() t1.join() t2.join() print(l)
Output when ERROR
Exception in thread Thread-63: Traceback (most recent call last): File "/Users/zup/.pyenv/versions/3.6.8/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/Users/zup/.pyenv/versions/3.6.8/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "<ipython-input-30-ecfbac1c776f>", line 13, in remove l.remove(i) ValueError: list.remove(x): x not in list
Version that uses locks
import threading import time count = 1000 l =  lock = threading.RLock() def add(): with lock: for i in range(count): l.append(i) time.sleep(0.0001) def remove(): with lock: for i in range(count): l.remove(i) time.sleep(0.0001) t1 = threading.Thread(target=add) t2 = threading.Thread(target=remove) t1.start() t2.start() t1.join() t2.join() print(l)
 # Empty list
As mentioned in the earlier answers while the act of appending or popping elements from the list itself is thread safe, what is not thread safe is when you append in one thread and pop in another