Is Python variable assignment atomic?
Question:
Let’s say I am using a signal handler for handling an interval timer.
def _aHandler(signum, _):
global SomeGlobalVariable
SomeGlobalVariable=True
Can I set SomeGlobalVariable without worrying that, in an unlikely scenario that whilst setting SomeGlobalVariable (i.e. the Python VM was executing bytecode to set the variable), that the assignment within the signal handler will break something? (i.e. meta-stable state)
Update: I am specifically interested in the case where a “compound assignment” is made outside of the handler.
(maybe I am thinking too “low level” and this is all taken care of in Python… coming from an Embedded Systems background, I have these sorts of impulses from time to time)
Answers:
Simple assignment to simple variables is “atomic” AKA threadsafe (compound assignments such as += or assignments to items or attributes of objects need not be, but your example is a simple assignment to a simple, albeit global, variable, thus safe).
Compound assignment involves three steps: read-update-write. This is a race condition if another thread is run and writes a new value to the location after the read happens, but before the write. In this case a stale value is being updated and written back, which will clobber whatever new value was written by the other thread. In Python anything that involves the execution of a single byte code SHOULD be atomic, but compound assignment does not fit this criteria. Use a lock.
Google’s Style Guide advises against it
I’m not claiming that Google styleguides are the ultimate truth, but the rationale in the "Threading" section gives some insight (highlight is mine):
Do not rely on the atomicity of built-in types.
While Python’s built-in data types such as dictionaries appear to have atomic operations, there are corner cases where they aren’t atomic (e.g. if __hash__ or __eq__ are implemented as Python methods) and their atomicity should not be relied upon. Neither should you rely on atomic variable assignment (since this in turn depends on dictionaries).
Use the Queue module’s Queue data type as the preferred way to communicate data between threads. Otherwise, use the threading module and its locking primitives. Learn about the proper use of condition variables so you can use threading.Condition instead of using lower-level locks.
So my interpretation is that in Python everything is dict-like and when you do a = b in the backend somewhere globals['a'] = b is happening, which is bad since dicts are not necessarily thread safe.
For a single variable, Queue is not ideal however since we want it to hold just one element, and I could not find a perfect pre-existing container in the stdlib that automatically synchronizes a .set() method. So for now I’m doing just:
import threading
myvar = 0
myvar_lock = threading.Lock()
with myvar_lock:
myvar = 1
with myvar_lock:
myvar = 2
It is interesting that Martelli does not seem to mind that Google style guide recommendation 🙂 (he works at Google)
I wonder if the CPython GIL has implications to this question: What is the global interpreter lock (GIL) in CPython?
This thread also suggests that CPython dicts are thread safe, including the following glossary quote that explicitly mentions it https://docs.python.org/3/glossary.html#term-global-interpreter-lock
This simplifies the CPython implementation by making the object model (including critical built-in types such as dict) implicitly safe against concurrent access.
you can try dis to see the underlying bytecode.
import dis
def foo():
a = 1
dis.dis(foo)
produces the bytecode:
# a = 1
5 0 LOAD_CONST 1 (1)
2 STORE_FAST 0 (a)
So the assignment is a single python bytecode (instruction 2), which is atomic in CPython since it executes one bytecode at a time.
whereas, adding one a += 1:
def foo():
a = 1
a += 1
produces the bytecode:
# a+=1
6 4 LOAD_FAST 0 (a)
6 LOAD_CONST 1 (1)
8 INPLACE_ADD
10 STORE_FAST 0 (a)
+= corresponds to 4 instructions, which is not atomic.
Let’s say I am using a signal handler for handling an interval timer.
def _aHandler(signum, _):
global SomeGlobalVariable
SomeGlobalVariable=True
Can I set SomeGlobalVariable without worrying that, in an unlikely scenario that whilst setting SomeGlobalVariable (i.e. the Python VM was executing bytecode to set the variable), that the assignment within the signal handler will break something? (i.e. meta-stable state)
Update: I am specifically interested in the case where a “compound assignment” is made outside of the handler.
(maybe I am thinking too “low level” and this is all taken care of in Python… coming from an Embedded Systems background, I have these sorts of impulses from time to time)
Simple assignment to simple variables is “atomic” AKA threadsafe (compound assignments such as += or assignments to items or attributes of objects need not be, but your example is a simple assignment to a simple, albeit global, variable, thus safe).
Compound assignment involves three steps: read-update-write. This is a race condition if another thread is run and writes a new value to the location after the read happens, but before the write. In this case a stale value is being updated and written back, which will clobber whatever new value was written by the other thread. In Python anything that involves the execution of a single byte code SHOULD be atomic, but compound assignment does not fit this criteria. Use a lock.
Google’s Style Guide advises against it
I’m not claiming that Google styleguides are the ultimate truth, but the rationale in the "Threading" section gives some insight (highlight is mine):
Do not rely on the atomicity of built-in types.
While Python’s built-in data types such as dictionaries appear to have atomic operations, there are corner cases where they aren’t atomic (e.g. if
__hash__or__eq__are implemented as Python methods) and their atomicity should not be relied upon. Neither should you rely on atomic variable assignment (since this in turn depends on dictionaries).Use the
Queuemodule’s Queue data type as the preferred way to communicate data between threads. Otherwise, use the threading module and its locking primitives. Learn about the proper use of condition variables so you can usethreading.Conditioninstead of using lower-level locks.
So my interpretation is that in Python everything is dict-like and when you do a = b in the backend somewhere globals['a'] = b is happening, which is bad since dicts are not necessarily thread safe.
For a single variable, Queue is not ideal however since we want it to hold just one element, and I could not find a perfect pre-existing container in the stdlib that automatically synchronizes a .set() method. So for now I’m doing just:
import threading
myvar = 0
myvar_lock = threading.Lock()
with myvar_lock:
myvar = 1
with myvar_lock:
myvar = 2
It is interesting that Martelli does not seem to mind that Google style guide recommendation 🙂 (he works at Google)
I wonder if the CPython GIL has implications to this question: What is the global interpreter lock (GIL) in CPython?
This thread also suggests that CPython dicts are thread safe, including the following glossary quote that explicitly mentions it https://docs.python.org/3/glossary.html#term-global-interpreter-lock
This simplifies the CPython implementation by making the object model (including critical built-in types such as dict) implicitly safe against concurrent access.
you can try dis to see the underlying bytecode.
import dis
def foo():
a = 1
dis.dis(foo)
produces the bytecode:
# a = 1
5 0 LOAD_CONST 1 (1)
2 STORE_FAST 0 (a)
So the assignment is a single python bytecode (instruction 2), which is atomic in CPython since it executes one bytecode at a time.
whereas, adding one a += 1:
def foo():
a = 1
a += 1
produces the bytecode:
# a+=1
6 4 LOAD_FAST 0 (a)
6 LOAD_CONST 1 (1)
8 INPLACE_ADD
10 STORE_FAST 0 (a)
+= corresponds to 4 instructions, which is not atomic.