How do I get a thread safe print in Python 2.6?
Question:
Answers:
I dont know if there is any better way instead this locking mechanism, but atleast it looks easy. I am also not sure if printing really isnt thread safe.
Edit: Okay tested it my self now, you are right, you can get really wierd looking output. And you dont need the future import, its just there, because i use Python 2.7.
from __future__ import print_function
from threading import Lock
print_lock = Lock()
def save_print(*args, **kwargs):
with print_lock:
print (*args, **kwargs)
save_print("test", "omg", sep='lol')
The issue is that python uses seperate opcodes for the NEWLINE printing and the printing of the object itself. The easiest solution is probably to just use an explicit sys.stdout.write with an explicit newline.
Interesting problem — considering all the things that happen within a print statement, including the setting and checking of the softspace attribute, making it “threadsafe” (meaning, actually: a thread that’s printing only yields “control of standard output” to another thread when it’s printing a newline, so that each entire line that’s output is guaranteed to come from a single thread) was a bit of a challenge (the usual easy approach to actual thread safety — delegating a separate thread to exclusively “own” and handle sys.stdout, communicate to it via Queue.Queue — isn’t all that useful, since the problem is not thread safety [[even with a plain print there is no risk of crashing and the characters that end up on standard output are exactly those which get printed]] but the need for mutual exclusion among threads for an extended range of operations).
So, I think I made it…:
import random
import sys
import thread
import threading
import time
def wait():
time.sleep(random.random())
return 'W'
def targ():
for n in range(8):
wait()
print 'Thr', wait(), thread.get_ident(), wait(), 'at', wait(), n
tls = threading.local()
class ThreadSafeFile(object):
def __init__(self, f):
self.f = f
self.lock = threading.RLock()
self.nesting = 0
def _getlock(self):
self.lock.acquire()
self.nesting += 1
def _droplock(self):
nesting = self.nesting
self.nesting = 0
for i in range(nesting):
self.lock.release()
def __getattr__(self, name):
if name == 'softspace':
return tls.softspace
else:
raise AttributeError(name)
def __setattr__(self, name, value):
if name == 'softspace':
tls.softspace = value
else:
return object.__setattr__(self, name, value)
def write(self, data):
self._getlock()
self.f.write(data)
if data == 'n':
self._droplock()
# comment the following statement out to get guaranteed chaos;-)
sys.stdout = ThreadSafeFile(sys.stdout)
thrs = []
for i in range(8):
thrs.append(threading.Thread(target=targ))
print 'Starting'
for t in thrs:
t.start()
for t in thrs:
t.join()
print 'Done'
The calls to wait are intended to guarantee chaotically mixed output in the absence of this mutual exclusion guarantee (whence the comment). With the wrapping, i.e., the above code exactly as it looks there, and (at least) Python 2.5 and up (I believe this may run in earlier versions, too, but I don’t have any easily at hand to check) the output is:
Thr W -1340583936 W at W 0
Thr W -1340051456 W at W 0
Thr W -1338986496 W at W 0
Thr W -1341116416 W at W 0
Thr W -1337921536 W at W 0
Thr W -1341648896 W at W 0
Thr W -1338454016 W at W 0
Thr W -1339518976 W at W 0
Thr W -1340583936 W at W 1
Thr W -1340051456 W at W 1
Thr W -1338986496 W at W 1
...more of the same...
The “serialization” efect (whereby the threads appear to “nicely round-robin” as above) is a side effect of the fact that the thread that gets to be the currently-printing one is seriously slower than the others (all those waits!-). Commenting out the time.sleep in wait, the output is instead
Thr W -1341648896 W at W 0
Thr W -1341116416 W at W 0
Thr W -1341648896 W at W 1
Thr W -1340583936 W at W 0
Thr W -1340051456 W at W 0
Thr W -1341116416 W at W 1
Thr W -1341116416 W at W 2
Thr W -1338986496 W at W 0
...more of the same...
i.e. a more typical “multithreaded output”… except for the guarantee that each line in the output comes entirely from one single thread.
Of course, a thread that does, e.g., print 'ciao', will keep “ownership” of standard output until it finally does perform a print without a trailing comma, and other threads wanting to print may sleep for quite a while (how else can one guarantee that each line in the output comes from a single thread? well, one architecture would be to accumulate partial lines to thread local storage instead of actually writing them to standard output, and only do the writing on receipt of the n… delicate to interleave properly with softspace settings, I fear, but probably feasible).
Through experimentation, I found that the following works, is simple, and suits my needs:
print "your string heren",
Or, wrapped in a function,
def safe_print(content):
print "{0}n".format(content),
My understanding is that the implicit newline of a normal print is actually output to stdout in a separate operation, causing the race condition with other print operations. By removing this implicit newline with the added ,, and instead including the newline in the string, we can avoid this problem.
2020 Edit: Here’s the Python 3 version of this (thanks to Bob Stein in the comments for the inspiration):
def safe_print(*args, sep=" ", end="", **kwargs):
joined_string = sep.join([ str(arg) for arg in args ])
print(joined_string + "n", sep=sep, end=end, **kwargs)
As pointed out by Bob Stein, relying on print to join multiple passed arguments results in garbled output, so we have to do it ourselves.
2017 Edit: this answer is starting to pick up some steam, so I just wanted to make a clarification. This doesn’t actually make print “thread safe” exactly. The output may be in the wrong order if the prints happen microseconds apart from each other. What this does do, however, is avoid garbled output coming from print statements executed from concurrent threads, which is what most people really want when asking this question.
Here is a test to show what I mean:
from concurrent.futures import ThreadPoolExecutor
def normal_print(content):
print content
def safe_print(content):
print "{0}n".format(content),
with ThreadPoolExecutor(max_workers=10) as executor:
print "Normal Print:"
for i in range(10):
executor.submit(normal_print, i)
print "---"
with ThreadPoolExecutor(max_workers=10) as executor:
print "Safe Print:"
for i in range(10):
executor.submit(safe_print, i)
Output:
Normal Print:
0
1
23
4
65
7
9
8
----
Safe Print:
1
0
3
2
4
5
6
7
8
9
I dont know if there is any better way instead this locking mechanism, but atleast it looks easy. I am also not sure if printing really isnt thread safe.
Edit: Okay tested it my self now, you are right, you can get really wierd looking output. And you dont need the future import, its just there, because i use Python 2.7.
from __future__ import print_function
from threading import Lock
print_lock = Lock()
def save_print(*args, **kwargs):
with print_lock:
print (*args, **kwargs)
save_print("test", "omg", sep='lol')
The issue is that python uses seperate opcodes for the NEWLINE printing and the printing of the object itself. The easiest solution is probably to just use an explicit sys.stdout.write with an explicit newline.
Interesting problem — considering all the things that happen within a print statement, including the setting and checking of the softspace attribute, making it “threadsafe” (meaning, actually: a thread that’s printing only yields “control of standard output” to another thread when it’s printing a newline, so that each entire line that’s output is guaranteed to come from a single thread) was a bit of a challenge (the usual easy approach to actual thread safety — delegating a separate thread to exclusively “own” and handle sys.stdout, communicate to it via Queue.Queue — isn’t all that useful, since the problem is not thread safety [[even with a plain print there is no risk of crashing and the characters that end up on standard output are exactly those which get printed]] but the need for mutual exclusion among threads for an extended range of operations).
So, I think I made it…:
import random
import sys
import thread
import threading
import time
def wait():
time.sleep(random.random())
return 'W'
def targ():
for n in range(8):
wait()
print 'Thr', wait(), thread.get_ident(), wait(), 'at', wait(), n
tls = threading.local()
class ThreadSafeFile(object):
def __init__(self, f):
self.f = f
self.lock = threading.RLock()
self.nesting = 0
def _getlock(self):
self.lock.acquire()
self.nesting += 1
def _droplock(self):
nesting = self.nesting
self.nesting = 0
for i in range(nesting):
self.lock.release()
def __getattr__(self, name):
if name == 'softspace':
return tls.softspace
else:
raise AttributeError(name)
def __setattr__(self, name, value):
if name == 'softspace':
tls.softspace = value
else:
return object.__setattr__(self, name, value)
def write(self, data):
self._getlock()
self.f.write(data)
if data == 'n':
self._droplock()
# comment the following statement out to get guaranteed chaos;-)
sys.stdout = ThreadSafeFile(sys.stdout)
thrs = []
for i in range(8):
thrs.append(threading.Thread(target=targ))
print 'Starting'
for t in thrs:
t.start()
for t in thrs:
t.join()
print 'Done'
The calls to wait are intended to guarantee chaotically mixed output in the absence of this mutual exclusion guarantee (whence the comment). With the wrapping, i.e., the above code exactly as it looks there, and (at least) Python 2.5 and up (I believe this may run in earlier versions, too, but I don’t have any easily at hand to check) the output is:
Thr W -1340583936 W at W 0
Thr W -1340051456 W at W 0
Thr W -1338986496 W at W 0
Thr W -1341116416 W at W 0
Thr W -1337921536 W at W 0
Thr W -1341648896 W at W 0
Thr W -1338454016 W at W 0
Thr W -1339518976 W at W 0
Thr W -1340583936 W at W 1
Thr W -1340051456 W at W 1
Thr W -1338986496 W at W 1
...more of the same...
The “serialization” efect (whereby the threads appear to “nicely round-robin” as above) is a side effect of the fact that the thread that gets to be the currently-printing one is seriously slower than the others (all those waits!-). Commenting out the time.sleep in wait, the output is instead
Thr W -1341648896 W at W 0
Thr W -1341116416 W at W 0
Thr W -1341648896 W at W 1
Thr W -1340583936 W at W 0
Thr W -1340051456 W at W 0
Thr W -1341116416 W at W 1
Thr W -1341116416 W at W 2
Thr W -1338986496 W at W 0
...more of the same...
i.e. a more typical “multithreaded output”… except for the guarantee that each line in the output comes entirely from one single thread.
Of course, a thread that does, e.g., print 'ciao', will keep “ownership” of standard output until it finally does perform a print without a trailing comma, and other threads wanting to print may sleep for quite a while (how else can one guarantee that each line in the output comes from a single thread? well, one architecture would be to accumulate partial lines to thread local storage instead of actually writing them to standard output, and only do the writing on receipt of the n… delicate to interleave properly with softspace settings, I fear, but probably feasible).
Through experimentation, I found that the following works, is simple, and suits my needs:
print "your string heren",
Or, wrapped in a function,
def safe_print(content):
print "{0}n".format(content),
My understanding is that the implicit newline of a normal print is actually output to stdout in a separate operation, causing the race condition with other print operations. By removing this implicit newline with the added ,, and instead including the newline in the string, we can avoid this problem.
2020 Edit: Here’s the Python 3 version of this (thanks to Bob Stein in the comments for the inspiration):
def safe_print(*args, sep=" ", end="", **kwargs):
joined_string = sep.join([ str(arg) for arg in args ])
print(joined_string + "n", sep=sep, end=end, **kwargs)
As pointed out by Bob Stein, relying on print to join multiple passed arguments results in garbled output, so we have to do it ourselves.
2017 Edit: this answer is starting to pick up some steam, so I just wanted to make a clarification. This doesn’t actually make print “thread safe” exactly. The output may be in the wrong order if the prints happen microseconds apart from each other. What this does do, however, is avoid garbled output coming from print statements executed from concurrent threads, which is what most people really want when asking this question.
Here is a test to show what I mean:
from concurrent.futures import ThreadPoolExecutor
def normal_print(content):
print content
def safe_print(content):
print "{0}n".format(content),
with ThreadPoolExecutor(max_workers=10) as executor:
print "Normal Print:"
for i in range(10):
executor.submit(normal_print, i)
print "---"
with ThreadPoolExecutor(max_workers=10) as executor:
print "Safe Print:"
for i in range(10):
executor.submit(safe_print, i)
Output:
Normal Print:
0
1
23
4
65
7
9
8
----
Safe Print:
1
0
3
2
4
5
6
7
8
9