Python multithreaded print statements delayed until all threads complete execution

Question

I have a piece of code below that creates a few threads to perform a task, which works perfectly well on its own. However I’m struggling to understand why the print statements I call in my function do not execute until all threads complete and the print 'finished' statement is called. I would expect them to be called as the thread executes. Is there any simple way to accomplish this, and why does this work this way in the first place?

def func(param):
    time.sleep(.25)
    print param*2

if __name__ == '__main__':
    print 'starting execution'
    launchTime = time.clock()
    params = range(10)
    pool=multiprocessing.Pool(processes=100) #use N processes to download the data
    _=pool.map(func,params)
    print 'finished'

Asked By: Hawkwing

||

Source

Answer 1

This happens due to stdout buffering. You still can flush the buffers:

import sys

print 'starting'
sys.stdout.flush()

You can find more info on this issue here and here.

Answered By: Yossi

Answer 2

For python 3 you can now use the flush param like that:

print('Your text', flush=True)

Answered By: Or Duan

Answer 3

Having run into plenty of issues around this and garbled outputs (especially under Windows when adding colours to the output..), my solution has been to have an exclusive printing thread which consumes a queue

If this still doesn’t work, also add flush=True to your print statement(s) as suggested by @Or Duan

Further, you may find the "most correct", but a heavy-handed approach to displaying messages with threading is to use the logging library which can wrap a queue (and write to many places asynchronously, including stdout) or write to a system-level queue (outside Python; availability depends greatly on OS support)

import threading
from queue import Queue

def display_worker(display_queue):
    while True:
        line = display_queue.get()
        if line is None:  # simple termination logic, other sentinels can be used
            break
        print(line, flush=True)  # remove flush if slow or using Python2


def some_other_worker(display_queue, other_args):
    # NOTE accepts queue reference as an argument, though it could be a global
    display_queue.put("something which should be printed from this thread")


def main():
    display_queue = Queue()  # synchronizes console output
    screen_printing_thread = threading.Thread(
        target=display_worker,
        args=(display_queue,),
    )
    screen_printing_thread.start()

    ### other logic ###

    display_queue.put(None)  # end screen_printing_thread
    screen_printing_thread.stop()

Answered By: ti7

Python multithreaded print statements delayed until all threads complete execution

Question:

Answers: