How do you get tkinter to work with asyncio?
Question:
How do you get tkinter to work with asyncio? My studies suggest this general question does resolve into the specific problem of getting tkinter to await a coroutine function.
Context
If tkinter’s event loop is blocked the loop will freeze until the blocking function returns. If the event loop also happens to be running a GUI that will freeze as well. The traditional solution to this problem is to move any blocking code into a thread.
The new asyncio module is able to schedule threaded calls using the coroutine function asyncio.to_thread(coro). I gather this avoids the difficulties of writing correct threaded code.
Baseline: blocked.py
As a starting point I wrote a baseline program (See code below). It creates a tkinter event loop which attempts to
destroy itself and end the program after 2000ms. That attempt is thwarted by a blocking function which runs for 4s.
The program output is:
08:51:57: Program started.
08:51:58: blocking_func started.
08:52:02: blocking_func completed.
08:52:02: Tk event loop terminated.
08:52:02: Program ended.
Process finished with exit code 0
1st try: async_blocked.py
The blocking code has been refactored as a coroutine function so there are two event loops – tkinter’s and asyncio’s. The function blocking_io_handler is scheduled onto tkinter’s event loop which runs it successfully. The coroutine function blocking_func is scheduled onto asyncio’s loop where it starts successfully.
The problem is it doesn’t start until after tkinter’s event loop has terminated. Asyncio’s loop was available throughout the execution of the coroutine function main so it was available when tk_root.mainloop() was executed. In spite of this asyncio was helpless because control was not yielded by an await statement during the execution of tk_root.mainloop. It had to wait for the await asyncio.sleep(3) statement which ran later and, by then, tkinter had stopped running.
At that time the await expression returns control to the async loop for three seconds — enough to start the four second blocking_func but not enough for it to finish.
08:38:22: Program started.
08:38:22: blocking_io_handler started.
08:38:22: blocking_io_handler completed.
08:38:24: Tk event loop terminated.
08:38:24: blocking_func started.
08:38:27: Program ended.
Process finished with exit code 0
2nd try: asyncth_blocked.py
This code replaces the function asyncio.create_task with the coroutine function asyncio.to_thread. This fails
with a runtime warning:
07:26:46: Program started.
07:26:47: blocking_io_handler started.
07:26:47: blocking_io_handler completed.
RuntimeWarning: coroutine 'to_thread' was never awaited
asyncio.to_thread(blocking_func)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
07:26:49: Tk event loop terminated.
07:26:49: Program ended.
> Process finished with exit code 0
3rd try: asyncth_blocked_2.py
asyncio.to_thread must be awaited because it is a coroutine function and not a regular function:
await asyncio.to_thread(blocking_func).
Since the await keyword is a syntax error inside a regular function, def blocking_io_handler has to be changed into a coroutine function: async def blocking_io_handler.
These changes are shown in asyncth_blocked_2.py which produces this output:
07:52:29: Program started.
RuntimeWarning:
coroutine 'blocking_io_handler' was never awaited
func(*args)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
07:52:31: Tk event loop terminated.
07:52:31: Program ended.
Process finished with exit code 0
Conclusion
For tkinter to work with asyncio the scheduled function call tk_root.after(0, blocking_io_handler) has to be somehow turned into a scheduled coroutine function call. This is the only way the asycio loop will have a chance to run scheduled async
tasks.
Is it possible?
Code
"""blocked.py"""
import time
import tkinter as tk
def timestamped_msg(msg: str):
print(f"{time.strftime('%X')}: {msg}")
def blocking_func():
timestamped_msg('blocking_func started.')
time.sleep(4)
timestamped_msg('blocking_func completed.')
def main():
timestamped_msg('Program started.')
tk_root = tk.Tk()
tk_root.after(0, blocking_func)
tk_root.after(2000, tk_root.destroy)
tk_root.mainloop()
timestamped_msg('Tk event loop terminated.')
timestamped_msg('Program ended.')
if __name__ == '__main__':
main()
"""async_blocked.py"""
import asyncio
import time
import tkinter as tk
def timestamped_msg(msg: str):
print(f"{time.strftime('%X')}: {msg}")
async def blocking_func():
timestamped_msg('blocking_func started.')
await asyncio.sleep(4)
timestamped_msg('blocking_func completed.')
def blocking_io_handler():
timestamped_msg('blocking_io_handler started.')
asyncio.create_task(blocking_func())
timestamped_msg('blocking_io_handler completed.')
async def main():
timestamped_msg('Program started.')
tk_root = tk.Tk()
tk_root.after(0, blocking_io_handler)
tk_root.after(2000, tk_root.destroy)
tk_root.mainloop()
timestamped_msg('Tk event loop terminated.')
await asyncio.sleep(3)
timestamped_msg('Program ended.')
if __name__ == '__main__':
asyncio.run(main())
"""asyncth_blocked.py"""
import asyncio
import time
import tkinter as tk
def timestamped_msg(msg: str):
print(f"{time.strftime('%X')}: {msg}")
async def blocking_func():
timestamped_msg('blocking_func started.')
await asyncio.sleep(4)
timestamped_msg('blocking_func completed.')
def blocking_io_handler():
timestamped_msg('blocking_io_handler started.')
asyncio.to_thread(blocking_func)
timestamped_msg('blocking_io_handler completed.')
async def main():
timestamped_msg('Program started.')
tk_root = tk.Tk()
tk_root.after(0, blocking_io_handler)
tk_root.after(2000, tk_root.destroy)
tk_root.mainloop()
timestamped_msg('Tk event loop terminated.')
timestamped_msg('Program ended.')
if __name__ == '__main__':
asyncio.run(main())
"""asyncth_blocked_2.py"""
import asyncio
import time
import tkinter as tk
def timestamped_msg(msg: str):
print(f"{time.strftime('%X')}: {msg}")
async def blocking_func():
timestamped_msg('blocking_func started.')
await asyncio.sleep(4)
timestamped_msg('blocking_func completed.')
async def blocking_io_handler():
timestamped_msg('blocking_io_handler started.')
await asyncio.to_thread(blocking_func)
timestamped_msg('blocking_io_handler completed.')
async def main():
timestamped_msg('Program started.')
tk_root = tk.Tk()
tk_root.after(0, blocking_io_handler)
tk_root.after(2000, tk_root.destroy)
tk_root.mainloop()
timestamped_msg('Tk event loop terminated.')
timestamped_msg('Program ended.')
if __name__ == '__main__':
asyncio.run(main())
Answers:
Tkinter’s Problem with Blocking IO Calls
The statement asyncio.sleep(60) will block tkinter for a minute if both are running in the same thread.
Blocking coroutine functions cannot run in the same thread as tkinter.
Similarly, the statement time.sleep(60) will block both tkinter and asyncio for a minute if all three are running in the same
thread.
Blocking non-coroutine functions cannot run in the same thread as either tkinter or asyncio.
Sleep commands have been used here to simplify this example of the blocking problem. The principles shown are applicable to internet or database accesses.
Solution
A solution is to create three distinct environments and take care when moving data between them.
Environment 1 – Main Thread
This is Python’s MainThread. It’s where Python starts and Tkinter lives. No blocking code can be allowed in this environment.
Environment 2 – Asyncio’s Thread
This is where asyncio and all its coroutine functions live. Blocking functions are only allowed if they are coroutine
functions.
Environment 3 – Multiple single use threads
This is where non-coroutine blocking functions run. Since these are capable of blocking each other each needs
its own thread.
Data
Data returned from blocking IO to tkinter should be returned in threadsafe queues using a producer/consumer pattern.
Arguments and return values should not be passed between environments using regular functions. Use the threadsafe calling protocols provided by Python as illustrated below.
Wrong code
func(*args, **kwargs)
return_value = func(*args, **kwargs)
print(*args, **kwargs)
Correct code
threading.Thread(func, *args, **kwargs).start()
The return_value is not directly available. Use a queue.
future = asyncio.run_coroutine_threadsafe(func(*args, **kwargs), loop)
return_value = future.result().
print: Use a threadsafe queue to move printable objects to a single print thread. (See the SafePrinter context
maanger in the code below).
The Polling Problem
With tkinter, asyncio, and threading all running together there are three event loops controlling different stuff. Bad things can
happen when they mix. For example threading’s Queue.get() will block environment 1 where tkinter’s loop is trying to
control events. In this particular case, Queue.get_nowait() has to be used with polling via tkinter’s after command. See the code below for other examples of unusual polling of queues.
GUI
Console output
0.001s In Print Thread of 2 without a loop: The SafePrinter is open for output.
0.001s In MainThread of 2 without a loop --- main starting
0.001s In Asyncio Thread of 3 without a loop --- aio_main starting
0.001s In MainThread of 3 without a loop --- tk_main starting
0.305s In Asyncio Thread of 3 with a loop --- manage_aio_loop starting
0.350s In MainThread of 3 without a loop --- tk_callbacks starting
0.350s In MainThread of 3 without a loop --- tk_callback_consumer starting
0.350s In Asyncio Thread of 3 with a loop --- aio_blocker starting. block=3.1s.
0.350s In MainThread of 3 without a loop --- aio_exception_handler starting. block=3.1s
0.351s In MainThread of 3 without a loop --- aio_exception_handler starting. block=1.1s
0.351s In Asyncio Thread of 4 with a loop --- aio_blocker starting. block=1.1s.
0.351s In IO Block Thread (3.2s) of 4 without a loop --- io_exception_handler starting. block=3.2s.
0.351s In IO Block Thread (3.2s) of 4 without a loop --- io_blocker starting. block=3.2s.
0.351s In IO Block Thread (1.2s) of 5 without a loop --- io_exception_handler starting. block=1.2s.
0.351s In IO Block Thread (1.2s) of 5 without a loop --- io_blocker starting. block=1.2s.
0.351s In MainThread of 5 without a loop --- tk_callbacks ending - All blocking callbacks have been scheduled.
1.451s In Asyncio Thread of 5 with a loop --- aio_blocker ending. block=1.1s.
1.459s In MainThread of 5 without a loop --- aio_exception_handler ending. block=1.1s
1.555s In IO Block Thread (1.2s) of 5 without a loop --- io_blocker ending. block=1.2s.
1.555s In IO Block Thread (1.2s) of 5 without a loop --- io_exception_handler ending. block=1.2s.
3.450s In Asyncio Thread of 4 with a loop --- aio_blocker ending. block=3.1s.
3.474s In MainThread of 4 without a loop --- aio_exception_handler ending. block=3.1s
3.553s In IO Block Thread (3.2s) of 4 without a loop --- io_blocker ending. block=3.2s.
3.553s In IO Block Thread (3.2s) of 4 without a loop --- io_exception_handler ending. block=3.2s.
4.140s In MainThread of 3 without a loop --- tk_callback_consumer ending
4.140s In MainThread of 3 without a loop --- tk_main ending
4.141s In Asyncio Thread of 3 with a loop --- manage_aio_loop ending
4.141s In Asyncio Thread of 3 without a loop --- aio_main ending
4.141s In MainThread of 2 without a loop --- main ending
4.141s In Print Thread of 2 without a loop: The SafePrinter has closed.
Process finished with exit code 0
Code
""" tkinter_demo.py
Created with Python 3.10
"""
import asyncio
import concurrent.futures
import functools
import itertools
import queue
import sys
import threading
import time
import tkinter as tk
import tkinter.ttk as ttk
from collections.abc import Iterator
from contextlib import AbstractContextManager
from dataclasses import dataclass
from types import TracebackType
from typing import Optional, Type
# Global reference to loop allows access from different environments.
aio_loop: Optional[asyncio.AbstractEventLoop] = None
def io_blocker(task_id: int, tk_q: queue.Queue, block: float = 0) -> None:
""" Block the thread and put a 'Hello World' work package into Tkinter's work queue.
This is a producer for Tkinter's work queue. It will run in a special thread created solely for running this
function. The statement `time.sleep(block)` can be replaced with any non-awaitable blocking code.
Args:
task_id: Sequentially issued tkinter task number.
tk_q: tkinter's work queue.
block: block time
Returns:
Nothing. The work package is returned via the threadsafe tk_q.
"""
safeprint(f'io_blocker starting. {block=}s.')
time.sleep(block)
# Exceptions for testing handlers. Uncomment these to see what happens when exceptions are raised.
# raise IOError('Just testing an expected error.')
# raise ValueError('Just testing an unexpected error.')
work_package = f"Task #{task_id} {block}s: 'Hello Threading World'."
tk_q.put(work_package)
safeprint(f'io_blocker ending. {block=}s.')
def io_exception_handler(task_id: int, tk_q: queue.Queue, block: float = 0) -> None:
""" Exception handler for non-awaitable blocking callback.
It will run in a special thread created solely for running io_blocker.
Args:
task_id: Sequentially issued tkinter task number.
tk_q: tkinter's work queue.
block: block time
"""
safeprint(f'io_exception_handler starting. {block=}s.')
try:
io_blocker(task_id, tk_q, block)
except IOError as exc:
safeprint(f'io_exception_handler: {exc!r} was handled correctly. ')
finally:
safeprint(f'io_exception_handler ending. {block=}s.')
async def aio_blocker(task_id: int, tk_q: queue.Queue, block: float = 0) -> None:
""" Asynchronously block the thread and put a 'Hello World' work package into Tkinter's work queue.
This is a producer for Tkinter's work queue. It will run in the same thread as the asyncio loop. The statement
`await asyncio.sleep(block)` can be replaced with any awaitable blocking code.
Args:
task_id: Sequentially issued tkinter task number.
tk_q: tkinter's work queue.
block: block time
Returns:
Nothing. The work package is returned via the threadsafe tk_q.
"""
safeprint(f'aio_blocker starting. {block=}s.')
await asyncio.sleep(block)
# Exceptions for testing handlers. Uncomment these to see what happens when exceptions are raised.
# raise IOError('Just testing an expected error.')
# raise ValueError('Just testing an unexpected error.')
work_package = f"Task #{task_id} {block}s: 'Hello Asynchronous World'."
# Put the work package into the tkinter's work queue.
while True:
try:
# Asyncio can't wait for the thread blocking `put` method…
tk_q.put_nowait(work_package)
except queue.Full:
# Give control back to asyncio's loop.
await asyncio.sleep(0)
else:
# The work package has been placed in the queue so we're done.
break
safeprint(f'aio_blocker ending. {block=}s.')
def aio_exception_handler(mainframe: ttk.Frame, future: concurrent.futures.Future, block: float,
first_call: bool = True) -> None:
""" Exception handler for future coroutine callbacks.
This non-coroutine function uses tkinter's event loop to wait for the future to finish.
It runs in the Main Thread.
Args:
mainframe: The after method of this object is used to poll this function.
future: The future running the future coroutine callback.
block: The block time parameter used to identify which future coroutine callback is being reported.
first_call: If True will cause an opening line to be printed on stdout.
"""
if first_call:
safeprint(f'aio_exception_handler starting. {block=}s')
poll_interval = 100 # milliseconds
try:
# Python will not raise exceptions during future execution until `future.result` is called. A zero timeout is
# required to avoid blocking the thread.
future.result(0)
# If the future hasn't completed, reschedule this function on tkinter's event loop.
except concurrent.futures.TimeoutError:
mainframe.after(poll_interval, functools.partial(aio_exception_handler, mainframe, future, block,
first_call=False))
# Handle an expected error.
except IOError as exc:
safeprint(f'aio_exception_handler: {exc!r} was handled correctly. ')
else:
safeprint(f'aio_exception_handler ending. {block=}s')
def tk_callback_consumer(tk_q: queue.Queue, mainframe: ttk.Frame, row_itr: Iterator):
""" Display queued 'Hello world' messages in the Tkinter window.
This is the consumer for Tkinter's work queue. It runs in the Main Thread. After starting, it runs
continuously until the GUI is closed by the user.
"""
# Poll continuously while queue has work needing processing.
poll_interval = 0
try:
# Tkinter can't wait for the thread blocking `get` method…
work_package = tk_q.get_nowait()
except queue.Empty:
# …so be prepared for an empty queue and slow the polling rate.
poll_interval = 40
else:
# Process a work package.
label = ttk.Label(mainframe, text=work_package)
label.grid(column=0, row=(next(row_itr)), sticky='w', padx=10)
finally:
# Have tkinter call this function again after the poll interval.
mainframe.after(poll_interval, functools.partial(tk_callback_consumer, tk_q, mainframe, row_itr))
def tk_callbacks(mainframe: ttk.Frame, row_itr: Iterator):
""" Set up 'Hello world' callbacks.
This runs in the Main Thread.
Args:
mainframe: The mainframe of the GUI used for displaying results from the work queue.
row_itr: A generator of line numbers for displaying items from the work queue.
"""
safeprint('tk_callbacks starting')
task_id_itr = itertools.count(1)
# Create the job queue and start its consumer.
tk_q = queue.Queue()
safeprint('tk_callback_consumer starting')
tk_callback_consumer(tk_q, mainframe, row_itr)
# Schedule the asyncio blocker.
for block in [3.1, 1.1]:
# This is a concurrent.futures.Future not an asyncio.Future because it isn't threadsafe. Also,
# it doesn't have a wait with timeout which we shall need.
task_id = next(task_id_itr)
future = asyncio.run_coroutine_threadsafe(aio_blocker(task_id, tk_q, block), aio_loop)
# Can't use Future.add_done_callback here. It doesn't return until the future is done and that would block
# tkinter's event loop.
aio_exception_handler(mainframe, future, block)
# Run the thread blocker.
for block in [3.2, 1.2]:
task_id = next(task_id_itr)
threading.Thread(target=io_exception_handler, args=(task_id, tk_q, block),
name=f'IO Block Thread ({block}s)').start()
safeprint('tk_callbacks ending - All blocking callbacks have been scheduled.n')
def tk_main():
""" Run tkinter.
This runs in the Main Thread.
"""
safeprint('tk_main startingn')
row_itr = itertools.count()
# Create the Tk root and mainframe.
root = tk.Tk()
mainframe = ttk.Frame(root, padding="15 15 15 15")
mainframe.grid(column=0, row=0)
# Add a close button
button = ttk.Button(mainframe, text='Shutdown', command=root.destroy)
button.grid(column=0, row=next(row_itr), sticky='w')
# Add an information widget.
label = ttk.Label(mainframe, text=f'nWelcome to hello_world*4.py.n')
label.grid(column=0, row=next(row_itr), sticky='w')
# Schedule the 'Hello World' callbacks
mainframe.after(0, functools.partial(tk_callbacks, mainframe, row_itr))
# The asyncio loop must start before the tkinter event loop.
while not aio_loop:
time.sleep(0)
root.mainloop()
safeprint(' ', timestamp=False)
safeprint('tk_callback_consumer ending')
safeprint('tk_main ending')
async def manage_aio_loop(aio_initiate_shutdown: threading.Event):
""" Run the asyncio loop.
This provides an always available asyncio service for tkinter to make any number of simultaneous blocking IO
calls. 'Any number' includes zero.
This runs in Asyncio's thread and in asyncio's loop.
"""
safeprint('manage_aio_loop starting')
# Communicate the asyncio loop status to tkinter via a global variable.
global aio_loop
aio_loop = asyncio.get_running_loop()
# If there are no awaitables left in the queue asyncio will close.
# The usual wait command — Event.wait() — would block the current thread and the asyncio loop.
while not aio_initiate_shutdown.is_set():
await asyncio.sleep(0)
safeprint('manage_aio_loop ending')
def aio_main(aio_initiate_shutdown: threading.Event):
""" Start the asyncio loop.
This non-coroutine function runs in Asyncio's thread.
"""
safeprint('aio_main starting')
asyncio.run(manage_aio_loop(aio_initiate_shutdown))
safeprint('aio_main ending')
def main():
"""Set up working environments for asyncio and tkinter.
This runs in the Main Thread.
"""
safeprint('main starting')
# Start the permanent asyncio loop in a new thread.
# aio_shutdown is signalled between threads. `asyncio.Event()` is not threadsafe.
aio_initiate_shutdown = threading.Event()
aio_thread = threading.Thread(target=aio_main, args=(aio_initiate_shutdown,), name="Asyncio's Thread")
aio_thread.start()
tk_main()
# Close the asyncio permanent loop and join the thread in which it runs.
aio_initiate_shutdown.set()
aio_thread.join()
safeprint('main ending')
@dataclass
class SafePrinter(AbstractContextManager):
_time_0 = time.perf_counter()
_print_q = queue.Queue()
_print_thread: threading.Thread | None = None
def __enter__(self):
""" Run the safeprint consumer method in a print thread.
Returns:
Thw safeprint producer method. (a.k.a. the runtime context)
"""
self._print_thread = threading.Thread(target=self._safeprint_consumer, name='Print Thread')
self._print_thread.start()
return self._safeprint
def __exit__(self, __exc_type: Type[BaseException] | None, __exc_value: BaseException | None,
__traceback: TracebackType | None) -> bool | None:
""" Close the print and join the print thread.
Args:
None or the exception raised during the execution of the safeprint producer method.
__exc_type:
__exc_value:
__traceback:
Returns:
False to indicate that any exception raised in self._safeprint has not been handled.
"""
self._print_q.put(None)
self._print_thread.join()
return False
def _safeprint(self, msg: str, *, timestamp: bool = True, reset: bool = False):
"""Put a string into the print queue.
'None' is a special msg. It is not printed but will close the queue and this context manager.
The exclusive thread and a threadsafe print queue ensure race free printing.
This is the producer in the print queue's producer/consumer pattern.
It runs in the same thread as the calling function
Args:
msg: The message to be printed.
timestamp: Print a timestamp (Default = True).
reset: Reset the time to zero (Default = False).
"""
if reset:
self._time_0 = time.perf_counter()
if timestamp:
self._print_q.put(f'{self._timestamp()} --- {msg}')
else:
self._print_q.put(msg)
def _safeprint_consumer(self):
"""Get strings from the print queue and print them on stdout.
The print statement is not threadsafe, so it must run in its own thread.
This is the consumer in the print queue's producer/consumer pattern.
"""
print(f'{self._timestamp()}: The SafePrinter is open for output.')
while True:
msg = self._print_q.get()
# Exit function when any producer function places 'None'.
if msg is not None:
print(msg)
else:
break
print(f'{self._timestamp()}: The SafePrinter has closed.')
def _timestamp(self) -> str:
"""Create a timestamp with useful status information.
This is a support function for the print queue producers. It runs in the same thread as the calling function
so the returned data does not cross between threads.
Returns:
timestamp
"""
secs = time.perf_counter() - self._time_0
try:
asyncio.get_running_loop()
except RuntimeError as exc:
if exc.args[0] == 'no running event loop':
loop_text = 'without a loop'
else:
raise
else:
loop_text = 'with a loop'
return f'{secs:.3f}s In {threading.current_thread().name} of {threading.active_count()} {loop_text}'
if __name__ == '__main__':
with SafePrinter() as safeprint:
sys.exit(main())
How do you get tkinter to work with asyncio? My studies suggest this general question does resolve into the specific problem of getting tkinter to await a coroutine function.
Context
If tkinter’s event loop is blocked the loop will freeze until the blocking function returns. If the event loop also happens to be running a GUI that will freeze as well. The traditional solution to this problem is to move any blocking code into a thread.
The new asyncio module is able to schedule threaded calls using the coroutine function asyncio.to_thread(coro). I gather this avoids the difficulties of writing correct threaded code.
Baseline: blocked.py
As a starting point I wrote a baseline program (See code below). It creates a tkinter event loop which attempts to
destroy itself and end the program after 2000ms. That attempt is thwarted by a blocking function which runs for 4s.
The program output is:
08:51:57: Program started.
08:51:58: blocking_func started.
08:52:02: blocking_func completed.
08:52:02: Tk event loop terminated.
08:52:02: Program ended.
Process finished with exit code 0
1st try: async_blocked.py
The blocking code has been refactored as a coroutine function so there are two event loops – tkinter’s and asyncio’s. The function blocking_io_handler is scheduled onto tkinter’s event loop which runs it successfully. The coroutine function blocking_func is scheduled onto asyncio’s loop where it starts successfully.
The problem is it doesn’t start until after tkinter’s event loop has terminated. Asyncio’s loop was available throughout the execution of the coroutine function main so it was available when tk_root.mainloop() was executed. In spite of this asyncio was helpless because control was not yielded by an await statement during the execution of tk_root.mainloop. It had to wait for the await asyncio.sleep(3) statement which ran later and, by then, tkinter had stopped running.
At that time the await expression returns control to the async loop for three seconds — enough to start the four second blocking_func but not enough for it to finish.
08:38:22: Program started.
08:38:22: blocking_io_handler started.
08:38:22: blocking_io_handler completed.
08:38:24: Tk event loop terminated.
08:38:24: blocking_func started.
08:38:27: Program ended.
Process finished with exit code 0
2nd try: asyncth_blocked.py
This code replaces the function asyncio.create_task with the coroutine function asyncio.to_thread. This fails
with a runtime warning:
07:26:46: Program started.
07:26:47: blocking_io_handler started.
07:26:47: blocking_io_handler completed.
RuntimeWarning: coroutine 'to_thread' was never awaited
asyncio.to_thread(blocking_func)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
07:26:49: Tk event loop terminated.
07:26:49: Program ended.
> Process finished with exit code 0
3rd try: asyncth_blocked_2.py
asyncio.to_thread must be awaited because it is a coroutine function and not a regular function:
await asyncio.to_thread(blocking_func).
Since the await keyword is a syntax error inside a regular function, def blocking_io_handler has to be changed into a coroutine function: async def blocking_io_handler.
These changes are shown in asyncth_blocked_2.py which produces this output:
07:52:29: Program started.
RuntimeWarning:
coroutine 'blocking_io_handler' was never awaited
func(*args)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
07:52:31: Tk event loop terminated.
07:52:31: Program ended.
Process finished with exit code 0
Conclusion
For tkinter to work with asyncio the scheduled function call tk_root.after(0, blocking_io_handler) has to be somehow turned into a scheduled coroutine function call. This is the only way the asycio loop will have a chance to run scheduled async
tasks.
Is it possible?
Code
"""blocked.py"""
import time
import tkinter as tk
def timestamped_msg(msg: str):
print(f"{time.strftime('%X')}: {msg}")
def blocking_func():
timestamped_msg('blocking_func started.')
time.sleep(4)
timestamped_msg('blocking_func completed.')
def main():
timestamped_msg('Program started.')
tk_root = tk.Tk()
tk_root.after(0, blocking_func)
tk_root.after(2000, tk_root.destroy)
tk_root.mainloop()
timestamped_msg('Tk event loop terminated.')
timestamped_msg('Program ended.')
if __name__ == '__main__':
main()
"""async_blocked.py"""
import asyncio
import time
import tkinter as tk
def timestamped_msg(msg: str):
print(f"{time.strftime('%X')}: {msg}")
async def blocking_func():
timestamped_msg('blocking_func started.')
await asyncio.sleep(4)
timestamped_msg('blocking_func completed.')
def blocking_io_handler():
timestamped_msg('blocking_io_handler started.')
asyncio.create_task(blocking_func())
timestamped_msg('blocking_io_handler completed.')
async def main():
timestamped_msg('Program started.')
tk_root = tk.Tk()
tk_root.after(0, blocking_io_handler)
tk_root.after(2000, tk_root.destroy)
tk_root.mainloop()
timestamped_msg('Tk event loop terminated.')
await asyncio.sleep(3)
timestamped_msg('Program ended.')
if __name__ == '__main__':
asyncio.run(main())
"""asyncth_blocked.py"""
import asyncio
import time
import tkinter as tk
def timestamped_msg(msg: str):
print(f"{time.strftime('%X')}: {msg}")
async def blocking_func():
timestamped_msg('blocking_func started.')
await asyncio.sleep(4)
timestamped_msg('blocking_func completed.')
def blocking_io_handler():
timestamped_msg('blocking_io_handler started.')
asyncio.to_thread(blocking_func)
timestamped_msg('blocking_io_handler completed.')
async def main():
timestamped_msg('Program started.')
tk_root = tk.Tk()
tk_root.after(0, blocking_io_handler)
tk_root.after(2000, tk_root.destroy)
tk_root.mainloop()
timestamped_msg('Tk event loop terminated.')
timestamped_msg('Program ended.')
if __name__ == '__main__':
asyncio.run(main())
"""asyncth_blocked_2.py"""
import asyncio
import time
import tkinter as tk
def timestamped_msg(msg: str):
print(f"{time.strftime('%X')}: {msg}")
async def blocking_func():
timestamped_msg('blocking_func started.')
await asyncio.sleep(4)
timestamped_msg('blocking_func completed.')
async def blocking_io_handler():
timestamped_msg('blocking_io_handler started.')
await asyncio.to_thread(blocking_func)
timestamped_msg('blocking_io_handler completed.')
async def main():
timestamped_msg('Program started.')
tk_root = tk.Tk()
tk_root.after(0, blocking_io_handler)
tk_root.after(2000, tk_root.destroy)
tk_root.mainloop()
timestamped_msg('Tk event loop terminated.')
timestamped_msg('Program ended.')
if __name__ == '__main__':
asyncio.run(main())
Tkinter’s Problem with Blocking IO Calls
The statement asyncio.sleep(60) will block tkinter for a minute if both are running in the same thread.
Blocking coroutine functions cannot run in the same thread as tkinter.
Similarly, the statement time.sleep(60) will block both tkinter and asyncio for a minute if all three are running in the same
thread.
Blocking non-coroutine functions cannot run in the same thread as either tkinter or asyncio.
Sleep commands have been used here to simplify this example of the blocking problem. The principles shown are applicable to internet or database accesses.
Solution
A solution is to create three distinct environments and take care when moving data between them.
Environment 1 – Main Thread
This is Python’s MainThread. It’s where Python starts and Tkinter lives. No blocking code can be allowed in this environment.
Environment 2 – Asyncio’s Thread
This is where asyncio and all its coroutine functions live. Blocking functions are only allowed if they are coroutine
functions.
Environment 3 – Multiple single use threads
This is where non-coroutine blocking functions run. Since these are capable of blocking each other each needs
its own thread.
Data
Data returned from blocking IO to tkinter should be returned in threadsafe queues using a producer/consumer pattern.
Arguments and return values should not be passed between environments using regular functions. Use the threadsafe calling protocols provided by Python as illustrated below.
Wrong code
func(*args, **kwargs)
return_value = func(*args, **kwargs)
print(*args, **kwargs)
Correct code
threading.Thread(func, *args, **kwargs).start()
The return_value is not directly available. Use a queue.
future = asyncio.run_coroutine_threadsafe(func(*args, **kwargs), loop)
return_value = future.result().
print: Use a threadsafe queue to move printable objects to a single print thread. (See the SafePrinter context
maanger in the code below).
The Polling Problem
With tkinter, asyncio, and threading all running together there are three event loops controlling different stuff. Bad things can
happen when they mix. For example threading’s Queue.get() will block environment 1 where tkinter’s loop is trying to
control events. In this particular case, Queue.get_nowait() has to be used with polling via tkinter’s after command. See the code below for other examples of unusual polling of queues.
GUI
Console output
0.001s In Print Thread of 2 without a loop: The SafePrinter is open for output.
0.001s In MainThread of 2 without a loop --- main starting
0.001s In Asyncio Thread of 3 without a loop --- aio_main starting
0.001s In MainThread of 3 without a loop --- tk_main starting
0.305s In Asyncio Thread of 3 with a loop --- manage_aio_loop starting
0.350s In MainThread of 3 without a loop --- tk_callbacks starting
0.350s In MainThread of 3 without a loop --- tk_callback_consumer starting
0.350s In Asyncio Thread of 3 with a loop --- aio_blocker starting. block=3.1s.
0.350s In MainThread of 3 without a loop --- aio_exception_handler starting. block=3.1s
0.351s In MainThread of 3 without a loop --- aio_exception_handler starting. block=1.1s
0.351s In Asyncio Thread of 4 with a loop --- aio_blocker starting. block=1.1s.
0.351s In IO Block Thread (3.2s) of 4 without a loop --- io_exception_handler starting. block=3.2s.
0.351s In IO Block Thread (3.2s) of 4 without a loop --- io_blocker starting. block=3.2s.
0.351s In IO Block Thread (1.2s) of 5 without a loop --- io_exception_handler starting. block=1.2s.
0.351s In IO Block Thread (1.2s) of 5 without a loop --- io_blocker starting. block=1.2s.
0.351s In MainThread of 5 without a loop --- tk_callbacks ending - All blocking callbacks have been scheduled.
1.451s In Asyncio Thread of 5 with a loop --- aio_blocker ending. block=1.1s.
1.459s In MainThread of 5 without a loop --- aio_exception_handler ending. block=1.1s
1.555s In IO Block Thread (1.2s) of 5 without a loop --- io_blocker ending. block=1.2s.
1.555s In IO Block Thread (1.2s) of 5 without a loop --- io_exception_handler ending. block=1.2s.
3.450s In Asyncio Thread of 4 with a loop --- aio_blocker ending. block=3.1s.
3.474s In MainThread of 4 without a loop --- aio_exception_handler ending. block=3.1s
3.553s In IO Block Thread (3.2s) of 4 without a loop --- io_blocker ending. block=3.2s.
3.553s In IO Block Thread (3.2s) of 4 without a loop --- io_exception_handler ending. block=3.2s.
4.140s In MainThread of 3 without a loop --- tk_callback_consumer ending
4.140s In MainThread of 3 without a loop --- tk_main ending
4.141s In Asyncio Thread of 3 with a loop --- manage_aio_loop ending
4.141s In Asyncio Thread of 3 without a loop --- aio_main ending
4.141s In MainThread of 2 without a loop --- main ending
4.141s In Print Thread of 2 without a loop: The SafePrinter has closed.
Process finished with exit code 0
Code
""" tkinter_demo.py
Created with Python 3.10
"""
import asyncio
import concurrent.futures
import functools
import itertools
import queue
import sys
import threading
import time
import tkinter as tk
import tkinter.ttk as ttk
from collections.abc import Iterator
from contextlib import AbstractContextManager
from dataclasses import dataclass
from types import TracebackType
from typing import Optional, Type
# Global reference to loop allows access from different environments.
aio_loop: Optional[asyncio.AbstractEventLoop] = None
def io_blocker(task_id: int, tk_q: queue.Queue, block: float = 0) -> None:
""" Block the thread and put a 'Hello World' work package into Tkinter's work queue.
This is a producer for Tkinter's work queue. It will run in a special thread created solely for running this
function. The statement `time.sleep(block)` can be replaced with any non-awaitable blocking code.
Args:
task_id: Sequentially issued tkinter task number.
tk_q: tkinter's work queue.
block: block time
Returns:
Nothing. The work package is returned via the threadsafe tk_q.
"""
safeprint(f'io_blocker starting. {block=}s.')
time.sleep(block)
# Exceptions for testing handlers. Uncomment these to see what happens when exceptions are raised.
# raise IOError('Just testing an expected error.')
# raise ValueError('Just testing an unexpected error.')
work_package = f"Task #{task_id} {block}s: 'Hello Threading World'."
tk_q.put(work_package)
safeprint(f'io_blocker ending. {block=}s.')
def io_exception_handler(task_id: int, tk_q: queue.Queue, block: float = 0) -> None:
""" Exception handler for non-awaitable blocking callback.
It will run in a special thread created solely for running io_blocker.
Args:
task_id: Sequentially issued tkinter task number.
tk_q: tkinter's work queue.
block: block time
"""
safeprint(f'io_exception_handler starting. {block=}s.')
try:
io_blocker(task_id, tk_q, block)
except IOError as exc:
safeprint(f'io_exception_handler: {exc!r} was handled correctly. ')
finally:
safeprint(f'io_exception_handler ending. {block=}s.')
async def aio_blocker(task_id: int, tk_q: queue.Queue, block: float = 0) -> None:
""" Asynchronously block the thread and put a 'Hello World' work package into Tkinter's work queue.
This is a producer for Tkinter's work queue. It will run in the same thread as the asyncio loop. The statement
`await asyncio.sleep(block)` can be replaced with any awaitable blocking code.
Args:
task_id: Sequentially issued tkinter task number.
tk_q: tkinter's work queue.
block: block time
Returns:
Nothing. The work package is returned via the threadsafe tk_q.
"""
safeprint(f'aio_blocker starting. {block=}s.')
await asyncio.sleep(block)
# Exceptions for testing handlers. Uncomment these to see what happens when exceptions are raised.
# raise IOError('Just testing an expected error.')
# raise ValueError('Just testing an unexpected error.')
work_package = f"Task #{task_id} {block}s: 'Hello Asynchronous World'."
# Put the work package into the tkinter's work queue.
while True:
try:
# Asyncio can't wait for the thread blocking `put` method…
tk_q.put_nowait(work_package)
except queue.Full:
# Give control back to asyncio's loop.
await asyncio.sleep(0)
else:
# The work package has been placed in the queue so we're done.
break
safeprint(f'aio_blocker ending. {block=}s.')
def aio_exception_handler(mainframe: ttk.Frame, future: concurrent.futures.Future, block: float,
first_call: bool = True) -> None:
""" Exception handler for future coroutine callbacks.
This non-coroutine function uses tkinter's event loop to wait for the future to finish.
It runs in the Main Thread.
Args:
mainframe: The after method of this object is used to poll this function.
future: The future running the future coroutine callback.
block: The block time parameter used to identify which future coroutine callback is being reported.
first_call: If True will cause an opening line to be printed on stdout.
"""
if first_call:
safeprint(f'aio_exception_handler starting. {block=}s')
poll_interval = 100 # milliseconds
try:
# Python will not raise exceptions during future execution until `future.result` is called. A zero timeout is
# required to avoid blocking the thread.
future.result(0)
# If the future hasn't completed, reschedule this function on tkinter's event loop.
except concurrent.futures.TimeoutError:
mainframe.after(poll_interval, functools.partial(aio_exception_handler, mainframe, future, block,
first_call=False))
# Handle an expected error.
except IOError as exc:
safeprint(f'aio_exception_handler: {exc!r} was handled correctly. ')
else:
safeprint(f'aio_exception_handler ending. {block=}s')
def tk_callback_consumer(tk_q: queue.Queue, mainframe: ttk.Frame, row_itr: Iterator):
""" Display queued 'Hello world' messages in the Tkinter window.
This is the consumer for Tkinter's work queue. It runs in the Main Thread. After starting, it runs
continuously until the GUI is closed by the user.
"""
# Poll continuously while queue has work needing processing.
poll_interval = 0
try:
# Tkinter can't wait for the thread blocking `get` method…
work_package = tk_q.get_nowait()
except queue.Empty:
# …so be prepared for an empty queue and slow the polling rate.
poll_interval = 40
else:
# Process a work package.
label = ttk.Label(mainframe, text=work_package)
label.grid(column=0, row=(next(row_itr)), sticky='w', padx=10)
finally:
# Have tkinter call this function again after the poll interval.
mainframe.after(poll_interval, functools.partial(tk_callback_consumer, tk_q, mainframe, row_itr))
def tk_callbacks(mainframe: ttk.Frame, row_itr: Iterator):
""" Set up 'Hello world' callbacks.
This runs in the Main Thread.
Args:
mainframe: The mainframe of the GUI used for displaying results from the work queue.
row_itr: A generator of line numbers for displaying items from the work queue.
"""
safeprint('tk_callbacks starting')
task_id_itr = itertools.count(1)
# Create the job queue and start its consumer.
tk_q = queue.Queue()
safeprint('tk_callback_consumer starting')
tk_callback_consumer(tk_q, mainframe, row_itr)
# Schedule the asyncio blocker.
for block in [3.1, 1.1]:
# This is a concurrent.futures.Future not an asyncio.Future because it isn't threadsafe. Also,
# it doesn't have a wait with timeout which we shall need.
task_id = next(task_id_itr)
future = asyncio.run_coroutine_threadsafe(aio_blocker(task_id, tk_q, block), aio_loop)
# Can't use Future.add_done_callback here. It doesn't return until the future is done and that would block
# tkinter's event loop.
aio_exception_handler(mainframe, future, block)
# Run the thread blocker.
for block in [3.2, 1.2]:
task_id = next(task_id_itr)
threading.Thread(target=io_exception_handler, args=(task_id, tk_q, block),
name=f'IO Block Thread ({block}s)').start()
safeprint('tk_callbacks ending - All blocking callbacks have been scheduled.n')
def tk_main():
""" Run tkinter.
This runs in the Main Thread.
"""
safeprint('tk_main startingn')
row_itr = itertools.count()
# Create the Tk root and mainframe.
root = tk.Tk()
mainframe = ttk.Frame(root, padding="15 15 15 15")
mainframe.grid(column=0, row=0)
# Add a close button
button = ttk.Button(mainframe, text='Shutdown', command=root.destroy)
button.grid(column=0, row=next(row_itr), sticky='w')
# Add an information widget.
label = ttk.Label(mainframe, text=f'nWelcome to hello_world*4.py.n')
label.grid(column=0, row=next(row_itr), sticky='w')
# Schedule the 'Hello World' callbacks
mainframe.after(0, functools.partial(tk_callbacks, mainframe, row_itr))
# The asyncio loop must start before the tkinter event loop.
while not aio_loop:
time.sleep(0)
root.mainloop()
safeprint(' ', timestamp=False)
safeprint('tk_callback_consumer ending')
safeprint('tk_main ending')
async def manage_aio_loop(aio_initiate_shutdown: threading.Event):
""" Run the asyncio loop.
This provides an always available asyncio service for tkinter to make any number of simultaneous blocking IO
calls. 'Any number' includes zero.
This runs in Asyncio's thread and in asyncio's loop.
"""
safeprint('manage_aio_loop starting')
# Communicate the asyncio loop status to tkinter via a global variable.
global aio_loop
aio_loop = asyncio.get_running_loop()
# If there are no awaitables left in the queue asyncio will close.
# The usual wait command — Event.wait() — would block the current thread and the asyncio loop.
while not aio_initiate_shutdown.is_set():
await asyncio.sleep(0)
safeprint('manage_aio_loop ending')
def aio_main(aio_initiate_shutdown: threading.Event):
""" Start the asyncio loop.
This non-coroutine function runs in Asyncio's thread.
"""
safeprint('aio_main starting')
asyncio.run(manage_aio_loop(aio_initiate_shutdown))
safeprint('aio_main ending')
def main():
"""Set up working environments for asyncio and tkinter.
This runs in the Main Thread.
"""
safeprint('main starting')
# Start the permanent asyncio loop in a new thread.
# aio_shutdown is signalled between threads. `asyncio.Event()` is not threadsafe.
aio_initiate_shutdown = threading.Event()
aio_thread = threading.Thread(target=aio_main, args=(aio_initiate_shutdown,), name="Asyncio's Thread")
aio_thread.start()
tk_main()
# Close the asyncio permanent loop and join the thread in which it runs.
aio_initiate_shutdown.set()
aio_thread.join()
safeprint('main ending')
@dataclass
class SafePrinter(AbstractContextManager):
_time_0 = time.perf_counter()
_print_q = queue.Queue()
_print_thread: threading.Thread | None = None
def __enter__(self):
""" Run the safeprint consumer method in a print thread.
Returns:
Thw safeprint producer method. (a.k.a. the runtime context)
"""
self._print_thread = threading.Thread(target=self._safeprint_consumer, name='Print Thread')
self._print_thread.start()
return self._safeprint
def __exit__(self, __exc_type: Type[BaseException] | None, __exc_value: BaseException | None,
__traceback: TracebackType | None) -> bool | None:
""" Close the print and join the print thread.
Args:
None or the exception raised during the execution of the safeprint producer method.
__exc_type:
__exc_value:
__traceback:
Returns:
False to indicate that any exception raised in self._safeprint has not been handled.
"""
self._print_q.put(None)
self._print_thread.join()
return False
def _safeprint(self, msg: str, *, timestamp: bool = True, reset: bool = False):
"""Put a string into the print queue.
'None' is a special msg. It is not printed but will close the queue and this context manager.
The exclusive thread and a threadsafe print queue ensure race free printing.
This is the producer in the print queue's producer/consumer pattern.
It runs in the same thread as the calling function
Args:
msg: The message to be printed.
timestamp: Print a timestamp (Default = True).
reset: Reset the time to zero (Default = False).
"""
if reset:
self._time_0 = time.perf_counter()
if timestamp:
self._print_q.put(f'{self._timestamp()} --- {msg}')
else:
self._print_q.put(msg)
def _safeprint_consumer(self):
"""Get strings from the print queue and print them on stdout.
The print statement is not threadsafe, so it must run in its own thread.
This is the consumer in the print queue's producer/consumer pattern.
"""
print(f'{self._timestamp()}: The SafePrinter is open for output.')
while True:
msg = self._print_q.get()
# Exit function when any producer function places 'None'.
if msg is not None:
print(msg)
else:
break
print(f'{self._timestamp()}: The SafePrinter has closed.')
def _timestamp(self) -> str:
"""Create a timestamp with useful status information.
This is a support function for the print queue producers. It runs in the same thread as the calling function
so the returned data does not cross between threads.
Returns:
timestamp
"""
secs = time.perf_counter() - self._time_0
try:
asyncio.get_running_loop()
except RuntimeError as exc:
if exc.args[0] == 'no running event loop':
loop_text = 'without a loop'
else:
raise
else:
loop_text = 'with a loop'
return f'{secs:.3f}s In {threading.current_thread().name} of {threading.active_count()} {loop_text}'
if __name__ == '__main__':
with SafePrinter() as safeprint:
sys.exit(main())