What's the recommended way to unittest Python GUI applications?
Question:
I’m currently foolish enough to try to maintaintain two parallel code bases for a Python desktop application, one using PyGObject introspection for GTK 3 and one using PyGTK for GTK 2. I work mainly on the PyGObject branch and then I port changes over to the PyGTK branch. Due to all the minor differences between these implementations, I often overlook things and cause breakage that I miss and accidentally release, only to be caught by the users.
I’m trying to figure out a good way to design some unittests that would, preferably, be suitable to run on both code bases. It’s not an overly complicated program, (it’s essentially a library management tool, imagine like iTunes):
- Main Window
|- Toolbar with some buttons (add/edit/remove items, configure the program)
|
|- VPaned
|--- Top HPaned
|------ ListView (listing values by which a library of items can be filtered)
|------ ListView (listing the contents of the library
|--- Bottom HPaned
|------ Image (displaying cover art for the currently selected item in the library)
|------ TextView (displaying formatted text describing the currently selected item)
- Edit dialog
- Configuration dialog
- About dialog
I’ve tried to separate views from models as much as possible. Each of those items is implemented in its own class (well, in classes which inherit from the GTK classes listed). The ListViews are coupled with other classes which inherit from ListStores. The library itself is handled by a different class. Nonetheless, there are interactions between the widgets that need to be tested. For example, if the user selects a particular item in the filter view, filtering the library, and then selects an item from the filtered results, the text view must display the information for the correct library entry, which is semi-complicated due to translating iters between TreeModelFilter and the original ListStore, etc etc.
So, I ask, what is the recommended way for writing robust unit tests for such a GUI application? I’ve seen that there are some libraries for this but the main ones for pygtk haven’t been updated in years and so they will almost certainly fail with PyGObject introspection. Perhaps I’m not creative enough to figure out a good way to do it using Python’s unittest module, so I’m open to suggestions.
Answers:
Are you sure you want to unit test the GUI? Your example of complexity involves more than 1 unit and is therefore an integration test.
If you really want to unit test, you should be able to instantiate a single class providing mocks or stubs for its dependencies, then call methods on it like the GUI framework would for e.g. a user click. This can be tedious and you have to know exactly how the GUI framework dispatches user input to your classes.
My advice is to put even more stuff in models. For your given example you could create a FilterManager which abstracts all the filter/select/display stuff behind a single method. Then unit test it.
In sticking with the theme that Jürgen was correct in that I’m not interested in unit testing, but actually interested in integration testing, I also found this framework from freedesktop.org:
http://ldtp.freedesktop.org/wiki/
It allows automating a variety of tests for Accessibility-enabled GUI applications (including GTK, Qt, Swing, etc).
There is a great way to test PyGTK functions and widgets directly, without going through acceptance/functional/integration testing frameworks that clikety their way into oblivion. I learned about this in this post which is fairly self explanatory. But the basic idea is that you treat your widgets as functions/classes, and you can test them directly. If you need to process callbacks and so on, there’s a neat trick that I will reproduce here:
import time
import gtk
# Stolen from Kiwi
def refresh_gui(delay=0):
while gtk.events_pending():
gtk.main_iteration_do(block=False)
time.sleep(delay)
As mentionned in the blog post, this code is LGPL. Otherwise, when you think about it, as long as you don’t show() windows or widgets, you can test them all you want and they should behave as if they were real because, in a way, they are. They are just not displayed.
Of course, you need to simulate interaction on buttons and interactive widgets yourself by calling clicked() on a button for example. See again Ali Afshar’s excellent post about unit testing in PyGTK.
You can use a X11 framebuffer:
Xvfb :119 -screen 0 1024x768x16 &
export DISPLAY=:119
... run your tests
Be sure, to not enter gtk.main(), since this would wait for mouse or keyboard input. You can use this pattern to let gtk handle all events:
def refresh_gui():
while gtk.events_pending():
gtk.main_iteration_do(block=False)
AFAIK you can’t see your application, but you can test your callbacks.
I’m currently foolish enough to try to maintaintain two parallel code bases for a Python desktop application, one using PyGObject introspection for GTK 3 and one using PyGTK for GTK 2. I work mainly on the PyGObject branch and then I port changes over to the PyGTK branch. Due to all the minor differences between these implementations, I often overlook things and cause breakage that I miss and accidentally release, only to be caught by the users.
I’m trying to figure out a good way to design some unittests that would, preferably, be suitable to run on both code bases. It’s not an overly complicated program, (it’s essentially a library management tool, imagine like iTunes):
- Main Window
|- Toolbar with some buttons (add/edit/remove items, configure the program)
|
|- VPaned
|--- Top HPaned
|------ ListView (listing values by which a library of items can be filtered)
|------ ListView (listing the contents of the library
|--- Bottom HPaned
|------ Image (displaying cover art for the currently selected item in the library)
|------ TextView (displaying formatted text describing the currently selected item)
- Edit dialog
- Configuration dialog
- About dialog
I’ve tried to separate views from models as much as possible. Each of those items is implemented in its own class (well, in classes which inherit from the GTK classes listed). The ListViews are coupled with other classes which inherit from ListStores. The library itself is handled by a different class. Nonetheless, there are interactions between the widgets that need to be tested. For example, if the user selects a particular item in the filter view, filtering the library, and then selects an item from the filtered results, the text view must display the information for the correct library entry, which is semi-complicated due to translating iters between TreeModelFilter and the original ListStore, etc etc.
So, I ask, what is the recommended way for writing robust unit tests for such a GUI application? I’ve seen that there are some libraries for this but the main ones for pygtk haven’t been updated in years and so they will almost certainly fail with PyGObject introspection. Perhaps I’m not creative enough to figure out a good way to do it using Python’s unittest module, so I’m open to suggestions.
Are you sure you want to unit test the GUI? Your example of complexity involves more than 1 unit and is therefore an integration test.
If you really want to unit test, you should be able to instantiate a single class providing mocks or stubs for its dependencies, then call methods on it like the GUI framework would for e.g. a user click. This can be tedious and you have to know exactly how the GUI framework dispatches user input to your classes.
My advice is to put even more stuff in models. For your given example you could create a FilterManager which abstracts all the filter/select/display stuff behind a single method. Then unit test it.
In sticking with the theme that Jürgen was correct in that I’m not interested in unit testing, but actually interested in integration testing, I also found this framework from freedesktop.org:
http://ldtp.freedesktop.org/wiki/
It allows automating a variety of tests for Accessibility-enabled GUI applications (including GTK, Qt, Swing, etc).
There is a great way to test PyGTK functions and widgets directly, without going through acceptance/functional/integration testing frameworks that clikety their way into oblivion. I learned about this in this post which is fairly self explanatory. But the basic idea is that you treat your widgets as functions/classes, and you can test them directly. If you need to process callbacks and so on, there’s a neat trick that I will reproduce here:
import time
import gtk
# Stolen from Kiwi
def refresh_gui(delay=0):
while gtk.events_pending():
gtk.main_iteration_do(block=False)
time.sleep(delay)
As mentionned in the blog post, this code is LGPL. Otherwise, when you think about it, as long as you don’t show() windows or widgets, you can test them all you want and they should behave as if they were real because, in a way, they are. They are just not displayed.
Of course, you need to simulate interaction on buttons and interactive widgets yourself by calling clicked() on a button for example. See again Ali Afshar’s excellent post about unit testing in PyGTK.
You can use a X11 framebuffer:
Xvfb :119 -screen 0 1024x768x16 &
export DISPLAY=:119
... run your tests
Be sure, to not enter gtk.main(), since this would wait for mouse or keyboard input. You can use this pattern to let gtk handle all events:
def refresh_gui():
while gtk.events_pending():
gtk.main_iteration_do(block=False)
AFAIK you can’t see your application, but you can test your callbacks.