What approach(es) have you used for lightweight Python unit-tests on App Engine?
Question:
I’m about to embark on some large Python-based App Engine projects, and I think I should check with Stack Overflow’s “wisdom of crowds” before committing to a unit-testing strategy. I have an existing unit-testing framework (based on unittest with custom runners and extensions) that I want to use, so anything “heavy-weight”/”intrusive” such as nose, webtest, or gaeunit doesn’t seem appropriate. The crucial unit tests in my worldview are extremely lightweight and fast ones, ones that run in an extremely short time, so I can keep running them over and over all the time without breaking my development rhythm (e.g., for a different project, I get 97% or so coverage for a 20K-lines project with several dozens of super-fast tests that take 5-7 seconds, elapsed time, for a typical run, overall — that’s what I consider a decent suite of small, fast unit-tests). I’ll have richer/heavier tests as well of course, all the way to integration tests with selenium or windmill, that’s not what I’m asking about;-) — my focus in this question (and in most of my development endeavors;-) is on the small, lightweight unit-tests that lightly and super-rapidly cover my code, not on the deeper ones.
So I think what I need is essentially a set of small, very lightweight simulations of the various key App Engine subsystems — data store, memcache, request/response objects and calls to webapp handlers, user handling, mail, &c, roughly in this order of priority. I haven’t found exactly what I’m looking for, so it seems to me that I should either rely on mox, as I’ve done often in the past, which basically means mocking each subsystem used in a given test and setting up all expectations &c (strong, but lots of work each time, and very sensitive to the tested-code’s internals, i.e. very “white-box”y), or rolling my own simulation of each subsystem (and doing asserts on the simulated subsystems’ states as part of the unit tests). The latters seems feasible, given GAE’s Python-side strong “stubs” architecture… but I can’t believe I need to roll my own, i.e., that nobody’s already written such simple-minded simulators!-) E.g., for the datastore, it looks like what I need is more or less the “datastore on file” stub that’s already part of the SDK, plus a way to mark it readonly and easy-to-use accessors for assertions about the datastore’s state; and so forth, subsystem by subsystem — each seems to need “just a bit more” than what’s already in the SDK, “perched on top” of the existing “stubs” architecture.
So, before diving in and spending a day or two of precious development time “rolling my own” simulations of GAE subsystems for unit testing purposes, I thought I’d double check with the SO crowd and see what y’all think of this… or, if there’s already some existing open source set of such simulators that I can simply reuse (or minimally tweak!-), and which I’ve just failed to spot in my searching!-)
Edit: to clarify, if I do roll my own, I do plan to leverage the SDK-supplied stubs where feasible; but for example there’s no stub for a datastore that gets initially read in from a file but then not saved at the end, so I need to subclass and tweak the existing one (which also doesn’t offer particularly convenient ways to do asserts on its state — same for the mail service stub, etc). That’s what I mean by “rolling my own” — not “rewriting from scratch”!-)
Edit: “why not GAEUnit” — GAEUnit is nice for its own use cases, but running dev_appserver and seeing results in my browser (or even via urllib.urlopen) is definitely not what I’m after — I want to use a fully automated setup, suitable for running within an existing test-running framework which is based on extending unittest, and no HTTP in the way (said framework defines a “fast” test as one that among other thing does no sockets and minimal disk I/O — we simulate or mock these — so via gaeunit I could do no better than “medium” tests) + no convenient way to prepopulate datastore for each test (and no OO structure to help customize things).
Answers:
I use GAEUnit for my Google App Engine App and I am quite happy with the speed of the tests. The thing that I like about GAEUnit,and I am sure Webtest does it, is that it creates its own version for stubs of everything for testing leaving your “live” versions alone for testing.
So your datastore that you may be using for development will be left as is when you run your GAETests.
You don’t need to write your own stubs – the SDK includes them, since they’re what it uses to emulate the production APIs. Not all of them are suitable for use in unit-tests, but most are. Check out this code for an example of the setup/teardown code you need to make use of the built in stubs.
NoseGAE is a nose plugin that support unittests by automatically setting up the development environment and a test datastore for you. Very useful when developing on dev_appserver.
I might also add that Fixture has been very useful in my unit tests. It lets you create models in a declarative syntax, which it converts into stored entities that you can load in your tests. This way you have the same data set at the beginning of every test case!, which saves you from having to create data by hand at the start of every test. Here is an example, from the Fixture documentation:
Given this model:
from google.appengine.ext import db
class Entry(db.Model):
title = db.StringProperty()
body = db.TextProperty()
added_on = db.DateTimeProperty(auto_now_add=True)
Your fixture would look like this:
from fixture import DataSet
class EntryData(DataSet):
class great_monday:
title = "Monday Was Great"
body = """
Monday was the best day ever.
"""
Note however, that I ran into the following issues:
1. This bug, but the included patch does remedy it.
2. The datastore is not -by default- reset between test cases. So I use this to force a reset for each test case:
class TycoonTest(unittest.TestCase):
def setUp(self):
# Clear out the datastore before starting the test.
apiproxy_stub_map.apiproxy._APIProxyStubMap__stub_map['datastore_v3'].Clear()
self.data = self.load_data()
self.data.setup()
os.environ['SERVER_NAME'] = "dev_appserver"
self.after_setUp()
def load_data(self):
return datafixture.data(*dset.__all__)
def after_setUp(self):
""" After setup
"""
pass
def tearDown(self):
# Teardown data.
try:
self.data.teardown()
except:
pass
Since 1.3.1 version of SDK there is the build-in unit test framework.
It is Java only right now but I feel like:
- it is much the same you talk about in your question (and much more – as running test in the cloud for example)
- and it is quite possible to portimplement the same on Python using SDK
So does the author of this framework – Max Ross and he explicitly tells us about it in his I/O presentation “Testing techniques for Google App Engine”
Does anyone have any updates on this topic?
The SDK 1.4.3 Testbed API provides easy configuration of stub libraries for local integration tests.
I’m about to embark on some large Python-based App Engine projects, and I think I should check with Stack Overflow’s “wisdom of crowds” before committing to a unit-testing strategy. I have an existing unit-testing framework (based on unittest with custom runners and extensions) that I want to use, so anything “heavy-weight”/”intrusive” such as nose, webtest, or gaeunit doesn’t seem appropriate. The crucial unit tests in my worldview are extremely lightweight and fast ones, ones that run in an extremely short time, so I can keep running them over and over all the time without breaking my development rhythm (e.g., for a different project, I get 97% or so coverage for a 20K-lines project with several dozens of super-fast tests that take 5-7 seconds, elapsed time, for a typical run, overall — that’s what I consider a decent suite of small, fast unit-tests). I’ll have richer/heavier tests as well of course, all the way to integration tests with selenium or windmill, that’s not what I’m asking about;-) — my focus in this question (and in most of my development endeavors;-) is on the small, lightweight unit-tests that lightly and super-rapidly cover my code, not on the deeper ones.
So I think what I need is essentially a set of small, very lightweight simulations of the various key App Engine subsystems — data store, memcache, request/response objects and calls to webapp handlers, user handling, mail, &c, roughly in this order of priority. I haven’t found exactly what I’m looking for, so it seems to me that I should either rely on mox, as I’ve done often in the past, which basically means mocking each subsystem used in a given test and setting up all expectations &c (strong, but lots of work each time, and very sensitive to the tested-code’s internals, i.e. very “white-box”y), or rolling my own simulation of each subsystem (and doing asserts on the simulated subsystems’ states as part of the unit tests). The latters seems feasible, given GAE’s Python-side strong “stubs” architecture… but I can’t believe I need to roll my own, i.e., that nobody’s already written such simple-minded simulators!-) E.g., for the datastore, it looks like what I need is more or less the “datastore on file” stub that’s already part of the SDK, plus a way to mark it readonly and easy-to-use accessors for assertions about the datastore’s state; and so forth, subsystem by subsystem — each seems to need “just a bit more” than what’s already in the SDK, “perched on top” of the existing “stubs” architecture.
So, before diving in and spending a day or two of precious development time “rolling my own” simulations of GAE subsystems for unit testing purposes, I thought I’d double check with the SO crowd and see what y’all think of this… or, if there’s already some existing open source set of such simulators that I can simply reuse (or minimally tweak!-), and which I’ve just failed to spot in my searching!-)
Edit: to clarify, if I do roll my own, I do plan to leverage the SDK-supplied stubs where feasible; but for example there’s no stub for a datastore that gets initially read in from a file but then not saved at the end, so I need to subclass and tweak the existing one (which also doesn’t offer particularly convenient ways to do asserts on its state — same for the mail service stub, etc). That’s what I mean by “rolling my own” — not “rewriting from scratch”!-)
Edit: “why not GAEUnit” — GAEUnit is nice for its own use cases, but running dev_appserver and seeing results in my browser (or even via urllib.urlopen) is definitely not what I’m after — I want to use a fully automated setup, suitable for running within an existing test-running framework which is based on extending unittest, and no HTTP in the way (said framework defines a “fast” test as one that among other thing does no sockets and minimal disk I/O — we simulate or mock these — so via gaeunit I could do no better than “medium” tests) + no convenient way to prepopulate datastore for each test (and no OO structure to help customize things).
I use GAEUnit for my Google App Engine App and I am quite happy with the speed of the tests. The thing that I like about GAEUnit,and I am sure Webtest does it, is that it creates its own version for stubs of everything for testing leaving your “live” versions alone for testing.
So your datastore that you may be using for development will be left as is when you run your GAETests.
You don’t need to write your own stubs – the SDK includes them, since they’re what it uses to emulate the production APIs. Not all of them are suitable for use in unit-tests, but most are. Check out this code for an example of the setup/teardown code you need to make use of the built in stubs.
NoseGAE is a nose plugin that support unittests by automatically setting up the development environment and a test datastore for you. Very useful when developing on dev_appserver.
I might also add that Fixture has been very useful in my unit tests. It lets you create models in a declarative syntax, which it converts into stored entities that you can load in your tests. This way you have the same data set at the beginning of every test case!, which saves you from having to create data by hand at the start of every test. Here is an example, from the Fixture documentation:
Given this model:
from google.appengine.ext import db
class Entry(db.Model):
title = db.StringProperty()
body = db.TextProperty()
added_on = db.DateTimeProperty(auto_now_add=True)
Your fixture would look like this:
from fixture import DataSet
class EntryData(DataSet):
class great_monday:
title = "Monday Was Great"
body = """
Monday was the best day ever.
"""
Note however, that I ran into the following issues:
1. This bug, but the included patch does remedy it.
2. The datastore is not -by default- reset between test cases. So I use this to force a reset for each test case:
class TycoonTest(unittest.TestCase):
def setUp(self):
# Clear out the datastore before starting the test.
apiproxy_stub_map.apiproxy._APIProxyStubMap__stub_map['datastore_v3'].Clear()
self.data = self.load_data()
self.data.setup()
os.environ['SERVER_NAME'] = "dev_appserver"
self.after_setUp()
def load_data(self):
return datafixture.data(*dset.__all__)
def after_setUp(self):
""" After setup
"""
pass
def tearDown(self):
# Teardown data.
try:
self.data.teardown()
except:
pass
Since 1.3.1 version of SDK there is the build-in unit test framework.
It is Java only right now but I feel like:
- it is much the same you talk about in your question (and much more – as running test in the cloud for example)
- and it is quite possible to portimplement the same on Python using SDK
So does the author of this framework – Max Ross and he explicitly tells us about it in his I/O presentation “Testing techniques for Google App Engine”
Does anyone have any updates on this topic?
The SDK 1.4.3 Testbed API provides easy configuration of stub libraries for local integration tests.