Python and Unit Testing

Earlier this month, Ian Bicking blogged on the inadequacies of the standard unittest module. Reading through the discussion and comments thread is enlightening. Ian’s complaints are:

• The intuitive thing to do—create a TestCase subclass for each test case—doesn’t quite work.
• The suite mechanism is inflexible—it is impossible to run part of a suite.
• The unittest module does not allow variable verbosity logging of test progress.
• There is far too much boiler-plate.
• The method names “setUp” and “tearDown” are misspelt.

While I can see Ian’s points, I don’t feel his pain; the unittest module is perfectly adequate for my unit testing needs. I think the big difference is in our approaches to unit testing.¹

When working on a system, I try to run all the unit tests every five or ten minutes. All I really need to know is whether the tests pass or fail. When they fail I know the bug must have been introduced in the code I was just working on.

The unittest module supports this quite well. I can a TestCase class for each major piece of functionality to be tested, with multiple testXXX() methods, each containing maybe five or ten individual tests. While not recommended by the unittest documentation, I find it handy to use the Python assert keyword rather than the TestCase testing methods.

Doing it this way:

A. Boiler-plate accounts for only a small percentage of the test code—maybe 10 or 15%, which is mostly the def and class statements.

B. The unittest.main() function supports running all the tests on a TestCase class, or an individual testXXX() method on a class, something you lose if you define your own TestCase constructor.

C. No logging of intermediate test results is necessary. The traceback is usually enough to diagnose the failure, otherwise it is time to insert a few print statements or break out the debugger.²

D. Nor is it necessary to see what the failures would have been had test continued on. Usually, I have done such little coding between test runs, that I have had time to introduce only one bug. If not, I’ll find out when I run the tests again.

This method relies upon is the ability to run the tests quickly. For larger systems, I need to split the tests into separate suites. All the tests are still run, but I run the suites pertaining to the current code more frequently.

All that said, one of Python’s strengths is the active, eclectic community it supports. It was great to see references to so many other unit test libraries and extensions to the unittest module on the comments to Ian’s blog entry. Among them were:

• TestiPy from Hans Nowak
• Sancho by MEMS Exchange
• PyLid from nullcube.
• A replacement test driver from the folks at Zope Corporation.
• Finally, Garth Kidd contributes a note on self stocking unit tests, useful for coping with huge numbers of simple tests.

—————

¹ I’m certainly not claiming that my unit testing method is better than Ian’s, just different.

² I drop ”import pdb ; pdb.set_trace()” into suspect code to start a debug session.