Unit Testing Reliability with Decorators

Charles’ recent blog entry on test decorators put in mind of a presentation I attended in a previous life.

The authors of the presentation had been writing code that managed a 99.999% available environments. IIRC, their software didn’t have to be 99.999% available itself, but it did have to be responsive and it could not destabilise any other software running nearby.

To help them achieve their goals, they wrote decorators for their unit tests that measured memory usage and execution times, then logged the results in a file.

The authors then graphed and compared the logged results between runs, allowing them to watch for changes that indicate a design defect in their algorithms or a bug in their memory management. In this way, they headed off several issues that were not obvious otherwise.

Two questions spring to mind:

A. How hard is this to do? With caching, pooling and all the really tricky stuff that goes on inside a J2EE server, pinning down memory usage in an EJB app would be difficult. However, recording the elapsed time for each unit test would be simple and useful. The big ‘trick’ would be to have a test decorator write out a line to a log file containing, say, the test name, date and time, user name and elapsed time. Maybe Charles would lend me his five line Ruby script for decorating Java tests, and a few moments work in Excel or StarCalc would make pretty graphs out of the log file. Easy-peasy.

B. If it’s that simple, why aren’t I doing this all the time? Ummmm… gee… I guess I just never thought of it before. It would be good to be paying more attention to performance from the get-go, and, since it looks to be almost free, I’ll have to start doing it.