November, 2003

Governments around the world are seriously considering an open source based office infrastructure over the Microsoft solution. Brazil, Germany, and France have gone as far as mandating that open source or open standard software be treated preferentially to completely proprietary solutions.

The office infrastructure market—including network server O/S, desktop O/S and office applications—was once highly competitive, with a typical site deploying products from three or four major vendors. Ten years on, Microsoft have a near monopoly. Open source software in this context is novel, but it is Microsoft’s only serious competition, and it is already causing them to consider the issues of interoperability, and pricing.

For Governments, open source offers several advantages over the Microsoft solution:

• Procurement is free (as in beer).

• Decisions to deploy new or upgraded software may be made solely on technical and business grounds, without being subject to the whims of Microsoft Sales and Marketing.

• Open source software is more trustworty than that purveyed by a monopoly.

• Open source suits marginal budgets; there are many applications that will run acceptably on older hardware.

• Citizens have free (as in speech) access to the tools required to review and comment on Government generated documentation.

To be sure, there are disadvantages to choosing Open Source solutions too:

• Open source software is generally less mature and well polished than the equivalent commercial packages.

• Government departments are somewhat locked-in to Microsoft’s non-standard, non-interoperable file formats.

• Open source products aren’t made by companies that run accrediation programs, so there aren’t many people officially certified to work with the technology.

Overall though, Governments using open source is a Good Thing. It is good for the department using the software, and it is good for the people they serve—you and me.

Government open source is also good increasing the quality and range of office software generally. I hope the new competition in this market continues to the point where users once again have a serious choice of products in this area.

When I first learnt Java, it seemed like a huge step forward. This last fortnight, I’ve been making that discovery all over again – I’m working on a C++ system for the first time in six years.

Case in point: just this morning, a co-worker and I figured out a trick that will save an error-prone, cut-and-paste duplication of thirty or forty struct definitions. The saving comes at a small cost: there will be strange #ifdefs dotted around the header files.

We were discussing whether or not to document the #ifdef trick for the developers who come after us. My co-worker took the position¹ that no documentation was needed. His argument went along these lines:

You know what’s going to happen… [the developers who come after us] are going to say “Well I tried taking it [the #ifdefs] out, but it didn’t work, so I put it back the way I found it.”

Since the #ifdefs will always be put back the way they were found, they will always keep working. Understanding of the mechanism the #ifdefs (ab)use, or even the intent of the #ifdefs, is not necessary. And, as the sole purpose of the proposed documentation would be to promote understanding, the proposed documentation is not necessary. Q.E.D.

---

¹ With his tongue in his cheek, of course.

KDE is still giving me pleasant surprises. Over the last week, I discovered three neat features in Konqueror.

First up, Konqueror supports the SFTP protocol. This means you can use Konqueror to browse remote file systems and transfer files to and from almost any host running sshd.

Type in a URL such as “sftp://yourhost.com”;, enter your password on the dialog box that appears, and – hey, presto! – you’re looking at the remote directory listing. From the directory listing, you can drag-and-drop, cut-and-paste, click-to-view, right click, and all the usual stuff.

The second feature is the embedded Cervisia mode. Cervisia is KDE‘s light-weight CVS tool. To turn it on, bring up Konqueror on a CVS directory on your local file system, then press the toolbar button that looks like a red-brick wall.

Cervisia shows which files are checked into CVS, their revision, and whether they have been modified. You can also do diffs, browse the CVS logs, commit changes or update from the repository.

Cervisia works particularly well with a third feature, the Midnight Commander mode. To get there, open a new Konqueror window, then select “Settings > Load View Profile > Midnight Commander” from the menus.

Konqueror will reconfigure itself with three panes. Below the horizontal split is a terminal window, open to your home directory. Above the horizontal split, the window is split vertically to show two file listings. You can copy and paste files by dragging and dropping them between the panes. You can even drop them on the terminal window to paste in file names to commands.

If that’s not sweet enough, you can press F9 to open the navigation panel, giving you a tree structured view of the file system as well. This is much better than the Windows Explorer.

Konqueror has me well and truly hooked.

To be fair, Konqueror is not perfect: (a) it crashes maybe once a fortnight,¹ (b) there are a few, rare incompatibilities² – though bloglines is the only one I have found recently, and© it doesn’t run on Windows, so my kids can’t use it on game sites that rely upon IE plugins.³

---

^1^ Show me a browser that doesn't.
^2^ Show me a browser that does work on every site.
^3^ Show me a browser that does allow IE plugins, and I'll show you a security hole.

Since I’m off to work with C++ for a few months, I went and investigated CppUnit. CppUnit started life as a port of JUnit, but has gone on to develop some neat, C++-style features, such as the test registry.

Unfortunately, CppUnit requires a fairly modern C++ compiler. Since my client is using HP-UX 10.20, CppUnit won’t be an option.

Partly to address this need for a unit test framework, and partly as an exercise in getting my head around C++ again, I wrote my own, minature unit test framework. The source code is online, the most important file being unittest.h. The generated documentation is also online.

A quick To Do list:

• Currently have just one CHECK macro, that checks an assertion is true. Should also make a CHECK_EQUAL, and possibly allow custom error messages.
• Get the category mechanism working so that subsets of the unit tests may be run.
• Factor out the test result display code from the UnitTestRegistry::run() method.
• Display failing line numbers and file names.
• Package appropriately for use in a larger project.

Being my first C++ code for quite some time, comments, suggestions and critique are very welcome.

Unless an SQL query specifies an ORDER BY clause, a database returns result rows in arbitrary order. If you don’t respect this in your unit testing, it causes a problem.

The problem arises because, for two identical queries over the same data, most databases will return result rows in the same order. Developers see the data come back in the same order twice and write unit tests that assume this order. Unfortunately, tests written like this will probably continue to work for quite some time.

Fast forward to the end of the project, when more is known about how the application uses the database, some poor bunny is giving some thought to the physical layout of the database – adding indexes, changing table storage parameters and the like. None of these physical layout changes affect the application, except perhaps to make it run faster.

But suddenly, all kinds of odd unit tests are breaking, even though the application itself still works fine.

The unit tests are breaking because the changes to the physical schema are affecting the order of the query result rows. Though it may take a few hour’s tedious, unnecessary, and high-pressure work to figure out what went wrong, and to fix the tests.

The best solution is to ensure that results returned from the database are always sorted; always use an ORDER BY clause in database queries. For unit testing, an arbitrary but stable order is preferable to a random ordering. If nothing else makes sense, try ordering by the primary key.

Sometimes it is not acceptable to order query results. In these cases the unit tests should either sort the resulting data in-memory, or scan through the result set to find the data it is looking for.

If you do these thing from the very start of your project, you will make the world a better place.

I’ve been calculating storage requirements for the small system our team is about to deliver. To hold the first three years of data, the Oracle database will require 400 Mb of storage, including overheads and indexes.

An 80Gb drive can be had for $110.00 (Aussie) dollars, which means our system is going to use a grand total of 55 cents worth of storage between now and 2007.

Between them, two or three individuals posted 15 blogspam to cardboard nu today, all advertising paedophile porno. I have trouble articulating my thoughts on this subject, but to have my blog co-opted in this manner left me feeling dirty and ill-used.

Free speech be damned; some things are just too wrong to be allowed to see the light of day. Words don’t exist in a vacuum.

In response to this and previous commercial comments, I have installed Jay Allen’s MT-Blacklist plugin. MT-Blacklist scans incoming comments for ‘inappropriate’ regexps. In particular, it targets the URLs that spammers promote. Turns out that there are only a few hundred common blog-spammed URLs, so the problem is quite tractable.

Of course, the Red Queen principle applies, and I suspect that I will be dealing with blogspam for as long as I run a blog.

I’ve been particularly enjoying my job over the last few weeks. There have been plenty of new products to look at and play with. I’ve been to visit clients. We’ve delivered working code. I’ve even done some fruitful pair programming.

Right now we’re in the middle of delivering a J2EE web app to a client. The system is functionally OK, so Charles and I are looking at the performance.

Our first task was to work out how to load a realistic number of records. We’re aiming for 75,000 people, a few more than the twenty or so we use for unit testing. We chose to load data through the applications web interface, modifying existing unit tests based on the httpunit library. That took an afternoon, and we can now load arbitrary data as fast as the server will handle it.

The problems we’ve come across so far have proven surprisingly tractable. A few log statements new database index here, and re-written Hibernate query there, and issues either disappear, or are at least reduced to acceptability.

One neat Oracle feature is the Function Based Index, which lets you create an index based on a function, such as UPPER(surname). In our case, it saved having to create a special column in the Person table to hold an uppercase version of the surname.

Meanwhile, my boss has sold me to another client. Next week I’ll be on-site for a few months with another good friend. The down-side is that the project is in C++, but the up-sides are many:

• only three people in the team;
• clearly defined scope;
• project has the support of the business;
• uses technologies I haven’t worked with before (Informix), or for a long time (HP-UX);
• client is keen to upgrade their tools and to take on unit testing and other techniques;
• it’s a green-field application—we get to build it however it needs to be build, so long as we keep it maintainable; and, thankfully,
• the project has nothing to do with insurance, superannuation or finance.

Woohoo!

Went and saw Matrix: Revolutions Saturday night. Thoroughly enjoyed it. Great piece of escapism.

I find myself agreeing with Andrew O’Hehir’s review at Salon.com. The Matrix is a good movie, but not a great movie, and many were hoping for a great movie.

Oh well. I gave up expecting much from sequels after George Lucas decided that Jar Jar Binks could be a General in the battle for Naboo.

Even so, Matrix: Revolutions is cool. Neo gets a social conscience—completing the transformation from lone hacker to saviour. Characters that were very important fade into the background and new ones come to the fore. You get the feeling that this is a big world.

I liked the ending too. It didn’t try to do too much, just completed the story without trying to tie up every loose end.

I look forward to more stories from the world of the Matrix. Maybe Animatrix 2?

Twisted LivePage lets a Python server handle any browser event such as onClick or onMouseOver and send arbitrary Javascript to the browser for execution.

The magic is done with a custom flash control embedded in the browser page, which opens a socket back to the server after the page has been loaded.

This could be really very interesting in a high-bandwidth, low-latency environment.

Via the DDJ Python URL!.

One of the benefits of going to my church is getting to hang out with people like Val and Adam, who volunteer a few hours a week to teach scripture in the local primary schools.

This year, Val and Adam taught a sixth-grade class. Val presents the lessons from the front, while Adam shepherds the class along from the back. One of the sixth graders, Jacob, was a real handful, and a large part of Adam’s energy was devoted to checking the Jacob was listening, or at least not disturbing the rest of the class.

One memorable week, Jacob turned up to class with a wristband bearing the initials “WWJD”. WWJD¹ stands for “What Would Jesus Do?”, a personal prompt that is popular with younger Christians. For a Christian, a WWJD wristband is a constant reminder to strive to do, not just the right thing, but the best possible thing, as God would judge.

And if Jacob was wearing that wristband, then Adam knew his job was about to get a lot easier.

The next time Jacob started to distract the class, Val continued with the lesson while Adam quietly asked, “Jacob, do you go to church?”

“Yes,” said Jacob.

“So, do you know what WWJD means?” Adam asked, pointing at Jacob’s wrist-band.

Jacob nodded and replied, “World Wide Jacob Domination!”

---

¹Unfortunately, WWJD has been commercialised to an early, tragic death.

Charles has seven reasons why ESR‘s Proposed Hacker Logo is stupid.

But I think it’s just perfect for hackers :

1. Only a hacker would be so engrossed in his computer as to think that black dots in white squares constitutes “Life”.

2. The logo serves to boost the ego of your typical, arrogant hacker. If you have explored Conway’s Game of Life, then the pattern is immediately recognisable. A hacker will nonchalantly display this logo, waiting for a normal person to ask about it. When somebody does ask, the hacker will give a technically correct but rudely obtuse answer, and later secretly congratulate himself on being smarter than his victim.

On the flip side. If you didn’t recoginise the pattern in the logo, then you are obviously too stupid to own a computer. For your own safety, and the common good, please hand your computer over to your nearest authorised hacker.

3. The glider moves down and to the right. Just the way ESR thinks¹ and would like all hackers to think.

More seriously: if the logo speaks positively to you, go ahead and use it. But I won’t be labelling myself with it.

---

¹ ESR has lots of good things to say, particularly about the art and science of making computers go. His politics, however, are another matter.