Twisted: Bye for Now

3 AM May 25, 2004

Back in February, I decided to write my blogging software—KangaPy—in Twisted.

For those who have it come across it before, Twisted is a single threaded, “event-driven networking framework.” The price of using Twisted is that your code must never block on I/O. Instead, the code must return a “Deferred” object that contains a callback which is fired when the I/O operation completes.

In return for writing your code inside-out, Twisted gives you a framework that scales well, freedom from multi-threading concurrency issues and a very clear idea of how your application interacts with the outside world.

Aside from that, Twisted has three1 big pluses:

  1. An active, helpful user community,
  1. Support for a wide range of internet protocols, and
  1. The cutest web framework/HTML generation/templating system I have seen, Nevow.

Unfortunately, a blog with a simple web interface on one end and an RDBMS on the other is not a good fit to Twisted. Since KangaPy is due a major refactoring, I had to decide whether to stay with Twisted (and start using Nevow) or move to a different framework.

I decided to port KangaPy to another framework. This will most likely be CherryPy with Cheetah templates, but time will tell.

I’m glad I decided to give Twisted a go, because the experience may well turn out to be useful in other contexts. I’ve already had one project for which Twisted would have been a great choice, except that my client needs it done in Java.

And finally, a big thanks to Matt Goodall and Donovan Preston for taking the time to point me to Nevow. I played with the tutorials, and it looks great. If you ever manage to make Nevow useful without Twisted, I’d be very interested.


1 This is just my opinion, formed in the light of limited experience.

By alang | # | Comments (17)
(Posted to Python)

Comments

At 07:23, 25 May 2004 opq wrote:

Did you know there are some binding Twisted for Java ?
That may be useful fo you !

(#)
At 09:37, 25 May 2004 Matt Goodall wrote:

The intention is to get Nevow working in CGI mode too, it's just a matter of time and, to some degree, interest. In theory, it's a matter of writing a CGI Request and making the Twisted support optional. It's never quite as easy as you think though ;-).

CherryPy is an interesting choice. I have played with it a little but was put off by the need for the compiler and the not-quite-Python-ness of it all. I should experiment with its AOP features more, they look interesting.

Have you looked at Quixote too (www.quixote.ca)? It's simple and lightweight but also very Pythonic.

Anyway, it's a shame you're not planning on using Nevow. Are you sure that not having to write the HTML for a form and validate it can't persuade you otherwise? ;-)

(#)
At 11:40, 25 May 2004 Jkx wrote:

I just finish my own system too, which use ZPT + SQLObject + Webware (http://www.larsen-b.com/About) And i'm a bit ok w/ you, thread is really hard to use for this kind of stuff.

This is mainly why i looked at twisted for a couple of hours.
Could you please explain deeply what enforce your choice ? (I guess that DB access in twisted are not really easy to use but ..)

And why did you choose Cherry ?

I think Cheetah is a great choice for template. I used ZPT for a couple of projects, but now i think all this HTML specific template engine have a big trouble, they can be used in something else, so building CSS for example don't apply to ZPT. (That's why zope keeps DTML ?)

(#)
At 12:56, 25 May 2004 John B. Cole wrote:

I am currently porting an application of mine from PHP to CherryPy. I have used Quixote in the past but seem to have some sort of cognitive problem with it -- it never seems to be as simple to me as it seems to be to others. Quixote may be more Pythonic, and the compile step in CherryPy is definitely un-Pythonic, but I find that CherryPy is a more comfortable tool for me (at least for now). I like the fact that you do not have to do a bunch of poking at Apache's configuration files to get CherryPy up and running.

(#)
At 13:59, 25 May 2004 Matt Goodall wrote:

Jack, database access in Twisted is really quite easy. See, www.twistedmatrix.com/documents/current/howto/enterprise. The adbapi module manages the allocation of database connections (from a connection pool) and spawns threads (from a thread pool) to run the query.

Nevow makes RDBMS access (and deferred processing in general) even easier. As a developer, you write two functions: one requests the data using the adbapi module; the other renders the result. Nevow takes care of calling the render function with the data once it is ready. (You don't always need a function to render the data but that is a different point).

John, yeah I agree. Having to use Apache is a real pain during development (and often at deployment time too ;-)). I'm using twisted.web for my current project and having a standalone, easily restartable web-server-application is just wonderful. CherryPy is nice like this too although I was a little concerned to see that it built on BaseHTTPServer. Does it work well enough under real use?

Incidentally, the Quixote developers were talking about distributing a stripped down medusa (async web server, based on ayncore) with Quixote but I'm not sure of the current status.

Whoa, didn't mean to do the whole Twisted+Nevow advocacy thing on your blog Alan! By the way, how do you post URLs here without it messing up the paragraph?

(#)
At 14:25, 25 May 2004 Jkx wrote:

"Jack, database access in Twisted is really quite easy". Yes i understand this, but i want to use things like Modeling, or SQLObject to map database.

After using any RDBM <-> object mapping i will never go back to old-fashion SQL :)

(#)
At 15:03, 25 May 2004 Remi wrote:

> I have played with it a little but was put off by the need for the compiler
[...]
> The compile step in CherryPy is definitely un-Pythonic

That's true, but in practice, this is never a problem... Let's say your "server" script is called "my-server.py"
In a regular "pythonic" development environment, you would start this script in a window ...
Then you would test your site, update your source file(s), go to the window where you started the script, hit "Control-C", probably use the "up-arrow" to repeat the last command and hit "enter" to start the server again with the new code.
Well, with CherryPy, you just need a 2-line script (called "compile-and-start" for instance that compiles your source files and then start the server.
Instead of typing "my-server.py" in the window, you just type "compile-and-start" and everything else is the same ...
Another important point is that when you get a python traceback, it is really easy to know which source file/lines of code it came from (otherwise, development would be a nightmare).

> I was a little concerned to see that it built on BaseHTTPServer. Does it work well enough under real use?

It is actually a heavily customised version of BaseHTTPServer and uses a thread-pool model, which works very well in production environments. CherryPy is now quite mature and it has been used in many projects, some of them with quite high-traffic websites.
I think one of the sites that gets a decent traffic is waypath.com
It has also been used for a site coinciding with a TV auction, so the site would get high peaks of traffic when the auction was on ...
The maintainer of the site reported that the CherryPy server "barely made a blip on the server" whereas "a previous version using webware brought the server down under the load" ...

(#)
At 17:01, 25 May 2004 Alex Levy wrote:

I have to ask, why insist on an RDBMS for just a blog? What were the problems you encountered with SQLite? (Also, wouldn't bsddb/atop, or just flat files, work "well enough"?)

I'm still attempting to write a blog in Nevow, so I'd be interested in seeing your Twisted-esque code -- I might steal an idea or two. Do you have any plans to open up the Kangapy source?

(Forgive me if the code is available somewhere; I couldn't find it.)

(#)
At 17:43, 25 May 2004 Ian Bicking wrote:

There's a poorly-documented feature in Webware that if you run "AppServer http" it will serve directly from 8080. I still prefer going through Apache (often using wkcgi, a small gateway-like CGI script), as it has some advantages during development and is a reliable way to deploy. Either way the configuration seems to be a lot less than with mod_python-based systems. Now if we could we just got rid of MakeAppWorkDir, then initial installation would be truly trivial. (I'd like to be able to run AppServer from within the default context and have it Just Work.)

Webware also will restart on your behalf when files are changed (use --AppServer.AutoReload=1, or change AppServer.config), so you are never working with stale files. So not only don't you need to recompile, you don't have to even restart. Webware is fairly conservative in a lot of ways (though maybe that's just limited scope), but I think there's a lot of details in it that make development pleasant.

(#)
At 17:55, 25 May 2004 has wrote:

Matt Goodall:

"Anyway, it's a shame you're not planning on using Nevow. Are you sure that not having to write the HTML for a form and validate it can't persuade you otherwise? ;-)"

Freeform is an idea that definitely has legs. I think it might become a terrific system for constructing browser-based UIs, kinda like Tkinter and wxPython do for the desktop GUI. It's something I'd definitely like to explore myself.

The other parts are of less interest to me: Renderer isn't as good as my own HTMLTemplate; Stan looks nice, but I can't convince myself that HTML generators provide sufficient advantage over raw HTML + string substitutions to justify the work; and I think formless'd make more sense split off as a project in its own right than locked into Nevow as it currently is. Freeform though... mmmm.

Alan:

"If you ever manage to make Nevow useful without Twisted, I’d be very interested."

I'll second that. The high coupling between Nevow's various modules and to the Webware framework makes it somewhat inflexible to use (i.e. it's "all-or-nothing") and the code hard to grok and hack. This has rather put me off exploring it any further for now, but I'll certainly be taking another look if/when it's fixed.

(#)
At 12:26, 26 May 2004 Alan wrote:

Alex:

I'm using the blog as means to explore web development with Python. Since most of the web apps that I develop have an RDBMS backend, I wanted to see how it all worked together.

(#)
At 12:30, 26 May 2004 Alan Green wrote:

Matt:

"Whoa, didn't mean to do the whole Twisted+Nevow advocacy thing on your blog Alan!"

So long as the advocacy doesn't descend into trolls, I really don't mind. Go for it!

(#)
At 14:00, 26 May 2004 Anthony wrote:

Did you have a look at mod_python? If you want to simply glue data to a web server with python this may be the fastest option. Sticking with LAMP makes is not bad idea. The new mod_python supports sessions and has a basic templating system. You could use

(#)
At 15:34, 26 May 2004 john wrote:

I faced the same problem and made the opposite choice: stay with Nevow. Yes, I want a great web framework and a great ORM in the same package, but if I can't have both I'll choose the web framework.

For a specific RDBMS and problem domain I can hack together an adequate ORM (a generalized ORM is a whole different animal of course). Having done that, I'm no longer writing raw SQL and the pain goes away.

If I switch to a web framework that chafes, even a little, or add a compiled language to the mix - well, that pain never goes away.

(#)
At 20:28, 26 May 2004 Steve wrote:

Yes, as Remi mentioned, we use CherryPy as the front end for Waypath. I can't say enough good about it. It's reliable, fast, and carries very little conceptual overhead.

RE: compiling, I find it convenient, rather than a burden. The CherryPy code for the site is spread over 7 files, so compiling makes it easier to move code up to a production environment. It's a "feature" not a "bug."

(#)
At 05:27, 12 Feb 2005 Sridhar Ratna wrote:

SVN Nevow works seemlessly in CGI. See
http://srid.bsdnerds.org/blog/2005/02/06/nevow-with-cgi-and-wsgi-adaptor/

Note: There will be some little annoyances (URL related exceptions), for which I have already made a couple of patches and submitted to DP who will apply it this weekend.

Please note that nevow 0.3 does not have wsgi adaptor, thus you need to get svn nevow.

(#)
At 15:50, 03 Apr 2005 Gabriel Birke wrote:

The un-pythonic character of CherryPy is now remedied with the 2.0 release. No need to compile *.cpy files, just import one file and you have a fully functional web server that maps urls to objects!

(#)

Add Comment




(Not displayed)






(Leave blank line between paragraphs. URLs converted to links. HTML stripped. Indented source code will be formatted with <pre> tags.)




© 2003-2006 Alan Green