What I've been doing for 10 months

Since September (September 2nd, to be exact, according to my Subversion logs), I've been working on a Django project in my spare time, and it's finally complete. I've kept a pretty complete log of my hours, split by activity, so I'm hoping it will be of some use to those trying to make realistic estimations of coding time.

The project is now live at www.cciw.co.uk. It is a website for a charity that runs outdoor Christian camps, which I've been involved in all my life (literally!). The website has all the details of the camps, camp sites, leaders etc, and a community of people who have been on the camps, based around a message board and photo gallery system. [Edit 2017: the site is now quite different in emphasis since this article in 2006, since the forums were removed after being killed by Facebook etc., but it is still Django and still actively developed].

Until a few days ago, the website was running under PHP with a flatfile database. The new website is mainly a re-implementation of the existing functionality, but I have added quite a few things and tidied quite a lot up, and I didn't bother trying to salvage any of the existing PHP code by porting it – I wrote everything from scratch.

Aims

My main aims were:

• to get rid of the ropey old PHP code (not to mention the flatfile database), and produce some clean, maintainable code.

• to make it easy to moderate the message boards. The main users of the website are 11 - 17 year olds, and the camps are strongly Christian in ethos and aims, so it's very important that the camp website is always a safe and fun place for the campers to interact.

• to make the website manageable by other people instead of just me, and hopefully write myself out of the picture.

• to add some fun new features.

• to generally increase usability.

• to get a reliable database, and a proper SQL one that would enable things like the 'stats roundups' I occasionally do.

• to do some test driven web-development.

Hours

I've spent a total of 240 hours on the project, or about 6 working weeks. That's quite a lot, and quite a bit more than you typically hear quoted for Django apps, but the project is probably fairly large compared to most of the ones you hear about:

It contains 22 database models, about 60 view functions (which vary massively in size – a handful are straight generic views, some of the message board ones are moderately complex), 15 Atom feeds and a total of 56 template files. It also has 14 custom template tags and various other bits and pieces as you would expect in a project of that size.

Also, the figure given above totals all my activities, including:

• learning Django (and Python to some extent)

• data migration (a lot of it – I was careful to ensure none of the old message boards were lost, and some of them go back 6 years to an even older system. I even managed to rewrite any embedded URLs in message board posts etc so that they are still correct)

• design mockups (I'm not much of a designer, but I'm the only person who is working on this), and then the actual designs in XHTML 1.0 strict, the highest quality stuff I've done so far in terms of semantic HTML and clean CSS.

• all the setup and deployment issues (setup was easy, but final deployment was harder because of a component I wrote that used the same database, and some of the same tables, but was deployed earlier, so I had to do a bit of a database merge).

• a fair amount of content editing

• oh, and writing the code (models, views and templates), testing and debugging, which accounted for about 75% of the time spent (perhaps a little less, it's not always easy to divide up the time correctly).

In terms of code there are about 2000 lines of template code, 6000 lines of Python, and 900 lines of migration scripts (done in Python). I know LOC aren't that accurate a measure of program size, but hopefully that's of some help.

Given those figures, it looks like I was reasonably productive – averaging over the complete time that comes to about 37 LOC/hour, which is reasonable. I also have been careful to avoid cut-and-paste, which can be an easy way to get stuff done, (and add LOC), but also an easy way to leave an unmaintainable mess behind! The great design of Django, including things like template inheritance, and the power of the Python language makes it possible to really keep duplication to a minimum.

Some of the things that added to the development time were:

• trying to handle my existing data properly (which added a fair amount of special casing etc)

• changing Django APIs – which meant sometimes I had to rewrite, and sometimes I avoided features that I knew were not stable, and went for something that might not have been optimal just to crack on.

• lack of ability in the design area

• deciding to change to Postgres part way through (though that was fairly trouble free)

• and probably a bit of perfectionism.

I should also point out that the framework is more mature than when I started! In fact, some of the things I coded have become part of Django – the hours where I was consciously hacking on Django itself I was careful to log separately, but nevertheless some of the code I wrote for CCIW ended being generic and has made it's way into the core framework – so that's code you won't have to write if you start with Django now.

On the other hand, there are some things which would increase a realistic estimate of the coding time. The main one is that for at least half of the code I was writing, especially the message board code, I had a very good idea of what I was doing, having implemented it once already, even if it was several years ago. It's very difficult to measure the effect of this – although the python code I wrote bears very little resemblance superficially to the original PHP code, it is very likely that my subconscious knowledge of how it would work in general helped me a lot.

Analysis

I'm quite pleased with my results! I'm not sure if I can really give a less biased view. I normally find with programming that by the time I've finished a project, I'm already quite unhappy with the quality of the code, and I have a list of 'cleanup' TODOs, or even 'rewrite this large chunk of it' TODOs, which usually never get done. By the time a few years have passed, I'm downright ashamed. So far I don't feel this way about any of the code – let's see how long that lasts!

The quality of the HTML is pleasing - the Django validator app I wrote (development time not included) made creating the entire site using only XHTML 1.0 strict really very easy – a task that I used to think was quite a challenge. The only part that proved tricky was writing a bbcode parser that would accept anything the users can throw at it and always produce valid XHTML that matched what the user would expect to get.

In terms of visual design, I'm reasonably pleased – though there are quite a few places that could do with a designer's eye. And as for end user experience, I can't really say yet. I've tried to slim down the interface and make the pages a bit simpler than they were before, but some new features, especially 'tagging', have added more things back in. [Edit 2017 - the site has had several major visual revamps since this article was written].

In terms of making the website easy for other people to manage, Django's admin has solved pretty much all of that. For the main models that other people will have to administer (which are details of the camps we run, camp sites they run at and people who run them), it's astonishing how well the automatic admin functionality caters for it – it does a lot more than I would have managed to create if I were writing a custom admin interface manually.

I've also made the website self-maintaining as much as possible – for instance, every year each camp that has just finished gets a new forum, and this now creates itself on first access. The website also has a concept of the 'next year', which depends on when camps finish etc, and the 'clock' for this ticks over automatically.

For moderation, everything now has a feed, so it is very easy to aggregate message board posts for the entire website, or for an individual user, and be aware of new topics etc. I discovered some nice patterns while doing the Atom feed work – detailed below.

On a downside, I did very little test driven development. I came to the conclusion that Django's view functions are very difficult to test. They take complex objects and return complex objects, and their output is highly dependent on what is in the database, so you have to do a lot of set up first. The view functions themselves often do very little – in fact some were just generic views, so testing them would just have been testing Django. Some do quite a lot however, but what exactly they do will depend on the validity of input and data in the database etc. I realised that unit tests are pretty inappropriate, but functional tests, using a tool like twill, would be perfect.

Unfortunately, after installing and playing around with twill, I never got around to writing tests with it, partly because of the pain of having to write setup code. I know that other Djangoers have done good work here, but with time constraints this was the first thing to go. I did, however, write tests the parts of the system which could be decoupled easily from the view functions – in particular most of the bbcode parser was developed in a test driven manner, which worked very well. [Edit 2017 - I now used django-functest for functional tests]

Feeds

I didn't use Django's “high level” feed framework, as it didn't fit very well, but the lower level one was just perfect for what I wanted. Feeds are all available at URLs with ?format=atom appended to the normal page. To handle this, I've got mini-framework involving a handle_feed_request() function and base CCIWFeed class that inherits from feeds.Feed.

With these in place (25 lines of code), my class for generating the feed for new message board posts, for example, looks like this:

class PostFeed(CCIWFeed):
    template_name = 'posts'
    title = "CCIW message boards posts"

    ## This is called by CCIWFeed.items()
    def modify_query(self, query_set):
        return
        query_set.order_by('-posted_at')[:POST_FEED_MAX_ITEMS]

    def item_author_name(self, post):
        return post.posted_by_id

    def item_author_link(self, post):
        return add_domain(get_member_href(post.posted_by_id))

    def item_pubdate(self, post):
        return post.posted_at

(Plus there are two templates to support this).

However, you can also get a PostFeed for a specific member – i.e. all posts that were created by that member. The only thing that needs to change is the title, so the implementation is just the following:

def member_post_feed(member):
    """Returns a Feed class suitable for the posts
    of a specific member."""
    class MemberPostFeed(PostFeed):
        title = "CCIW - Posts by %s" % member.user_name
    return MemberPostFeed

The view code has to call member_post_feed() with a specific member, and passes the generated class to the feed handling code. It doesn't require a special view – it just requires two lines in the existing HTML view for a specific member's posts.

This pattern is repeated quite a number of times, and I think it is wonderfully elegant – it's so easy to see what it is supposed to do, and using a class in the same way you use closures is so expressive.

Conclusion

I've enjoyed this project, and I'm very pleased with the result, but I am also happy to get it over with, as it has been dragging on in my spare time since September. The launch of the new website has been a bit of a non-event. The amount of traffic on the site varies enormously – after the camps in the summer when everyone has just met up again, there is a massive surge in activity – last August and September the very small active user base managed to create up to 280 posts a day. But right now, there is almost no activity and it's been like that for months. I have to tell myself that it hasn't been a wasted effort :-). [Edit 2017 - the forums, a major focus of the site originally, have been killed off due to lack of use. However, the site has become hugely useful in other ways, gaining online bookings for campers and a raft of staff functions.]

If anyone wants to play with it, you can use the forum at http://www.cciw.co.uk/website/forum/8/ and log in using user name and password 'guest'. I'll probably clear out everything that 'guest' creates every week or so, so create whatever you like.

Finally, if anyone is interested in any of the code, I'd be happy to make it available. Most of it is probably quite specific to CCIW, and not very re- usable, but I made the tagging functionality very generic (I posted to django-devs about this already), and there may be other bits people would want to glean. [Edit 2017 - complete source code, including history, here: https://github.com/cciw-uk/cciw.co.uk/ Most of the 'forum' related code talked about here was removed in Nov 2015]