All Unkept

Bundling dependencies

2013-04-15T11:08:10Z

This post is about maintenance programming and the issue of Open Source dependencies that may need customising. It compiles some of my current thoughts, but I'm also eager to find out what other people do.

3 approaches to dependencies

Pure dependency

The source code of the dependency does not become a part of your project in any way. For a web project with Python and virtualenv/pip, you would just list the project name and version in requirements.txt, and it will be installed when you deploy your project.

This is by far the easiest approach to dependencies.
Forked dependency

You create a fork of the library (usually hosted publicly, but not necessarily) and add to it the changes you need. You then use this fork from your main project.

This is done either in the hope that bug fixes and feature additions that you make will be merged into the original, so that you won't have to maintain your fork forever, or with the aim of keeping your changes small enough that it will always be easy to merge in fixes from upstream.
Bundled dependency

You take a copy of the library, and include it directly into your own source code, so that it becomes a part of your source code, so that you can make whatever modifications you need. The code becomes a part of your source code forever.

This post is about number 3 — the bundled dependency.

(There are, of course, variants and mixtures of these — for example, Django has often bundled dependencies, but this was purely because of the confusing state of packaging, and the code was never modified for use in Django. These libraries have been or will be un-bundled as soon as possible.)

Avoid it if you possibly can

The first thing to say about bundling dependencies is that you should avoid doing so if at all possible:

It can result in large increases in code base.
You won't get critical fixes from upstream, and it can be hard to merge them in.

Bundling a dependency can be a drastic decision — you are taking on all the technical debt and maintenance burden of the code you are adding. Some developers look at Open Source libraries and think “wow, all this free source code I can just add to my project”. Your attitude needs to be exactly the opposite: “Wow, look at all that code I'm going to have to maintain and debug”.

An external dependency is often much worse from a maintenance point of view than code you have written yourself:

You may not understand the code very well at all, and you may not have access to the original reasons for the way it is.
When you add it to your project, you typically lose its history, making it harder to track down reasons for its current state.
Library code can often be over-generalised and complex. It copes with all kinds of situations that you don't need, but you will have to understand and maintain that complexity.
The code will not ‘fit’ into your project well — there may be all kinds of conventions and decisions that make it alien to your project, but now it is part of your project and needs to fit.

Alternatives

To avoid bundling a dependency, you can go for the ‘forked dependency’ above. For the missing features you need, attempt to add extension points that will give you the flexibility you need, rather than simply hard code something very specific to your project that will never get merged upstream.

Another alternative is to build what you need yourself, or very selectively add parts of the dependency into your own source. This may seem more work, but could be easier to maintain long term.

Finally, you could consider a monkey patch. But be very careful, and make sure you know all the places where you are doing that kind of thing, so that you can assess at what point you should be switching strategy.

When you should consider it

However, there are times when you should consider bundling the dependency:

When the changes you want to make are more than bug fixes.
When the changes can't be easily made by adding extension points to the original.
When the number/size of extensions is going to severely inhibit a developer's ability to understand the code.

I recently took on a project that had bundled a copy of Satchmo. It was a bit of a shock, because requirements.txt also listed Satchmo as a dependency, making me think I was in situation 1, when actually I was in situation 3, which is much worse.

Sometimes, however, it is unavoidable. e.g. you need multiple fields adding to DB different models, or you need to make invasive changes in other ways. As I looked at the number of modifications made to the bundled Satchmo, I realised there was no way that strategies 1 or 2 would be any good. Strategy 3 had already been chosen, it was impossible to turn back the clock, and with hindsight it was probably the right decision.

But implementation of that decision was lacking in lots of ways.

So how do you cope when you are forced to bundle? Here are my hints so far.

Recognise that you have done a really bad thing, and you need to take equally drastic action to cope with it. The bigger the dependency you've bundled, the more likely it is that you have seriously damaged your ability to maintain the project long term.
Make sure you include the tests of the original dependency, and integrate them as part of your test suite.

Sounds obvious, but in the project that inspired this post, the opposite had been done — they had copied all the source code, with the exception of every file called 'tests.py' or directory called 'tests'. I do not know what possessed them to do this, but this decision was an extremely expensive one for their client, and has caused massive damage to the project.
Maintain the test suite properly.

Again, sounds obvious, but tests are extremely valuable to a project, and in this situation it is vital that you keep them maintained.

It is acceptable to delete tests if they are checking requirements that you no longer have. But you should be deleting the code that supports those tests as well.
Take complete ownership of the code.

Having made the decision to bundle, don't treat the code like an external dependency. It is your code now, only you can fix it. Don't pretend you are going to merge in upstream changes.

The code should live at the same ‘level’ as the rest of your code — for example, it should be in the same directory, not off in some 'libs' directory that makes it harder to find. You need to embrace the fact that it is part of your maintenance burden.

On the other hand, it is your code now, you can do what you want with it. So don't be afraid of making changes. A tentative approach will leave you with the worst of both worlds — a library that doesn't really do what you want, but that you have to maintain. Make it do what you want.

Obviously, there can be some value in maintaining a separation between "your stuff" and the "framework stuff" or "library stuff", but this is just good coding practice — you wouldn't hard code something very specific into a function that is supposed to be generic.
Delete, delete, delete.

If there is code that you don't need, just delete it. The more code you can remove, the better. There can be a case for keeping some code around if:
1. It is causing very little nuisance to maintenance efforts.
2. It is fairly likely to be needed in the near future.
3. It is not causing runtime weaknesses (e.g. security problems), because there is no entrance point to the code.
But note that just the existence of code is a maintenance problem. If, for example, you need to change the signature of a function, you will do a search for sites that call it. Every hit you get is something you have to investigate, which takes time. If, in the process of this kind of investigation, you find some code that might be unused, find out if it is, and delete aggressively where appropriate.

And code that might be needed one day is better deleted. By the time you come to need it, it might be horribly broken, or broken in subtle ways that will take you longer to debug than to write, or too complex or badly performing for the context of your evolved application.

This applies to all kinds of code, including templates etc.
Clean aggressively.

If you delete unused code, you'll find that you may well end up with code that has essentially unused generality, or various other things that no longer make sense for your specific project.

This is my golden rule for maintenance:

Leave the code looking as if it had always been designed that way.

This is a general maintenance principle, but it is especially important for the situation where you are trying to go from a larger code base to a smaller one.

Ideally, there should never be artefacts that can only be explained by talking about the history of the project. This applies to every detail, including:
- names of models
- names of fields
- names of variables and functions
Altering models is not hard if you have a good database migration tool e.g. South for Django.

This principle may seem like it adds to the load of the maintenance programming, but long term it reduces the load, and reduces the likelihood that a project will collapse under its own weight. Even with this principle, projects tend to become unmaintainable — the natural tendency of a project is towards chaos, and you have to be very proactive about reversing that.

Example 1: after deleting some classes, you end up with a class hierarchy where each base class is only used once. This adds a lot of overhead when reading the code. You should clean aggressively — fold the classes together (unless keeping them separate increases the clarity of the code).

Example 2: The code I'm maintaining uses livesettings (and uses it far too much in my opinion, for things that ought to be in settings.py). It includes some options that are unlikely to change for a given project, or are likely to become ignored easily. For example, there is an "Only authenticated users can check out" setting. In a project with an overridden login form or login view (which can easily happen), it's very easy for this switch to become (at least partly) broken. When you are working on some code that branches on the value of this switch, there is no point fixing both branches — you won't have decent tests to ensure that the unused branch is really working.

Instead, find out what the current value is, and just delete the other branch. Then find all instances of the setting being used, and clean up similarly. Finally, delete the code that defines the switch in the first place. Remove every trace — you always have the history if you really need to see how something was done before.
Lather, rinse, repeat.

The aggressive process of deleting and cleaning leads to more, and you should follow this up. You may not have the time to do it right now, but you should be doing as you go — whenever some coding has turned up something that can be cleaned/deleted, first do the necessary commit for whatever you were working on. Then do a round of cleaning/deleting, finding all the code paths that are now dead or can be simplified, commit the change, and repeat,

These things have to go together. Aggressive deleting and cleaning can be made a lot easier if you have a good test suite. Of course, when deleting code, you will do a search for sites that might call it. But it ought to be possible to check if you can delete code simply by running the test suite with it absent.

What other approaches or hints do you have for dealing with this situation?

Never fix a bug twice

2012-06-19T13:42:56Z

Noah Sussman wrote a post on Things you should test, “A checklist of things that are worth testing in pretty much any software system.”

Many of the things on the list are helpful reminders. However, I think the mindset it encourages is essentially wrong.

The mindset is basically this:

Identify common mistakes that developers make, and ensure you are writing tests that check you haven't made them.

The problem with this approach is that it is essentially whack-a-mole debugging. There is a never ending supply of bugs to kill.

A much more helpful approach is found in this post on easy programming that advocates “Never fix a bug twice” (about one-third of the way down).

If you come across a bug or class of bugs that often occur, you should not be thinking first of all “better add that to my list of classes of bugs that need testing against”. You should rather be thinking “how can I change the system so that this class of bugs disappears entirely?”.

So, to take some of items listed for testing:

Input handling bugs, such SQL injection and XSS attacks.

In Django apps, I never write tests for SQL injection attacks or XSS attacks. The reason for this is that these types of bugs are very rare. The reason for that is that the APIs that are easiest to use for executing SQL or generating HTML are secure by default. If there are any errors of this type in my programs, they will be very rare and obscure, and trying to catch bugs by a black box scatter-gun approach will be a waste of time.

With XSS, there are times when I am slightly more tempted to generate HTML in ways that might be vulnerable to XSS — for example, using Python string interpolation. At this point, however, you still shouldn't reach for a test to ensure you did it correctly. Rather: stop writing HTML generation the stupid way!

To do that, you first have to ask “why?”. “Why am I tempted to write HTML this way?”. The answer is usually “there isn't a convenient API for the particular way I want to write this”. The correct solution now becomes obvious: create the API that removes the temptation to introduce potentially buggy code.

So, here is some code in a project I'm writing, that uses string interpolation to implement a template tag for formatting a link in a standard way:
```
from django.core.urlresolvers import reverse
from django.template import Library
from django.utils.html import escape
from django.utils.safestring import mark_safe

register = Library()

@register.filter
def account_link(account):
    return mark_safe(u'%s" title="%s %s">%s' % (
            escape(reverse('account_stats', args=(account.username,))),
            escape(account.first_name),
            escape(account.last_name),
            escape(account.username),
            ))
```
The problem with this is that I have to remember to use escape on each variable. The reason I wrote it this way is that Django's Template API is kind of bulky for this use case. So, I should instead have written this:
```
from django.core.urlresolvers import reverse
from django.template import Library

from somewhere import html_fragment

register = Library()

@register.filter
def account_link(account):
    return html_fragment(u'%s" title="%s %s">%s',
            reverse('account_stats', args=(account.username,)),
            account.first_name,
            account.last_name,
            account.username,
            )
```
And then write the API, html_fragment, that makes this work, which is just:
```
from django.utils.html import escape
from django.utils.safestring import mark_safe, conditional_escape

def html_fragment(template, *args):
    return mark_safe(template % tuple(map(conditional_escape, args)))
```
EDIT: After some discussion on django-devs and subsequent modification, this is now in Django 1.5 as 'django.utils.html.format_html', so use that instead of the above

I'm now no longer tempted to write it the vulnerable way, and I don't need a test (though I may want a test or two for html_fragment). I have two ways of doing it - html_fragment for very small snippets, and Django templates for bigger chunks, both of them secure by default.

So, if you find yourself needing tests for specific SQL injection or XSS attacks in your code, you are probably doing it wrong. Fix the underlying API that makes the mistake likely in the first place.
Input handling type checking - “Invalid values such as null and NaN. Strings instead of integers, arrays instead of strings.”

Input handling should generally only occur on trust boundaries, but it should occur systematically on such boundaries. If you have a problem with the wrong type of value being passed to an internal function, due to an external input being passed in, you shouldn't be dealing with it in the function, you should be asking “how was this even possible?”.

One solution of course is a static type system. Of course, that might not be possible/practical in your situation, but you should be thinking about it.
Math-related bugs: in many of the listed instances in Noah's post, the underlying bug is that the language does funny things.

So you should be asking “why am I using this language? Is it the right tool for the job?”.

Or at least “will a library solve this problem?”.

If you are handling decimal values, the solution is a library that does it well, and doesn't make it easy to make mistakes. For example, the Python 'decimal' library does not allow multiplication of floats and decimals - which sounds inconvenient, but stops you from making all kinds of mistakes.
Units of measurement.

If you have potential bugs with this, you should be thinking “why are these possible? How can I eliminate the bug entirely?”.

Different languages have different solutions, often involving including the unit of measurement in the value itself. Haskell has various solutions that make it impossible to add "3 years" to "2 meters", for example, and the compiler will stop you at compile time.

For Python, there are things like magnitude and units that work at run time.

Of course, even if you use these internally, you'll still have boundaries where you need to do some conversions from inputs etc., and at this point you might need to know what units the inputs are in. But:
1. You should try to insist that the external system sends the unit information along with the numerical value, so that you never have to rely on common knowledge, and you will know immediately if something changed. And your internal APIs have helped you do this by forcing the issue.
2. Even if the external systems can't be changed, you'll still have very few places to check - just your input and output boundaries, each of which should be managed in a single place. And writing tests probably won't help you very much — you should write at most one, as a single integration test. Your time will be better spent manually checking the boundaries.
Text - “Are Unicode inputs treated differently than ASCII?”

Again, you should be using proper internal datatypes — unicode everywhere — to eliminate potential bugs.

Python 2.x is notorious here - its ‘forgiving’ conversion between bytestrings and unicode causes endless bugs that are found in production, not in development.

And the correct solution is not just to write tests — which you may have to do for now — but rather to fix the underlying cause. This is what has been done in Python 3.

The same type of things applies to almost everything on the list, and in the cases where I can't see how it applies, I imagine that this is due to my own lack of imagination :-) If I can't think of a way of completely eliminating a class of bug, then I should think harder, rather than assume it is impossible.

Notice that from the perspective of an older programmer, there are some notable omissions from the list - things like buffer overflows, for example. Presumably that's because the author uses languages/frameworks where those bugs are very unlikely. And that has happened not because previous generations of programmers wrote lots of tests, but because they wrote languages and systems where those bugs are impossible or almost impossible.

So, in all this I'm not saying that you won't ever have to write tests for these kinds of bugs. But every time you identify a class of bugs, tests are a very poor answer. You should try to ensure that you never fix the same bug twice, and so never write the same test twice. If you are writing essentially the same test more than once, you are proving that you haven't fixed the underlying issue yet.

If a bug is worth adding to a list of common bugs, then you have identified a systemic problem with your platform, and it is worth eliminating entirely. We should be striving for libraries/languages/systems which do that.

A prayer to the programming gods

2011-09-19T12:52:12Z

O gods of software development and operations, I have sinned.

Your anger falls on me, and I feel your wrath.

The web site I have inherited has no unit tests.
It has no deployment script, and no README.
Or database migration tool.
It makes no use of virtualenv or requirements.txt or buildout,
    nor has any description of dependencies.
It has most of the VCS history missing.
Source dependencies are in random folders,
    clearly checked out from private SVN clones of
        proprietary and open source projects,
    but forked at unknown date with no history.

And I cry, “Why me?”

Have I not used a fabfile for projects I have started?
Have I not included a setup.py for my open source apps?
Have I not written helpful docs, or at least a README.rst?
Have I not written correct commit messages, with carefully
    constructed patches that didn't mix features and fixes?
Are the projects I hand on not covered by automated tests,
    at least for the critical functions?

But then I consider the sins of my youth,
    and I confess: You are just.

You could have given me the VBA project I wrote when I was 18.
    or some of the web apps I have written since.
It could have been the thousands of ASP.NET lines I cranked out in two short years,
    like the proverbial monkeys trying to produce Shakespeare.
It could be raw SQL in the frontend code,
    and HTML mixed with business logic.
You could have given me a PHP project.

But it is Python, and Django at that, and it is easily fixed.
Your chastisement is light indeed.

My Bash prompt

2011-02-28T23:32:42Z

I like my bash prompt:

to include the current Mercurial/Git branch if applicable — otherwise I would quickly get in big trouble with committing things to the wrong branch in Django etc.
to show the full current working directory - for similar reasons to above, as I often end up lost in a maze of twisty little directories, all alike.
to be always instantaneous (tricky, when 'hg branch' can take up to a second on a bad day due to startup time)
to work well with virtualenv (which adds the environment name to the beginning of the prompt)
to show the time (so that if I have a long running job that I didn't realise was going to be long running, all the info is there to find out how long it took).
to do all the above without confusing me!

All of these together mean I need two lines, but this has advantages - it means the commands I type always start at the same column. I achieve (1) by a bash function that looks for .hg or .git recursively, and looks for '.hg/branch' where applicable. I achieve (6) by using some colours distinguish the different bits.

The end product looks like this - first without a virtualenv, then with:

The code looks like this (in ~/.bashrc)

# Prints " $branchname" if in a hg or git repo, otherwise nothing.
print_branch_name() {
    if [ -z "$1" ]
    then
        curdir=`pwd`
    else
        curdir=$1
    fi
    if [ -d "$curdir/.hg" ]
    then
        echo -n " "
        if [ -f  "$curdir/.hg/branch" ]
        then
            cat "$curdir/.hg/branch"
        else
            echo "default"
        fi
        return 0
    elif [ -d "$curdir/.git" ]
    then
        echo -n " "
        git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\1/'
    fi
    # Recurse upwards
    if [ "$curdir" == '/' ]
    then
        return 1
    else
        print_branch_name `dirname "$curdir"`
    fi
}

e=\\\033
export PS1="\[$e[1;36m\][\u@\h \t]\[$e[1;33m\]\$(print_branch_name) \[$e[0m\]\w\n\[$e[1;37m\]——> \[$e[0m\]"

Docs or it doesn't exist

2011-01-10T16:43:18Z

tl;dr

Writing good quality documentation for the software libraries you publish always matters. Otherwise, you are doing the world a disservice by publishing.

Definitions

I realise I am talking mainly in the context of libraries published for public consumption, rather than in-house libraries. Some of the reasons given below don't apply so much for the in-house library that people may be forced to use.

By publishing, I specifically mean package repositories like PyPI, djangopackages, Hackage, CPAN — the places where developers will go looking for your library.

By ‘good quality documentation’ I do not mean ‘lots of documentation’. Often documentation can be ruined by thoughtless quantity. I remember some API docs produced by the NDoc documentation tool, which, when I last used it (admittedly about 5 years ago), flagged as warnings any methods or properties that were not documented. This led to developers jumping through the hoops, documenting the identity property of the Person class with such epiphany-inducing revelations as “Gets/sets the identity of the Person object”. The tool, however, doesn't flag in red the missing parts of the documentation that you really need, like how you would go about populating a Person object from the database or why you might want to use it etc., nor does it flag poor quality docs.

The result is documentation that is worse than useless — it promises usefulness by the fact of its existence, but instead wastes hours of your time, as you search it in vain to find anything which could not be deduced from looking at a class browser. Even if there is some documentation that is worth having, I will give up looking, and never find it, due to all the almost-auto-generated dross that surrounds it. Developers need to be persuaded of the importance of documentation, rather than simply instructed to eradicate build warnings.

Also, blog posts are not documentation. Blog posts are adverts. Some blog posts contain ‘tutorial style’ documentation. This is a valid type of documentation, but it belongs in one place, with your software, not on a blog that might disappear one day. If your documentation isn't part of your software package, I have no idea it exists. Googling for it is not a solution — docs found in blog posts are typically fragmented in content, scattered over several blogs, frustratingly incorrect because they are out of date, and always leave you wondering "is that it, or should I look somewhere else?"

Intro

I'm prompted to write this rant by the astonishing number of packages on PyPI and Hackage that have virtually no documentation or very poor documentation — not even a README sometimes. I presume that people put libraries on these repositories because they would like to think that they might be useful to other people. Often they like to think they are doing ‘Open Source’ development by publishing in this way.

However, I would argue that releasing software libraries without documentation is like dumping all your old junk on the street “because someone might want it”. You are not, in reality, being helpful or contributing to society — you are just littering. The same is true of libraries without documentation.

Arguments

A library is a risk

The problem with the person who dumps their junk on the street is that he is not considering his ‘contribution’ from the perspective of the people who might be looking for such things — or from the perspective of the people who might want to use the street.

People who are looking for old fridges etc. have places that they will go for that — junk yards, freecycle, ebay, etc. And people walking down streets have expectations of what they will find on those streets. Once you take those things into account, you won't leave your old fridge on the street if you actually care about other people.

Now, people looking for software libraries have a specific task that they need to achieve. They are looking for a library because they believe the task can be abstracted into a library somehow, and they don't want to have to write the code themselves. They then go to Google or a framework/programming language specific repository with keywords for that task.

Every result they find is an avenue that might need exploring. And everything they find which is not suitable is junk that is just getting in their way. The developer has to do 3 things:

Evaluate whether a certain package could possibly do what they need.
Install the package, possibly including some kind of configuration.
Write the code to get the package to do what they need.

And then, possibly, the developer may need to patch/fork your code to produce a version that does what they need.

Every step represents increasing commitment in terms of time. At each step, every second that I have to spend to find something out is time that I am spending, and therefore potentially wasting, because of your library. I will not know until I have actually finished the last step whether your library will help me — it could easily have some flaw that makes it useless for me.

So your library represents risk to me, especially as I always have alternatives — another library, or just writing the code myself. At each point, I've got to assess whether it is likely I will succeed with this package. I've made an investment in this library already, but will it reward continuing investment, or will it turn out to be another dead end? Should I get out now?

So, at the very least I need an overview that tells me what a library does. Without that, every second I spend looking at a library is probably time wasted. It seems ridiculous that someone would publish a library like this, where there is simply nothing to help you know how you are supposed to use it, but many instances exist (I picked that one at random), and, thinking that you must be missing something obvious, you actually spend time searching for any docs, before concluding that there is no documentation whatsoever, not even a single source code comment that might give you a clue.

I hate your package already, and wish it didn't exist, just like the fridge on the street. You have probably not helped anyone, and you have certainly hindered me.

But I need to know more than what is covered by the overview, since it is very unlikely that the default, simple case will cover all my needs. I also need to know what customisation or extension points there are. If none are documented, I can only presume that none exist.

Even if they do exist, I cannot know from reading the source whether they are essentially accidental or not, and this leads to another point:

Documentation is an API contract

With small libraries in particular, the author is not going to write a document explaining exactly how they see the code evolving. The only thing that people can rely on is what has been documented to work. Anything else may be something that is considered an implementation detail, and relying on it is increasing the risk all the time. Even with languages that clearly distinguish between public and private, often these distinctions are not fine-grained enough for the current purpose, and I still want some assurance from the author that something isn't public by accident.

And most people will not want to fork the code to get a version they know works — they want to know they will get bug fixes.

In addition, the existence of documentation tells you not only what the author is thinking about what is really public/private, it shows that the author actually cares about this library. It shows the kind of pride in your work that helps other people to know that this library is likely to get bug fixes.

Responses

It's open source, and so people can just read the source

If you only have the source, it can take a huge investment of reading and understanding before you can conclude whether the library is even attempting to solve the problem you need to solve. I have to fit all the code inside my head and mentally execute it — or I have to download it and try to use it. Both of these represent a huge investment.

My software is too small to be worth documenting

On the contrary, the smaller the software library, the more important it is to have good documentation.

If there is a relatively small task that I want to find a library for, I am not going to spend a huge amount of time researching the libraries, simply because it is not worth it. Any library will likely have all kinds of disadvantages compared to writing my own code — overhead for features I don't need, bugs in features I do need but the author obviously didn't, the need to add customisations (like subclassing, which serves to make things harder for a maintenance programmer) etc. So for small features, I'm very tempted to write my own solution anyway, since I know that it will fit my needs.

Therefore, I need to know even faster whether the library will do what I want it to do, or whether it can be easily extended to do so. I am much less likely to get as far as step 2 or 3 above, and I am more annoyed with every library that doesn't make step 1 easy — because the time wasted represents an even bigger fraction of my time allowance for this task.

So, if your software is too small to be worth documenting, it is too small to be wasting other people's time with it, and putting it in any public package repositories is doing the world a disservice.

Conclusion

If you don't have time to write documentation, don't bother publishing on some package repository — you are simply fooling yourself into thinking you might have done someone a favour, when all you have done is waste resources.

I think we need a new mantra for software libraries, especially in the Open Source world, that applies to entire projects or features of individuals libraries:

Docs or it doesn't exist.

Perhaps, in the context of public package repositories, we should go further:

Docs or delete.

UPDATE:

Of course, I should have also mentioned that with projects like Read The Docs and Sphinx there is even less excuse for creating great quality docs.

Class Based Views and Dry Ravioli

2010-11-20T04:29:53Z

Django's new class based views are very cool. I am starting to clean up an existing project using them, and lots of existing views are turning into declarative code. But it makes me worried about the ravioli effect.

Take my feedback form as an example. It is a simple view, and typical of code in my project — a couple of calls to some standard utilities. One of the them, standard_extra_context, adds some standard context variables depending on what it passed in (like title, description etc.) which are used in the HTML metadata and in the page itself. Another, json_validation_request allows a form view to be re-used for AJAX validation. The form declaration and view look like this:

class FeedbackForm(CciwFormMixin, forms.Form):
    email = forms.EmailField(label="Email address", max_length=320)
    name = forms.CharField(label="Name", max_length=200, required=False)
    message = forms.CharField(label="Message", widget=forms.Textarea)

def feedback(request):
    c = standard_extra_context(title=u"Contact us")

    if request.method == 'POST':
        form = FeedbackForm(request.POST)

        json = utils.json_validation_request(request, form)
        if json: return json

        if form.is_valid():
            send_feedback(form.cleaned_data['email'],
                          form.cleaned_data['name'],
                          form.cleaned_data['message'])
            c['message'] = u"Thank you, your message has been sent."
    else:
        form = FeedbackForm()

    c['form'] = form
    return shortcuts.render_to_response('cciw/feedback.html',
                context_instance=template.RequestContext(request, c))

It does one bad thing, namely it doesn't redirect on success, but apart from that it is fine. But there is this annoying boilerplate of flow control. So, very quickly, after creating some standard mixins and base classes that are equivalent to my standard utilities, I now have the following (which now does the redirect-on-success):

class FeedbackForm(CciwFormMixin, forms.Form):
    email = forms.EmailField(label="Email address", max_length=320)
    name = forms.CharField(label="Name", max_length=200, required=False)
    message = forms.CharField(label="Message", widget=forms.Textarea)

class FeedbackBase(DefaultMetaData):
    metadata_title = u"Contact us"

class FeedbackFormView(FeedbackBase, AjaxyFormView):
    form_class = FeedbackForm
    template_name = 'cciw/feedback.html'

    def get_success_url(self):
        return reverse('cciwmain.misc.feedback_done')

    def form_valid(self, form):
        send_feedback(form.cleaned_data['email'],
                      form.cleaned_data['name'],
                      form.cleaned_data['message'])
        return super(FeedbackFormView, self).form_valid(form)

class FeedbackDone(FeedbackBase, TemplateView):
    template_name = 'cciw/feedback_done.html'

feedback = FeedbackFormView.as_view()
feedback_done = FeedbackDone.as_view()

(DefaultMetaData is a TemplateResponseMixin subclass that replaces standard_extra_context, FeedbackBase defines common attributes for FeedbackFormView and FeedbackDone. AjaxyFormView is a FormView subclass and replaces utils.json_validation_request. The last two lines are simply there for the sake of not adding imports to my urls.py).

Now, this is a big improvement in one way. All standard flow control has been removed, so it is very DRY in that sense, and the details shown are the details which are relevant to the one task of handling a feedback form. The way to read this view is as declarative code, similar to how you read models and forms:

FeedbackFormView:

is a feedback view
- so it has the title 'Contact us'
  - and other default metadata
is a form processing view (which has been ajaxified)
has FeedbackForm as the form
has "cciw/feedback.html" as the template.
has the 'success url' equal to the 'feedback_done' URL.
does the following with a valid form:
- sends a feedback message.

Only the last of these is really imperative, and this is quite a nice way to think about a chunk of code.

However, suppose we have a maintenance programmer who needs, let's say, to make the template display a certain message on the basis of a certain URL parameter.

In the first case, it is pretty obvious what to do — a couple of lines inserted before the render_to_response call will work fine, with a corresponding change to the template.

In the second, it is also fairly easy to do — you would just have to define an appropriate get_context_data() method. But to find that out, you've got a fairly intimidating stack of mixins and base classes to investigate. In fact, it's more like a tree not a stack (though I think I may already have some diamond inheritance in there by accident).

This is classic ravioli code — “thousands of little classes everywhere and no one knows how to find the places where things really happen”. As one person on that c2 page says, “this style maximizes maintainability by old hands at the expense of comprehensibility by newcomer”.

Put it this way: suppose you want to go from the current version to the previous version, in a precise and controlled way. You could trace through all the functions that would be called, starting with as_view() and working through till you collected all the code paths that were being used, but it would take you a lot longer than the forward transformation just took me. And this is exactly the transformation you might need to do if, for example, you needed to make a fundamental change to the flow control while keeping most of the code intact. It is faster to start from scratch with the declarative reading above — but that will only work if the view really is declarative, and there are no sneaky surprises in the base classes, and even then it may take some digging.

So, what am I saying? Don't do this? I guess I'm just saying “be aware of this trade-off”. As Shannon Behrens recently blogged, there are times when fixing DRY violations are not worth it.

Mercurial tree visualisation

2010-02-12T15:51:23Z

I discovered hgview, and wondered whether it could make better sense of my Django repository than hgk.

Which of these two would you rather look at?

hgk:

hgview:

Eclipse and Mercurial with existing checkout

2010-02-12T14:53:00Z

One very nice thing about Emacs — that you don't appreciate until you try alternatives — is that it is very unobtrusive. It doesn't try to take control of any of your projects. It takes your code as it finds it, and works from there.

I'm trying to get MercurialEclipse or HgEclipse to work with my existing project. In switching to Eclipse, it was actually relatively easy to persuade it that my code can stay where it is on my disk, and doesn't need to live inside the 'workspace' folder — you can use symlinks or the builtin 'linked resources' feature.

But now I want to persuade it to recognise one of these source folders as a Mercurial checkout. It seems the only way to do this is to use the "import" feature, or "share" the project - both of which do the wrong thing.

My setup is that inside a workspace folder (call it workspace1), I have a project defined (e.g. 'Django'). This project has a .project file with a 'linkedResources' to include the actual source, which is elsewhere:

devel/
 django/
   hg/
     trunk/
 workspaces/
   workspace1/
     Django/
       src/ (defined in .project as a link to devel/django/hg/trunk)
     python2.5

I simply want the Mercurial plugin to recognise that 'src' folder as a Mercurial checkout, which it is already.

All attempts at reverse engineering settings files etc. have failed. Has anyone achieved this?

It would certainly make IDEs much more attractive if they didn't insist on trying to manage everything for you. I'm a big boy, I don't want them to hold my hand as if this was the first time I had used Mercurial or worked on a project. And I'm certainly not going to move my code about on the disk to satisfy some IDE - I might change my mind and use something else next month, or I might want to use more than one IDE on the same project. IDE metadata needs to be completely separate from my source code.

Python and copyright

2009-07-15T22:04:42Z

There has been lots of interesting discussion about dynamic languages and the GPL, but much of it seems to miss the main point: with many dynamic languages, the terms of the GPL are an irrelevance.

In the comments on the above page, Peter Gerdes seems to be one of the few who does get this:

@Morty Yidel

It doesn't matter what the GPL says in situations 2 & 3. What matters here is copyright law.

The GPL is a license that lets you create and distribute copies of certain software. You are only bound by the terms of that license when you need the license as a matter of copyright law.

In 2 & 3 you aren't distributing any GPLed software thus you need only abide by the GPL if it is a derivitive work of a GPLed program. As I said that's a complex question but it's one that can only be answered by looking at copyright law and precedent not the GPL. Moreover, if merely using an API makes your code a derivitive work of the library servicing the API it's hard to see how programs like WINE aren't infringing.

If someone wants to apply the terms of the GPL, they first have to demonstrate something that would otherwise be copyright infringement. As the GPL (v2) says:

5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it.

So, if I sit down at my computer and type (for the sake of argument):

echo 'import qt' > test.py

and then distribute test.py, have I created a derivative work of Qt (or any other GPL library)? Since I could type all of that without having a drop of Qt source code on my computer, I'd argue it's pretty difficult to say that this is a derivative work. Of course, once I go further and actually start using more the API than just the name 'qt', there is the chance that someone important might decide that by virtue of my use of an API, the program constitutes a derivative work of the library. (If they did, that would have huge implications — every Linux program, not to mention Linux itself, would become a derivative of Unix itself, for example, because it uses Unix API's, and every Win32 program would be a Windows derivative...)

This, of course, is completely different from C/C++, where in order to compile an executable, the compiler #includes parts of the actual source code of the library into the executable, thereby automatically triggering the need for an exemption from copyright laws.

With Python and Javascript, nothing of the sort occurs for the user to get the developer's software. If the developer decides to test their program before distributing, they may well need copies of the library code, either in source or object form, on their machine, but that has nothing to do with their distribution of their own code.

Now, the GPL (or whatever), may have terms explicitly forbidding you from finding some sneaky way around the licence, such as dynamic languages or web interfaces etc. But it doesn't matter. The licence could require you to sacrifice your first-born child to the gods of Jupiter for even thinking of doing such a thing, but the licence is irrelevant, since you haven't done anything that would require you to accept its terms.

So, unless judges are going to rule that use of APIs constitutes copyright infringement, all software licenses of this kind are simply irrelevant in this case.

This is one of the things that makes me shy away from the GPL. I'm a really grateful beneficiary of all that it has helped to safeguard, but as it relies on copyright law which may well be completely irrelevant for dynamic languages, it seems somewhat unwise to lean on it for any of my own financial security. More and more it seems that being paid for writing code, and not for the code itself, is the way programmers should make their living. Just because copyright law may have supported certain practices in the past doesn't mean we have a right to that continuing.

Managing Django patches using Mercurial

2008-07-31T21:51:37Z

Update: this is probably superseded by: http://code.djangoproject.com/wiki/MercurialBranches.

For any slightly longlived Django patch, for instance if I want to work on something in stages before committing, or if it might never be committed to trunk, here is the pattern I started using today:

Setup §

Do this once:

In a subversion checkout:

$ hg init
$ cat > .hgignore
\.svn/
\.hgignore$
\.pyc
~$
^D

(^D == Control-D, EOF)

$ hg ci -Am "Initializing from subversion checkout"
$ hg qinit -c

NB: make sure you do 'hg qinit -c'. Your patches live in .hg/patches, and this command makes that directory into a Mercurial repository, so all your changes to the patches are versioned. You have to remember to do hg qcommit after hg qrefresh.

Usage §

Commit all work to mercurial queues, not to mercurial directly.

New patch:

$ hg qnew name_of_patch

(Hack away, using hg diff etc. to see changes)

Commit changes to the current patch:

$ hg qrefresh && hg qcommit

Pull in new subversion changes:

$ hg qpop -a
$ svn update
$ hg addremove -s 75
$ hg commit -m upstream
$ hg qpush -a

(Of course, you could put that lot in a little script)

Once done, commit to subversion, and delete patches:

$ svn commit
$ hg qdel -r qbase:qtip
$ hg qcommit -m "Pushed patches upstream"

(Your patches are still there, of course, in the mercurial repository in .hg/patches, should you need the history)

Evaluation §

I'm not yet sure if this is the best way to do this (see the other methods on the Mercurial Working with Subversion page), but it is a lot better than Subversion on its own, and it isn't too complicated.

One advantage of the system is that it is very easy to ignore it or use it as required. Most of my Django work I will just hack directly and commit. If I realise part way through that it is going to take I while, or that I'm doing two sets of changes that I want to split up, hg qrecord is just the ticket -- interactively record changes to a new patch. If I then need to work on something else, I can just do hg qnew, doing hg qpop first if necessary/desired.

It is quite nice that the patches are just simple files in .hg/patches -- you can just go and look at them with normal tools etc. If your patch has got messy (i.e. accidental whitespace changes), you can just hg qpop it, fix it up manually, hg qcommit and hg qpush it again.

One of the downsides is that patches are managed as a stack -- if you want to add a new patch, you have to put in somewhere in the series of patches (not necessarily at the 'top' i.e. most recent end, but you do need to think about where to insert it). I imagine things can get tricky if you put it in the wrong order initially, though you could always just delete it and put it somewhere else. As long as you have versioned everything, you can get at it easily.

On the other hand, given that your patches directory is a full hg repository, I imagine you can actually do very powerful things -- you can branch your patches etc. very easily. I've discovered recently that Mercurial branching is actually very powerful -- you don't have to create clones, you can do everything inside one working directory.

Anyway, hope that is useful to someone else.