This post is about maintenance programming and the issue of Open Source dependencies that may need customising. It compiles some of my current thoughts, but I'm also eager to find out what other people do.
Pure dependency
The source code of the dependency does not become a part of your project in any way. For a web project with Python and virtualenv/pip, you would just list the project name and version in requirements.txt, and it will be installed when you deploy your project.
This is by far the easiest approach to dependencies.
Forked dependency
You create a fork of the library (usually hosted publicly, but not necessarily) and add to it the changes you need. You then use this fork from your main project.
This is done either in the hope that bug fixes and feature additions that you make will be merged into the original, so that you won't have to maintain your fork forever, or with the aim of keeping your changes small enough that it will always be easy to merge in fixes from upstream.
Bundled dependency
You take a copy of the library, and include it directly into your own source code, so that it becomes a part of your source code, so that you can make whatever modifications you need. The code becomes a part of your source code forever.
This post is about number 3 — the bundled dependency.
(There are, of course, variants and mixtures of these — for example, Django has often bundled dependencies, but this was purely because of the confusing state of packaging, and the code was never modified for use in Django. These libraries have been or will be un-bundled as soon as possible.)
The first thing to say about bundling dependencies is that you should avoid doing so if at all possible:
Bundling a dependency can be a drastic decision — you are taking on all the technical debt and maintenance burden of the code you are adding. Some developers look at Open Source libraries and think “wow, all this free source code I can just add to my project”. Your attitude needs to be exactly the opposite: “Wow, look at all that code I'm going to have to maintain and debug”.
An external dependency is often much worse from a maintenance point of view than code you have written yourself:
To avoid bundling a dependency, you can go for the ‘forked dependency’ above. For the missing features you need, attempt to add extension points that will give you the flexibility you need, rather than simply hard code something very specific to your project that will never get merged upstream.
Another alternative is to build what you need yourself, or very selectively add parts of the dependency into your own source. This may seem more work, but could be easier to maintain long term.
Finally, you could consider a monkey patch. But be very careful, and make sure you know all the places where you are doing that kind of thing, so that you can assess at what point you should be switching strategy.
However, there are times when you should consider bundling the dependency:
I recently took on a project that had bundled a copy of Satchmo. It was a bit of a shock, because requirements.txt also listed Satchmo as a dependency, making me think I was in situation 1, when actually I was in situation 3, which is much worse.
Sometimes, however, it is unavoidable. e.g. you need multiple fields adding to DB different models, or you need to make invasive changes in other ways. As I looked at the number of modifications made to the bundled Satchmo, I realised there was no way that strategies 1 or 2 would be any good. Strategy 3 had already been chosen, it was impossible to turn back the clock, and with hindsight it was probably the right decision.
But implementation of that decision was lacking in lots of ways.
So how do you cope when you are forced to bundle? Here are my hints so far.
Recognise that you have done a really bad thing, and you need to take equally drastic action to cope with it. The bigger the dependency you've bundled, the more likely it is that you have seriously damaged your ability to maintain the project long term.
Make sure you include the tests of the original dependency, and integrate them as part of your test suite.
Sounds obvious, but in the project that inspired this post, the opposite had been done — they had copied all the source code, with the exception of every file called 'tests.py' or directory called 'tests'. I do not know what possessed them to do this, but this decision was an extremely expensive one for their client, and has caused massive damage to the project.
Maintain the test suite properly.
Again, sounds obvious, but tests are extremely valuable to a project, and in this situation it is vital that you keep them maintained.
It is acceptable to delete tests if they are checking requirements that you no longer have. But you should be deleting the code that supports those tests as well.
Take complete ownership of the code.
Having made the decision to bundle, don't treat the code like an external dependency. It is your code now, only you can fix it. Don't pretend you are going to merge in upstream changes.
The code should live at the same ‘level’ as the rest of your code — for example, it should be in the same directory, not off in some 'libs' directory that makes it harder to find. You need to embrace the fact that it is part of your maintenance burden.
On the other hand, it is your code now, you can do what you want with it. So don't be afraid of making changes. A tentative approach will leave you with the worst of both worlds — a library that doesn't really do what you want, but that you have to maintain. Make it do what you want.
Obviously, there can be some value in maintaining a separation between "your stuff" and the "framework stuff" or "library stuff", but this is just good coding practice — you wouldn't hard code something very specific into a function that is supposed to be generic.
Delete, delete, delete.
If there is code that you don't need, just delete it. The more code you can remove, the better. There can be a case for keeping some code around if:
But note that just the existence of code is a maintenance problem. If, for example, you need to change the signature of a function, you will do a search for sites that call it. Every hit you get is something you have to investigate, which takes time. If, in the process of this kind of investigation, you find some code that might be unused, find out if it is, and delete aggressively where appropriate.
And code that might be needed one day is better deleted. By the time you come to need it, it might be horribly broken, or broken in subtle ways that will take you longer to debug than to write, or too complex or badly performing for the context of your evolved application.
This applies to all kinds of code, including templates etc.
Clean aggressively.
If you delete unused code, you'll find that you may well end up with code that has essentially unused generality, or various other things that no longer make sense for your specific project.
This is my golden rule for maintenance:
Leave the code looking as if it had always been designed that way.
This is a general maintenance principle, but it is especially important for the situation where you are trying to go from a larger code base to a smaller one.
Ideally, there should never be artefacts that can only be explained by talking about the history of the project. This applies to every detail, including:
Altering models is not hard if you have a good database migration tool e.g. South for Django.
This principle may seem like it adds to the load of the maintenance programming, but long term it reduces the load, and reduces the likelihood that a project will collapse under its own weight. Even with this principle, projects tend to become unmaintainable — the natural tendency of a project is towards chaos, and you have to be very proactive about reversing that.
Example 1: after deleting some classes, you end up with a class hierarchy where each base class is only used once. This adds a lot of overhead when reading the code. You should clean aggressively — fold the classes together (unless keeping them separate increases the clarity of the code).
Example 2: The code I'm maintaining uses livesettings (and uses it far too much in my opinion, for things that ought to be in settings.py). It includes some options that are unlikely to change for a given project, or are likely to become ignored easily. For example, there is an "Only authenticated users can check out" setting. In a project with an overridden login form or login view (which can easily happen), it's very easy for this switch to become (at least partly) broken. When you are working on some code that branches on the value of this switch, there is no point fixing both branches — you won't have decent tests to ensure that the unused branch is really working.
Instead, find out what the current value is, and just delete the other branch. Then find all instances of the setting being used, and clean up similarly. Finally, delete the code that defines the switch in the first place. Remove every trace — you always have the history if you really need to see how something was done before.
Lather, rinse, repeat.
The aggressive process of deleting and cleaning leads to more, and you should follow this up. You may not have the time to do it right now, but you should be doing as you go — whenever some coding has turned up something that can be cleaned/deleted, first do the necessary commit for whatever you were working on. Then do a round of cleaning/deleting, finding all the code paths that are now dead or can be simplified, commit the change, and repeat,
These things have to go together. Aggressive deleting and cleaning can be made a lot easier if you have a good test suite. Of course, when deleting code, you will do a search for sites that might call it. But it ought to be possible to check if you can delete code simply by running the test suite with it absent.
What other approaches or hints do you have for dealing with this situation?
This blog post is about deploying Django apps. It is addressed primarily to WebFaction, but applies to any other hosts else with one-click installers for this kind of app.
First, many thanks to WebFaction for their great service and support. I've got very few complaints about them in general.
However, I think their approach to setting up Django projects is far from ideal, and makes life harder than it needs to be.
The WebFaction Django docs tell us to create a new Django app by using the control panel and choosing an app of type ‘Django’ and then choosing your Django version. This will install an Apache instance with mod_wsgi, and a copy of the Django sources, and set up a basic Django app skeleton.
There are lots of problems with this:
What happens when there is a security issue with Django?
The user will need to have an upgrade mechanism. This is covered in the docs, but it is an 11 step process.
And this process is a process they will first have to do in their development environment for testing, in some way, and then again on their live box. How many people will actually do that, especially for security releases that they don't feel they need?
Experienced developers, of course, don't do this. For Django, I've used virtualenv and pip, and removed the existing copy of Django that WebFaction put there. Of course, I needed to use virtualenv and pip anyway, and so will 99% of all Django users. So my upgrade procedure for Django, like for any dependency, looks like this:
Note that this includes local upgrade as well as remote. It is much, much easier and much less error prone than the process WebFaction describes.
It uses an Apache instance.
Apache is a massive overkill for this kind of thing. Just figuring out the mod_wsgi settings for threads and processes nearly made my head explode, and I'm a reasonably experienced web developer.
Most experienced Django developers are not recommending Apache. They use gunicorn or uWSGI or similar.
And what happens if there is a security issue with the Apache version installed? Would users get any notifcation of this? How would they upgrade Apache? I have no idea—I know how to compile Apache from source, but I've no idea what compilation settings WebFaction uses etc. So far the only solution I've come up with to this worry is to migrate my projects to gunicorn.
It goes against the grain of how people will want to upload their apps.
New Django developers will follow the tutorial, or something similar like a django-cms setup guide, so that when they first come to deploy their app they will have a working app, with settings file etc. that they just want to deploy.
Experienced Django developers will not be following a tutorial, but will have a very similar set of files that they want to upload.
They will not start with a settings file or project layout generated by WebFaction. That is just a confusion that gets in the way.
Rather, they will want to customise their settings.py file, or create a new one for deployment, and insert into it the database settings, and maybe a few other tweaks.
Then they will simply want to upload it, including a requirements.txt file, and have it run.
As a contrast to the process WebFaction have documented, here is my process for deploying a new app on WebFaction:
(I've actually got a script that automates this, but that's another matter).
To upgrade Django or any other dependency, including gunicorn, I do:
For initial setup, this is easier than the process WebFaction have documented. For upgrades, it is much easier. As an experienced developer, I'm happy writing this myself, and I've got some template fabfile.py files I can use for new projects.
But newbies get the much harder processes, and they are still left with big hurdles when it comes to:
The ‘one-click installers’ WebFaction features sound great in theory, but in practice they are doing their users a disservice. Please give us, and especially newbies, a method that we will want to use long term.
Designing such a method is hard, and will probably involve a set of instructions rather than a ‘one-click’ experience, especially when different Django developers do things different ways. But newbies often don't have opinions like experienced developers do, and, providing the process allows customisation, experienced developers will be able to customise it. IMO, the default process should leave newbies with a setup that encourages the following best practices and de-facto standards:
I have a starter fabfile.py I used a while back, and have tried to update according to the way I now use gunicorn. It isn't properly tested in its own right, but it could be a useful starting point.
Perhaps one approach is to have a public repository (mirrored in different popular VCS systems), which includes some basic files (requirements.txt, fabfile.py, settings_webfaction.py, wsgi.py) that can be merged into a developer's project sources. Picky developers can override what they want, but newbies will have something that works well.
With that kind of process, I'd feel happy to recommend Django newbies to use WebFaction. At the moment, I could only recommend experienced developers to use it for Django deployments, and tell them to ignore WebFaction's own docs about deployment.
]]>I just watched Jacob's talk on "Porting Django apps to Python 3", and realised it was time to tackle my own small Django apps.
The problem with porting is that you need your dependencies to be ported first. But now that Django has Python 3 support, the finger is no longer pointing at Django — it is pointing at all of us with Django apps that have only Django as a dependency (or other dependencies that are already ported to Python 3). As Jacob put it at the end of the talk, it’s your turn.
So, I took the challenge, and here is a walk through of what you need to do, and what I had to do:
Find an app/library you've written that has very few dependencies, or all dependencies already ported to Python 3. In my case, django-easyfilters. Not a massively popular library, but it has had over 1000 downloads, and I know some people use it.
Install tox and create a tox.ini file to run your test suite on more than one version. Start with all the Django versions you want to support, with Python 2.x combinations (Python 2.6 and 2.7 recommended), and Python 3.3.
My tox.ini file looks like this.
Run tox and watch it pass with Python 2.x.
Of course, if you get failures at this point, fix them first. Because I am an exceptionally good boy, I got no failures, even for Django 1.5 which I had not tried before with this library. This is my reward for having Done Things Right (with this particular app :-). As well as a pretty complete test suite, I even created a small demo app, and left instructions about how to use it. I may be the only person to have ever read these instructions, but it was well worth it.
Watch the tests fail with Python 3.3
OK, now to start fixing it.
Activate the Python 3.3 virtualenv that tox created. In my case:
. .tox/py33-django15/bin/activate
Then run your test command. In my case:
./manage.py test django_easyfilters
You may need to do some work even to get it to run at all.
On the first run, out of 45 tests only 7 passed. Gulp.
Now go read a porting guide, using the 'single source' method, and check out Armin’s Python 3 porting redux.
You'll probably want to install 'six' and add it to your project's dependencies (unless you are targetting only Django 1.4 and later, in which case you can use django.utils.six). And add that dependency to your setup.py file, and your tox.ini file, and set tox running, because it will need to rebuild all your virtualenvs if six wasn't a dependency before.
Now iterate on fixing your tests. Each time you find a problem, grep the code base for other instances of it.
I found a relatively small bunch of problems which caused most of my tests to fail:
Use of implicit relative imports
Use of Decimal._rescale which is removed in Python 3.3
map() had to be replaced with list(map()) a few times
I needed from six.moves import xrange to use xrange
s/unicode/six.text_type/ and similar
I copied python_2_unicode_compatible from Django 1.5 for fixing __unicode__ and __str__:
def python_2_unicode_compatible(klass): """ A decorator that defines __unicode__ and __str__ methods under Python 2. Under Python 3 it does nothing. To support Python 2 and 3 with a single code base, define a __str__ method returning text and apply this decorator to the class. """ if not six.PY3: klass.__unicode__ = klass.__str__ klass.__str__ = lambda self: self.__unicode__().encode('utf-8') return klass
__cmp__ no longer supported.
This was a bit of pain to fix. For some of my classes __cmp__ was a much more natural way to define sorting, and it broke my head trying to rewrite. So I did this:
if six.PY3: # Support for __cmp__ implementation below def cmp(a, b): return (a > b) - (a < b) from functools import total_ordering else: total_ordering = lambda c: c # no-op
And in the classes, just added this:
@total_ordering class MyClass(object): # ... def __eq__(self, other): return self.__cmp__(other) == 0 def __lt__(self, other): return self.__cmp__(other) < 0
You could even write a decorator to do all of this.
When I migrate fully away from Python 2.x I may get round to rewriting these.
You also need to check your setup.py file. If you use setuptools, it should work fine - under Python 3 this is supported by 'distribute', a setuptools fork that is a drop-in replacement. Tools like pip and virtualenv install distribute for Python 3 environments. But you do need to check you don't have syntax errors under Python 3.
Finally, add the Python 3.3 trove category to your setup.py:
"Programming Language :: Python :: 3.3",
Overrall, this took me about 2.5 hours to complete. However, the app is small, and has a pretty good test suite, which makes things much, much easier. On the other hand, I've given you a clear plan of attack, which is a big part of the battle.
Now it's your turn
]]>Many programs build up sentences using bits - often a template into which different things might be substituted. However, the things you substitute into a sentence can change the sentence, and vice-versa, in ways that are not anticipated by the programmer.
For example, plurals. In English, you might try code like this:
if n == 1: return "I have 1 pig" else: return "I have %s pigs" % n
Localising these strings gives problems, because the rules for how to create plural forms is different in every language.
This specific problem is generally considered 'solved' by the use of gettext, but many more exist.
For example, we have another problem as soon as we start substituting nouns:
"Delete selected %s?" % object_name
Various attributes about the noun could affect the sentence. In French, the adjective "selected" needs to agree in gender with the noun being substituted in. So you cannot lookup the translations for "Delete selected %s" and for object_name separately. (This is a real example picked from Django source code).
Further, depending on how the sentence uses the noun, the form of the noun might need to change. For example, the noun might appear in the accusative position for a given sentence and language, which requires a different form of the noun to be used compared to the nominative form.
Several other examples of this appeared in Django ticket 11688. One proposed solution on that ticket would require a huge amount of knowledge and effort on the part of Django programmers, and almost certainly would not work anyway.
This post is an attempt to come up with a better solution, or at least kick start discussion. I haven't been able to find any solutions to this problem online, and most people seem to be just using gettext, which is a 95% solution — and maybe that is good enough for most people.
[Update 2013-02-19 - ‘Richard’ pointed me to Locale::Maketext article, which has in essence a similar approach to what I've done here]
We will assume that a sentence is a composable unit of meaning, such that sentences can be translated independently. So, if in language A we have sentences 1 and 2, in that order, we can translate these into language B by translating sentence 1 and sentence 2 independently, and putting them together in the same order.
This is, no doubt a simplification. In some languages, the two sentences might make more sense if re-ordered, or combined, or split in various ways. Indeed, some languages may not have a truly equivalent concept of 'sentence' at all.
However, we have to do something, and this is a reasonable approximation.
We need a powerful way of defining sentences in a given human language. It must be powerful enough that the person doing the translation can do anything they need, without the programmer needing to be aware of all the things in the language that will cause difficulty.
So, we'll start with a full programming language, and chop out the things we shouldn't need.
We shouldn't need side effects - translation should be a pure function. So we'll use a purely functional programming language without side effects.
We need something fairly readable, because translators are going to have to use it. It should be as close as possible to declarative in style.
Pattern matching seems like a great fit for some of our needs.
Given the above requirements, let's start with a Haskell-like pure functional language, whose pattern matching will be extremely helpful. It will obviously have IO removed, and no type signatures (but that won't stop us inferring them and being able to statically type-check the code). Everything else will be borrowed directly from Haskell, so that I can avoid having to make up my own syntax and semantics.
If the concept works, we can argue about better or simpler syntax for some constructs, or helper functions that aren't part of the Haskell prelude.
Hopefully, we will find a relatively small subset of Haskell that is needed to give us all the power we need to solve this problem - a subset small enough that we could guarantee non-termination ideally, to avoid problems with translations created by malicious agents.
This will be an example based exploration.
Let's assume that every sentence can be generated by a function. The function will take as parameters all the substutions that are needed, and return the translated string.
So, suppose we have the English sentence "I have some pigs". For every different language we need, we would have a translation file which contains the function iHaveSomePigs, which in this case takes zero parameters. So for French:
iHaveSomePigs = "J'ai des cochons"
(The mapping between the English sentence "I have some pigs" and the function name iHaveSomePigs hasn't been defined, and we'll skate over that detail for now).
If we have a variable number of pigs, for French we might have:
iHaveNPigs 0 = "Je n'ai pas de cochon" iHaveNPigs 1 = "J'ai un cochon" iHaveNPigs n = "J'ai " ++ show n ++ " cochons"
For English we could do this:
iHaveNPigs 1 = "I have 1 pig" iHaveNPigs n = "I have " ++ show n ++ " pigs"
(For those unfamiliar with Haskell, the way that pattern matching works is that the first definition that matches the arguments is used. Since n is not a literal, but a variable, it can match any argument.)
We can cope with more complicated rules, such as those used in Polish, perhaps something like this:
iHaveNFiles n = "Mam " ++ show n ++ " " ++ pluralize n "file" plurals "file" = [ "plik" , "pliki" , "plików" ] pluralize n word = plurals word !! pluralForm n pluralForm n | n == 1 = 0 | n `mod` 10 >= 2 && n `mod` 10 <= 4 && (n `mod` 100 < 10 || n `mod` 100 >= 20) = 1 | otherwise = 2
Note that the complex logix in pluralForm and pluralize only has to be defined once. Adding more words simply requires additional plurals lines. It's not the nicest syntax, but could probably be improved, and it's pretty easy to copy.
Let's add in gender, using the sentences "Delete this %s?" (singular) and "Delete selected %s?" (plural). We can use guards:
deleteThisThing thing | isMasculine thing = "Supprimer ce " ++ singular thing ++ "?" | otherwise = "Supprimer cette " ++ singular thing ++ "?" -- (Ignoring the problem with 'ce' followed by vowel for now...) deleteSelectedThings thing | isMasculine thing = "Supprimer les " ++ plural thing ++ " sélectionnés" | otherwise = "Supprimer les " ++ plural thing ++ " sélectionnées" isMasculine thing = elem thing [ "pig" , "man" -- anything else masculine ] singular thing = pluralForm 1 thing plural thing = pluralForm 2 thing pluralForm 1 "pig" = "cochon" pluralForm 2 "pig" = "cochons" pluralForm 1 "man" = "homme" pluralForm 2 "man" = "hommes"
Note that the only thing required by this system is that the functions deleteThisThing and deleteSelectedThings exist. Everything else is at the freedom of the translator, and better ways of defining any of these functions are possible.
Of course, it isn't expected that a translator would be able to produce this by himself/herself. However, once the basic logic has been set up, this syntax is readable enough that a translator could easily add more of the same. Lines like:
pluralForm 1 "pig" = "cochon"
are actually pretty readable. The lack of parentheses in Haskell function calls is also a bonus (though, as I said earlier, exact syntax could be debated). This is not really that much harder than editing a .po file if you are just wanting to add more of the same.
Also, we've got flexibility. If we really don't care about getting the gender right, we can just do "sélectioné(e)s" and be done with it.
Let's make it harder - we'll add case. I'll use NT Greek as an example, because it has nouns that decline with case (and I don't know any similar modern languages well enough). I'm going to introduce an enum for the different cases, using data for now, and for the different genders. I could also do the same for number ("Singular" and "Plural"), but just using 1 and 2 seems easier.
Our sentence will be "You like the %s.". For this in Greek, we need to choose the accusative singular form of the thing we pass in. We also need to pick the word for "the" (the definite article) which matches the gender and number of the noun, and it has to match the accusative case too. So, if we pass in a masculine word, we need the singular accusative masculine definite article (having fun yet?):
data Case = Nominative | Accusative | Genitive | Dative data Gender = Masculine | Feminine | Neuter youLikeTheThing thing = "φιλεις " ++ definiteArticle 1 Accusative (genderOf thing) ++ " " ++ accusativeSingular thing ++ "." accusativeSingular thing = nounForm 1 Accusative thing nounForm 1 Nominative "book" = "βιβλιον" nounForm 1 Accusative "book" = "βιβλιον" nounForm 1 Genitive "book" = "βιβλιου" nounForm 1 Dative "book" = "βιβλιω" nounForm 2 Nominative "book" = "βιβλια" nounForm 2 Accusative "book" = "βιβλια" nounForm 2 Genitive "book" = "βιβλιων" nounForm 2 Dative "book" = "βιβλιοις" genderOf "book" = Neuter genderOf "man" = Masculine -- etc definiteArticle 1 Nominative Masculine = "ο" definiteArticle 1 Accusative Masculine = "τον" definiteArticle 1 Genitive Masculine = "του" definiteArticle 1 Dative Masculine = "τω" definiteArticle 1 Nominative Neuter = "το" definiteArticle 1 Accusative Neuter = "το" definiteArticle 1 Genitive Neuter = "του" definiteArticle 1 Dative Neuter = "τω" -- feminine etc -- definiteArticle 2 (plurals) etc.
Of course, you can easily define shorter aliases to avoid some typing here, and there may be better ways to generate the tables, though as written above they are pretty readable, and should be familiar to anyone who knows Greek.
The function youLikeTheThing here is no longer very readable, although it could be much worse. Some kind of substitution syntax/function could be used.
The code above actually works, BTW, and it actually ran first time I tried - the only correction I needed to make its output correct was to add a space after the definite article. You just need to put it in a file test.hs, add the following line:
main = putStrLn $ youLikeTheThing "book"
and do:
$ runhaskell test.hs
There is not a type signature in sight, but you have compile time guarantees. This is all a testimony to the clarity of Haskell's syntax.
The features of Haskell we've used are:
We haven't used recursion. I can imagine circumstances where it might be useful, but if deemed too risky, you could add some rules that would disallow it (e.g. by requiring a function mustn't call itself directly, and must only call functions that exist prior to it in the source code, to avoid mutual recursion.) This would be helpful to ensure termination.
You might also want a module system, to be able to pull in some common definitions and functions for a given language, for consistency across different projects.
This whole approach has the advantage of being able to refine and special case as much as you want. Take the sentence "you like the %s": suppose that if the thing is a human being e.g. "man" or "woman", you need to use a completely different verb. Then you just add a special case first:
isAPerson "man" = True isAPerson "woman" = True isAPerson n = False youLikeTheThing thing | isAPerson thing = ... -- fall through to the normal case here
In the other direction, if you just don't have the time to care about any of this, you can just use a really simple (and often wrong) formula:
youLikeTheThing thing = "φιλεις τον " ++ greek thing greek "book" = "βιβλιον"
Notice that the programmer of the main project does not know anything about plural forms, gender, case etc., or put any of that into the source code. The only thing he/she would do is call a function with all the things to be substituted. We could have some mapping from English strings to function names, or we could just use the function name as a string, e.g. from a Python project we might call the function like so:
prompt = translate("doYouWantToDelete", n, object_name)
This would call the translation function doYouWantToDelete with the parameters n and object_name.
As a refinement, we can provide a version which will work when the whole localisation machinery is turned off i.e. we allow the programmer to provide their own version of the translation function which returns the default language:
prompt = translate("doYouWantToDelete", n, object_name, lambda n, object_name: "Do you want to delete these %s %s(s)" % (n, object_name))
As before, the provided function can be correct or simplistic as desired for English.
There are a few questions in my mind:
Would a solution like this work for the languages you know? What additional features would be needed to cope with other human languages?
Is this vaguely practical? Could you get translators to be able to edit code like this? If not, and only programmers would be able to do this, are there enough programmer-translators to make it a viable solution, at least for some big projects?
I'm aware that the string concatenation gets ugly fairly quicky, and some kind of interpolation might be needed (including the ability to call functions within that interpolation). With that in place, I think you could achieve a reasonable level of readability.
A translation tool could also have language-specific templates to quickly insert the code for common forms.
Is it possible to have a simpler language that would still be able to cope with the examples here?
The examples I've come up with suggest to me that you need a full programming language, and that attempting to start from the other direction (e.g. build up from the current gettext approach) will produce a monstrosity.
gettext already does a 95% job, and we are at the point of diminishing returns. So if we are going to try to tackle the final bit, we need to err on the side of enough power to get all the of that 5%, rather than put a lot of effort in and discover we've only arrived at 96%.
You also cover the case of having a client who insists that the program should output "cet homme" and not "ce homme" - while it might make your translation file ugly, you've got the power to be able to do it if you want.
Comments?
I decided to make an Android app for my Bible memory verse site (created using Django). The main motivation for the app is to get rid of the unnecessary and annoying address bars and status bars when using the site on an Android phone. The site is already designed to adapt to mobiles, so I'm not creating a native app — I just want a better way to access the web app from an Android phone.
I tried appsgeyser, but discovered they put adverts on your site, which I definitely don't want — this is an entirely free app, for a free (and ad-free) site.
My requirements are:
There are lots of pages and wizards with solutions for bits of these, but putting them together turned out to be harder — for example, it seems that the normal way of showing a progress bar for the whole window doesn't work if you've gone full screen.
Anyway, I've completed the learn scripture app (ha, my first Java app!), and thought I'd share the complete source code.
If you wanting a similar app, you are probably best creating your basic app structure using a wizard, but it is helpful to see a complete solution. The important bits are:
The right way to handle issues with untrusted data is:
Filter on input, escape on output
This means that you validate or limit data that comes in (filter), but only transform (escape or encode) it at the point you are sending it as output to another system that requires the encoding. It has been standard best practice since just about forever [citation required].
An alternative is “escape on input”: at the point that data enters your system, you apply a transformation to it to avoid a problem further down the line when the data is used.
It's come to my attention that some serious web developers (or at least, they take themselves seriously and are taken seriously by others) are still suggesting the practice of escape-on-input.
For example, with escape-on-input, to avoid XSS any data that enters your system has HTML escaping applied to it immediately, before your application code touches it.
I chose that example deliberately, because people are actually recommending it:
which, in turn, linked to a page by Rasmus Lerdorf recommending escape-on-input as a sensible way to deal with XSS. The page, admittedly, is describing a ‘toy’, a ‘no-framework PHP framework’, yet he does seem to be serious about the usefulness of escape-on-input.
The page is from 2006, and uses the pecl/filter extension, but the extension has since made it into core (PHP 5.2), and the docs for it suggest a configuration that is clearly intended for XSS prevention. As recently as 2008, and probably to this day, Lerdorf is still defending and recommending this approach, and it appears to be part of his reason for thinking that PHP templating doesn't need an autoescape mechanism.
Just as significantly, Etsy are using and recommending escape-on-input (slide 18 onward). As a very successful modern company using PHP, people will look up to them and copy them.
So, this approach, unfortunately, is popular amongst some, and I can't find a decent post explaining why it's such a terrible idea both in theory and practice. Here is my attempt. It should be applicable to almost any system and any language, although I'll mainly be using examples from web development.
First of all, escape-on-input is just wrong — you've taken some input and applied some transformation that is totally irrelevant to that data. If, taking our example, you have some data collected by HTTP POST or GET parameters, applying HTML escaping to it is a layering violation — it mixes an output formatting concern into input handling. Layering violations make your code much harder to understand and maintain, because you have to take into account other layers instead of letting each component and layer do its own job.
Doing things ‘right’ is very important, even if doing them ‘wrong’ seems to work and you are tempted to be dismissive of ‘theoretical’ concerns about purity etc. When you have to maintain code, you will be very glad if things are in the right place, and not full of hacks and surprises.
You have corrupted your data by default. The system (or the most convenient API) is now lying about what data has come in. As you have applied a transformation to the data itself, the layering violation is not an isolated problem in one part of the code, but infects every part of your code, especially if you store the corrupted data in a database.
Your data is everything. As I read recently, “data matures like wine, applications like fish”. You can always rewrite your application, but if you corrupt your data, you've done the worst thing you can to your system.
This is exacerbated by the fact that many encodings are one-way — you cannot losslessly or unambiguously convert them back. If at a later point you need the original data, you might be in a pickle.
Escaping your data for one output backend will only deal with that output. A typical web app might deal with at least the following backends, which have different characters that are dangerous, and have different requirements for dealing with them:
Any number of others could be added, and all could have security implications. Using escape-on-input will only fix one of these (apart from happy coincidences where it might fix more than one), but for the others you will still need a sensible solution to the problem. Why not have a sensible solution for all of them?
Escaping on input will not only fail to deal with the problems of more than one output, it will actually make your data incorrect for many outputs.
Suppose you decide to do HTML escaping, and someone enters Jack & Jill as a title for something. Your escape-on-input turns this to Jack & Jill and that goes in the DB. Suppose you want to email people and put this title in the subject line. You now have to apply the reverse transformation to get a sensible subject line in the email, and you have to remember to do this for every output that is not HTML.
Sometimes, the bug is significantly more annoying than an email with an incorrect title.
You also have daft bugs like the fact that doing a search on that field for the string ‘amp’ (or ‘quot’, ‘apos’, ‘lt’, ‘gt’ etc. or any substrings) will get various false matches.
I have seen some people respond to this by saying “it's better to have the occasional double-encoding bug or incorrect query result than an XSS exploit". Well, first, that depends on your business. XSS is a problem because it costs time and money, and so does corrupting your data. Many people have data that actually matters, and corrupt data is a big deal, and much harder to cope with than an XSS bug, because data lives on and on, while your code can get replaced easily.
Second, this decision affects frameworks that are used to handle data of all kinds, and the decision affects the entire code base of your application and beyond, as described below. Data-handling frameworks that work on the assumption that your data is not important are insanity. If the foundations be destroyed, what can the righteous do?
Third, it's entirely unnecessary. XSS is not hard to fix given decent programming tools.
At what point does data ‘enter’ your system?
It might sound like a simple question, but it's tricky in reality, and I'll illustrate using an HTTP request.
In most web apps, the GET and POST parameters are your ‘raw input’. However, using most normal web framework APIs, data in GET and POST parameters has already been interpreted. The ‘raw’ data is really the bytes that make up the HTTP request, which typically will use URL encoding for GET query parameters and a choice of encodings for POST data (URL encoding or MIME multipart attachment format).
The framework may also do another level of decoding — interpreting the series of bytes as a series of unicode code points.
Both parts of this initial transformation makes sense and are appropriate, because they are reversing the encoding already applied to the data by the protocol involved. The web browser takes the data you type in — unicode code points — and applies a series of transformations to it, according to the HTTP protocol, and your web framework reverses these to get the data back.
Now, if you want to avoid XSS problems, you have to apply the escaping after this initial decoding has been done. But this highlights another possibility. What if the data requires further decoding before you get the ‘real’ raw data? For example, some data might be sent base64 encoded for a variety of reasons, or any other type of encoding.
This extra level of encoding gives two problems:
your automatic HTML escaping may have corrupted the encoded data so that it now cannot be decoded. For example, you had a GET parameter that held a URL, which itself had parameters in the query string:
GET /foo?bar=1&url=http%3A%2F%2Fexample.com%2F%3Fx%3D1%26y%3D2 HTTP/1.1
Your framework's HTTP handling will produce a query dictionary that looks something like the following:
{"bar": 1,
"url": "http://example.com/?x=1&y=2"
}
But your automatic escaping turns that into:
{"bar": 1,
"url": "http://example.com/?x=1&y=2"
}
If you want to extract the 'y' parameter from 'url', you are stuck. You can't correctly interpret the data in the 'url' parameter, because it has been corrupted. You're going to have to re-decode the input, and you might not even notice this problem.
Even if the data comes through your automatic escaping unscathed (e.g. base64 under HTML escaping), or you can undo the corruption and get it properly decoded, after decoding you will have to manually apply HTML escaping to make it match all the other automatically escaped data. If you don't, you've potentially got a bug and an XSS exploit.
So your automatic escape-on-input has missed data, and this happens because you can't really define the point at which the data has ‘entered’ your system and needs the escaping applied.
This problem means that the escape-on-input approach is inherently flawed and cannot be fixed. You just have to patch it up on a case-by-case basis, which is exactly what escape-on-input is supposed to avoid.
And then, what about other sources of data — data on the file system, in a cache etc. Are these entry points? Well, it depends on how the data was put there. You have to manually follow this all the way through your app; get it wrong and you've got double escaping bugs or security flaws.
(By contrast, escape on output always works, because you apply it at the point where you know it is needed — in the backend that knows the escaping rules.)
Other systems putting data into your database, or getting data out, have to abide by your data transformation rules.
These systems might have nothing to do with your primary domain (e.g. a web site). Making them understand and obey rules that have nothing to do with the data itself is insanity and extremely short sighted.
You can't deal with this problem when you come to it, because you don't have to just fix your code, you've got to fix all your data too, and by the time you cross this bridge you might have a lot of data and might need a very delicate database migration to get it right. The data may even have escaped your control (e.g. been copied into other systems), or backwards compatibility concerns might stop you from making the change you need to make.
Within your main application, the decision to escape on input affects your whole code base.
If you want to use any libraries, you need to make sure that they are using all the same assumptions that you have in your main code base.
For example, if you've got a form/widget library in your web app, it will very often need to echo user input back to them in the case of a form that has validation errors. This library has to know if you already escaped the input.
Writing the library to work in two modes is asking for trouble. Rather, you need it to have been written from the beginning to assume the same escaping rules.
This kills code re-use — you can only use code that assumes the same input escaping — or it means that you will end up with tons of bugs due to incompatibilities between the assumptions made in your application code and the library.
Essentially, this is the problem of a global configuration setting, but worse since it affects the operand of your entire application (the data going through it), not just the functionality of various operators.
The confusion caused by the above is likely to increase security problems. “Keep It Simple, Stupid” remains a very good maxim for developers.
To continue an example used above: you want to send an email with some data that has already been HTML escaped, and so you need to unescape the data to avoid emails with the subject "Jack & Jill" when the user entered "Jack & Jill". You decide it's not sensible for the mail sending functions to do this internally, (or maybe they're provided by a 3rd party who made that decision for you), so the calling code does the unescaping.
You later decide to switch to HTML emails, and the developer who implements it thinks that since data is already escaped, there is no problem including it without extra escaping in the body of the HTML email, leading to a vulnerability (not classic XSS in this case, but still a problem).
There is also the example I gave above where an extra layer of encoding/decoding in the raw data makes it likely you'll forget to apply the escaping.
The confusion caused by escape-on-input means your entire code base becomes a potential source not only of double-escaping bugs but of security problems as well.
Thankfully, we don't just have to rely on the above analysis to conclude that escape-on-input is a terrible idea. PHP, always willing to help when it comes to “examples of how not to do it”, provides us with a perfect case study.
PHP used to have a feature called magic quotes. It was an escape-on-input feature that escaped single quotes (') with backslashes. This was to protect you from SQL injection attacks, by making the data safe for interpolation into a SQL query.
This caused all kinds of problems.
First, if you are not first passing something through a database, and using string interpolation to build up SQL queries, you have to remember to strip those slashes using the function stripslashes().
If you don't, you get double encoding. It looks like \'this\', you\'ve almost certainly seen it across the web, though it seems we\'re thankfully past the worst of it.
Second, even if you remember, you've added some hideous cruft to your code. In the bit of code which is handling form validation (and is therefore echoing user input back to the user without the database being involved), you've got these bizarre stripslashes() calls. What on earth does ‘reverse transforming a string for SQL statement preparation’ have to do with the task of input validation?
Third, it turns out that different databases need different escaping mechanisms to do things fully correctly. So you now have to do stripslashes() on data even if you are passing it to a database using string-interpolated queries!
Then, since the above problems are common (building up SQL queries by string interpolation was always a bad idea, and very often you pass on the data to outputs that don't want SQL escaping at all), it's desirable to have a way to turn this behaviour off completely.
To handle this, there is a php.ini setting to turn it on/off.
And there were more complications, for example:
And so now you've got even more problems. Since these are global settings, you can't have library code mess with them, since other code might set the global to a different value or assume a certain value.
You could try making all code detect the current setting and have different code paths depending on the result. This works very badly — having multiple code paths is a recipe for code duplication and bug proliferation. It's extremely easy to forget to do it, or get one of the paths wrong, since you will likely only test one configuration value and one set of code paths in reality.
Alternatively, you can make one bit of code responsible for fixing the setting to a sensible value (the only one being 'off'), and then make all code assume that from then on. (If you can't turn it off, you can use the code included here as a horrible kludge to reverse it's behaviour).
Eventually, this final approach was the one taken by all significant projects. Turn the whole feature off, and assume it is off from then on. (Which means the feature is useless, of course).
And of course, thankfully, the PHP developers realised that this entire thing was a huge mistake that caused nothing but a vast amount of confusion and bugs, and removed the whole thing for good in PHP 5.4.
Magic quotes, as eevee put it, were “so close to secure-by-default, and yet so far from understanding the concept at all.”
To digress for a moment: we keep getting told that PHP is improving, and the community has learnt from its mistakes. Unfortunately it seems the leaders in the community are bent on recreating old mistakes.
According to Lerdorf, the much newer PHP 'filter' extension is “magic_quotes done right”. But it still suffers from almost all the problems described here, for all the reasons described. Global HTML escaping on input is essentially the same as magic quotes, and just as tragically bad.
In researching for this post, I came across this ticket for Elgg, an open source social networking engine. Just read through the ticket and see the mess they are in. It's clear they strongly regret the decision they made to escape-on-input, and, in their own words, have created “horrendous” problems for themselves, especially as their application has grown to include other interfaces such as JSON REST APIs.
However, fixing it is very hard. They have to coordinate many changes across their code base with a big database migration. If data has leaked from the databases and tables they control into other systems, such as denormalised tables, other databases, caches etc., or if there is other code by third parties that makes the old assumptions about encoded data, they are in even more of a pickle. And both of those things are probably inevitable in something like an open source framework, which is designed for other people to build on and extend.
This is the pain that comes from mixing input handling and output encoding, and from corrupting the data in your database.
According to their security presentation, Etsy are using escape-on-input for XSS protection.
They claim that this is a much more secure option, as it is secure by default. (They do note, however, the problem with input that is encoded in some other way, like base64, so they are aware of the problems.)
Their presentation goes on to describe an elaborate system for detecting and fixing XSS attacks (the slides don't give enough detail for me to understand what exactly they are doing, but it's clearly a lot of work).
And their system does indeed catch XSS bugs in the wild and allow them to fix them within hours.
Wait, what?
They've corrupted their database by doing escape-on-input, they've inflicted themselves with all the development pain described above, and they've still got XSS bugs?
Granted, they've got impressive ways of dealing with these problems. But it's like virus checkers on voting machines. Advanced ways of dealing with problems that shouldn't even be possible tells you that you are doing it wrong. They've become very fast at re-tying their shoelaces, instead of working out how to tie shoelaces so they don't come undone.
They claim that with escape-on-input, XSS problems are now greppable, but it doesn't sound like it. If they were, code audits would be a massively more efficient way to find XSS problems than the methods they are using.
The main problem is almost certainly that they are using an output system for HTML that doesn't do HTML escaping by default (I'm guessing they are using PHP as their template language). If the backend that deals with HTML actually deals with HTML then you eliminate the vast majority of these problems overnight.
I'm willing to bet that large sites that use Django (or other frameworks that have basically solved the XSS problem by HTML escaping on output by default) don't have teams and automated systems dedicated to this problem, and don't need them. In Django apps, XSS problems are greppable - you grep for mark_safe in Python and the |safe filter in templates (and then, obviously, you may have to recursively grep for any functions that call mark_safe on inputs). Since all data which isn't ‘mark_safe()’d gets escaped by the templating engine, and all HTML comes out of the template engine, that's basically all you need to do.
How did this happen to Etsy?
Are the Etsy devs stupid? I suspect not. Etsy is clearly doing well, and I imagine they have enough money to hire top-notch developers. Some of their careers pages show they are happy using a variety of languages and technologies, and their engineering blog seems to be sane and competent. Even their security presentation showed considerable ingenuity and technical ability in dealing with security problems (in entirely the wrong way, unfortunately, but still impressive).
I doubt they are low quality developers. Rather, I suspect that use of PHP has addled their brains. They have become far too accustomed to working in an environment in which insanity reigns — an environment in which the less than operator pretends to work correctly with strings but it's just a trap.
When I programmed in a Windows environment, I theorised that use of Windows itself contributed to the poor quality of the programming in the code base, and the fact that developers thought nothing or writing tons of tedious code. Because Windows was so unscriptable, I imagined that Windows programmers developed a high tolerance for tedium and repetition, which is exactly the opposite of qualities needed by a programmer to make a computer do everything efficiently and reliably. (Since then, I've found that Sturgeon's law was probably a better explanation for the quality of the code, but I still think the fundamental idea applies).
With PHP, the fact that it comes with a template language that is simply not fit for purpose — because it doesn't do HTML escaping by default, or even easily — has somehow made the Etsy developers believe that it is normal to struggle with XSS, that it is perfectly reasonable that even after taking the drastic action of corrupting their entire database by HTML escaping it, they should still need elaborate XSS-catching systems.
Instead of trying to fix XSS, they should just fix it. Like this in Django. Or this in Turbogears and Jinja. Or this in Yesod. Or even this in PHP (though due to limitations of the language you won't be able to have the convenience of things like mark_safe in Django). But living with an environment of pain and madness makes you think that it ought to be hard.
Right the way up to Rasmus Lerdorf at the top, many people in the PHP community live with the insanity of their tools, and add more insanity to cope with it, rather than fix their tools or choose better ones.
Bashing other people is fun, but when I do so I always try to get something more valuable out of it by using the opportunity to examine myself. The problem I discussed in the last section (which is just a manifestation of the broken windows theory) applies to other communities, and I'll attempt to apply it to the Python community.
Refusing to live with stupidity is one of the reasons that Python 3 is really important.
Python 3 does not represent a massive leap forward in terms of additions to the language. Mainly it just fixes a bunch of mistakes in Python 2, and introduces a whole lot of backwards incompatibilities in the process. One of the biggest is unicode/bytes. Python 2 was stupid here — it went directly against the Zen of Python, and said “in the face of ambiguity about what encoding to use, guess.” This caused a world of pain.
Now, you can work around it in most cases by some sensible conventions and a certain amount of discipline. You can also cope with the fact the "a" < 1 doesn't raise an exception. You can live with next() being a method in the iterator protocol, when it should be a method called __next__() and a builtin function next(). You can live with the fact that print is a totally unnecessary keyword, since it should just be a builtin function. You can get used to the fact that class Foo: means something subtly but significantly different from class Foo(object):. You can work around or ignore dozens of other little niggles, gotchas and inconsistencies.
But all the while, you are training yourself to tolerate stupidity, inconsistency and brokenness. Removing these warts is really important, and worth all the pain of the migration. The alternative is for Python to become the next PHP.
On top of these things, there are other types of brokenness in Python that people in the community seem less willing to acknowledge or tackle. For some of these I think we need exposure to completely different languages — languages where you can spawn thousands of ‘threads’ easily and get performance benefits, for example, or languages where you can write code that is both very high level and extremely fast. If we live entirely with Python and its set of limitations, we'll think that those problems are normal and unavoidable.
Updates:
I always find it fascinating to observe conversations in which people's arguments fail to convince each other. A few days ago we witnessed some PHP debates, kicked off by Jeff Attwood.
I foolishly got slightly involved on one ‘rebuttal’. At church last Sunday I also chatted with a friend who has used PHP and likes it, and this time I tried to put myself in his shoes. It is often much more helpful talking to people in the flesh, and I think it is always enlightening to look at why we fail to convince.
One reason we can't rule out is simple irrationality. All of us are vulnerable to confirmation bias, and people will often go to great lengths to convince themselves that they are doing the right thing and do not need to change their views or practices. You see this all the time when two groups of people share experiences after having made different decisions about how to spend some leisure time. Both groups are often desperate to believe that they haven't missed out, and will seek to persuade each other (but in reality, persuade themselves) that they were in the group that had the most fun.
However, just assuming the other person is being irrational doesn't really help you, and can in fact hinder communication. Below I will attempt to be more constructive in looking at the ways we can fail to convince people. I'll try not to turn it into a rebuttal to the “PHP isn't so bad” posts!
In his great rant PHP: a fractal of bad design, ‘Eevee’ has lots of great arguments against PHP, and there were others in different posts. But there are reasons why some will fail to hit home. (This is not a criticism, by the way — many of these problems are unavoidable if you are addressing an audience as mixed as “all the PHP developers in the world”).
Expert understanding.
Eevee writes:
empty($var) is so extremely not-a-function that anything but a variable, e.g. empty($var || $var2), is a parse error. Why on Earth does the parser need to know about empty?
To an experienced programmer, empty() is rather surprising. If you know — or at least have some idea — about how to implement a programming language, you'll understand terminology like lexer, parser, interpreter etc. So when you try empty($var || $var2), and it returns a parse error, even though it looks like a function, you think “this programming language must have been designed by complete amateurs — I don't want anything to do with it”. [ EDIT: changed this paragraph, previously I was gettting mixed up between isset() and empty() ]
For a less experienced or able programmer, however, none of this is a problem. Programming languages are entirely magic, and one type of magic is no more surprising than another.
Talking about what the parser is doing is completely incomprehensible to such a developer. You cannot communicate the reaction you feel, because it requires a deeper level of understanding of how things are supposed to work. Less able or experienced developers are simply unable to assess the quality of the tools they work with.
Craftsmanship
For many coders, the only thing that matters is whether PHP allows them to get something done. The quality of the tools or materials being used does not matter.
Not only are many coders unable to assess the quality of the tools they using, they wouldn't care even if they could, because programming is simply about getting something done. Any thought of taking pride in your work is absent. Such a person will never be convinced by arguments that talk about the quality of tools.
Amateur vs professional
Eevee accused the PHP world of being created by and filled with amateurs. I suspect that to many PHP developers, that has the same effect as Java people saying to Pythonistas that Python is not ‘enterprise ready’. Many Python developers don't care about “the Enterprise” in the first place, and the word may even have negative conatations — associations of massively overengineered and poorly written bloatware written in Java or C#, which could be replaced by a Python project 10 times smaller.
Many PHP users simply do not aspire to be professional, because they are happy to be called amateurs — they are amateurs, doing stuff just for fun, pure hobbyists who are not making money out of what they are doing, nor relying on PHP to behave in a sane manner with important data. PHP has brought joy to their life. Does it matter to them that PHP doesn't live up to some standard they don't need?
Many developers simply do not care that much about the data that goes through their website. “So what if PHP doesn't have any decent way of handling decimal values? It's close enough for my needs.” For these people not only is the quality of the tool unimportant, the quality of the result is of little consequence. No-one will die or sue them even if it has major bugs.
Also, professionals often have (or feel) obligations to support the code they write, whereas amateurs do not. Therefore amateurs will always trade long-term maintainability for initial deployment speed.
Defending your day job
The previous argument doesn't cover a lot of users, however. Many are using PHP ‘professionally’, and even have customers who may demand that work is done with PHP. (It is perhaps the essential problem of PHP that a language that was designed to be a simple template language for non-programmers has turned into the work-horse of the web, and the network effects caused by adoption amongst amateurs have made it a language for professionals.)
Now, most people want to feel good about the work they are doing. And most people are not in a position to have much influence on whether or not they use PHP. If you tell someone “the tool you are using is Bad For The World”, they will get defensive, even if they would rather be using something else.
I'm extremely fortunate in the work I do that I get to choose my projects carefully, and then charge a high rate for work that I can take real pride in. It's actually fairly unkind of me to berate people for using a language that they didn't really choose — even if they think they are choosing it and will defend it to the death. They are only doing so because the alternative is to say “yes it sucks, and yes I'm doing the world a disservice by continuing to promote it, but it pays the bills”. I've occasionally been in a similar situation in the past, and know how demoralising and depressing it is, and it is more comforting to rationalise your current situation.
In fact, when I was last in a situation like this, I did come up with a whole bunch of rationalisations, which I still think are valid to some degree.
Engineers must be pragmatists at some level. You're paid to achieve things, not for the internal beauty of your code.
I should take pride in customer satisfaction, because that is what matters.
The language itself is not the only factor. You also must consider:
All of these things depend on human factors and network effects, and can be used to justify a choice on the basis that “lots of other people are choosing this”.
(If you are a “PHP developer” reading this, I apologise for being patronising — this post is not meant to insult, and is not aimed at you but at others).
Experience
The vast majority of PHP developers that I've come across have not used any other serious web frameworks. That is why they can say things like “PHP is the best web platform... ever.”
To make such a statement, you need to have in-depth knowledge of a large number of alternative web platforms — if not all the web platforms in existence. However, the author did not demonstrate any knowledge of any alternatives. You are never going to persuade anyone that way. Rather, it will lead people simply to dismiss your opinions entirely. Being ready to make pronouncements on subjects for which you clearly don't have a fraction of the required knowledge is a sign that nothing you say is trustworthy.
However, the problem goes both ways. Eevee's rant, and Jeff's, both contain statements that are to some degree out of date (or appear out of date to someone on the other side who knows the 'standard solution' for the problems highlighted). The reason for this is obvious. If you feel passionately about how bad something is, you will stop using it, and your level of knowledge of the technology will go down. This does lead to a bit of a problem. Those who are unwilling to work with technology X because of how bad it is can always be dismissed as being ignorant of it.
A facet of this problem is that it is extremely easy to underestimate something that you don't have knowledge of — for instance, you can simply underestimate the size of a competing community.
I'll admit I was surprised to learn that PHP has a package manager with 1900 packages in its main repository. However, the author who pointed that out might be more surprised to know some of the following figures:
Python:
Perl:
Let's add Haskell, as an example of a minority language if ever there was one, and not the first language you'd think of for building web sites:
I'm sure other languages can boast similar or much better figures. I don't even know where to look with Java — it's quite possibly so large that the idea of a central repository doesn't even make sense.
PHP, despite having huge numbers of developers, looks rather small in comparison. Now, all of these statistics are flawed in a variety of ways, PHP's included, and the bigger a community is, the more they will be flawed — for example, PyPI download stats will often be way out because people are using mirrors etc — but this doesn't affect the point I'm making.
The point is that all the communities are large, and these figures are just the tip of the iceberg in terms of how much is going on in each and every community. And from within a community, you can see some big figures and think “well I doubt anyone could seriously be competing with that!”. This is true no matter what community you belong to. And it makes it difficult to communicate in a meaningful way. Very few can honestly say that they've evaluated the alternatives in a fair way, because the alternatives are so huge.
I'm not sure I have a conclusion actually. These are just some of the pitfalls we face in communicating across different technology communities. I often forget these things, especially when communicating on the internet, and this post is here mainly to help me remember!
Because of the broad spectrum of PHP users, I do think that arguments against PHP are going to have to be more individual to be effective.
For example, suppose you have a customer that wants PHP for a website that deals with money in some way e.g. a shop. I might attempt to shock them by using phpsh to demonstrate PHP's inability to get some basic arithmetic questions correct. I would explain that all languages have problems with floating point arithmetic, which is why other languages have good solutions to this problem e.g. Python's decimal module, which makes it easy to get things right. PHP has nothing approaching this (and yes I know about BCMath and GNU Multiple Precision bindings). Rather, PHP's fundamental attitude is to 1) silently ignore errors, 2) attempt to paper over errors and 3) silently convert your data even if that means losing information. Is this the langauge you want to handle your data?
For a different situation, I think you'll need a completely different argument. And for many situations, don't even try if you think your words are going to come over as insulting, since that will be counter-productive. That often applies to the internet, and I need to remember that!
Noah Sussman wrote a post on Things you should test, “A checklist of things that are worth testing in pretty much any software system.”
Many of the things on the list are helpful reminders. However, I think the mindset it encourages is essentially wrong.
The mindset is basically this:
Identify common mistakes that developers make, and ensure you are writing tests that check you haven't made them.
The problem with this approach is that it is essentially whack-a-mole debugging. There is a never ending supply of bugs to kill.
A much more helpful approach is found in this post on easy programming that advocates “Never fix a bug twice” (about one-third of the way down).
If you come across a bug or class of bugs that often occur, you should not be thinking first of all “better add that to my list of classes of bugs that need testing against”. You should rather be thinking “how can I change the system so that this class of bugs disappears entirely?”.
So, to take some of items listed for testing:
Input handling bugs, such SQL injection and XSS attacks.
In Django apps, I never write tests for SQL injection attacks or XSS attacks. The reason for this is that these types of bugs are very rare. The reason for that is that the APIs that are easiest to use for executing SQL or generating HTML are secure by default. If there are any errors of this type in my programs, they will be very rare and obscure, and trying to catch bugs by a black box scatter-gun approach will be a waste of time.
With XSS, there are times when I am slightly more tempted to generate HTML in ways that might be vulnerable to XSS — for example, using Python string interpolation. At this point, however, you still shouldn't reach for a test to ensure you did it correctly. Rather: stop writing HTML generation the stupid way!
To do that, you first have to ask “why?”. “Why am I tempted to write HTML this way?”. The answer is usually “there isn't a convenient API for the particular way I want to write this”. The correct solution now becomes obvious: create the API that removes the temptation to introduce potentially buggy code.
So, here is some code in a project I'm writing, that uses string interpolation to implement a template tag for formatting a link in a standard way:
from django.core.urlresolvers import reverse from django.template import Library from django.utils.html import escape from django.utils.safestring import mark_safe register = Library() @register.filter def account_link(account): return mark_safe(u'<a href="%s" title="%s %s">%s</a>' % ( escape(reverse('account_stats', args=(account.username,))), escape(account.first_name), escape(account.last_name), escape(account.username), ))
The problem with this is that I have to remember to use escape on each variable. The reason I wrote it this way is that Django's Template API is kind of bulky for this use case. So, I should instead have written this:
from django.core.urlresolvers import reverse from django.template import Library from somewhere import html_fragment register = Library() @register.filter def account_link(account): return html_fragment(u'<a href="%s" title="%s %s">%s</a>', reverse('account_stats', args=(account.username,)), account.first_name, account.last_name, account.username, )
And then write the API, html_fragment, that makes this work, which is just:
from django.utils.html import escape from django.utils.safestring import mark_safe, conditional_escape def html_fragment(template, *args): return mark_safe(template % tuple(map(conditional_escape, args)))
EDIT: After some discussion on django-devs and subsequent modification, this is now in Django 1.5 as 'django.utils.html.format_html', so use that instead of the above
I'm now no longer tempted to write it the vulnerable way, and I don't need a test (though I may want a test or two for html_fragment). I have two ways of doing it - html_fragment for very small snippets, and Django templates for bigger chunks, both of them secure by default.
So, if you find yourself needing tests for specific SQL injection or XSS attacks in your code, you are probably doing it wrong. Fix the underlying API that makes the mistake likely in the first place.
Input handling type checking - “Invalid values such as null and NaN. Strings instead of integers, arrays instead of strings.”
Input handling should generally only occur on trust boundaries, but it should occur systematically on such boundaries. If you have a problem with the wrong type of value being passed to an internal function, due to an external input being passed in, you shouldn't be dealing with it in the function, you should be asking “how was this even possible?”.
One solution of course is a static type system. Of course, that might not be possible/practical in your situation, but you should be thinking about it.
Math-related bugs: in many of the listed instances in Noah's post, the underlying bug is that the language does funny things.
So you should be asking “why am I using this language? Is it the right tool for the job?”.
Or at least “will a library solve this problem?”.
If you are handling decimal values, the solution is a library that does it well, and doesn't make it easy to make mistakes. For example, the Python 'decimal' library does not allow multiplication of floats and decimals - which sounds inconvenient, but stops you from making all kinds of mistakes.
Units of measurement.
If you have potential bugs with this, you should be thinking “why are these possible? How can I eliminate the bug entirely?”.
Different languages have different solutions, often involving including the unit of measurement in the value itself. Haskell has various solutions that make it impossible to add "3 years" to "2 meters", for example, and the compiler will stop you at compile time.
For Python, there are things like magnitude and units that work at run time.
Of course, even if you use these internally, you'll still have boundaries where you need to do some conversions from inputs etc., and at this point you might need to know what units the inputs are in. But:
Text - “Are Unicode inputs treated differently than ASCII?”
Again, you should be using proper internal datatypes — unicode everywhere — to eliminate potential bugs.
Python 2.x is notorious here - its ‘forgiving’ conversion between bytestrings and unicode causes endless bugs that are found in production, not in development.
And the correct solution is not just to write tests — which you may have to do for now — but rather to fix the underlying cause. This is what has been done in Python 3.
The same type of things applies to almost everything on the list, and in the cases where I can't see how it applies, I imagine that this is due to my own lack of imagination :-) If I can't think of a way of completely eliminating a class of bug, then I should think harder, rather than assume it is impossible.
Notice that from the perspective of an older programmer, there are some notable omissions from the list - things like buffer overflows, for example. Presumably that's because the author uses languages/frameworks where those bugs are very unlikely. And that has happened not because previous generations of programmers wrote lots of tests, but because they wrote languages and systems where those bugs are impossible or almost impossible.
So, in all this I'm not saying that you won't ever have to write tests for these kinds of bugs. But every time you identify a class of bugs, tests are a very poor answer. You should try to ensure that you never fix the same bug twice, and so never write the same test twice. If you are writing essentially the same test more than once, you are proving that you haven't fixed the underlying issue yet.
If a bug is worth adding to a list of common bugs, then you have identified a systemic problem with your platform, and it is worth eliminating entirely. We should be striving for libraries/languages/systems which do that.
]]>See end for updates to my ideas on this
I've written before about the somewhat doubtful advantages of Class-Based Views.
Since then, I've done more work as a maintenance programmer on a Django project, and I've been reminded that library and framework design must take into account the fact that not all developers are experts. Even if you only hire the best, no-one can be an expert straight away.
Thinking through things more from the perspective of a maintenance programmer, my doubts about CBVs have increased, to the point where I recently tweeted that CBVs were a mistake.
So I thought I'd explain my reasons here. First, I'll look at the motivation behind CBVs, how they are doing at solving what they are supposed to solve, and then analyse the problems with them in terms of the Zen of Python.
People kept wanting more functionality and more keyword arguments to the list_detail views (and others, but that those especially, as I remember). The alternative was large copy-and-paste of the code, so people were understandably wanting to avoid that (and avoid writing any code themselves).
So, we replaced them with classes that allows people to override just the bit they need to override. This eliminates the need for code duplication, and removes the burden of lots of feature requests for generic views.
Or does it? Instead of tickets for keyword arguments to list_detail, it seems we have a bunch of other tickets asking for changes to CBVs, many of which can't be implemented as mixins or subclasses.
One of the problems is that if the view calls anything else (e.g. a paginator class, or a form class), you have to provide hooks for how it calls it, which means implementing methods that can be overridden. If you forget any, or if the thing you are calling gains some new keyword arguments, you've got feature requests, or duplication because someone had to override a larger method just to change one aspect of it. If you don't forget any, you've got dozens of little methods to document.
Also, there are problems like this attempt to mix FormView with ListView functionality. Fixing this will end up with similar amounts of copy-paste, but in this case it requires a fair bit of debugging first to realise you have a problem.
So I'm not convinced CBVs have made much difference here.
The classic example is editing using a form. You see this pattern again and again using function based views (FBVs from now on):
from django.shortcuts import render def contact(request): if request.method == 'POST': form = ContactForm(request.POST) if form.is_valid(): send_contact_message(form.cleaned_data['email'], form.cleaned_data['message']) return HttpResponseRedirect('/thanks/') else: form = ContactForm() return render(request, 'contact.html', {'form': form})
Without question this is tedious and annoying. CBVs reduce this to:
from django.views.generic.edit import ProcessFormView class ContactView(ProcessFormView): form_class = ContactForm template_name = 'contact.html' success_url = '/thanks/' def form_valid(self, form): send_contact_message(form.cleaned_data['email'], form.cleaned_data['message']) return super(ContactView, self).form_valid(form)
Much better!
However...
It's not really that much shorter. 8 lines compared to 11, ignoring imports. But now let's make it more realistic. We're going to have:
FBV:
from django.core.urlresolvers import reverse from django.shortcuts import render def contact(request): high_priority_user = (not request.user.is_anonymous() and request.user.get_profile().high_priority) form_class = HighPriorityContactForm if high_priority_user else ContactForm if request.method == 'POST': form = form_class(request.POST) if form.is_valid(): email, message = form.cleaned_data['email'], form.cleaned_data['message'] send_contact_message(email, message) if high_priority_user and form.cleaned_data['urgent']: send_text_message(email, message) return HttpResponseRedirect(reverse('contact_thanks')) else: form = form_class(initial={'email': request.user.email} if not request.user.is_anonymous() else {}) return render(request, 'contact.html', {'form': form, 'high_priority_user': high_priority_user})
CBV:
from django.core.urlresolvers import reverse_lazy from django.views.generic.edit import ProcessFormView class ContactView(ProcessFormView): template_name = 'contact.html' success_url = reverse_lazy('contact_thanks') def dispatch(self, request, *args, **kwargs): self.high_priority_user = (not request.user.is_anonymous() and request.user.get_profile().high_priority) return super(ContactView, self).dispatch(request, *args, **kwargs) def get_form_class(self): return HighPriorityContactForm if self.high_priority_user else ContactForm def get_initial(self): initial = super(ContactView, self).get_initial() if not request.user.is_anonymous(): initial['email'] = request.user.email return initial def form_valid(self, form): email, message = form.cleaned_data['email'], form.cleaned_data['message'] send_contact_message(email, message) if self.high_priority_user and form.cleaned_data['urgent']: send_text_message(email, message) return super(ContactView, self).form_valid(form) def get_context_data(self, **kwargs): context = super(ContactView, self).get_context_data(**kwargs) context['high_priority_user'] = self.high_priority_user return context
(A few lines could be shaved by not using super() in a number of places, but only at the expense of future confusion/problems if a maintainer was expecting the normal declarative behaviour.)
Notice:
I really want high_priority_user to be a local variable that is calculated once, and used in a couple of places. With a function, that's what I have. With a CBV, I have to simulate it using an attribute on self. I also need to override the dispatch() method just to create it. These are both ugly hacks.
The CBV version is extremely noisy, due to all the calls to super(). This would be improved in Python 3 (at the expense of a rather magical super() builtin), but is still far from perfect. Every time you override a method, you have to mention it twice.
The CBV version is now significantly longer - 24 non-blank lines compared to 17. Since I'm using the same APIs for requests and forms, and doing the same thing, this can only mean that the amount of boilerplate has significantly increased. Sure, I've removed the flow control boilerplate, but must have added some other type.
Even if you know the Form API very well, you are going to have to look up the docs or the source code to find get_initial() etc. and ensure you get the signature correct. You have to know two sets of APIs to use a form, instead of one.
In the CBV, flow control is totally hidden. In the process of abstracting away the duplication, we've also hidden the order of execution. I've ordered my methods in the order they are called (I think), but that may or may not be obvious to anyone else, and nothing forced me to do it.
Because of the last point, it is massively more difficult to debug. Which of the two would you rather maintain? Which do you think a maintenance programmer, who has never seen this code before, would rather maintain?
To understand what is going on, you've got an intimidating stack of base classes to navigate, compared to a single function.
The fundamental problem here is that Python sucks at implementing custom flow control. Not many languages shine here. Ruby has blocks, which help. Haskell has a pretty good story due to a combination of succinct function definition, lazy evaluation and the way that IO works. Lisp has macros. But with Python, we are limited to: abusing generators, abusing the with statement, or classes and the template method design pattern (which is basically what CBVs use).
I think we decided to use the latter because it's one of the only options we've got, but failed to notice that it's just not very good.
There are worse things that can happen with shifting requirements. What if you want two different forms on the same page? I've done this on more than one occasion, and it can be a useful thing — when you present the user with two different courses of action, and you've got completely different info they need to fill in.
It would be a nightmare to get CreateView or UpdateView to do this. You will either produce a monstrosity, or you'll have to start from scratch with an FBV. You've been seduced down a tempting, calm stretch of water, and then left high and dry when you come to the end of what CBVs offer. If you start with an FBV, it's an easy change.
Overall, when I work on CBVs, even views I've created, I start to feel like I'm working on a classic ASP.NET Page class. It is bringing back painful memories! We’re not as bad as that yet, but methods that simply modify some data on self are a code-smell that we are heading in that direction.
Another way of looking at it is that with CBVs, views have become an instance of the ‘framework pattern’ instead of the ‘library pattern’. With a framework, your application code gets called by the framework code, and you can easily end up having to understand how the framework is implemented.
With a library, the library code gets called by your application code, and this is in general much more flexible, much easier to document and much easier to debug. We ought to be moving Django more in the direction of a library.
With the comparison to ASP.NET, I'm also basically saying CBVs are not Pythonic. I'll back up this claim in terms of selected parts of the Zen of Python.
This is a subjective one, but the noise of all the super() calls, get_context_data() compared to a simple {} etc. is ugly to me.
Let's notice, first off, that FBVs are simpler than CBVs, according to the basic meaning of having fewer parts.
A class is a datastructure with attributes and functions attached. A function is just a function. So a class is more complex than a function. If we ever use a class where a function would work, we have to justify the additional complexity.
Are CBVs complicated? Just read the Django source if you think they are not.
In one app I wrote, I had a mixin that implemented a bit of common flow control for forms — namely it returned a JSON response with validation errors for certain types of requests. This meant I didn't need separate views for the AJAX validation, and with CBVs I eliminated all the boilerplate. Great!
Well, it was, until I found a crazy problem to do with the order in which different base classes set things up. To solve my problem, I ended up writing this:
# MRO problem: we need BaseCreateView.post to come first in MRO, to # provide self.object = None, then AjaxyFormMixin must be called before # ProcessFormView, so that the right thing happens for AJAX. class AjaxMroFixer(type): def mro(cls): classes = type.mro(cls) # Move AjaxyFormMixin to one before last that has a 'post' defined. new_list = [c for c in classes if c is not AjaxyFormMixin] have_post = [c for c in new_list if 'post' in c.__dict__] last = have_post[-1] new_list.insert(new_list.index(last), AjaxyFormMixin) return new_list
It's a metaclass, and implements the mro() method that allows you to override the method order resolution. I am not proud of this code. (For those who don't get English understatement, please understand what I'm saying: this code is horrific). Before writing this code, I didn't know that the mro() method existed. Although I value the education, if a framework forces you to learn about type.mro() and implement it, it is doing something wrong.
The hierarchy of classes is certainly a form of nesting, whereas functions are flat. I think CBVs might be considerably better if you started by writing your own, so that you had base classes that did everything you needed, but nothing more, and ended up with a very flat hierarchy.
Get some people who don't know Django to read ContactView above and figure out what it does, and some others to feed the function version, and see who has the easier time. Ask them what happens, for example, when the form is invalid. There is no contest here.
With a CBV, by inheriting from a class you are inheriting all the behaviour of that class, and all its parent classes. You could argue this is explicit, since you've explicitly indicated the base class, but you haven't indicated all the parent classes — they come automatically. So it's more like an implicit request in practice.
Of course, this is always true with OOP to some extent — you are inheriting a bunch of behaviour precisely because you don't want to define it again. But let's notice that it does have its downsides.
For example, let's play spot the difference between the following two views — an FBV:
from django.shortcuts import render def my_view(request): return render(request, "my_template.html", {})
and a CBV:
from django.views.generic import TemplateView class MyView(TemplateView): template_name = "my_template.html"
Can you see the important difference? Try to answer before reading on.
I'm talking only about the functional differences when this view is accessed by a client, not differences of code organisation or re-use or performance.
Well, TemplateView inherits from View which provides a dispatch() method, and TemplateView provides a get() method which will handle all GET (and HEAD) requests, by the logic defined in View.dispatch(). However, neither defines a post() method, which means that you will get a "405 Method Not Allowed" error if you try POST or other HTTP verbs, to the CBV view, whereas you will get a 200 with the FBV.
In the CBV, all of this logic has been invoked implicitly. Nothing in what I wrote was an explicit request for 405s for POST requests, but I got it because I inherited from TemplateView — even though using a template does not imply that behaviour.
(By the way, this issue caused a real problem in a site I wrote, which was easily debugged because I knew a lot about the internals of CBVs, and therefore easily fixed, but it still adds noise to my class — code that can only be explained by reference to some inherited behaviour I didn't really want).
Of course, as already mentioned, you can say the same thing whenever you inherit from a class. But, in my opinion, it is one of the disadvantages of OOP, and it is showing itself here. The question is: do the advantages of inherited behaviour outweigh the disadvantages of implicit behaviour?
I already mentioned one case where the CBV method is just not going to work for you, or is not going to be worth it — needing two different forms on the same page. But in the real views I write, I suspect probably the majority would not fit easily into a CBV.
So, in your project you now need both FBVs and CBVs — you've got two ways to do it. When a maintenance programmer comes along, and needs to do some similar work that involves a form, they will be confused. Which pattern do they start from?
The simple way to avoid this is to avoid the less flexible solution — avoid using CBVs.
So simply having two different ways of building up views is a violation of this principle, but within CBVs there are also instances.
First, there are also problems with the attempt to use declarative style, which is perhaps the most attractive feature of the CBV API. So, you need initial arguments to your form? Just define the initial attribute on your class. Unless, however, you need it to be dynamic — you'll have to define get_initial() instead.
Also, it often isn't obvious where to add certain bits of code. There is often more than one choice of method to override when you come to add a little bit of functionality e.g. get() or dispatch(), and which you choose sometimes has subtle implications, and sometimes doesn't matter. With FBVs, two different Django developers tasked with the same modification to an existing view would often produce identical or nearly identical patches, but I suspect that would be rarer with CBVs.
Looking at the implementation, you'll find things like MultipleObjectTemplateResponseMixin, as well as MultipleObjectMixin and TemplateResponseMixin, and the former is not just the composition of the other two. This is just one of many signs that things are going wrong. If you can really compose behaviour just by adding mixins, MultipleObjectTemplateResponseMixin should not be needed. Explaining things like this is hard, and the reason is that you just can't build up complex views using classes and mixins.
Overall, I think CBVs make:
You only gain for the simple views, but they were simple anyway, just slightly tedious. Is this advantage really enough to outweigh the disadvantages I've listed?
To be honest, I regret dropping the function based generic views for CBVs. There was an alternative solution to the tickets asking for them to do more: WONTFIX.
Generic views were simple shortcuts for common problems. They didn't actually involve very much code, and if you needed to duplicate some of it you weren't duplicating much. We should have just said: this is what generic views do, if you need them to do something else then write your own. Because that is what you have to say with class based generic views anyway, just slightly later.
Regarding going forward, I say: stop the rot.
If you're starting a new project, I recommend avoiding CBVs, or just wrapping them in thin functional wrappers, like this one for ListView (or, with django-pagination a simple 3-line view function is probably all you need and is actually easier than subclassing ListView).
For Django core, let's fix up the main problems and bugs CBVs have, and then leave them as solutions to simple problems.
But please: let's not move everything in Django-world — whether Django core/contrib or resuable apps — to CBVs. Just use a function, and stop writing classes.
Update: 2013-03-09
My opinions have changed slightly since I wrote the above, due to comments below and other helpful blog posts. I'd summarise by saying:
I needed to add a boolean field to a model. For many web apps, this typically involves:
Using Django, from a cold start (no editor/IDE open), this just took me 1 minute 45 seconds of work for steps 1 - 5, and an additional 45 seconds waiting for step 6, total 2 minutes 30 seconds, and I wasn't rushing.
Step 1 is a one line code addition. Pretty much everything else can and should be generated automatically.
Step 2 is taken care of by a one line command using South, as is step 3 and the database part of step 6 (which is run de-rigueur from my deployment scripts).
Step 4 is taken care of by Django's admin, which introspects the model and generates the right form for you.
This is one of the reasons I love Django. It's not so much the time it saves, although that is pretty awesome, it's the tedium it saves.
This is also one of the reasons I'm not very tempted by schema-less or schema-light databases, because with Django a nice strict schema brings so little administrative overhead. I was going to have to add something about the change to the model anyway, even if it was only documentation, and having done that in one place, the other additional changes required by a relational DB with strong schema placed virtually no burden on me.
(Of course, things could be more complex on bigger apps, especially if the table is large or sharded. But then again, there's no reason why rolling out your DB change shouldn't be just as automated - it's only the 'waiting' stage that has to take longer for a simple change like adding a column. If the coding/work part is taking much longer than the above example, your tools probably need fixing or replacing.)
]]>