I decided to make an Android app for my Bible memory verse site (created using Django). The main motivation for the app is to get rid of the unnecessary and annoying address bars and status bars when using the site on an Android phone. The site is already designed to adapt to mobiles, so I'm not creating a native app — I just want a better way to access the web app from an Android phone.
I tried appsgeyser, but discovered they put adverts on your site, which I definitely don't want — this is an entirely free app, for a free (and ad-free) site.
My requirements are:
There are lots of pages and wizards with solutions for bits of these, but putting them together turned out to be harder — for example, it seems that the normal way of showing a progress bar for the whole window doesn't work if you've gone full screen.
Anyway, I've completed the learn scripture app (ha, my first Java app!), and thought I'd share the complete source code.
If you wanting a similar app, you are probably best creating your basic app structure using a wizard, but it is helpful to see a complete solution. The important bits are:
The right way to handle issues with untrusted data is:
Filter on input, escape on output
This means that you validate or limit data that comes in (filter), but only transform (escape or encode) it at the point you are sending it as output to another system that requires the encoding. It has been standard best practice since just about forever [citation required].
An alternative is “escape on input”: at the point that data enters your system, you apply a transformation to it to avoid a problem further down the line when the data is used.
It's come to my attention that some serious web developers (or at least, they take themselves seriously and are taken seriously by others) are still suggesting the practice of escape-on-input.
For example, with escape-on-input, to avoid XSS any data that enters your system has HTML escaping applied to it immediately, before your application code touches it.
I chose that example deliberately, because people are actually recommending it:
which, in turn, linked to a page by Rasmus Lerdorf recommending escape-on-input as a sensible way to deal with XSS. The page, admittedly, is describing a ‘toy’, a ‘no-framework PHP framework’, yet he does seem to be serious about the usefulness of escape-on-input.
The page is from 2006, and uses the pecl/filter extension, but the extension has since made it into core (PHP 5.2), and the docs for it suggest a configuration that is clearly intended for XSS prevention. As recently as 2008, and probably to this day, Lerdorf is still defending and recommending this approach, and it appears to be part of his reason for thinking that PHP templating doesn't need an autoescape mechanism.
Just as significantly, Etsy are using and recommending escape-on-input (slide 18 onward). As a very successful modern company using PHP, people will look up to them and copy them.
So, this approach, unfortunately, is popular amongst some, and I can't find a decent post explaining why it's such a terrible idea both in theory and practice. Here is my attempt. It should be applicable to almost any system and any language, although I'll mainly be using examples from web development.
First of all, escape-on-input is just wrong — you've taken some input and applied some transformation that is totally irrelevant to that data. If, taking our example, you have some data collected by HTTP POST or GET parameters, applying HTML escaping to it is a layering violation — it mixes an output formatting concern into input handling. Layering violations make your code much harder to understand and maintain, because you have to take into account other layers instead of letting each component and layer do its own job.
Doing things ‘right’ is very important, even if doing them ‘wrong’ seems to work and you are tempted to be dismissive of ‘theoretical’ concerns about purity etc. When you have to maintain code, you will be very glad if things are in the right place, and not full of hacks and surprises.
You have corrupted your data by default. The system (or the most convenient API) is now lying about what data has come in. As you have applied a transformation to the data itself, the layering violation is not an isolated problem in one part of the code, but infects every part of your code, especially if you store the corrupted data in a database.
Your data is everything. As I read recently, “data matures like wine, applications like fish”. You can always rewrite your application, but if you corrupt your data, you've done the worst thing you can to your system.
This is exacerbated by the fact that many encodings are one-way — you cannot losslessly or unambiguously convert them back. If at a later point you need the original data, you might be in a pickle.
Escaping your data for one output backend will only deal with that output. A typical web app might deal with at least the following backends, which have different characters that are dangerous, and have different requirements for dealing with them:
Any number of others could be added, and all could have security implications. Using escape-on-input will only fix one of these (apart from happy coincidences where it might fix more than one), but for the others you will still need a sensible solution to the problem. Why not have a sensible solution for all of them?
Escaping on input will not only fail to deal with the problems of more than one output, it will actually make your data incorrect for many outputs.
Suppose you decide to do HTML escaping, and someone enters Jack & Jill as a title for something. Your escape-on-input turns this to Jack & Jill and that goes in the DB. Suppose you want to email people and put this title in the subject line. You now have to apply the reverse transformation to get a sensible subject line in the email, and you have to remember to do this for every output that is not HTML.
Sometimes, the bug is significantly more annoying than an email with an incorrect title.
You also have daft bugs like the fact that doing a search on that field for the string ‘amp’ (or ‘quot’, ‘apos’, ‘lt’, ‘gt’ etc. or any substrings) will get various false matches.
I have seen some people respond to this by saying “it's better to have the occasional double-encoding bug or incorrect query result than an XSS exploit". Well, first, that depends on your business. XSS is a problem because it costs time and money, and so does corrupting your data. Many people have data that actually matters, and corrupt data is a big deal, and much harder to cope with than an XSS bug, because data lives on and on, while your code can get replaced easily.
Second, this decision affects frameworks that are used to handle data of all kinds, and the decision affects the entire code base of your application and beyond, as described below. Data-handling frameworks that work on the assumption that your data is not important are insanity. If the foundations be destroyed, what can the righteous do?
Third, it's entirely unnecessary. XSS is not hard to fix given decent programming tools.
At what point does data ‘enter’ your system?
It might sound like a simple question, but it's tricky in reality, and I'll illustrate using an HTTP request.
In most web apps, the GET and POST parameters are your ‘raw input’. However, using most normal web framework APIs, data in GET and POST parameters has already been interpreted. The ‘raw’ data is really the bytes that make up the HTTP request, which typically will use URL encoding for GET query parameters and a choice of encodings for POST data (URL encoding or MIME multipart attachment format).
The framework may also do another level of decoding — interpreting the series of bytes as a series of unicode code points.
Both parts of this initial transformation makes sense and are appropriate, because they are reversing the encoding already applied to the data by the protocol involved. The web browser takes the data you type in — unicode code points — and applies a series of transformations to it, according to the HTTP protocol, and your web framework reverses these to get the data back.
Now, if you want to avoid XSS problems, you have to apply the escaping after this initial decoding has been done. But this highlights another possibility. What if the data requires further decoding before you get the ‘real’ raw data? For example, some data might be sent base64 encoded for a variety of reasons, or any other type of encoding.
This extra level of encoding gives two problems:
your automatic HTML escaping may have corrupted the encoded data so that it now cannot be decoded. For example, you had a GET parameter that held a URL, which itself had parameters in the query string:
GET /foo?bar=1&url=http%3A%2F%2Fexample.com%2F%3Fx%3D1%26y%3D2 HTTP/1.1
Your framework's HTTP handling will produce a query dictionary that looks something like the following:
{"bar": 1,
"url": "http://example.com/?x=1&y=2"
}
But your automatic escaping turns that into:
{"bar": 1,
"url": "http://example.com/?x=1&y=2"
}
If you want to extract the 'y' parameter from 'url', you are stuck. You can't correctly interpret the data in the 'url' parameter, because it has been corrupted. You're going to have to re-decode the input, and you might not even notice this problem.
Even if the data comes through your automatic escaping unscathed (e.g. base64 under HTML escaping), or you can undo the corruption and get it properly decoded, after decoding you will have to manually apply HTML escaping to make it match all the other automatically escaped data. If you don't, you've potentially got a bug and an XSS exploit.
So your automatic escape-on-input has missed data, and this happens because you can't really define the point at which the data has ‘entered’ your system and needs the escaping applied.
This problem means that the escape-on-input approach is inherently flawed and cannot be fixed. You just have to patch it up on a case-by-case basis, which is exactly what escape-on-input is supposed to avoid.
And then, what about other sources of data — data on the file system, in a cache etc. Are these entry points? Well, it depends on how the data was put there. You have to manually follow this all the way through your app; get it wrong and you've got double escaping bugs or security flaws.
(By contrast, escape on output always works, because you apply it at the point where you know it is needed — in the backend that knows the escaping rules.)
Other systems putting data into your database, or getting data out, have to abide by your data transformation rules.
These systems might have nothing to do with your primary domain (e.g. a web site). Making them understand and obey rules that have nothing to do with the data itself is insanity and extremely short sighted.
You can't deal with this problem when you come to it, because you don't have to just fix your code, you've got to fix all your data too, and by the time you cross this bridge you might have a lot of data and might need a very delicate database migration to get it right. The data may even have escaped your control (e.g. been copied into other systems), or backwards compatibility concerns might stop you from making the change you need to make.
Within your main application, the decision to escape on input affects your whole code base.
If you want to use any libraries, you need to make sure that they are using all the same assumptions that you have in your main code base.
For example, if you've got a form/widget library in your web app, it will very often need to echo user input back to them in the case of a form that has validation errors. This library has to know if you already escaped the input.
Writing the library to work in two modes is asking for trouble. Rather, you need it to have been written from the beginning to assume the same escaping rules.
This kills code re-use — you can only use code that assumes the same input escaping — or it means that you will end up with tons of bugs due to incompatibilities between the assumptions made in your application code and the library.
Essentially, this is the problem of a global configuration setting, but worse since it affects the operand of your entire application (the data going through it), not just the functionality of various operators.
The confusion caused by the above is likely to increase security problems. “Keep It Simple, Stupid” remains a very good maxim for developers.
To continue an example used above: you want to send an email with some data that has already been HTML escaped, and so you need to unescape the data to avoid emails with the subject "Jack & Jill" when the user entered "Jack & Jill". You decide it's not sensible for the mail sending functions to do this internally, (or maybe they're provided by a 3rd party who made that decision for you), so the calling code does the unescaping.
You later decide to switch to HTML emails, and the developer who implements it thinks that since data is already escaped, there is no problem including it without extra escaping in the body of the HTML email, leading to a vulnerability (not classic XSS in this case, but still a problem).
There is also the example I gave above where an extra layer of encoding/decoding in the raw data makes it likely you'll forget to apply the escaping.
The confusion caused by escape-on-input means your entire code base becomes a potential source not only of double-escaping bugs but of security problems as well.
Thankfully, we don't just have to rely on the above analysis to conclude that escape-on-input is a terrible idea. PHP, always willing to help when it comes to “examples of how not to do it”, provides us with a perfect case study.
PHP used to have a feature called magic quotes. It was an escape-on-input feature that escaped single quotes (') with backslashes. This was to protect you from SQL injection attacks, by making the data safe for interpolation into a SQL query.
This caused all kinds of problems.
First, if you are not first passing something through a database, and using string interpolation to build up SQL queries, you have to remember to strip those slashes using the function stripslashes().
If you don't, you get double encoding. It looks like \'this\', you\'ve almost certainly seen it across the web, though it seems we\'re thankfully past the worst of it.
Second, even if you remember, you've added some hideous cruft to your code. In the bit of code which is handling form validation (and is therefore echoing user input back to the user without the database being involved), you've got these bizarre stripslashes() calls. What on earth does ‘reverse transforming a string for SQL statement preparation’ have to do with the task of input validation?
Third, it turns out that different databases need different escaping mechanisms to do things fully correctly. So you now have to do stripslashes() on data even if you are passing it to a database using string-interpolated queries!
Then, since the above problems are common (building up SQL queries by string interpolation was always a bad idea, and very often you pass on the data to outputs that don't want SQL escaping at all), it's desirable to have a way to turn this behaviour off completely.
To handle this, there is a php.ini setting to turn it on/off.
And there were more complications, for example:
And so now you've got even more problems. Since these are global settings, you can't have library code mess with them, since other code might set the global to a different value or assume a certain value.
You could try making all code detect the current setting and have different code paths depending on the result. This works very badly — having multiple code paths is a recipe for code duplication and bug proliferation. It's extremely easy to forget to do it, or get one of the paths wrong, since you will likely only test one configuration value and one set of code paths in reality.
Alternatively, you can make one bit of code responsible for fixing the setting to a sensible value (the only one being 'off'), and then make all code assume that from then on. (If you can't turn it off, you can use the code included here as a horrible kludge to reverse it's behaviour).
Eventually, this final approach was the one taken by all significant projects. Turn the whole feature off, and assume it is off from then on. (Which means the feature is useless, of course).
And of course, thankfully, the PHP developers realised that this entire thing was a huge mistake that caused nothing but a vast amount of confusion and bugs, and removed the whole thing for good in PHP 5.4.
Magic quotes, as eevee put it, were “so close to secure-by-default, and yet so far from understanding the concept at all.”
To digress for a moment: we keep getting told that PHP is improving, and the community has learnt from its mistakes. Unfortunately it seems the leaders in the community are bent on recreating old mistakes.
According to Lerdorf, the much newer PHP 'filter' extension is “magic_quotes done right”. But it still suffers from almost all the problems described here, for all the reasons described. Global HTML escaping on input is essentially the same as magic quotes, and just as tragically bad.
In researching for this post, I came across this ticket for Elgg, an open source social networking engine. Just read through the ticket and see the mess they are in. It's clear they strongly regret the decision they made to escape-on-input, and, in their own words, have created “horrendous” problems for themselves, especially as their application has grown to include other interfaces such as JSON REST APIs.
However, fixing it is very hard. They have to coordinate many changes across their code base with a big database migration. If data has leaked from the databases and tables they control into other systems, such as denormalised tables, other databases, caches etc., or if there is other code by third parties that makes the old assumptions about encoded data, they are in even more of a pickle. And both of those things are probably inevitable in something like an open source framework, which is designed for other people to build on and extend.
This is the pain that comes from mixing input handling and output encoding, and from corrupting the data in your database.
According to their security presentation, Etsy are using escape-on-input for XSS protection.
They claim that this is a much more secure option, as it is secure by default. (They do note, however, the problem with input that is encoded in some other way, like base64, so they are aware of the problems.)
Their presentation goes on to describe an elaborate system for detecting and fixing XSS attacks (the slides don't give enough detail for me to understand what exactly they are doing, but it's clearly a lot of work).
And their system does indeed catch XSS bugs in the wild and allow them to fix them within hours.
Wait, what?
They've corrupted their database by doing escape-on-input, they've inflicted themselves with all the development pain described above, and they've still got XSS bugs?
Granted, they've got impressive ways of dealing with these problems. But it's like virus checkers on voting machines. Advanced ways of dealing with problems that shouldn't even be possible tells you that you are doing it wrong. They've become very fast at re-tying their shoelaces, instead of working out how to tie shoelaces so they don't come undone.
They claim that with escape-on-input, XSS problems are now greppable, but it doesn't sound like it. If they were, code audits would be a massively more efficient way to find XSS problems than the methods they are using.
The main problem is almost certainly that they are using an output system for HTML that doesn't do HTML escaping by default (I'm guessing they are using PHP as their template language). If the backend that deals with HTML actually deals with HTML then you eliminate the vast majority of these problems overnight.
I'm willing to bet that large sites that use Django (or other frameworks that have basically solved the XSS problem by HTML escaping on output by default) don't have teams and automated systems dedicated to this problem, and don't need them. In Django apps, XSS problems are greppable - you grep for mark_safe in Python and the |safe filter in templates (and then, obviously, you may have to recursively grep for any functions that call mark_safe on inputs). Since all data which isn't ‘mark_safe()’d gets escaped by the templating engine, and all HTML comes out of the template engine, that's basically all you need to do.
How did this happen to Etsy?
Are the Etsy devs stupid? I suspect not. Etsy is clearly doing well, and I imagine they have enough money to hire top-notch developers. Some of their careers pages show they are happy using a variety of languages and technologies, and their engineering blog seems to be sane and competent. Even their security presentation showed considerable ingenuity and technical ability in dealing with security problems (in entirely the wrong way, unfortunately, but still impressive).
I doubt they are low quality developers. Rather, I suspect that use of PHP has addled their brains. They have become far too accustomed to working in an environment in which insanity reigns — an environment in which the less than operator pretends to work correctly with strings but it's just a trap.
When I programmed in a Windows environment, I theorised that use of Windows itself contributed to the poor quality of the programming in the code base, and the fact that developers thought nothing or writing tons of tedious code. Because Windows was so unscriptable, I imagined that Windows programmers developed a high tolerance for tedium and repetition, which is exactly the opposite of qualities needed by a programmer to make a computer do everything efficiently and reliably. (Since then, I've found that Sturgeon's law was probably a better explanation for the quality of the code, but I still think the fundamental idea applies).
With PHP, the fact that it comes with a template language that is simply not fit for purpose — because it doesn't do HTML escaping by default, or even easily — has somehow made the Etsy developers believe that it is normal to struggle with XSS, that it is perfectly reasonable that even after taking the drastic action of corrupting their entire database by HTML escaping it, they should still need elaborate XSS-catching systems.
Instead of trying to fix XSS, they should just fix it. Like this in Django. Or this in Turbogears and Jinja. Or this in Yesod. Or even this in PHP (though due to limitations of the language you won't be able to have the convenience of things like mark_safe in Django). But living with an environment of pain and madness makes you think that it ought to be hard.
Right the way up to Rasmus Lerdorf at the top, many people in the PHP community live with the insanity of their tools, and add more insanity to cope with it, rather than fix their tools or choose better ones.
Bashing other people is fun, but when I do so I always try to get something more valuable out of it by using the opportunity to examine myself. The problem I discussed in the last section (which is just a manifestation of the broken windows theory) applies to other communities, and I'll attempt to apply it to the Python community.
Refusing to live with stupidity is one of the reasons that Python 3 is really important.
Python 3 does not represent a massive leap forward in terms of additions to the language. Mainly it just fixes a bunch of mistakes in Python 2, and introduces a whole lot of backwards incompatibilities in the process. One of the biggest is unicode/bytes. Python 2 was stupid here — it went directly against the Zen of Python, and said “in the face of ambiguity about what encoding to use, guess.” This caused a world of pain.
Now, you can work around it in most cases by some sensible conventions and a certain amount of discipline. You can also cope with the fact the "a" < 1 doesn't raise an exception. You can live with next() being a method in the iterator protocol, when it should be a method called __next__() and a builtin function next(). You can live with the fact that print is a totally unnecessary keyword, since it should just be a builtin function. You can get used to the fact that class Foo: means something subtly but significantly different from class Foo(object):. You can work around or ignore dozens of other little niggles, gotchas and inconsistencies.
But all the while, you are training yourself to tolerate stupidity, inconsistency and brokenness. Removing these warts is really important, and worth all the pain of the migration. The alternative is for Python to become the next PHP.
On top of these things, there are other types of brokenness in Python that people in the community seem less willing to acknowledge or tackle. For some of these I think we need exposure to completely different languages — languages where you can spawn thousands of ‘threads’ easily and get performance benefits, for example, or languages where you can write code that is both very high level and extremely fast. If we live entirely with Python and its set of limitations, we'll think that those problems are normal and unavoidable.
Updates:
I needed to add a boolean field to a model. For many web apps, this typically involves:
Using Django, from a cold start (no editor/IDE open), this just took me 1 minute 45 seconds of work for steps 1 - 5, and an additional 45 seconds waiting for step 6, total 2 minutes 30 seconds, and I wasn't rushing.
Step 1 is a one line code addition. Pretty much everything else can and should be generated automatically.
Step 2 is taken care of by a one line command using South, as is step 3 and the database part of step 6 (which is run de-rigueur from my deployment scripts).
Step 4 is taken care of by Django's admin, which introspects the model and generates the right form for you.
This is one of the reasons I love Django. It's not so much the time it saves, although that is pretty awesome, it's the tedium it saves.
This is also one of the reasons I'm not very tempted by schema-less or schema-light databases, because with Django a nice strict schema brings so little administrative overhead. I was going to have to add something about the change to the model anyway, even if it was only documentation, and having done that in one place, the other additional changes required by a relational DB with strong schema placed virtually no burden on me.
(Of course, things could be more complex on bigger apps, especially if the table is large or sharded. But then again, there's no reason why rolling out your DB change shouldn't be just as automated - it's only the 'waiting' stage that has to take longer for a simple change like adding a column. If the coding/work part is taking much longer than the above example, your tools probably need fixing or replacing.)
]]>A friend of mine has developed a command-line utility for managing many WordPress installations - WordShell.
I don't use WordPress myself, but having been totally converted to tools like fabric for managing remote sites, I'm sure this tool would be invaluable if you need to manage WordPress sites.
]]>Sentry really ought to use UDP, not TCP, because you don't want logging functionality to stall or even slow down your main application. At the moment, it doesn't support that, although there have been some promising commits.
For my usage (a web application), this means that you can really only use Sentry for logging exceptions, and not for anything less important.
However, there are some alternatives to UDP that make Sentry usable for more than exceptions. You could use a queue process like Celery or RabbitMQ (apparently what they use at Disqus).
A more light weight alternative, however, is an asynchronous client that does its work in the background, and so doesn't block your web server thread.
There is some hopeful looking code in raven.contrib.async, but unfortunately it currently has a critical bug (Raven 1.4.2).
However, using that code I cobbled together my own, and this one subclasses DjangoClient, which is what I need:
from raven.contrib.django import DjangoClient from raven.contrib.async import AsyncWorker class AsyncDjangoClient(DjangoClient): """ This client uses a single background thread to dispatch errors. """ def __init__(self, *args, **kwargs): self.worker = AsyncWorker() super(AsyncDjangoClient, self).__init__(*args, **kwargs) def send_sync(self, **kwargs): super(AsyncDjangoClient, self).send(**kwargs) def send(self, **kwargs): self.worker.queue.put_nowait((self.send_sync, kwargs))
Then you need to set SENTRY_CLIENT in your settings to point to this class.
(If you're not using Django, you should be able to do something similar.)
This is working fine for me - I can now enable the Sentry 404 middleware and not see any slowdown on my app, as opposed to the synchronous client which was slowing down 404 responses massively because my Sentry server is not on the same box as my main web app.
I should say this is use at own risk - the AsyncClient in Raven is undocumented as well as broken, so I don't know if it is considered a sensible approach or not!
]]>I recently used django-fiber for a simple project.
It's a nice CMS, a bit more lightweight than django-cms, with a slightly slicker frontend editing experience, and a bit easier when it comes to sharing content between different pages.
I found however, that it was doing a rather large number of queries to render pages, and some that pulled back lots of data, especially when you were logged in (which enables frontend editing).
So, I set to work and created a query count reduction branch. Below are the results so far, and some lessons.
I used the example django-fiber project for testing my changes (as well as my own project), and tried them out on both the home page and a more deeply nested page.
As you can see, there was a lot of unnecessary work. Django makes it extremely easy to generate database queries, which is both a good and bad thing, and here it is bad.
Here are some lessons for avoiding this unnecessary work:
Use django-debug-toolbar when developing, right from the beginning.
Seriously, use django-debug-toolbar.
Use django-debug-toolbar and keep it turned on. And look at it regularly. OK, point made.
This project was quite a lot easier to fix than django-cms, because it is much newer, and so has less code, and - more importantly - far fewer compatibility issues to worry about with 3rd parties who depend on certain features.
You should think about big O scaling issues fairly early, because you can easily put yourself into a situation where things are hard to fix, due to:
For example, django-fiber has a concept of 'current' pages (pages which would form part of the bread crumb for the page you are on) and, in addition to the obvious ones (the 'ancestors' of the active page in the tree of pages), it has a feature which allows any page in the database to be a candidate 'current page' for any other page, based on a regex field. And so you have to check all these pages when rendering any page.
This does not scale well. Thankfully, it's not too much of a problem if you don't use this feature, since you can do DB level filtering to eliminate most of the pages as potential candidates for this.
But if you do use it, you have a scaling problem - for every time you use it, then the amount of work you've got to do to render any page increases. (And, if the DB filtering isn't efficient, you may still be paying an increasing penalty for every page added to the system even if you never use the feature).
EDIT: I should have mentioned on the positive side that django-fiber has obviously given thought to general scaling issues, and used django-mptt for the tree structure of their Page model. This made it relatively easy to fix the 'show_menu' template tag to do everything in 2 queries. Otherwise it would have been a nightmare.
Create tests with assertNumQueries, and add them fairly early. You'll then be alerted to performance regressions that affect scaling.
The tests will need to include tests for whole pages, not just bits, if you are going to analyse Big O scaling correctly, because a set of objects might be retrieved from the DB efficiently, but each one can easily make more database calls when it is actually used.
Some more specific lessons:
Read and understand the Django docs on optimizing DB access.
Understand when QuerySets are evaluated (which is part of the above, but worth mentioning).
There were some examples in the fiber code of really inefficient use of QuerySets, e.g. if obj in MyModel.objects.filter(foo=bar) (example). This code will load all the MyModel records with foo=bar and create MyModel instances. It does not do MyModel.objects.filter(foo=bar, pk=obj.pk).exists(), and it certainly does not do if obj.foo == bar.
Although it could do the first of these, Django deliberately does not make this optimisation. Django's QuerySets are deliberately dumb. The rule of thumb is this: a QuerySet will only do one query - the query you have told it to do using methods such as filter(), order_by() and slicing. It does not respond 'intelligently' to any Python builtins such as len(), bool() or the in operator - these simply force the QuerySet to be evaluated. There is efficiency in the way it uses its cache, but there is no 'cleverness'.
(BTW, I remember the discussions we had about this a long time ago on django-dev, and I'm convinced with hindsight we made the right discussion. It might seem like a nice opportunity to do some clever queries, but since the cleverness does not extend to actual mind-reading, it will fail and it will get in the way. For instance, in the template example in the docs, cleverness on the part of QuerySet would only result in unnecessary work, and it would be much harder to get it to do the right thing).
Don't do queries when you've already got the information you need. There were multiple examples of this.
This will often mean that you will have some duplication of logic - a manager method that defines some filtering for a set of objects, and an instance level method which answers the question 'do I belong to that set of objects'.
This duplication is often unavoidable if you want any kind of performance. Just put a comment on the two methods indicating that they need to be synced with each other, and never use one when the other is what you need.
(I can conceive of a cleverer system that would allow one of these to be automatically created from the other, but it would be limited to the subset of what can be expressed in both SQL and Python, and it doesn't exist in Django).
When defining special values that you need to search for, ensure that you can do efficient DB filtering.
Beware this common bug - writing:
if not foo:
when what you mean is:
if foo is not None:
These are completely different. If None is being used as a sentinel value, the first will treat things like the empty list or empty dictionary incorrectly. If you mean if foo is None or if foo is not None, always write just that, never assume that no other false-y values will be passed in.
This is not just a performance-related bug, but it can cause a massive amount of repeated work where None is a sentinel value meaning "the work has not been done yet", which is very common. This bug resulted in dozens of unneeded queries (including database UPDATEs being made for every request) in django-fiber.
Note: I now recommend using gunicorn for deployment, various things about this need changing for that setup
With the new 256 Mb limit for WebFaction Shared accounts, at a squeeze you could now host up to 10 Django apps on a single account without upgrades, for a very low price.
This was a game changer for me, and I realised I need to be able to start a new Django project quickly, including setup on WebFaction, either to be able to host a small client website for them, or to generate a demo.
There are a few parts required, one of which is a fabfile for fabric deployment. Since this is often a sticking point for other people, I created one that should be generic enough for most Django apps that run on shared hosting:
It requires some customisation, and is fairly simplistic, but should be a good starting point for more complex setups.
For quick project setup, there are other things you need:
#!/usr/bin/env python import sys import xmlrpclib import random import string def main(domain, shortname): server = xmlrpclib.ServerProxy('https://api.webfaction.com/', verbose=0) session_id, account = server.login('MYUSERNAME', 'MYPASSWORD') print("Creating apps on webfaction") server.create_app(session_id, shortname + "_django", "djangotrunk_mw33_27", False, "") server.create_app(session_id, shortname + "_static", "static_only", False, "") server.create_app(session_id, shortname + "_usermedia", "static_only", False, "") print("Creating database on webfaction") l = list(string.letters + string.digits) random.shuffle(l) password = ''.join(l[0:16]) db_name = "MYUSERNAME_%s" % shortname print("Remote database: %s" % db_name) print("Password: %s" % password) server.create_db(session_id, db_name, "postgresql", password) print("Creating domain on webfaction") server.create_domain(session_id, domain) server.create_domain(session_id, domain, 'www') print("Creating website entry on webfaction") server.create_website(session_id, shortname, 'MY.IP.ADD.RESS', False, [domain, 'www.' + domain], (shortname + '_django', '/'), (shortname + '_static', '/static'), (shortname + '_usermedia', '/usermedia')) if __name__ == '__main__': if len(sys.argv) != 3: print "Usage: create_webfaction_site.py $DOMAIN $SHORTNAME" sys.exit(1) main(sys.argv[1], sys.argv[2])
Scripts for setup of your local database. Mine looks like this:
#!/bin/sh sudo -u postgres psql -U postgres -d template1 -c "CREATE DATABASE $1;" sudo -u postgres psql -U postgres -d template1 -c "CREATE USER $1 WITH PASSWORD 'foo';" sudo -u postgres psql -U postgres -d template1 -c "GRANT ALL ON DATABASE $1 TO $1;" # Need create DB privileges to run tests. sudo -u postgres psql -U postgres -d template1 -c "ALTER USER $1 CREATEDB;" echo "Local database: $1" echo "Password: foo"
A script to bind it all together. Mine takes just two arguments - a short name for the project and domain name. A lot is fairly specific to my environment, so I won't share it, but the sake of completeness, I'll list what it does:
runs mkvirtualenv and creates some directories for the project
runs django-admin.py startproject
copies in the fabfile above, and modifies some settings
creates a basic requirements.txt
runs the local database creation script
prints a reminder for some Django settings.py that would be hard to fix automatically, and anything else which is still to be done, which includes: * Fixing the WebFaction Apache conf to point to myproject/wsgi.py * Fixing the WebFaction Apache start script to include the virtualenv generated by the fabfile
Some of these things could be done automatically in the future
Having this written allows me to be massively more efficient with very small sites, so it is now feasible to use something like django-cms for a 5 page site, because the setup costs are so low.
]]>Yesterday I needed to play Family Fortunes (a.k.a. Family Feud) at our church youth group. Our church doesn't have internet connection, so I needed something that would work offline.
There were quite a few versions you could download — for a fee. But as well as the price, getting the computer to do it all is often not so much fun as something with humans more in control, and you're limited to what the computer can do — not easy to fix when you type something in wrong.
So in the end, with a couple of hours to go before the club started, I reckoned I could code up a simple implementation of the score board in a web browser. I scraped some questions from here, used a bit of Python to parse and convert to JSON, then used jQuery and HTML5 audio to get something that works pretty well for our purposes and doesn't need an internet connection.
Usage and source code on bitbucket. In brief:
We set up my laptop as the screen, with some speakers for increased volume, and one of the other leaders used a USB keyboard to control it and decide on correct answers, while I was the game show host. It ended up being pretty fun, and I can imagine re-using on a church holiday etc, with a projector perhaps.
I was also impressed by just how much you can do in a browser these days, with so little time and effort. Recently the brilliant flot library made me realise the same thing — more and more, web development is beating desktop development in terms of APIs, speed of development, ease of development, and even the performance of the result. I very quickly produced a great set of graphs gathering some stats on a website — both attractive and interactive — and realised that even with many of the easiest charting solutions for the desktop, I would not have been able to produce anything like the results.
]]>It is common practice in develpment to use a database that is very similar in content to the real data. It is tempting to download a copy of the production database for smaller apps (or a trimmed version). The problem is that this can lead to having copies of sensitive customer data on development machines and other placers (like automatic backups). This Django app provides an easy and customizable way to anonymize data in your database, while leaving the structure intact and the data looking sensible.
Using some manage.py commands, it will introspect your models to generate basic 'anonymizers' for you, attempting to guess the type of data in fields based on Django field type and field name (e.g. 'email', 'name', 'address' etc), and generate appropriate fake data. These guesses will probably need to be tweaked with knowledge of what is actually stored in your fields. You then run the anonymizers to change your data.
It is currently in a working state, although not all model fields are supported. It has support for the 'unique' constraint, to avoid database IntegrityErrors, and the 'max_length' attribute, but does not yet support 'unique_together'.
Download: PyPI
Source: bitbucket
Bugs: bitbucket issues
The README has fairly good documentation, I don't know when I'll get around to writing some proper docs!
It uses the Faker package by Dylan Clendenin (thanks!).
UPDATE: Fuller docs available now
]]>