All Unkept
Posted in: Django  —  30 November 2010

Fuzzy testing with assertNumQueries

The yet-to-be-released Django 1.3 has an assertNumQueries method which will allows you to simply specify the number of queries you expect.

Sometimes, however, specifying the exact number of queries is overkill, and makes the test too brittle:

  1. I might not actually care if some global change to my app means I get one more request on a whole bunch of non-critical views, for example.

  2. Caching of various queries might mean that the number of queries in a block of code can vary by a few. The correct way to fix this is to track down all the queries that are being cached, and to ensure the cache is populated before you start counting the queries. But this can be tedious in practice.

Sometimes, I simply want to ensure that a change to a template or view function that deals with a set of N objects hasn't gone from O(1) queries to O(N) queries. I want to set up a test in which N is some reasonably big number, and then test that the number of queries is less than N, and reasonably close to what it currently is.

With an appropriate subclass of int, I can still use assertNumQueries to do this:

class FuzzyInt(int):
    def __new__(cls, lowest, highest):
        obj = super(FuzzyInt, cls).__new__(cls, highest)
        obj.lowest = lowest
        obj.highest = highest
        return obj

    def __eq__(self, other):
        return other >= self.lowest and other <= self.highest

    def __repr__(self):
        return "[%d..%d]" % (self.lowest, self.highest)

class MyFuncTests(TestCase):
    def test_1(self):
        with self.assertNumQueries(FuzzyInt(5,8)):
            my_func(some_args)

This will ensure that the number of queries is between 5 and 8 (inclusive). The error message is slightly confused if the test fails, but it still has all the info you need:

AssertionError: 10 != [5..8] : 10 queries executed, 8 expected

This trick also works with the count parameter to assertContains.

Comments §

§ On 30 November 2010, Mikhail Korobov wrote: 967
The landed assertNumQueries can't be used as a decorator unfortunately.

FuzzyInt is a very clever trick!
Thanks for the correction, applied. — luke

§ On 27 December 2010, Wes Winham wrote: 976
Love this idea. I don't yet have a really good way of testing and tracking performance and in my experience, a big majority of our performance problems have been a result of too many database queries.

If we used this kind of fuzzy validation to ensure that we're getting the big-o query performance that we think we're getting, it would go a long way towards eliminating those problems where a page is snappy with test data and then falls down with real data because of scale.

Add comment

Format:

  • Javascript has to be on to get past my spam protection, and cookies, and there is a delay, sorry for any inconvenience!
  • I reserve the right to moderate comments.