Luke Plant's home page (Posts about Django)

Help my website is too small

2025-12-19T13:45:33Z

A jobs web site I belong to just emailed me, telling me that some of the links in my public profile on their site are “broken” and “thus have been removed”.

The evidence that these sites are broken? They are too small:

https://www.djangoproject.com/: response body too small (6220 bytes)

https://www.cciw.co.uk/: response body too small (3033 bytes)

The first is the home page of the Django web framework, and is, unsurprisingly, implemented using Django (see the djangoproject.com source code). The second is one of my own projects, and also implemented using Django (source also available for anyone who cares).

Checking in webdev tools on these sites gives very similar numbers to the above for the over-the-wire size of the initial HTML (though I get slightly higher figures), so this wasn’t a blip caused by downtime, as far as I can see.

Apparently, if your HTML is less than 7k, that obviously can’t be a real website, let alone something as ridiculously small as 3k. Even with compression turned up all the way, it’s clearly impossible to return more than an error message with less than at least 4k, right?

So please can Django get it sorted and add some bloat to their home page, and to their framework, and can someone also send me tips on bloating my own sites, so that my profile links can be counted as real websites? Thanks!

Keeping things in sync: derive vs test

2024-06-28T10:15:00+01:00

An extremely common problem in programming is that multiple parts of a program need to be kept in sync – they need to do exactly the same thing or behave in a consistent way. It is in response to this problem that we have mantras like “DRY” (Don’t Repeat Yourself), or, as I prefer it, OAOO, “Each and every declaration of behaviour should appear Once And Only Once”.

For both of these mantras, if you are faced with possible duplication of any kind, the answer is simply “just say no”. However, since programming mantras are to be understood as proverbs, not absolute laws, there are times that obeying this mantra can hurt more than it helps, so in this post I’m going to discuss other approaches.

Most of what I say is fairly language agnostic I think, but I’ve got specific tips for Python and web development.

The essential problem

To step back for a second, the essential problem that we are addressing here is that if making a change to a certain behaviour requires changing more than one place in the code, we have the risk that one will be forgotten. This results in bugs, which can be of various degrees of seriousness depending on the code in question.

To pick a concrete example, suppose we have a rule that says that items in a deleted folder get stored for 30 days, then expunged. We’re going to need some code that does the actual expunging after 30 days, but we’re also going to need to tell the user about the limit somewhere in the user interface. “Once And Only Once” says that the 30 days limit needs to be defined in a single place somewhere, and then reused.

There is a second kind of motivating example, which I think often crops up when people quote “Don’t Repeat Yourself”, and it’s really about avoiding tedious things from a developer perspective. Suppose you need to add an item to a menu, and you find out that first you’ve got to edit the MENU_ITEMS file to add an entry, then you’ve got to edit the MAIN_MENU constant to refer to the new entry, then you’ve got to define a keyboard shortcut in the MENU_SHORTCUTS file, then a menu icon somewhere else etc. All of these different places are in some way repeating things about how menus work. I think this is less important in general, but it is certainly life-draining as a developer if code is structured in this way, especially if it is difficult to discover or remember all the things that have to be done.

The ideal solution: derive

OAOO and DRY say that we aim to have a single place that defines the rule or logic, and any other place should be derived from this.

Regarding the simple example of a time limit displayed in the UI and used in the backend, this might be as simple as defining a constant e.g. in Python:

from datetime import timedelta

EXPUNGE_TIME_LIMIT = timedelta(days=30)

We then import and use this constant in both our UI and backend.

An important part of this approach is that the “deriving” process should be entirely automatic, not something that you can forget to do. In the case of a Python import statement, that is very easy to achieve, and relatively hard to get wrong – if you change the constant where it is defined in one module, any other code that uses it will pick up the change the next time the Python process is restarted.

Alternative solution: test

By “test”, I mean ideally an automated test, but manual tests may also work if they are properly scripted. The idea is that you write a test that checks the behaviour of code is synced. Often, it may be that for one (or more) instances that need the behaviour will define it using some constant as above, let’s say the “backend” code. Then, for one instance, e.g. the UI, you would hard code “30 days” without using the constant, but have a test that uses the backend constant to build a string, and checks the UI for that string.

Examples

In the example above, it might be hard to see why you want to use the fundamentally less reliable, less automatic method I’m suggesting. So I now have to show some motivating examples where the “derive” method ends up losing to the cruder, simpler alternative of “test”.

Example 1 - external data sources

My first example comes from the project I’m currently working on, which involves creating CAM files from input data. Most of the logic for that is driven using code, but there are some dimensions that are specified as data tables by the engineers of the physical product.

These data tables look something like below. The details here aren’t important, and I’ve changed them – it’s enough to know that we’ve are creating some physical “widgets” which need to have specific dimensions specified:

Widgets have length 150mm unless specified below
Widget id	Location	Length (mm)
A	start	100
A	end	120
F	start	105
F	end	110

These tables are supplied at design-time rather than run-time i.e. they are bundled with the software and can’t be changed after the code is shipped. But it is still convenient to read them in automatically rather than simply duplicate the tables in my code by some process. So, for the body of the table, that’s exactly what my code does on startup – it reads the bundled XLSX/CSV files.

So we are obeying “derive” here — there is a single, canonical source of data, and anywhere that needs it derives it by an entirely automatic process.

But what about that “150mm” default value specified in the header of that table?

It would be possible to “derive” it by having a parser. Writing such a parser is not hard to do – for this kind of thing in Python I like parsy, and it is as simple as:

import parsy as P

default_length_parser = (
  P.string("Widgets have length ") >>
  P.regex(r"\d+").map(int)
  << P.string("mm unless specified below")
)

In fact I do something similar in some cases. But in reality, the “parser” here is pretty simplistic – it can’t deal with the real variety of English text that might be put into the sentence, and to claim I’m “deriving” it from the table is a bit of a stretch – I’m just matching a specific, known pattern. In addition, it’s probably not the case that any value for the default length would work – most likely if it was 10 times larger, there would be some other problem, and I’d want to do some manual checking.

So, let’s admit that we are really just checking for something expected, using the “test” approach. You can still define a constant that you use in most of the code:

DEFAULT_LENGTH_MM = 150

And then you test it is what you expect when you load the data file:

assert worksheets[0].cell(1, 1).value == f"Widgets have length {DEFAULT_LENGTH_MM}mm unless specified below"

So, I’ve achieved my aim: a guard against the original problem of having multiple sources of information that could potentially be out of sync. But I’ve done it using a simple test, rather than a more complex and fragile “derive” that wouldn’t have worked well anyway.

By the way, for this specific project – we’re looking for another contract developer! It’s a very worthwhile project, and one I’m really enjoying – a small flexible team, with plenty of problem solving and fun challenges, so if you’re a talented developer and interested give me a shout.

Example 2 - defining UI behaviour for domain objects

Suppose you have a database that stores information about some kind of entity, like customers say, and you have different types of customer, represented using an enum of some kind, perhaps a string enum like this in Python:

from enum import StrEnum


class CustomerType(StrEnum):
    ENTERPRISE = "Enterprise"
    SMALL_FRY = "Small fry"  # Let’s be honest! Try not to let the name leak…
    LEGACY = "Legacy"

We need to a way edit the different customer types, and they are sufficiently different that we want quite different interfaces. So, we might have a dictionary mapping the customer type to a function or class that defines the UI. If this were a Django project, it might be a different Form class for each type:

CUSTOMER_EDIT_FORMS = {
    CustomerType.ENTERPRISE: EnterpriseCustomerForm,
    CustomerType.SMALL_FRY: SmallFryCustomerForm,
    CustomerType.LEGACY: LegacyCustomerForm,
}

Now, the DRY instinct kicks in and we notice that we now have two things we have to remember to keep in sync — any addition to the customer enum requires a corresponding addition to the UI definition dictionary. Maybe there are multiple dictionaries like this.

We could attempt to solve this by “deriving”, or some “correct by construction” mechanism that puts the creation of a new customer type all in one place.

For example, maybe we’ll have a base Customer class with get_edit_form_class() as an abstractmethod, which means it is required to be implemented. If I fail to implement it in a subclass, I can’t even construct an instance of the new customer subclass – it will throw an error.

from abc import abstractmethod

class Customer:
    @abstractmethod
    def get_edit_form_class(self):
        pass


class EnterpriseCustomer(Customer):
    def get_edit_form_class(self):
        return EnterpriseCustomerForm

class LegacyCustomer(Customer):
    ...  # etc.

I still need my enum value, or at least a list of valid values that I can use for my database field. Maybe I could derive that automatically by looking at all the sublclasses?

CUSTOMER_TYPES = [
    cls.__name__.upper().replace("CUSTOMER", "")
    for cls in Customer.__subclasses__()
]

Or maybe an __init_subclass__ trick, and I can perhaps also set up the various mappings I’ll need that way?

It’s at this point you should stop and think. In addition to requiring you to mix UI concerns into the Customer class definitions, it’s getting complex and magical.

The alternative I’m suggesting is this: require manual syncing of the two parts of the code base, but add a test to ensure that you did it. All you need is a few lines after your CUSTOMER_EDIT_FORMS definition:

CUSTOMER_EDIT_FORMS = {
    # etc as before
}

for c_type in CustomerType:
    assert (
        c_type in CUSTOMER_EDIT_FORMS
    ), f"You've defined a new customer type {c_type}, you need to add an entry in CUSTOMER_EDIT_FORMS"

You could do this as a more traditional unit test in a separate file, but for simple things like this, I think an assertion right next to the code works much better. It really helps local reasoning to be able to look and immediately conclude “yes, I can see that this dictionary must be exhaustive because the assertion tells me so.” Plus you get really early failure – as soon as you import the code.

This kind of thing crops up a lot – if you create a class here, you’ve got to create another one over there, or add a dictionary entry etc. In these cases, I’m finding simple tests and assertions have a ton of advantages when compared to clever architectural contortions (or other things like advanced static typing gymnastics):

they are massively simpler to create and understand.
you can write your own error message in the assertion. If you make a habit of using really clear error messages, like the one above, your code base will literally tell you how to maintain it.
you can easily add things like exceptions. “Every Customer type needs an edit UI defined, except Legacy because they are read only” is an easy, small change to the above.
- This contrasts with cleverer mechanisms, which might require relaxing other constraints to the point where you defeat the whole point of the mechanism, or create more difficulties for yourself.
the rule about how the code works is very explicit, rather than implicit in some complicated code structure, and typically needs no comment other than what you write in the assertion message.
you express and enforce the rule, with any complexities it gains, in just one place. Ironically, if you try to enforce this kind of constraint using type systems or hierarchies to eliminate repetition or the need for any kind of code syncing, you may find that when you come to change the constraint it actually requires touching far more places.
temporarily silencing the assertion while developing is easy and doesn’t have far reaching consequences.

Of course, there are many times when being able to automatically derive things at the code level, including some complex relationships between parts of the code, can be a win, and it’s the kind of thing you can do in Python with its many powerful techniques.

But my point is that you should remember the alternative: “synchronise manually, and have a test to check you did it.” Being able to add any kind of executable code at module level – the same level as class/function/constant definitions – is a Python super-power that you should use.

Example 3 - external polymorphism and static typing

A variant of the above problem is when, instead of an enum defining different types, I’ve got a set of classes that all need some behaviour defined.

Often we just use polymorphism where a base class defines the methods or interfaces needed and sub-classes provide the implementation. However, as in the previous case, this can involve mixing concerns e.g. user interface code, possibly of several types, is mixed up with the base domain objects. It also imposes constraints on class hierarchies.

Recently for these kind of cases, I’m more likely to prefer external polymorphism to avoid these problems. To give an example, in my current project I’m using the Command pattern or plan-execute pattern extensively, and it involves manipulating CAM objects using a series of command objects that look something like this:

@dataclass
class DeleteFeature:
    feature_name: str


@dataclass
class SetParameter:
    param_name: str
    value: float


@dataclass
class SetTextSegment:
    text_name: str
    segment: int
    value: str


Command: TypeAlias = DeleteFeature | SetParameter | SetTextSegment

Note that none of them share a base class, but I do have a union type that gives me the complete set.

It’s much more convenient to define the behaviour associated with these separately from these definitions, and so I have multiple other places that deal with Command, such as the place that executes these commands and several others. One example that requires very little code to show is where I’m generating user-presentable tables that show groups of commands. I convert each of these Command objects into key-value pairs that are used for column headings and values:

def get_command_display(command: Command) -> tuple[str, str | float | bool]:
    match command:
        case DeleteFeature(feature_name=feature_name):
            return (f"Delete {feature_name}", True)
        case SetParameter(param_name=param_name, value=value):
            return (param_name, value)
        case SetTextSegment(text_name=text_name, segment=segment, value=value):
            return (f"{text_name}[{segment}]", value)

This is giving me a similar problem to the one I had before I had before: if I add a new Command, I have to remember to add the new branch to get_command_display.

I could split out get_command_display into a dictionary of functions, and apply the same technique as in the previous example, but it’s more work, a less natural fit for the problem and potentially less flexible.

Instead, all I need to do is add exhaustiveness checking with one more branch:

match command:
    ...  # etc
    case _:
        assert_never(command)

Now, pyright will check that I didn’t forget to add branches here for any new Command. The error message is not controllable, in contrast to hand-written asserts, but it is clear enough.

The theme here is that additions in one part of the code require synchronised additions in other parts of the code, rather than being automatically correct “by construction”, but you have something that tests you didn’t forget.

Example 4 - generated code

In web development, ensuring consistent design and keeping different things in sync is a significant problem. There are many approaches, but let’s start with the simple case of using a single CSS stylesheet to define all the styles.

We may want a bunch of components to have a consistent border colour, and a first attempt might look like this (ignoring the many issues of naming conventions here):

.card-component, .bordered-heading {
   border-color: #800;
}

This often becomes impractical when we want to organise by component, rather than by property, which introduces duplication:

.card-component {
   border-color: #800;
}

/* somewhere far away … */

.bordered-heading {
   border-color: #800;
}

Thankfully, CSS has variables, so the first application of “derive” is straightforward – we define a variable which we can use in multiple places:

:root {
    --primary-border-color: #800;
}

/* elsewhere */

.bordered-heading {
    border-bottom: 1px solid var(--primary-border-color);
}

However, as the project grows, we may find that we want to use the same variables in different contexts where CSS isn’t applicable. So the next step at this point is typically to move to Design Tokens.

Practically speaking, this might mean that we now have our variables defined in a separate JSON file. Maybe something like this (using a W3C draft spec):

{
  "primary-border-color": {
    "$value": "#800000",
    "$type": "color"
  }
  "primary-hightlight-color": {
    "$value": "#FBC100",
    "$type": "color"
  }
}

From this, we can automatically generate CSS fragments that contain the same variables quite easily – for simple cases, this isn’t more than a 50 line Python script.

However, we’ve got some choices when it comes to how we put everything together. I think the general assumption in web development world is that a fully automatic “derive” is the only acceptable answer. This typically means you have to put your own CSS in a separate file, and then you have a build tool that watches for changes, and compiles your CSS plus the generated CSS into the final output that gets sent to the browser.

In addition, once you’ve bought into these kind of tools you’ll find they want to do extensive changes to the output, and define more and more extensions to the underlying languages. For example, postcss-design-tokens wants you to write things like:

.foo {
     color: design-token('color.background.primary');
 }

And instead of using CSS variables in the output, it puts the value of the token right in to every place in your code that uses it.

This approach has various problems, in particular that you become more and more dependent on the build process, and the output gets further from your input. You can no longer use the Dev Tools built in to your browser to do editing – the flow of using Dev Tools to experiment with changing a single spacing or colour CSS variable for global changes is broken, you need your build tool. And you can’t easily copy changes from Dev Tools back into the source, because of the transformation step, and debugging can be similarly difficult. And then, you’ll probably want special IDE support for the special CSS extensions, rather than being able to lean on your editor simply understanding CSS, and any other tools that want to look at your CSS now need support etc.

It’s also a lot of extra infrastructure and complexity to solve this one problem, especially when our design tokens JSON file is probably not going to change that often, or is going to have long periods of high stability. There are good reasons to want to be essentially build free. The current state of the art in this space is that to get your build tool to compile your CSS you add import './styles.css' in your entry point Javascript file! What if I don’t even have a Javascript file? I think I understand how this sort of thing came about, but don’t try to tell me that it’s anything less than completely bonkers.

Do we have an alternative to the fully automatic derive?

Using the “test” approach, we do. We can even stick with our single CSS file – we just write it like this:

/* DESIGN TOKENS START */
/* auto-created block - do not edit */
:root {
    --primary-border-color: #800000;
    --primary-highlight-color: #FBC100;
}
/* DESIGN TOKENS END */

/* the rest of our CSS here */

The contents of this block will be almost certainly auto-generated. We won’t have a process that fully automatically updates it, however, because this is the same file where we are putting our custom CSS, and we don’t want any possibility of lost work due to the file being overwritten as we are editing it.

On the other hand we don’t want things to get out of sync, so we’ll add a test that checks whether the current styles.css contains the block of design tokens that we expect to be there, based on the JSON. For actually updating the block, we’ll need some kind of manual step – maybe a script that can find and update the DESIGN TOKEN START block, maybe cog – which is a perfect little tool for this use case — or we could just copy-paste.

There are also slightly simpler solutions in this case, like using a CSS import if you don’t mind having multiple CSS files.

Conclusion

For all the examples above, the solutions I’ve presented might not work perfectly for your context. You might also want to draw the line at different place to me. But my main point is that we don’t have to go all the way with a fully automatic derive solution to eliminate any manual syncing. Having some manual work plus a mechanism to test that two things are in sync is a perfectly legitimate solution, and it can avoid some of the large costs that come with structuring everything around “derive”.

pyastgrep and custom linting

2024-05-23T20:07:34+01:00

A while back I released pyastgrep, which is a rewrite of astpath. It’s a tool that allows you to search for specific Python syntax elements using XPath as a query language.

As part of the rewrite, I separated out the layers of code so that it can now be used as a library as well as a command line tool. I haven’t committed to very much API surface area for library usage, but there is enough.

My main personal use of this has been for linting tasks or enforcing of conventions that might be difficult to do otherwise. I don’t always use this – quite often I’d reach for custom Semgrep rules, and at other times I use introspection to enforce conventions. However, there are times when both of these fail or are rather difficult.

Examples

Some examples of the kinds of rules I’m thinking of include:

Boolean arguments to functions/methods should always be “keyword only”.

Keyword-only arguments are a big win in many cases, and especially when it comes to boolean values. For example, forcing delete_thing(True, False) to be something like delete_thing(permanent=True, force=False) is an easy win, and this is common enough that applying this as a default policy across the code base will probably be a good idea.

The pattern can be distinguished easily at syntax level. Good:
```
def foo(*, my_bool_arg: bool):
    ...
```
Bad:
```
def foo(my_bool_arg: bool):
    ...
```
Simple coding conventions like “Don’t use single letter variables like i or j as a loop variables, use index or idx instead”.

This can be found by looking for code like:
```
for i, val in enumerate(...):
    ...
```
You might not care about this, but if you do, you really want the rule to be applied as an automated test, not a nit-picky code review.
A Django-specific one: for inclusion tags, the tag names should match the template file name. This is nice for consistency and code navigation, plus I actually have some custom “jump to definition” code in my editor that relies on it for fast navigation.

The pattern can again be seen quite easily at the syntax level. Good:
```
@inclusion_tag("something/foo.html")
def foo():
    ...
```
Bad:
```
@inclusion_tag("something/bar.html")
def foo():
    ...
```
Any ’task’ (something decorated with @task) should be named foo_task in order to give a clue that it works as an asynchronous call, and its return value is just a promise object.

There are many more examples you’ll come up with once you start thinking like this.

Method

Having identified the bad patterns we want to find and fix, my method for doing so looks as follows. It contains a number of tips and refinements I’ve made over the past few years.

First, I open a test file, e.g. tests/test_conventions.py, and start by inserting some example code – at least one bad example (the kind we are trying to fix), and one good example.

There are a few reasons for this:

First, I need to make sure I can prove life exists on earth, as John D. Cook puts it. I’ll say more about this later on.
Second, it gives me a deliberately simplified bit of code that I can pass to pyastdump.
Third, it provides some explanation for the test I’m going to write, and a potentially rather hairy XPath expression.

I’ll use my first example above, keyword-only boolean args. I start by inserting the following text into my test file:

def bad_boolean_arg(foo: bool):
    pass


def good_boolean_arg(*, foo: bool):
    pass

Then, I copy both of these in turn to the clipboard (or both together if there isn’t much code, like in this case), and pass them through pyastdump. From a terminal, I do:

$ xsel | pyastdump -

I’m using the xsel Linux utility, you can also use xclip -out, or pbpaste on MacOS, or Get-Clipboard in Powershell.

This gives me some AST to look at, structured as XML:

<Module>
  <body>
    <FunctionDef lineno="1" col_offset="0" type="str" name="bad_boolean_arg">
      <args>
        <arguments>
          <posonlyargs/>
          <args>
            <arg lineno="1" col_offset="20" type="str" arg="foo">
              <annotation>
                <Name lineno="1" col_offset="25" type="str" id="bool">
                  <ctx>
                    <Load/>
                  </ctx>
                </Name>
              </annotation>
            </arg>
          </args>
          <kwonlyargs/>
          <kw_defaults/>
          <defaults/>
        </arguments>
      </args>
      <body>
        <Pass lineno="2" col_offset="4"/>
      </body>
      <decorator_list/>
    </FunctionDef>
  </body>
  <type_ignores/>
</Module>

In this case, the current structure of Python’s AST has helped us out a lot – it has separated out posonlyargs (positional only arguments), args (positional or keyword), and kwonlyargs (keyword only args). We can see the offending annotation containing a Name with id="bool" inside the args, when we want it only to be allowed as a keyword-only argument.

(Do we want to disallow boolean-annotated arguments as positional only? I’m leaning towards “no” here, as positional only is quite rare and usually a very deliberate choice).

I now have to construct an XPath expression that will find the offending XML nodes, but not match good examples. It’s pretty straightforward in this case, once you know the basics of XPath. I test it out straight away at the CLI:

pyastgrep './/FunctionDef/args/arguments/args/arg/annotation/Name[@id="bool"]' tests/test_conventions.py

If I’ve done it correctly, it should print my bad example, and not my good example.

Then I widen the net, omitting tests/test_conventions.py to search everywhere in my current directory.

At this point, I’ve probably got some real results that I want to address, but I might also notice there are other variants of the same thing I need to be able to match, and so I iterate, adding more bad/good examples as necessary.

Now I need to write a test. It’s going to look like this:

def test_boolean_arguments_are_keyword_only():
    assert_expected_pyastgrep_matches(
        """
        .//FunctionDef/args/arguments/args/arg/annotation/Name[@id="bool"]
        """,
        message="Function arguments with type `bool` should be keyword-only",
        expected_count=1,
    )

Of course, the real work is being done inside my assert_expected_pyastgrep_matches utility, which looks like this:

from pathlib import Path
from boltons import iterutils
from pyastgrep.api import Match, search_python_files

SRC_ROOT = Path(__file__).parent.parent.resolve()  # depends on project structure


def assert_expected_pyastgrep_matches(xpath_expr: str, *, expected_count: int, message: str):
    """
    Asserts that the pyastgrep XPath expression matches only `expected_count` times,
    each of which must be marked with `pyastgrep_exception`

    `message` is a message to be printed on failure.

    """
    xpath_expr = xpath_expr.strip()
    matches: list[Match] = [item for item in search_python_files([SRC_ROOT], xpath_expr) if isinstance(item, Match)]

    expected_matches, other_matches = iterutils.partition(
        matches, key=lambda match: "pyastgrep: expected" in match.matching_line
    )

    if len(expected_matches) < expected_count:
        assert False, f"Expected {expected_count} matches but found {len(expected_matches)} for {xpath_expr}"

    assert not other_matches, (
        message
        + "\n Failing examples:\n"
        + "\n".join(
            f"  {match.path}:{match.position.lineno}:{match.position.col_offset}:{match.matching_line}"
            for match in other_matches
        )
    )

There is a bit of explaining to do now.

Being sure that you can “find life on earth” is especially important for a negative test like this. It would be very easy to have an XPath query that you thought worked but didn’t, as it might just silently return zero results. In addition, Python’s AST is not stable – so a query that works now might stop working in the future.

It’s like you have a machine that claims to be able to find needles in haystacks – when it comes back and says “no needles found”, do you believe it? To increase your confidence that everything works and continues to work, you place a few needles at locations that you know, then check that the machine is able to find those needles. When it claims “found exactly 2 needles”, and you can account for those, you’ve got much more confidence that it has indeed found the all the needles (no false negatives).

Additionally, to ensure we don’t have false positives, you can also place some things in the haystack that are similar to needles, but slightly different, and ensure that these are not counted as needles.

So, it’s important to leave my bad (and good) examples in there.

But, I obviously don’t want the bad examples to cause the test to fail! In addition, I want a general mechanism for exceptions. A simple mechanism I’ve chosen is to add the text pyastgrep: expected as a comment.

So, I need to change my bad example like this:

def bad_boolean_arg(foo: bool):  # pyastgrep: expected
    pass

I also pass expected_count=1 to indicate that I expect to find at least one bad example (or more, if I’ve added more bad examples).

Hopefully that explains everything assert_expected_pyastgrep_matches does. A couple more notes:

it uses boltons, a pretty useful set of Python utilities
it requires a SRC_ROOT folder to be defined, which will depend on your project, and might be different depending on which folder(s) you want to apply the convention too.

Now, everything is set up, and I run the test for real, hopefully locating all the bad usages. I work through them and fix, then leave the test in.

Tips

pyastgrep works strictly at the syntax level, so unlike Semgrep you might get caught out by aliases if you try match on specific names:

from foo import bar
from foo import bar as foo_bar
import foo

# These all call the same function but look different in AST:
foo.bar()
bar()
foo_bar()

There is however, an advantage to this – you don’t need a real import to construct your bad examples, you can just use a Mock. e.g. for my inclusion_tag example above, I have code like:
```
from unittest.mock import Mock

register = Mock()

@register.inclusion_tag(filename="something/not_bad_tag.html")
def bad_tag():  # pyastgrep: expected
    pass
```
You can see the full code on GitHub.
You might be able to use a mixture of techniques:
- A Semgrep rule avoids one set of bad patterns using some thirdparty.func, and requiring everyone to use your own wrapper, which is then constructed in such a way to make it easier to apply a pyastgrep rule
- Some introspection that produces a list of classes or functions to which some rule applies, then dynamically generates XPath expression to pass to pyastgrep.

Conclusion

Syntax level searching isn’t right for every job, but it can be a powerful addition to your toolkit, and with a decent query language like XPath, you can do a surprising amount. Have a look at the pyastgrep examples for inspiration!

Enforcing conventions in Django projects with introspection

2024-04-01T16:05:03+01:00

Naming conventions can make a big difference to maintenance issues in software projects. This post is about how we can use the great introspection capabilities in Python to help enforce naming conventions in Django projects.

Let’s start with an example problem and the naming convention we’re going to use to solve it. There are many other applications of the techniques here, but it helps to have something concrete.

The problem: DateField and DateTimeField confusion

Over several projects I’ve found that inconsistent or bad naming of DateField and DateTimeField fields can cause various problems.

First, poor naming means that you can confuse them for each other, and this can easily trip you up. In Python, datetime is a subclass of date, so if you use a field called created_date assuming it holds a date when it actually holds a datetime, it might be not obvious initially that you are mishandling the value, but you’ll often have subtle problems down the line.

Second, sometimes you have a field named like expired which is actually the timestamp of when the record expired, but it could easily be confused for a boolean field.

Third, not having a strong convention, or having multiple conventions, leads to unnecessary time wasted on decisions that could have been made once.

Finally, inconsistency in naming is just confusing and ugly for developers, and often for users further down the line, because names tend to leak.

Even if you do have an established convention, it’s possible for people not to know. It’s also very easy for people to change a field’s type between date and datetime without also changing the name. So merely having the convention is not enough, it needs to be enforced.

For this specific example, the convention I quite like is:

field names should end with _at for timestamp fields that use DateTimeField, like expires_at or deleted_at.
field names should end with _on or _date for fields that use DateField, like issued_on or birth_date.

This is based on the English grammar rule that we use “on” for dates but “at” for times – “on the 25th March”, but “at 7:00 pm” – and conveniently it also needs very few letters and tends to read well in code. The _date suffix is also helpful in various contexts where _on seems very unnatural. You might want different conventions, of course.

To get our convention to be enforced with automated checks we need a few tools.

The tools

Introspection

Introspection means the ability to use code to inspect code, and typically we’re talking about doing this when our code is already running, from within the same program and using the same programming language.

In Python, this starts from simple things like isinstance() and type() to check the type of an object, to things like hasattr() to check for the presence of attributes and many other more advanced techniques, including the inspect module and many of the metaprogramming dunder methods.

Django app and model introspection

Django is just Python, so you can use all normal Python introspection techniques. In addition, there is a formally documented and supported set of functions and methods for introspecting Django apps and models, such as the apps module and the Model _meta API.

Django checks framework

The third main tool we’re going to use in this solution is Django’s system checks framework, which allows us to run certain kinds of checks, at both “warning” and “error” level. This is the least important tool, and we could in fact switch it out for something else like a unit test.

The solution

It’s easiest to present the code, and then discuss it:

from django.apps import apps
from django.conf import settings
from django.core.checks import Tags, Warning, register


@register()
def check_date_fields(app_configs, **kwargs):
    exceptions = [
        # This field is provided by Django's AbstractBaseUser, we don't control it
        # and we’ll break things if we change it:
        "accounts.User.last_login",
    ]
    from django.db.models import DateField, DateTimeField

    errors = []
    for field in get_first_party_fields():
        field_name = field.name
        model = field.model

        if f"{model._meta.app_label}.{model.__name__}.{field_name}" in exceptions:
            continue

        # Order of checks here is important, because DateTimeField inherits from DateField

        if isinstance(field, DateTimeField):
            if not field_name.endswith("_at"):
                errors.append(
                    Warning(
                        f"{model.__name__}.{field_name} field expected to end with `_at`, "
                        + "or be added to the exceptions in this check.",
                        obj=field,
                        id="conventions.E001",
                    )
                )
        elif isinstance(field, DateField):
            if not (field_name.endswith("_date") or field_name.endswith("_on")):
                errors.append(
                    Warning(
                        f"{model.__name__}.{field_name} field expected to end with `_date` or `_on`, "
                        + "or be added to the exceptions in this check.",
                        obj=field,
                        id="conventions.E002",
                    )
                )
    return errors


def get_first_party_fields():
    for app_config in get_first_party_apps():
        for model in app_config.get_models():
            yield from model._meta.get_fields()


def get_first_party_apps() -> list[AppConfig]:
    return [app_config for app_config in apps.get_app_configs() if is_first_party_app(app_config)]


def is_first_party_app(app_config: AppConfig) -> bool:
    if app_config.module.__name__ in settings.FIRST_PARTY_APPS:
        return True
    app_config_class = app_config.__class__
    if f"{app_config_class.__module__}.{app_config_class.__name__}" in settings.FIRST_PARTY_APPS:
        return True
    return False

We start here with some imports and registration, as documented in the “System checks” docs. You’ll need to place this code somewhere that will be loaded when your application is loaded.

Our checking function defines some allowed exceptions, because there are some things out of our control, or there might be other reasons. It also mentions the exceptions mechanism in the warning message. You might want a different mechanism here, but I think having some way of dealing with exceptions, and advertising its existence in the warnings, is often pretty important. Otherwise, you can end up with worse consequences when people just slavishly follow rules. Notice how in the exception list above I’ve given a comment detailing why the exception is there though – this helps to establish a precedent that exceptions should be justified, and the justification should be there in the code.

We then loop through all “first party” model fields, looking for DateTimeField and DateField instances. This is done using our get_first_party_fields() utility, which is defined in terms of get_first_party_apps(), which in turn depends on:

the get_app_configs() function.
the AppConfig.get_models() method
the _meta get_fields() method

a custom setting FIRST_PARTY_APPS which I’ve created in my settings.py like this:

FIRST_PARTY_APPS = ["myapp", "myotherapp"]

INSTALLED_APPS = [
 "django.contrib.auth",
 "django.contrib.sessions",
] + FIRSTY_PARTY_APPS + [ ... ]

You may have a different way of recognising your own apps.

The id values passed to Warning here are examples – you should change according to your needs. You might also choose to use Error instead of Warning.

Output

When you run manage.py check, you’ll then get output like:

 System check identified some issues:

 WARNINGS:
 myapp.MyModel.created: (conventions.E001) MyModel.created field expected to end with `_at`,
 or be added to the exceptions in this check.

 System check identified 1 issue (0 silenced).

As mentioned, you might instead want to run this kind of check as a unit test.

Conclusion

There are many variations on this technique that can be used to great effect in Django or other Python projects. Very often you will be able to play around with a REPL to do the introspection you need.

Where it is possible, I find doing this far more effective than attempting to document things and relying on people reading and remembering those docs. Every time I’m tripped up by bad names, or when good names or a strong convention could have helped me, I try to think about how I could push people towards a good convention automatically – while also giving a thought to unintended bad consequences of doing that prematurely or too forcefully.

Re-using CSS for the wrong HTML with Sass

2023-06-01T20:44:15Z

Recently, while writing up some examples and pattern for using htmx with Django for form validation, I discovered a new trick for using externally defined CSS without having to change the HTML you are working with.

To make it concrete, an example might be that you are using some CSS from a CSS library or framework that requires your HTML to look a certain way. In the Bulma framework, for instance, you have to add the right class attribute directly on an element that needs styling.

At the same time, you might be working with another system that is generating the HTML for you, and modifying that output might be hard or impossible or just tedious and a potential maintenance burden going forward. For instance, in Django forms, there is an ErrorList class whose output can be overridden, but by default renders like this:

<ul class="errorlist">
  <li>Enter a valid email address.</li>
</ul>

Now I have these requirements:

I want this error list to be coloured using a Bulma colour utility as if it had class="has-text-danger" when it appears within a field row (which are <div class="field"> elements).
When it appears at the top of the form where it has an extra nofield class, I want it to instead be styled like a Bulma notification as if it had class="notification is-danger is-light".

But I want to do these without changing the HTML we’re given by Django, or changing existing CSS – only by adding some CSS rules.

The “best” way to do this is if your CSS framework provides its styles as a set of Sass mixins, or something equivalent. Bulma, as it happens, usually does this, but sometimes we’re not so lucky, and we just have CSS.

The trick I learnt requires you to use Sass/SCSS and the @extend directive. This powerful directive takes rules relating to one selector, and pulls them into whatever rule you are writing.

(If you are, like me, put off using things like CSS pre-processors because of the need for a separate build step, or needing to use Node.js/npm, see my post on How to use Sass/SCSS in a Django project without needing Node.js/npm or running a build process)

The one thing you have to do is rename the base CSS file you want to re-use from .css to .scss. This works because SCSS is a CSS superset. Then, for the example above, you can write your own SCSS file like this:

@import "path/to/bulma.scss";

.field ul.errorlist {
    @extend .has-text-danger;
}

ul.errorlist.nonfield {
    @extend .notification;
    @extend .is-danger;
    @extend .is-light;
}

This technique can be very powerful e.g. make all input[type=text] inside a <form class="bulma"> have the normal Bulma input appearance:

form.bulma {
    input[type=text] {
        @extend .input;
    }
}

This will include all related rules like .input:focus etc.

As mentioned, it may not always be the best technique, but it’s a great one to have in your toolbox.

Django and Sass/SCSS without Node.js or a build step

2023-06-01T19:54:15Z

Although they are less necessary than in the past, I like to use a CSS pre-processor when doing web development. I used to use LessCSS, but recently I’ve found that I can use Sass without needing either a separate build step, or a package that requires Node.js and npm to install it. The heart of the functionality is provided by libsass, an implementation of Sass as a C++ library.

On Linux systems, this can be installed as a package libsass or similar, but even better is that you can pip install it as a Python package, libsass.

When it comes to using it from a Django project, the first step is to install django-compressor.

Then, you need to add django-libsass as per its instructions.

That’s about it. As per the django-libsass instructions, somewhere in your base HTML templates you’ll have something like this:

{# at the top #}
{% load compress %}
{% load static %}

{# in the <head> element #]
{% compress css %}
  <link rel="stylesheet" type="text/x-scss" href="{% static "myapp/css/main.scss" %}" />
{% endcompress %}

You write your SCSS in that main.scss file (it doesn’t have to be called that), and it can @import other SCSS files of course.

Then, when you load a page, django-compressor will take care of running the SCSS files through libsass, saving the output CSS to a file and inserting the appropriate HTML that references that CSS file into your template output. It caches things very well so that you don’t incur any penalty if files haven’t changed — and libsass is a very fast implementation for when the processing does need to happen.

What this means is that you have eliminated both the need for Node.js/npm, and the need for a build step/process, if you only needed these things for CSS pre-processing.

Of course, the SCSS → CSS compilation still has to happen, but it happens on demand in the same process that runs the web app, and it’s both fast enough and reliable enough that you simply never have to think about it again. So this is “build-less” in the same way that “server-less” means you don’t have to think about servers, and the same way that Python “doesn’t have a compilation step”.

Future proofing

On the Sass-lang page about libsass, they say it is “deprecated”, and on the project page page it says:

While it will continue to receive maintenance releases indefinitely, there are no plans to add additional features or compatibility with any new CSS or Sass features.

In other words, this is what I prefer to call “mature software” 😉. libsass already has everything I need. If it does eventually fail to be maintained or I need new features, it’s not a problem:

Switch to Dart Sass, which can be installed as a standalone binary.

Set your django-compressor settings like this:

COMPRESS_PRECOMPILERS = [
    ("text/x-scss", "sass {infile} {outfile}"),
]

This covers the basic case. If you want all the features of django-libsass, which includes looking in your other static file folders for SCSS, you’ll probably need to fork the code and make it work by calling Dart Sass using subprocess — a small amount of work, and nothing that will fundamentally break this approach.

Python’s “Disappointing” Superpowers

2023-02-01T13:44:15Z

In Hillel Wayne’s post “I am disappointed by dynamic typing”, he expresses his sense that the Python ecosystem doesn’t really make the most of the possibilities that Python provides as a dynamically typed language. This is an important subject, since every Python program pays a very substantial set of costs for Python’s highly dynamic nature, such as poor run-time performance, and maintainability issues. Are we we getting anything out of this tradeoff?

I think Hillel makes some fair points, and this post is intended as a response rather than a rebuttal. Recently there has been a significant influence of static type systems which I think might be harmful. The static type system we have in the form of mypy/pyright (which is partly codified in PEP 484 and following) seems to be much too heavily inspired by what is possible to map to other languages, rather than the features that Python provides.

(As a simple example to support that claim, consider the fact that Python has had support for keyword arguments since as long as I can remember, and for keyword-only arguments since Python 3.0. But typing.Callable has zero support for them, meaning they can’t be typed in a higher-order context. . This is bad, since they are a key part of Python’s excellent reputation for readability, and we want more keyword-only arguments, not fewer. [EDIT: it looks like there is another way to do it, it’s just about 10 times more work, so the point kind of stands.] I can give more examples, but that will have to wait for another blog post).

I’m worried that a de-facto move away from dynamic stuff in the Python ecosystem, possibly motivated by those who use Python only because they have to, and just want to make it more like the C# or Java they are comfortable with, could leave us with the very worst of all worlds.

However, I also think there are plenty of counter-examples to Hillel’s claim, and that’s what this post will explore.

Hillel was specifically thinking about, in his own words:

“runtime program manipulation”
“programs that take programs and output other programs”
“thinking of the whole runtime environment in the same way, where everything is a runtime construct”

…and he gave some examples that included things like:

run-time type modification
introspection/manipulation of the stack
passing very differently typed objects through normal code to collect information about it.

I’m going to give lots of examples of this kind of thing in Python, and they will all be real world examples. This means that either I have used them myself to solve real problems, or I’m aware that other people are using them in significant numbers.

Before I get going, there are some things to point out.

First, I don’t have the exact examples Hillel is looking for – but that’s because the kind of problems I’ve needed to solve have not been exactly the same as his. My examples are all necessarily limited in scope: since Python allows unrestricted side-effects in any function, including IO and being able to modify other code, there are obviously limits into how well these techniques can work across large amounts of code.

I do think, however, that my examples are in the same general region, and some of them very close. On both sides we’ve got to avoid semantic hair-splitting – you can argue that every time you use the class keyword in Python you are doing “run-time type creation”, rather than “compile-time type creation”, because that’s how Python’s classes work. But that’s not what Hillel meant.

Second, many of these more magical techniques involve what is called monkey patching. People are often confused about the difference between monkey patching and “dynamic meta-programming”, so I’ve prepared a handy flow chart for you:

There are, however, many instances of advanced, dynamic techniques that never get to the point of the chart above, and that’s because you never know about them. What you know is that the code does something useful, and it does so reliably enough that you don’t need to know what techniques contributed to it. And this is, I think, the biggest problem in what Hillel is asking for. The best examples of these techniques will be reliable enough that they don’t draw attention to themselves, and you immediately take them for granted.

Which is also to say that you cannot discount something I mention below just because it is so widely used that you, too, have taken it for granted – that would effectively be saying that the only examples that count are the ones that have proved to be so wild and wacky that everyone has decided they are a bad idea.

Third, you might also discount these examples as “just using features the language provides”, rather than “hyper-programming” or something exotic. On the one hand, it would be true, but also unfair in the context of this debate. The most obvious example is eval. This is clearly a very powerful technique not available to many statically typed languages, and exactly the kind that Hillel is looking for – you are literally creating more of your program as your program is running. On the other hand, it’s nothing more than a builtin function.

Finally, a number of these examples don’t involve “production” code i.e. the code is typically run only on developer machines or in CI. These still count, however – just like many of Hillel’s examples are in the area of testing. The reasons they still count are 1) developers are humans too, and solving their problems is still important and 2) the techniques used by developers on their own machines are useful in creating high quality code for running on other people’s machines, where we don’t want to incur the performance or robustness penalties of the techniques used.

So, here are my examples. The majority are not my own code, but I’ve also taken the opportunity to do some fairly obvious bragging about cool things I’ve done in Python.

Gooey

Gooey is a library that will re-interpret argparse entry points as if they were specifying a GUI. In other words, you do “import gooey”, add a decorator and it transforms your CLI program into a GUI program. Apparently it does this by re-parsing your entry point module, for reasons I don’t know and don’t need to know. I do know that it works for programs I’ve tried it with, when I wanted to make something that I was using as a CLI, but also needed to be usable by other family members. A pretty cool tool that solves real problems.

Werkzeug’s interactive debugger

Werkzeug provide a debugger middleware which works with any WSGI-compliant Python web framework (which is most of them) with the following extremely useful behaviour:

Crashing errors are automatically intercepted and an error page is shown with a stack trace instead of a generic 500 error.
For any and every frame of the stack trace, you can, right from your web browser, start a Python REPL at that frame – i.e. you can effectively continue execution of the crashed program at any point in the stack, or from multiple points simultaneously.

This is extremely useful, to say the least.

(For Django users – you can use this most easily using django-extensions)

Hybrid attributes in SQLAlchemy

I’m sure there are many examples of advanced dynamic techniques in SQLAlchemy, and I’m not the best qualified to talk about them, but here is a cool one I came across that helps explain the kind of thing you can do in Python.

Suppose you have an ORM object with some attributes that come straight from the database, along with some calculated properties. In the example below we’ve got a model representing an account that might have payments against it:

class Account(Base):
    # DB columns:
    amount_paid: Mapped[Decimal]
    total_purchased: Mapped[Decimal]

    # Calculated properties:
    @property
    def balance_due(self) -> Decimal:
        return self.total_purchased - self.amount_paid

    @property
    def has_payment_outstanding(self) -> bool:
        return self.balance_due > 0

Very often you find yourself in a situation like this:

Sometimes you have already loaded an object from the DB, and want to know a calculated value like “does this account have an outstanding payment?”. This shouldn’t execute any more database queries, since you’ve already loaded everything you need to answer that question.
But sometimes, you want to re-use this logic to do something like “get me all the accounts that have outstanding payments”, and it is vital for efficiency that we do the filtering in the database as a SQL WHERE clause, rather than loading all the records into a Python process and filtering there.

How could we do this in SQLAlchemy without duplicating the logic regarding balance_due and has_outstanding_payment?

The answer is hybrid attributes:

from sqlalchemy.ext.hybrid import hybrid_property
replace property with hybrid_property on the two properties.

That is all. Then you can do:

select(Account).where(Account.has_payment_outstanding == True)

This will generate a SQL query that looks like this:

SELECT account.id, account.amount_paid, account.total_purchased
FROM account
WHERE (account.total_purchased - account.amount_paid > 0) = 1

What’s going on here? If you have a normal model instance an_account, retrieved from a database query, and you do an_account.has_payment_outstanding, then in the has_payment_outstanding function body above, everything is normal: self is bound to an_account, attributes like total_purchased will be Decimal objects that have been loaded from the database.

However, if you use Account.has_payment_outstanding, the self variable gets bound to a different type of object (the Account class or some proxy), and so things like self.total_purchased instead resolve to objects representing columns/fields. These classes have appropriate “dunder” methods defined, (__add__, __gt__ etc) so that operations done on them, such as maths and comparisons, instead of returning values immediately, return new expression objects that track what operations were done. These can then be compiled to SQL later on. So we can execute the filtering as a WHERE clause in the DB.

The point here is: we are passing both “normal” and “instrumented” types through the same code in order to completely change our execution strategy. This allows us to effectively compile our Python code into SQL on the fly. This is essentially identical to Hillel’s example of passing instrumented objects (“Replacer” class) through normal code to extract certain information about what operations were done.

This is a very neat feature in SQLAlchemy that I’m rather jealous of as a Django user. If you want the same efficiency in Django, you have to define the instance properties and the database filtering separately, and usually physically not next to each other in the code. The closest we have is Query expressions but they don’t work quite the same.

Pony ORM

This ORM has a way of writing SQL select queries that appears even more magical. Using an example from their home page, you write code like this:

select(c for c in Customer if sum(c.orders.price) > 1000)

The result of this is a SQL query that looks like this:

SELECT "c"."id"
FROM "customer" "c"
  LEFT JOIN "order" "order-1"
    ON "c"."id" = "order-1"."customer"
GROUP BY "c"."id"
HAVING coalesce(SUM("order-1"."total_price"), 0) > 1000

A normal understanding of generator expressions suggests that the select function is consuming a generator. But that couldn’t explain the behaviour here. Instead, it actually introspects the frame object of the calling code, then decompiles the byte code of the generator expression object it finds, and builds a Query based on the AST objects.

PonyORM doesn’t advertise all that, of course. It advertises a “beautiful” syntax for writing ORM code, because that’s what matters.

Django

This is the web framework I know well, as I used to contribute significantly, and I’ll pick just a few important examples from the ORM, and then from the broader ecosystem.

ForeignKey

Suppose, to pick one example of many, you are writing django-otp, a third party library that provides a One Time Password implementation for 2FA requirements. You want to create a table of TOTP devices that are linked to user accounts, and so you have something like this:

class TOTPDevice(models.Model):
    user = models.ForeignKey('auth.User', related_name='totp_devices')

Later on, you have code that starts with a User object and retrieves their TOTP devices, and it looks something like this:

devices = user.totp_devices.all()

This is interesting, because my user variable is an instance of a User model that was provided by core Django, which has no knowledge of the third party project that provides the TOTPDevice model.

In fact it goes further: I may not be using Django’s User at all, but my own custom User class, and the TOTPDevice model can easily support that too just by doing this:

user = models.ForeignKey(getattr(settings, "AUTH_USER_MODEL", "auth.User"))

This means that my User model has no knowledge of the TOTPDevice class, nor the other way around, yet instances of these classes both get wired up to refer to each other.

What is actually going on to enable this?

When you import Django and call setup(), it imports all the apps in your project. When it comes to the TOTPDevice class, it sees the "auth.User" reference and finds the class it refers to. It then modifies that class, adding a totp_devices descriptor object to the class attributes.

This is run-time type modification.

The result is that when you do user.totp_devices, you get a Manager instance that does queries against the TOTPDevice table. It is a specific kind of manager, known as a RelatedManager, with the special property that it automatically does the correct filter() calls to limit returned values to those related to the model instance, among other things.

RelatedManager

The RelatedManager class is interesting in a number of ways. First, it is created as a closure – meaning the class itself is created inside a function that is called for each relationship. This is run-time type creation.

Second, there are some additional things it needs to support. In Django, projects often override Manager classes, and the related QuerySet classes, to provide a lot of model layer functionality. This is Django’s answer to the “Repository” pattern – popular in some languages, but laborious to create and painful to use compared to what we have.

Because of this, it’s important that the RelatedManager preserves any custom Manager and QuerySet behaviour defined on the target model. So, the solution is simply that Django makes the created RelatedManager class inherit from your custom Manager. This is dynamic sub-classing – sub-classing of a class that is discovered at run-time. In other OOP languages, you inherit from framework classes. In Python, framework inherits you!

ManyToMany models

One common need in database applications is to have many-to-many relationships between two models. Typically this can be modelled with a separate table that has foreign keys to the two related tables.

To make this easy, Django provides a ManyToManyField. For simple cases, it’s tedious to have to create a model for the intermediate table yourself, so of course Django just creates it for you if you don’t provide your own, using type() with 3 arguments. This is again run-time type creation.

Consequences

These examples of run-time type modification or creation are perhaps not the most extreme or mind-bending. But they are something even better: useful. It’s these features, and things like them, that enable an ecosystem of third party Django libraries that can integrate with your own code without any problems.

Python also always gives us enough flexibility to have a good backwards compatibility story – so that, for example, the swappable User model was introduced with an absolute minimum of fuss for both projects and pluggable Django apps.

I’m interested in functional programming, Haskell in particular – this blog even ran on Haskell for a time – so I always take interest in developments in the Haskell web framework world. I see lots of cool things, but it always seems that the ecosystems around Haskell web frameworks are at least 10 years behind Django. One key issue is that in contrast to Django or other Python frameworks, Haskell web frameworks almost always have some kind of code generation layer. This can be made to work well for the purposes envisaged by the framework authors, but it never seems to enable the ecosystem of external packages that highly dynamic typing supports.

Please note that I’m not claiming here that Python is better than Haskell or anything so grand. I’m simply claiming that Python does enable very useful things to be built, and those things are made possible and easy because of Python’s design, rather than despite it.

I think this is important to say. Python has become massively more popular than it was when I first started to use it, and there are increasing numbers of people who use it only because of network effects, and don’t understand why it got so popular in the first place. These people can sometimes assume that it’s fundamentally a poorly designed language that we are just lumped with – today’s PHP – whose best trajectory is to make it more like Java or C# or Rust etc. I think that would be a big mistake.

Baserow

One project that takes Python’s run-time type creation further is Baserow. In their case, their customers create database applications, and the metadata for those tables is stored in … tables. They like Django and want to use it as much as possible. But they also want their customers’ actual data tables to be normal RDBMS tables, and therefore benefit from all the typical RDBMS features to make their tables fast and compact etc. (I’ve seen and worked on systems that took the opposite approach, where customer schema was relegated to second class storage – essentially a key-value table – and the performance was predictably awful).

And they want plug-in authors to be able to use Django too! Some people are just greedy! They have a nice article describing how they achieved all this: in short, they use type() for run-time type creation and then leverage everything Django gives them.

This has the interesting effect that the metadata tables, along with their own business tables, live at the same level as their customers’ tables which are described by those metadata tables. This “meta and non-meta living at the same level” is a neat illustration of what Python’s type system gives you:

When you first discover the mind-bending relationships around type(type) == type, you might think of an infinitely-recursive relationship. But actually, an infinite relationship has been flattened to being just 3 layers deep – instance, class, metaclass. The last layer just recurses onto itself. The infinity has been tamed, and brought into the same structures that you can already deal with, and without changing language or switching to code generation techniques. This is one reason why many examples of the “hyper-programming” that Hillel talks about can just be dismissed as normal programming – but they are simply hyper-programming that you are now taking for granted.

CCiW data retention policy

CCiW is a small charity I’ve been involved with for a long time. When I came to implement its GDPR and data retention policies, I found another example of how useful it is having access to Django’s meta-layer (generic framework code) on the same level as my normal data layer (business specific classes and tables), in ways that often aren’t the case for statically typed languages that resort to code-generation techniques for some of these things.

I wanted to have a data retention policy that was both human readable and machine readable, so that:

We don’t have keep two separate documents in sync.
the CCiW committee and other interested parties would be able to read the policy that actually gets applied, rather than merely what the policy was supposed to be.
I could have machine level checking of the exhaustiveness of the policy.

My solution was to split the data retention policy into two parts:

a heavily commented, human-and-machine readable Literate YAML file with a nicely formatted version, that I can genuinely claim is our data retention policy, and that it is automatically applied,
and a Python implementation that reads this file and applies it, along with some additional logic.

A key part of the neatness of this solution is that the generic, higher level code (which reads in a YAML file, and therefore has to treat field names and table names as strings), and the business/domain specific logic can sit right next to each other. The end result is something that’s both efficient and elegant, with great separation of concerns, and virtually self-maintaining – it complains at me automatically if I fail to update it when adding new fields or tables.

In terms of performance, the daily application for the data retention policy for the entire database requires, at the moment, just 5 UPDATE and 3 DELETE queries, run once a day. This is made possible by:

using the power of an ORM,
using generic code to build up **kwargs to pass to QuerySet.update(),
seamlessly integrating these two with business specific logic.

Here is one of the queries the ORM generates, which is complex enough that I wouldn’t attempt to write this by hand, but it correctly applies business logic like not erasing any data of people who still owe us money, and combines all the erasure that needs to be done into a single query.

Query tracing

A common need in database web applications is development tools that monitor what database queries your code is generating and where they are coming from in the code. In Python this is made very easy thanks to sys._getframe which gives you frame objects of the currently running program.

For Django, the go-to tool that uses this is django-debug-toolbar, which does an excellent job of pinpointing where queries are coming from.

There have been times when it has failed me, however. In particular, when you are working with generic code, such as the Django admin or Django REST framework, in which the fields and properties that will be fetched may be defined as strings in declarative code, a stack trace alone is not enough to work out what is triggering the queries. For example, you might have an admin class defined like this:

class MyModelAdmin(admin.ModelAdmin):
    list_display = ["field1", "field2", "field3", ...]

And the stack trace points you to this code:

for field_index, field_name in enumerate(cl.list_display):
    f, attr, value = lookup_field(field_name, result, cl.model_admin)

It’s correct, but not helpful. I need to know what the value of the local variable field_name is in that loop to work out what is actually causing these queries.

In addition, in one case I was actually working with DRF endpoints, not the HTML endpoints the debug toolbar is designed for.

So, I wrote my own utilities that, in addition to extracting the stack, would also include certain local variables for specified functions/methods. I then needed to add some aggregation functionality and pretty-printing for the SQL queries too. Also, I wrote a version of assertNumQueries that used this better reporting.

This was highly effective, and enabled me and members of my team to tackle these DRF endpoints that had got entirely out of hand, often taking them from 10,000+ database queries (!) to 10 or 20.

This is relatively advanced stuff, but not actually all that hard, and it’s within reach of many developers. It doesn’t require learning a whole new language or deep black magic. You can call sys._getframe interactively from a REPL and find out what it does. The biggest hurdle is actually making the mental leap that says “I need to build this, and with Python, I probably can”.

time-machine and pyfakefs

As an example of “entire program transformation”, time-machine is an extremely useful library that mocks out date/time functions across your entire program, and pyfakefs is one that does the same thing for file-system calls.

These contrast with libraries like unittest.mock do that do monkey patching on a more limited, module-by-module basis.

This technique is primarily useful in automated test suites, but it has a profound impact on the rest of your code base. In other languages, if you want to mock out “all date/time access” or “all filesystem access”, you may end up with a lot of tedious and noisy code to pass these dependencies through layers of code, or complex automatic dependency injection frameworks to avoid that. In Python, those things are rarely necessary, precisely because of things like time-machine and pyfakefs – that is, because your entire program can be manipulated at run-time. Your code base then has the massive benefit of a direct and simple style.

Environment detection

My current employer is Datapane who make tools for data apps. Many of our customers use Jupyter or similar environments. To make things work really smoothly, our library codes detects the environment it is running in and responds, and in some cases interacts with this environment (courtesy of Shahin, our Jupyter guy). This is an application of Python’s great support for introspection of the running program. There are a bunch of ways you can do this kind of thing:

checking the system environment in os.environ
checking the contents of sys.modules
using sys._getframe to examine how you are being called.
attempting to use get_ipython and seeing if it works etc.

This is an example of the “whole runtime environment” being dynamic and introspectable, and Jupyter Notebook and its huge ecosystem make great use of this.

With some of the bigger features we’re working on at the moment at Datapane, we’re needing more advanced ways of adjusting to the running environment. Of course, as long it works, none of the implementation matters to our customers, so we don’t advertise any of that. Our marketing tagline for this is “Jupyter notebook to a shareable data app in 10 seconds”, not “we’re in your Python process, looking at your sys.modules”.

After doing a grep through my site-packages, I found that doing sys._getframe for different kinds of environment detection is relatively common – often used for things like “raise a deprecation warning, but not if we are being called from these specific callees, like our own code”. Here’s just one more example:

boltons provides a make_sentinel function. The docs state that if you want “pickleability”, the sentinel must be stored in a module-level constant. But the implementation goes further and checks you are doing that using a sys._getframe trick. This is just a simple usability enhancement in which code checks that it is being used correctly, made possible by Python’s deep introspection support, but this kind of thing adds up. You will find many similar things in small amounts scattered across different libraries.

fluent-compiler

Fluent is a localisation system by Mozilla. I wrote and contributed the initial version of the official fluent.runtime Python implementation, which is an interpreter for the Fluent language, and I also wrote a second implementation, fluent-compiler.

Of all the libraries I’ve written, this was the one I enjoyed most, and it’s also the least popular it seems – not surprising, since GNU gettext provides a great 90% solution, which is enough for just about everyone, apart from Mozilla and, for some reason I can’t quite remember, me. However, I do know that Mozilla are actually using my second implementation in some of their web projects, via django-ftl, and I’m using it, and it has a few GitHub stars, so that counts as real world!

Here are some of the Hillel-worthy Python techniques I used:

Compile-to-Python

In fluent-compiler, the implementation strategy I took was to compile the parsed Fluent AST to Python code, and exec it. I actually use Python AST nodes rather than strings, for various security reasons, but this is basically the same as doing eval, and that same technique is used by various other projects like Jinja and Mako.

If anything qualifies as “programs that create programs on the fly”, then using exec, eval or compile must do so! The main advantage of this technique here is speed. It works particularly well with Fluent, because with a bit of static analysis, we can often completely eliminate the overhead that would otherwise be caused by its more advanced features, like terms and parameterized terms, so that at run-time they cost us nothing.

This works even better when combined with PyPy. For the simple and common cases, under CPython 3.11 my benchmarks show a solution using fluent-compiler is about 15% faster than GNU gettext, while under PyPy it’s more than twice as fast. You should take these numbers with a pinch of salt, but I am confident that the result is not slow, despite having far more advanced capabilities than GNU gettext, which is not true for the first implementation – the compiler is about 5-10x faster than the interpreter for common cases on CPython.

Additionally, there are some neat tricks you can do when implementing a compiler using the same language that you are compiling to, like evaluating some things ahead of time that you know are constants.

Dynamic test methods

While developing the second implementation, I used the first implementation as a reference. I didn’t want to duplicate every test, or really do anything manually to every test, I just wanted a large sub-set of the test suite to automatically test both implementations. I also wanted failures to clearly indicate which implementation had failed, i.e. I wanted them to run as separate test cases, because the reference implementation could potentially be at fault in some corner cases.

As I was using unittest, my solution was this: I added a class decorator that modified the test classes by removing every method that started with test_, replacing it with two methods, one for each implementation.

This provided almost exactly the same functionality as one of Hillel’s wished-for examples:

Add an output assertion to an optimized function in dev/testing, checking that on all invocations it matches the result of an unoptimized function

I just used a slightly different technique that better suited my needs, but also made great use of run-time program manipulation.

`morph_into`

As part of the Fluent-to-Python compilation process, I have a tree of AST objects that I want to simplify. Simplifications include things like replacing a “string join” operation that has just one string, with that single string – so we need completely different types of objects. Even in a language that has mutation this can be a bit of a pain, because we’ve got to update the parent object and tell it to replace this child with a different child, and there are many different types of parent object with very different shapes. So my solution was morph_into:

def morph_into(item, new_item):
    """
    Change `item` into `new_item` without changing its identity
    """
    item.__class__ = new_item.__class__
    item.__dict__ = new_item.__dict__

With this solution, we leave the identity of the object the same, so none of the pointers to it need to be updated. But its type and all associated data is changed into something else, so that, other than id(), the behaviour of item will now be indistinguishable from new_item. Not many languages allow you to do this!

I spent quite a lot of time wondering if I should be ashamed or proud of this code. But it turned out there was nothing to be ashamed of – it saved me writing a bunch of code and has had really no downsides.

Now, this technique won’t work for some things, like builtin primitives, so it can’t be completely generalised. But it doesn’t need that in order to be useful – all the objects I want to do this on are my own custom AST classes that share an interface, so it works and is “type safe” in its own way.

I’m far from the first person to discover this kind of trick when implementing compilers. In this comparison of several groups of people working on a compiler project, one of the most impressive results came from a single-person team who chose Python. She used way less code, and implemented way more features than the other groups, who all had multiple people on their teams and were using C++/Rust/Haskell etc. Fancy metaprogramming and dynamic typing were a big part of the difference, and by the sounds of it she used exactly the same kinds of things I used:

Another example of the power of metaprogramming and dynamic typing is that we have a 400 line file called visit.rs that is mostly repetitive boilerplate code implementing a visitor on a bunch of AST structures. In Python this could be a short ~10 line function that recursively introspects on the fields of the AST node and visits them (using the __dict__ attribute).

Again, I’m not claiming “dynamic typing is better than static typing” – I don’t think it’s even meaningful to do that comparison. I’m claiming that highly dynamic meta-programming tricks are indeed a significant part of real Python code, and really do make a big difference.

Pytest

Pytest does quite a few dynamic tricks. Hillel wishes that pytests functionality was more easily usable elsewhere, such as from a REPL. I’ve no doubt this is a legitimate complaint – as it happens, my own use cases involve sticking a REPL in my test, rather than sticking a pytest in my REPL. However, you can’t claim that pytest isn’t a valid example, or isn’t making use of Python’s dynamism – it does, and it provides a lot of useful functionality as a result, including:

Assert rewriting

The most obvious is perhaps their assert rewriting, which relies on modifying the AST of test modules to inject sub-expression information for when asserts fail. It makes test assertions often much more immediately useful.

Automatic dependency injection of fixtures

Pytest provides one of the few cases of automatic dependency injection in Python where I’ve thought it was a good idea. It also makes use of Python’s dynamism to make this dependency injection extremely low ceremony. All you need to do is add a parameter to your test function, giving the parameter the name of the fixture you want, and pytest will find that fixture in its registry and pass it to your function.

Others

This post is way too long already, and I’ve done very little actual searching for this stuff – almost all my examples are things that I’ve heard about in the past or done myself, so there must be far more than these in the real world out there. Here are a bunch more I thought of but didn’t have time to expand on:

PyTorch automatic differentiation which uses instrumented objects (similar to the SQLAlchemy example I presume), plus some kind of pattern matching on function calls that I haven’t had time to investigate.

Understanding Automatic Differentiation in 30 lines of Python is a great article on how you can build this kind of thing. Crucially, Python’s dynamism makes this kind of thing very accessible to mere mortals.
VCR.py: monkey patch all HTTP functions and record interactions, so that the second time we run a test we can use canned responses and avoid the network.
CCiW email tests: monkey patch Django’s Atomic decorator and mail sending functions to ensure we are using “queued email” appropriately inside transactions.
Lots of tricks in django-tagulous to improve usability for developers.
numba: JIT compile and run your Python code on a GPU with a single decorator.
drf-spectacular: iterate over all endpoints in a DRF project, introspecting serializers and calling methods with dummy request objects where necessary, to produce an OpenAPI schema.
In the stdlib, @total_ordering will look at your class and add missing rich comparison methods.
depending on an environment flag, automatically wrap all UI test cases in a decorator that takes a screenshot if the test fails.

EDIT: And some more I discovered after publishing this post, which look interesting:

mandala - “Computations that save, query and version themselves”
latexify - pretty print Python functions using LaTeX
JAX which has a JIT compiler of Python code and Autograd which implements automatic differentiation of Python code, possibly similar to PyTorch.

Conclusion

Why don’t we talk about these much? I think a large part of the answer is that the Python community cares about solving problems, and not about how clever your code is. Clever code, in fact, is looked down on, which is the right attitude – cleverness for the sake of it is always bad. Problem solving is good though. So libraries and projects that do these things don’t tend to brag about their clever techniques, but the problem that they solve.

Also, many libraries that use these things wrap them up so that you don’t have to know what’s going on – It Just Works. As a newbie, everything about computers is magical and you have to just accept that that’s how they work. Then you take it for granted, and just get on with using it.

On the other hand, for the implementer, once you understand the magic, it stops being magic, it’s just a feature that the language has.

Either way, pretty soon none of these things count as “hyper-programming” any more – in one sense, they are just normal Python programming, and that’s the whole point: Python gives you super powers which are not super powers, they are normal powers. Everyone gets to use them, and you don’t need to learn a different language to do so.

Perhaps we do need to talk about them more, though. At the very least, I hope my examples have sparked some ideas about the kinds of things that are possible in Python.

Happy hacking!

Test factory functions in Django

2022-11-25T16:07:02Z

When writing tests for Django projects, you typically need to create quite a lot of instances of database model objects. This page documents the patterns I recommend, and the ones I don’t.

Before I get going, I should mention that a lot of this can be avoided altogether if you can separate out database independent logic from your models. But you can only go so far without serious contortions, and you’ll probably still need to write a fair number of tests that hit the database.

The aim

We want the following:

Every test should specify each detail about database state it depends on
The test should not specify any detail it doesn’t depend on
We should be able to conveniently and succinctly write “what we mean”, without having to worry about lower level details, especially database schema details that are not intrinsic to the test.

These things are important so that you can understand tests in isolation, and so that changes not relevant to a test should not break that test. Otherwise you will spend a lot of your time fixing broken tests rather than actually doing the changes you need to do.

Using Django ORM create calls directly in your tests is not a great solution, because database constraints often force you to specify fields that you are not interested in.

Custom factory functions

The answer to this is simply to create your own “factory” functions, with optional keyword arguments (preferably keyword only) for almost everything. You can add parameters by hand as and when you need them.

Here are some simple but real examples from the Christian Camps in Wales booking system, which has a BookingAccount model and includes the ability to pay by cheque which is a ManualPayment object:

def create_booking_account(
    *,
    name: str = "A Booker",
    address_line1: str = "",
    address_post_code: str = "XYZ",
    email: str = Auto,
) -> BookingAccount:
    return BookingAccount.objects.create(
        name=name,
        email=email or next(BOOKING_ACCOUNT_EMAIL_SEQUENCE),
        address_line1=address_line1,
        address_post_code=address_post_code,
    )

def create_manual_payment(
    *,
    account: BookingAccount = Auto,
    amount: int = 1,
) -> ManualPayment:
    return ManualPayment.objects.create(
        account=account or create_booking_account(),
        amount=amount,
        payment_type=ManualPaymentType.CHEQUE,
    )

You can find the rest of this project’s test factory functions with this search on GitHub

A few patterns to note:

The Auto sentinel

A number of places here we used a default value of Auto, which is a custom object defined as follows:

class _Auto:
    """
    Sentinel value indicating an automatic default will be used.
    """

    def __bool__(self):
        # Allow `Auto` to be used like `None` or `False` in boolean expressions
        return False


Auto: Any = _Auto()

We use Auto instead of None or something else, because:

Sometimes you need to specify None as an actual value (for nullable DB fields), but not want it as a default.
Often the correct default needs to be defined dynamically:
- you need to create another object at runtime, as in the account: BookingAccount = Auto line above
- a sensible and correct default depends on some other argument, so requires some logic in the body of the function.

We create a singleton value Auto so we can do if foo is Auto checks.

We also give it a type Any so that type checkers don’t complain about using it as a default value. It doesn’t break type checking for the functions calling our factory functions.

Constraints and sequences

Often you have the problem that a unique constraint on a field makes it difficult to provide a static default. As in the example above, I’m using a really simple technique to deal with this – generate a sequence of values that are unlikely to be specified manually in a test. In the above code, you can see BOOKING_ACCOUNT_EMAIL_SEQUENCE which is defined like this at the module level:

BOOKING_ACCOUNT_EMAIL_SEQUENCE = sequence(lambda n: f"booker_{n}@example.com")

Every time we call next() on this object, we get a distinct value, so we avoid issues with constraints.

The sequence utility is actually super simple, but presented here in all it’s type-hinted glory:

import itertools
from typing import Any, Callable, Generator, TypeVar

T = TypeVar("T")


def sequence(func: Callable[[int], T]) -> Generator[T, None, None]:
    """
    Generates a sequence of values from a sequence of integers starting at zero,
    passed through the callable, which must take an integer argument.
    """
    return (func(n) for n in itertools.count())

You could do something even simpler though – just use a generator expression at the top level:

BOOKING_ACCOUNT_EMAIL_SEQUENCE = (f"booker_{n}@example.com" for n in itertools.count())

There can be some cases where you need something more complicated than this (for example to be able to reset sequences) but they are rare in my experience and fairly easy to write [1].

Delegation and sub-objects

Factory functions often delegate to other factory functions, as in the examples above.

It’s also quite common to want to specify something about a sub-object. Rather than build up a tree of objects as the caller, I often add a parameter to the top-level factory itself. This gives you some independence from the actual schema.

Special purpose factories

You aren’t limited to one factory function per model, you can have as many as you like. For example you might have create_staff_user and create_customer which take different parameters, but both happen to return the same User model.

Sensible and minimal defaults

As far as possible, the factory function should pick sensible defaults, based on what parameters were passed in if any. If it can’t because the caller contradicted themselves, it should raise an exception.

I normally take the approach that the defaults should produce minimal and pristine objects, while being complete and usable.

For example, if your model supports soft-delete via deactivation, active=False would be a bad default. On the other hand, creating lots of related objects in order to be “realistic” would not be a good idea.

You should be pragmatic. For example, for a User object, if a brand new, “pristine” user is always forced to go through an on-boarding flow on your website, meaning that every single page but the on-boarding page is blocked until they complete it, then has_onboarded=True is probably a more sensible default – only a few of your tests will want has_onboarded=False.

In many cases, your main business logic may already have functions that initialise database objects into sensible states when creating them, or when changing their states. Test factory functions will often delegate to them, so that things are set up as close as possible to how they would be normally.

Simplified interface

A good factory function will often simplify things for the caller.

For example, in the CCiW project mentioned, the Camp model has a leaders relationship, which is a many-to-many. For several good reasons, the leaders are not User objects, but Person objects, where Person has some metadata and another many-to-many (!) with User objects. However, when I’m writing a test, I might want to be able to say something like:

user = create_user()
camp = create_camp(leader=user)
login(user)

Here, I just care that the user is conceptually the leader of the camp. I don’t care:

that a camp can have more than one leader
that the Camp is actually related to the User object via a Person object.

Sometimes I don’t care about specifying who the leader actually is, just that there is one, so I might want to pass leader=True.

My factory function ends up looking like this:

def create_camp(
    *,
    leader: Person | User | bool = Auto,
    leaders: list[Person | User] = Auto,
) -> Camp:
    ...

It’s redundant, but it’s easy to use, and this approach means you isolate many of your tests from needing changing. Sometimes my factory functions end up having a lot of parameters, and they’re unlikely to win any beauty contests — but who really cares? They are easy to understand and modify.

Type hints

Type hints are great for getting good help in your editor when writing tests. Use them!

Don’t depend on defaults

If a test requires a certain value, and it happens to be the default that the factory will use, the test should still specify it. This makes the test more robust, and allows the factory to change the defaults. If a test doesn’t specify it, it means it doesn’t care, and it should work with any value the factory happens to choose.

Enhancements

If you are using pytest (which I recommend, along with pytest-django), Haki Benita has nice post that explains how to use factory functions as pytest fixtures.

What not to do

Now for the anti-patterns. If you’re happy with the answer above, you don’t need to read this bit.

JSON/YAML fixtures

Django docs used to encourage you to define models in JSON/YAML fixtures for use in tests. Don’t do that! I’ll let Carl Meyer tell you why.

There are some legitimate cases for using these kinds of fixtures in tests – in particular, where you might use the same/similar fixture files for loading data in a production environment. This is typically when you have essentially static data that is defined by some external reality, which happens to be stored in a database table in your app – such as a list of countries and their ISO codes.

`**kwargs`

When writing factory functions, rather than adding loads of parameters, it may be tempting to just let them accept **kwargs and pass those on to the underlying model. I usually prefer not to do that, because:

you get much less help when writing tests
you tend to end up overly tied to the actual schema

django-dynamic-fixture

I used to use django-dynamic-fixture to avoid the tedium of manual factory functions, but have since moved away from that. You are just introducing a layer between yourself and the code that you actually need to write, and have to stop it from doing things you don’t want etc. It also doesn’t understand the “business logic” needed to come up with sensible defaults.

factory_boy

OK, factory_boy, this is like my comments for django-dynamic-fixture, only more so.

Let me put it this way:

You’ve been tasked with providing a procedure for creating model instances, where that procedure will have sensible defaults, but will allow the caller to override them. You have to decide what are the appropriate language features of Python to use. Do you:

Create a function or a method, with parameters for overriding defaults, or,
Define a new class that inherits from Factory, and use the body of the class statement to define a procedure?

If you chose A), congratulations, you got the right answer! You will be rewarded for using the language as it was meant to be used, by things like:

Automatic help inside your editor, both for the parameters and the returned value.
Static type checking if you want it.
Everyone being able to modify your code without looking up some documentation.

If you chose B), you get points for novelty. But you will be punished as follows:

You will have to invent things like:
- nested class Meta for essential configuration of FactoryOptions
- nested class Params
- Trait
- PostGeneration
- @post_generation
- LazyAttribute
- @lazy_attribute
- @lazy_attribute_sequence
- LazyFunction
- SubFactory
- RelatedFactory
- SelfAttribute
- a debug mode (of course)
- and much, much more!
You will have to write thousands of lines of code (1700+), thousands more of tests (5000+), and page after page of documentation (16,000+ words) to support all this.
You will have to get people to read that documentation. Instead of which, they will spend their evenings writing snarky blog posts complaining about all your hard work!
You will have an Open Source side project with hundreds of open issues, fun!
You will get less than zero help from your editor when using these factories – not only will it just display **kwargs for inputs, it will think the output is a Factory instance, which it is not.
For people to find what parameters they can pass to a Factory, they will have to look up the model, and inspect the Factory definition and decipher its “traits” etc.

I don’t want to add any further to the burden of the authors – they have suffered enough already! But I do want to deal with a few objections:

But factory_boy can also create instances without saving them!

This is useful if you want to avoid hitting the DB while being able to test a model method that doesn’t need the DB. In Django, it’s extremely easy to do that without help, because if you aren’t going to save a model instance, you don’t need to worry about any attributes other than the ones you specify – models don’t run validation in the constructor – and so you don’t need factories at all:

def test_address_formatted():
    address = Address(line1="123 Main St", line2="London")
    assert address.formatted() == "123 Main St\nLondon\n")

If you really need it, you could always add a commit: bool = True parameter to your factory functions.

But factory_boy has faker integration!

If you want randomized and realistic looking data, you can use faker directly with almost exactly the same amount of code:

from faker import Faker

faker = Faker()

def create_user():
    return User.objects.create(
        name=faker.name(),
    )

But factory_boy has a create_batch method!

If you need to create a bunch of things, you can just do this:

payments = [create_manual_payment() for i in range(0, 100)]

which really isn’t very hard, and also means you can have arguments that vary depending on the loop variable.

But, because I’m very generous, I will write you a create_batch function for free. Not only that, I’ll add type hints for free, and I’ll leave it right here where you can find it, in the public domain:

from typing import Callable, TypeVar

T = TypeVar("T")


def create_batch(factory: Callable[..., T], count, /, **kwargs) -> list[T]:
    """
    Use `factory` callable to create `count` objects, passing along kwargs
    """
    return [factory(**kwargs) for i in range(0, count)]

Now you can do the following, and your editor and static type checker will know exactly what type of objects payment_1 and payment_2 are:

payment_1, payment_2 = create_batch(create_manual_payment, 2, amount=10)

Conclusion

You don’t need to install anything to create factory functions. Just use built-in language features, and maybe a few tiny helpers like I’ve shown, and you’re good!

The only real issue with my approach is that sometimes it can feel a bit tedious adding another parameter. But slightly tedious code that is extremely easy to understand and modify, and helps you in all the ways I’ve described, is still a big win in my book. There will be many days when you long for slightly tedious code that just works.

Happy testing!

Footnotes

[1]

Advanced sequences:

Sometimes, you might want to reset your sequences, and perhaps automatically between every test case. I would implement that as follows. Replace the previous sequence implementation with:

from __future__ import annotations
import itertools
from typing import Generic, Iterator, TypeVar


T = TypeVar("T")


class sequence(Generic[T]):
    instances: list[sequence] = []

    def __init__(self, func: Callable[[int], T]) -> None:
        self.func = func
        self.reset_sequence()
        self.instances.append(self)

    def reset_sequence(self):
        self.seq: Iterator[T] = (self.func(n) for n in itertools.count())

    def __next__(self) -> T:
        return next(self.seq)

To reset automatically between each test case, assuming use of pytest, add the following autouse fixture to conftest.py:

@pytest.fixture(autouse=True)
def reset_all_sequences():
    from myproject.factory_utils import sequence  # or wherever

    for instance in sequence.instances:
        instance.reset_sequence()

Raising exceptions or returning error objects in Python

2022-06-06T11:29:35+01:00

The other day I got a question about some old code I had written which, instead of raising an exception for an error condition as the reader expected, returned an error object:

With your EmailVerifyTokenGenerator class, why do you return error classes instead of raising custom errors? You could still pass the email to a custom VerifyExpired exception.

https://github.com/cciw-uk/cciw.co.uk/blob/eae8005feb95a5383663e69e92d80e11effe5ee6/cciw/bookings/email.py#L41

I think I'm too eager to raise errors but maybe there's something I'm missing with classes 😁!

The code in question is below (slightly modified and with several uninteresting methods removed). It is part of a system for doing email address verification via magic links in emails.

from dataclasses import dataclass

class VerifyFailed:
    pass


VerifyFailed = VerifyFailed()  # singleton sentinel value


@dataclass
class VerifyExpired:
    email: str


class EmailVerifyTokenGenerator:
    def token_for_email(self, email):
        ...

    def email_from_token(self, token):
        """
        Extracts the verified email address from the token, or a VerifyFailed
        constant if verification failed, or VerifyExpired if the link expired.
        """
        max_age = settings.EMAIL_VERIFY_TIMEOUT
        try:
            unencoded_token = self.url_safe_decode(token)
        except (UnicodeDecodeError, binascii.Error):
            return VerifyFailed
        try:
            return self.signer.unsign(unencoded_token, max_age=max_age)
        except (SignatureExpired,):
            return VerifyExpired(self.signer.unsign(unencoded_token))
        except (BadSignature,):
            return VerifyFailed

To sum up, we have a function that extracts an email address from a token, checking the HMAC signature that it is bundled with. There are 3 possibilities we want to deal with:

The happy case – we’ve got a valid HMAC code, we just need the email address returned.
We’ve got an invalid signature.
We’ve got a valid but expired signature. We want to handle this separately, because we’d like to streamline the user experience for getting a new token generated and sent to them, which means we need to return the email address.

It’s using Django’s signer functions to do the heavy lifting, but that doesn’t matter for our purposes, because we are wrapping it up.

To get going on designing our API for this bit of code, here are some bad options:

We could have a pair of methods or functions: extract_email_from_token and check_signature, which can be used independently. This is bad because you could easily use extract_email_from_token and completely forget to use check_signature.

The principle here is that we want the developer using this API to fall into the pit of success. Either the developer should get their code perfectly correct, or if they don’t, it either will be obviously broken and not work at all, or at least not subtly flawed with some nasty bug, like a security issue.
We could have email_from_token() method or function with a return value of a tuple containing (email_address: str, valid_and_not_expired_signature: bool).

This has a similar issue to above – the calling code could use email_address and forget to check the validity boolean.

Having ruled those out, we’ve got two main contenders for how to design email_from_token():

We could make it raise exceptions for the “invalid” or “expired” cases. We need to pass extra data for the latter, but we can put it inside the exception object – as noted by the original questioner.
We could make it return error objects for the error cases, as coded above.

Both of these satisfy the “pit of success” criterion. If the developer accidentally does not handle the error cases, they won’t have a bug where we verified an email address that should not be verified. We will instead probably have a crasher of some kind, which in the case of a web app, like this one, means a 500 error page being seen, and something in our logs that makes it pretty clear what happened.

If we choose to raise exceptions, naive code which doesn’t check for the exceptions will simply get no further – the exception will propagate up and terminate the handler. With the second option where we return error objects, those objects can’t be accidentally converted into success values – the VerifyExpired object contains the email address, but it is a completely different shape of value from the happy case.

Both of these approaches, to some degree, respect the principle that can be summed up as Parse Don’t Validate. Instead of merely validating a token and extracting an email address as two independent things, we are parsing a token, and encoding the result of the validation in the type of objects that will then flow through the program.

But which is better?

One of the influences on my thinking is the way types work in Haskell and other similar language which make it very easy to create types and constructors. In Haskell, the following is all the code you need to define a return type for this kind of function, and the 3 different data constructors you need, which then do double duty for pattern matching:

data EmailVerificationResult = EmailVerified string
                             | VerifyFailed
                             | VerifyExpired string

Now, Python is not nearly as succinct, but dataclasses were a big improvement for defining things like VerifyExpired.

In Haskell, due to static type checking, this pattern makes it pretty much impossible for the calling code to accidentally fail to handle the return value correctly. But even in Python, which doesn’t have that built in, I think there are some compelling advantages:

We expect the calling code to handle all the different return values at some point, and at the same point. (This is unlike some code where we can raise an exception that we never expect the calling code to specifically handle – it will be handled by more generic methods at a different layer). It therefore makes sense that we treat all 3 values as the same kind of thing — they are just different return values.
If you instead raise exceptions, you are immediately forcing the calling code into a special control flow structure, namely the try/except dance, which can be inconvenient.
In particular, if you want to hand off processing of the value to some other function or code for handling, you can’t do it easily. For example, code like this would be fine with the “return error object” method, but significantly complicated by the “raise exception” method:
```
verify_result = verifier.email_from_token(token)
log_verify_result(request.ip_address, verify_result)
# etc.
```

In the years since I wrote the code, however, some perhaps more compelling arguments have come along for the error object method.

First, with some small changes (specifically, removing the sentinel singleton value), we can now add a type signature for email_from_token:

def email_from_token(self, token, max_age=None) -> str | VerifyFailed | VerifyExpired:
    ...

(You may need typing.Union for older Python versions)

This is a benefit in itself from a documentation point of view, and for better IDE/editor help.

We can go further with mypy. We can structure our calling code as follows to make use of mypy exhaustiveness checking:

from typing_extensions import assert_never

verified_email = EmailVerifyTokenGenerator().email_from_token(token)
if isinstance(verified_email, VerifyFailed):
    ...
elif isinstance(verified_email, VerifyExpired):
    ...
elif isinstance(verified_email, str):
    ...
else:
    assert_never(verified_email)

Now, if we remove one of these blocks, let’s say the VerifyExpired one (or if we added another option to email_from_token), mypy will catch it for us:

error: Argument 1 to "assert_never" has incompatible type "VerifyExpired"; expected "NoReturn"

With the error object method, we could also write our handling code using structural pattern matching. The equivalent code, including our mypy exhaustiveness check, now looks like this:

verified_email = EmailVerifyTokenGenerator().email_from_token(token)
match verified_email:
    case VerifyFailed():
        ...
    case VerifyExpired(expired_token_email):
        ...
    case str():
        ...
    case _:
        assert_never(verified_email)

This has destructuring of the email address in VerifyExpired built in – it is bound to the name expired_token_email in that branch.

Hopefully this gives a good justification for the approach I took with this code. There are times when exceptions are better – generally when the things mentioned above don’t apply, or the opposite applies – but I think error objects also have their place, and sometimes are a much better solution.

REPL Python programming and debugging with IPython

2022-05-04T07:26:56+01:00

When programming in Python, I spend a large amount of time using IPython and its powerful interactive prompt, not just for some one-off calculations, but for significant chunks of actual programming and debugging. I use it especially for exploratory programming where I’m unsure of the APIs available to me, or what the state of the system will be at a particular point in the code.

While it looks like I’ve been doing this for 12 years now, I’m not sure how widespread this method of working is, as I rarely hear other people talk about it. So I thought it would be worth sharing in some detail.

If you like videos and want to see this method in action for writing a test, you could have a look at the django-functest video about writing tests interactively, or skip to the bit where I start using the REPL.

Setup

You normally need IPython installed into your current virtualenv for it to work properly:

pip install ipython

or:

uv add --dev ipython

(See Tips section below if installing IPython is not possible)

Methods

There are basically two ways I open an IPython prompt. The first is by running it directly from a terminal:

$ ipython
Python 3.9.5 (default, Jul  1 2021, 11:45:58)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

In a Django project, ./manage.py shell can also be used if you have IPython installed, with the advantage that it will properly initialise Django for you.

This works fine if you want to explore writing some “top level” code – for example, a new bit of functionality where the entry points have not been created yet. However, most code I write is not like that. Most of the time I find myself wanting to write code when I am already 10 levels of function calls down – for example:

I’m writing some view code in a Django application, which has a request object – an object you could not easily recreate if you started from scratch at an IPython prompt.
or, model layer code such as inside a save() method that is itself being called by some other code you have not written, like the Django admin or some signal.
or, inside a test, where the setup code has already created a whole bunch of things that are not available to you when you open IPython.

For these cases, I use the second method:

Find the bit of code I want to modify, explore or debug. This will often be my own code, but could equally be a third party library. I’m always working in a virtualenv, so even with third party libraries ,“go to definition” in my editor will take me straight to a writable copy of the code (apart from code not written in Python).
Insert the code for an IPython prompt and save the file:
```
import IPython; IPython.embed()
```
I have this bound to a function key in my editor.

So the code might end up looking like this, if it was a Django view for example:
```
def contact_us(request):
    if request.method == "POST":
        form = ContactUsForm(request.POST)
        if form.is_valid():
            import IPython; IPython.embed()

        # …
```
I sometimes also might put the snippet inside a new if clause that I add to catch a particular condition, especially when using this for debugging.
Trigger the code in the appropriate way. For the above case, it would involve first running the Django development server in a terminal, then opening the web page, filling out the form and pressing submit. For a test, it would be running the specific test from a terminal. For command line apps it would be running the app directly.
In the terminal, I will now find myself in the IPython REPL, and I can go ahead and:
- work out what code I need to write
- or debug the code that I’m confused about.

Note that you can write and edit multi-line code at this REPL – it’s not quite as comfortable as an editor, but it’s OK, and has good history support. There’s much more to say about IPython and its features that I won’t write here, you can learn about it in the docs.

For those with a background in other languages, it might also be worth pointing out that a Python REPL is not a different thing from normal Python. Everything you can do in normal Python, like defining functions and classes, is possible right there in the REPL.

Once I’m done with my exploring, I can copy any useful snippets back from the REPL into my real code, using the history to scan back through what I typed.

Advantages

The advantages of this method are:

You can explore APIs and objects much more easily when you actually have the object, rather than docs about the object, or what your editor’s auto-complete tools believe to be true about the object. For example, what attributes and methods are available on Django’s HttpRequest? You don’t have to ensure you’ve got correct type annotations, and hope they are complete, or make assumptions about what the values are - you’ve got the object right there, you can inspect it, with extensive and correct tab completion. You can actually call functions and see what they do.

For example, Django’s request object typically has a user attribute which is not part of the HttpRequest definition, because of how it is added later. It’s visible in a REPL though.
You can directly explore the state of the system. This can be a huge advantage for both exploratory programming and debugging.

For debugging, pdb and similar debugging tools and environments will often provide you with “the state of the system”, and they are much better at being able to step through multiple layers of code. But I often find that the power and comfort of an IPython prompt is much nicer for exploring and finding solutions.

The feel of this kind of environment is not quite a smooth as REPL-driven programming in Lisp, but I still find it hugely enjoyable and productive. Compared to many other methods, like iterating on your code followed by manual or automated testing, it cuts the latency of the feedback loop from seconds or minutes to milliseconds, and that is huge.

Tips and gotchas

IPython has tons of cool features that will help you in a REPL environment, like %autoreload (thanks haki), and many other cool magics. You should spend the time getting to know them!
In a multi-threaded (or multi-process) environment, IPython prompts won’t play nice. Turn off multi-threading if possible, or otherwise ensure that you don’t hit that gotcha.
If you do get messed up in a terminal, you may need to manually find the processes to kill and do reset in your terminal.
With the Django development server:
- It’s multi-threaded by default, so either ensure that you don’t hit the view code multiple times, or use --nothreading.
- Beware of auto-reloading, which will mess you up if you are still in an IPython prompt when it kicks in. Either use --noreload or just ensure you exit IPython cleanly before doing anything that will trigger a reload.
Beware of environments that capture standard input/output, that will break this technique.
pytest captures standard input and breaks things by default. You can turn it off using -s. Also if you are using pytest-xdist you should remember to do -n0 to turn off multiple processes.
When using IPython.embed() there’s an annoying bug involving closures and undefined names due to Python limitations. It often shows itself when using generator expressions, but at other times too. It can often be worked around by doing:
```
globals().update(locals())
```
If for some reason you can’t use IPython, but only have access to the standard library, the one-liner you need to run a (basic) REPL at any point in your code is this:
```
import code; code.interact(local=locals())
```

End

That’s it, I hope you found it useful. Do you have any other tips for using this technique?

Luke Plant's home page (Posts about Django)

Help my website is too small

Links

Keeping things in sync: derive vs test

pyastgrep and custom linting

Examples

Method

Tips

Conclusion

Links

Enforcing conventions in Django projects with introspection

Re-using CSS for the wrong HTML with Sass

Django and Sass/SCSS without Node.js or a build step

Future proofing

Python’s “Disappointing” Superpowers

Test factory functions in Django

Raising exceptions or returning error objects in Python

Links

REPL Python programming and debugging with IPython

Setup

Methods

Advantages

Tips and gotchas

End

Links