Luke Plant's home page (Posts about Python type hints)

Statically checking Python dicts for completeness

2025-06-27T11:09:03+01:00

In Python, I often have the situation where I create a dictionary, and want to ensure that it is complete – it has an entry for every valid key.

Let’s say for my (currently hypothetical) automatic squirrel-deterring water gun system, I have a number of different states the water tank can be in, defined using an enum:

from enum import StrEnum

class TankState(StrEnum):
    FULL = "FULL"
    HALF_FULL = "HALF_FULL"
    NEARLY_EMPTY = "NEARLY_EMPTY"
    EMPTY = "EMPTY"

In a separate bit of code, I define an RGB colour for each of these states, using a simple dict.

TANK_STATE_COLORS = {
    TankState.FULL: 0x00FF00,
    TankState.HALF_FULL: 0x28D728,
    TankState.NEARLY_EMPTY: 0xFF9900,
    TankState.EMPTY: 0xFF0000,
}

This is deliberately distinct from my TankState code and related definitions, because it relates to a different part of the project - the user interface. The UI concerns shouldn’t be mixed up with the core logic.

This dict is fine, and currently complete. But I’d like to ensure that if I add a new item to TankState, I don’t forget to update the TANK_STATE_COLORS dict.

With a growing ability to do static type checks in Python, some people have asked how we can ensure this using static type checks. The short answer is, we can’t (at least at the moment).

But the better question is “how can we (somehow) ensure we don’t forget?” It doesn’t have to be a static type check, as long as it’s very hard to forget, and if it preferably runs as early as possible.

Instead of shoe-horning everything into static type checks, let’s just make use of the fact that this is Python and we can write any code we want at module level. All we need to do is this:

TANK_STATE_COLORS = {
    # …
}
for val in TankState:
    assert val in TANK_STATE_COLORS, f"TANK_STATE_COLORS is missing an entry for {val}"

That’s it, that’s the whole technique. I’d argue that this is a pretty much optimal, Pythonic solution to the problem. No clever type tricks to debug later, just 2 lines of plain simple code, and it’s impossible to import your code until you fix the problem, which means you get the early checking you want. Plus you get exactly the error message you want, not some obscure compiler output, which is also really important.

It can also be extended if you want to do something more fancy (e.g. allow some values of the enum to be missing), and if it does get in your way, you can turn it off temporarily by just commenting out a couple of lines.

That’s not quite it

OK, in a project where I’m using this a lot, I did eventually get bored of this small bit of boilerplate. So, as a Pythonic extension of this Pythonic solution, I now do this:

TANK_STATE_COLORS: dict[TankState, int] = {
    TankState.FULL: 0x00FF00,
    TankState.HALF_FULL: 0x28D728,
    TankState.NEARLY_EMPTY: 0xFF9900,
    TankState.EMPTY: 0xFF0000,
}
assert_complete_enumerations_dict(TANK_STATE_COLORS)

Specifically, I’m adding:

a type hint on the constant
a call to a clever utility function that does just the right amount of Python magic.

This function needs to be “magical” because we want it to produce good error messages, like we had before. This means it needs to get hold of the name of the dict in the calling module, but functions don’t usually have access to that.

In addition, it wants to get hold of the type hint (although there would be other ways to infer it without a type hint, there are advantages this way), for which we also need the name.

The specific magic we need is:

the clever function needs to get hold of the module that called it
it then looks through the module dictionary to get the name of the object that has been passed in
then it can find type hints, and do the checking.

So, because you don’t want to write all that yourself, the code is below. It also supports:

having a tuple of Enum types as the key
allowing some items to be missing

using Literal as the key. So you can do things like this:

X: dict[typing.Literal[-1, 0, 1], str] = {
    -1: "negative",
    0: "zero",
    1: "positive",
}
assert_complete_enumerations_dict(X)

It’s got a ton of error checking, because once you get magical then you really don’t want to be debugging obscure messages.

Enjoy!

I hereby place the following code into the public domain - CC0 1.0 Universal.

import inspect
import itertools
import sys
import typing
from collections.abc import Mapping, Sequence
from enum import Enum

from frozendict import frozendict


def assert_complete_enumerations_dict[T](the_dict: Mapping[T, object], *, allowed_missing: Sequence[T] = ()):
    """
    Magically assert that the dict in the calling module has a
    value for every item in an enumeration.
    The dict object must be bound to a name in the module.
    It must be type hinted, with the key being an Enum subclass, or Literal.
    The key may also be a tuple of Enum subclasses

    If you expect some values to be missing, pass them in `allowed_missing`
    """
    assert isinstance(the_dict, Mapping), f"{the_dict!r} is not a dict or mapping, it is a {type(the_dict)}"

    frame_up = sys._getframe(1)  # type: ignore[reportPrivateUsage]
    assert frame_up is not None
    module = inspect.getmodule(frame_up)
    assert module is not None, f"Couldn't get module for frame {frame_up}"
    msg_prefix = f"In module `{module.__name__}`,"

    module_dict = frame_up.f_locals
    name: str | None = None
    # Find the object:
    names = [k for k, val in module_dict.items() if val is the_dict]
    assert names, f"{msg_prefix} there is no name for {the_dict}, please check"

    # Any name that has a type hint will do, there will usually be one.
    hints = typing.get_type_hints(module)
    hinted_names = [name for name in names if name in hints]
    assert (
        hinted_names
    ), f"{msg_prefix} no type hints were found for {', '.join(names)}, they are needed to use assert_complete_enumerations_dict"
    name = hinted_names[0]

    hint = hints[name]
    origin = typing.get_origin(hint)
    assert origin is not None, f"{msg_prefix} type hint for {name} must supply arguments"
    assert origin in (
        dict,
        typing.Mapping,
        Mapping,
        frozendict,
    ), f"{msg_prefix} type hint for {name} must be dict/frozendict/Mapping with arguments to use assert_complete_enumerations_dict, not {origin}"

    args = typing.get_args(hint)
    assert len(args) == 2, f"{msg_prefix} type hint for {name} must have two args"

    arg0, _ = args
    arg0_origin = typing.get_origin(arg0)
    if arg0_origin is tuple:
        # tuple of Enums
        enum_list = typing.get_args(arg0)
        for enum_cls in enum_list:
            assert issubclass(
                enum_cls, Enum
            ), f"{msg_prefix} type hint must be an Enum to use assert_complete_enumerations_dict, not {enum_cls}"

        items = list(itertools.product(*(list(enum_cls) for enum_cls in enum_list)))
    elif arg0_origin is typing.Literal:
        items = typing.get_args(arg0)
    else:
        assert issubclass(
            arg0, Enum
        ), f"{msg_prefix} type hint must be an Enum to use assert_complete_enumerations_dict, not {arg0}"
        items = list(arg0)

    for item in items:
        if item in allowed_missing:
            continue

        # This is the assert we actually want to do, everything else is just error checking:
        assert item in the_dict, f"{msg_prefix} {name} needs an entry for {item}"

Python Type Hints: pyastgrep case study

2023-10-05T09:45:02Z

In a previous post, I did a case study on my attempts to add type hints to parsy. In this post, I’m continuing the series, but in a very different project.

A while back I forked an existing tool called astpath to create my own tool pyastgrep, fixing various bugs and usability issues. In the process I rewrote significant parts of the existing code, and added quite a lot of my own. This was a pretty good opportunity for me to attempt to use static typing throughout, since I was not limited by backwards compatibility in the design. Plus it’s a relatively very small amount of code, making it much easier than many of the larger projects I maintain, while still being big enough to be much more than a toy.

There are at least 5 different ways that type hints can be used in Python, but this post focuses on static type checking and interactive programming help. In particular, I wanted to get mypy to catch errors for me, and I was incorporating it in my CI/testing workflows.

About pyastgrep

This tool is a utility to allow you to grep Python code for specific syntax elements using XPath as a powerful query language. The main functions are:

intelligently work out which files to search (respecting .gitignore files etc.)
parse the Python files as AST and convert to XML
apply a user-supplied XPath expression to search for specific AST elements
print the results (with the complexity of handling different context strategies and colouring)

Here is an example showing pyastgrep searching its own code base for all usages of names (variables etc) that contain the substring idx:

$ pyastgrep './/Name[contains(@id, "idx")]'
src/pyastgrep/files.py:60:5:    current_idx = 0
src/pyastgrep/files.py:64:9:        linebreak_idx = python_file_bytes.find(b"\n", current_idx)
src/pyastgrep/files.py:64:55:        linebreak_idx = python_file_bytes.find(b"\n", current_idx)
src/pyastgrep/files.py:65:12:        if linebreak_idx < 0:
src/pyastgrep/files.py:66:38:            line = python_file_bytes[current_idx:]
src/pyastgrep/files.py:68:38:            line = python_file_bytes[current_idx:linebreak_idx]
src/pyastgrep/files.py:68:50:            line = python_file_bytes[current_idx:linebreak_idx]
src/pyastgrep/files.py:72:12:        if linebreak_idx < 0:
src/pyastgrep/files.py:75:13:            current_idx = linebreak_idx + 1
src/pyastgrep/files.py:75:27:            current_idx = linebreak_idx + 1
src/pyastgrep/printer.py:244:9:        start_line_idx = line_index - before_context
src/pyastgrep/printer.py:245:9:        end_line_idx = line_index + after_context
src/pyastgrep/printer.py:246:9:        stop_line_idx = end_line_idx + 1
src/pyastgrep/printer.py:246:25:        stop_line_idx = end_line_idx + 1
src/pyastgrep/printer.py:248:19:        if (path, end_line_idx) in self.printed_context_lines:
src/pyastgrep/printer.py:252:19:        if (path, start_line_idx - 1) not in self.printed_context_lines:
src/pyastgrep/printer.py:253:57:            header = self.formatter.format_header(path, start_line_idx)
src/pyastgrep/printer.py:257:44:        code = "\n".join(result.file_lines[start_line_idx:stop_line_idx])
src/pyastgrep/printer.py:257:59:        code = "\n".join(result.file_lines[start_line_idx:stop_line_idx])
src/pyastgrep/printer.py:260:13:        for idx in range(start_line_idx, stop_line_idx):
src/pyastgrep/printer.py:260:26:        for idx in range(start_line_idx, stop_line_idx):
src/pyastgrep/printer.py:260:42:        for idx in range(start_line_idx, stop_line_idx):
src/pyastgrep/printer.py:261:51:            self.printed_context_lines.add((path, idx))

Things I liked

There were some things I really liked about the flow of using a type checker, and I was able to lean on the checker pretty heavily for some things.

One example of using type-driven programming was going between the layer of my code that found matches and the layer that printed them. I found that the search function needed to go from returning a simple Iterable[Match] eventually all the way to Iterable[Match | MissingPath | ReadError | NonElementReturned | FileFinished]. In each case, I could do something like:

add the new return value, something like yield MissingPath(...), in the body of the function.
fix up the function type signature, in response to the type error that mypy would now report.
then respond to the type error that mypy would report in the places that consumed this iterable, usually by implementing handling of the new result type. isinstance checks can be used to drive type narrowing and satisfy mypy that everything is fine.

It was nice when this went through multiple layers, knowing that something was checking it for me and driving me to the next bit of code that needed fixing. Having the types there explicitly also helps to make you conscious of decisions you are making about how the layers are working, which was helpful in keeping the layers straight, so that I can use this code both as a library and a command line tool.

Being able to convert various exceptions, errors and corner cases to sum types in this way was also great, and I leaned on this heavily — probably more than if I hadn’t been using a type checker. I quite like these kind of ad-hoc sum types – in some ways they often work nicer than sum types in Haskell etc.

Use of mypy has encouraged me to use lots of small custom classes, because I can use “Find references” to search the code base for everything related to a particular type, or a particular attribute or method. For custom classes I create and use within the code base, this works perfectly, which is really nice. The same goes for renaming things using IDE tools (I’m using Emacs and the pyright LSP server).

I have a high degree of confidence that I’ll be able to come back to this code base after years without touching and be able to navigate it and make changes very easily.

Issues

However, I have a long list of complaints about issues I found too!

mypy just isn’t checking that code.

You have to turn on at least:

check_untyped_defs = true
disallow_untyped_calls = true
disallow_untyped_defs = true
disallow_incomplete_defs = true

Otherwise you should not be expecting mypy to actually find typing errors. In general, it can be frustrating not knowing whether the lack of red is because you haven’t got errors, or because mypy just isn’t checking code, or can’t meaningfully check anything.

IOBase vs BinaryIO

pyastgrep, in common with many similar tools, allows you to process standard input as well as grepping files named on the command line or found by directory walking. So I had to have branches for that, and I had tests for it too.

I hit a bug regarding encodings, where files not encoded in UTF-8 were causing the tool to crash. I ended up doing an internal change that switched a bunch of types from str to bytes. The code worked, the tests passed, and mypy reported no errors. But later – thankfully before a release – I noticed that I had broken stdin handling.

It turned out that according to mypy, IOBase.read() returns Any, and not the actual type bytes or str. I had been using IOBase as a type for some of my arguments, which meant that mypy didn’t pick up the problem – if Any appears anywhere, it’s like throwing a “silently disable everything this touches” bomb into the type checker.

Now, I had been alerted to the problem earlier – mypy thinks that sys.stdin is of type typing.TextIO, not IOBase. However, typing.TextIO is not something you test for at run-time, so it interacts badly with the very useful isinstance type guards and type narrowing. So I had ended up using IOBase as that seemed less problematic.

In other words, I had added type hints, and I had added correct type hints. But they weren’t correct enough, and therefore turned out to be “wrong”. It was very disappointing that despite the effort I had put in, this kind of type error still got through.

Fixing the bug involved writing a better, more accurate test that more closely emulated actual stdin handling, and then a very simple change. Fixing my type hints, however, was a much bigger task.

It involved a long journey of understanding regarding type guards and stricter type guards, because non-strict type guards (which is what we have at the moment) turned out not to work how I thought. The eventual refactoring now uses the typing.BinaryIO type hint, and some code that seems somewhat fragile in terms of type checking – because there is no way of doing a “negative type guard” for BinaryIO, I have to order my if/elif/else clauses in exactly the right way.

It’s also closing the barn door after the horse has bolted – I had already fixed the bug, and added much more thorough tests to prove it was fixed. I was hoping that static type checking would have caught this before I had to do that.

Missing types for imports

I saw a bug I hadn’t fixed, and one that again I thought mypy might have caught.

It turns out I had added ignore_missing_imports = true early in development to reduce the errors to a manageable set. This silenced errors relating to lxml and effectively gave me a bunch of Any types floating around instead of something useful.

Again, this was “my fault”, but I feel it’s fairly typical of what will happen in the real world. Switching to ignore_missing_imports = false can cause so many problems that it will be hard to justify the cost in many cases. In this case, the type stubs it wants me to install require me to “fix” a whole load of static type checking issues relating to lxml that don’t correspond to real run-time issues.

Equality checks

I switched a bunch of code paths from str to Path at one point. mypy gave me some help, but less than I wanted. For example, this reports no error:

path: Path = Path(...)

if path == "-":
    ...

A comparison between a str and a Path always returns False, so it’s not a useful thing to do, and therefore a developer error. I meant to do the comparison to "-" before I converted the input str objects to internal Path objects. It’s conceptually a TypeError, but not actually one. Thankfully I had tests that failed.

mypy caching bugs

Several times I had to blow away .mypy_cache to get errors to appear. This is not a fundamental problem with the idea of static typing, but it makes very big difference to the whole flow of leaning on mypy. I often noticed only when I knew that mypy should be reporting errors due to a change I just made, but it wasn’t – I have no idea how many other times it was happening. When interpreting “mypy reports no errors”, there are now about half a dozen reasons why that might be the case, only one of which is “you fixed everything and your code is correct”.

Third party types

Are types provided by third party libs or typeshed reliable?

No, they are not. For example, I discovered this one in typeshed/stdlib/_ast.pyi among many others:

class AST:
    ...
    # TODO: Not all nodes have all of the following attributes

This is probably not typeshed’s “fault” — it’s a problem trying to retro-fit static types to a language and stdlib that wasn’t designed for them.

Duck typing

As soon as you want to use duck typing, which I did want to, you’ve got more work ahead of you, work that isn’t really to do with solving your actual problem. There are solutions such as Protocol, but I’m simply noting you do have significant amounts of extra work for the type checker to understand idiomatic Python.

False security

I fairly often got that sense of “it type checks, and everything works first time I run it, cool!”

Sometimes, it was an illusion though – take this code:

if args.color == UseColor.AUTO:
    colorer = make_default_colorer()
elif args.colors == UseColor.NEVER:
    colorer = NullColorer()

I had typed colors instead of the correct color in the second branch, but I got no squiggly red lines. This was because of a lurking Any – the argparse args object is actually an Any. This tripped me up, because I didn’t have any tests for that line of code.

Having strict = true in your mypy config doesn’t fix this. I think I’d need a way to say “warn me for about anywhere that Any leaks into my code base”, but even if it existed I imagine I would not like it.

Exhaustiveness checking

mypy fails to find the obvious issue in this bit of code:

def foo() -> None:
    if 1 + 1 == 3:
        x = "hello"
    print(x)

I hoped I’d at least get a warning for a potential unbound variable. This comes up with cases where you want exhaustiveness checking, like:

def print_greeting(username: str, type: Greeting) -> None:
    if type == Greeting.HELLO:
        greeting = "hello"
    elif type == Greeting.GOODBYE:
        greeting = "goodbye"

    print(f"{greeting} {username}")

You can get this right using an else branch with assert_never, but it’s annoying that you have to remember to do this.

[Update 2024: while mypy doesn’t see the potential unbound variable error, even with --strict, I’ve found pyright does spot it, and these days I’m using pyright more and more]

Decorators

Type hints for decorators are … bad. If you want parameterised decorators, or other people’s decorators that don’t have types

[Apologies for the unfinished sentence above. I don’t want to risk a repeated head-against-table moment that the first attempt triggered, once was painful enough]

pyright

More recently I’ve tried pyright as an alternative to mypy. Generally I’ve found it to be significantly less buggy. However, mypy has a lot going for it in terms of features and extensions, and I don’t really want to have two different type checkers. At the moment I’m experimenting with mainly using pyright for interactive checks in my editor, and using mypy for pre-commit/CI checks.

The overlapping feature sets can be kind of annoying though. For example, for the potential unbound variable error above, the latest version of pyright does warn you. It also has built-in exhaustiveness checking without needing the assert_never technique. However, in one case it wasn’t working for me, until I finally tracked down the issue — mypy was able to handle this code and correctly deduce the base class of my enum, but pyright wasn’t:

try:
    from enum import StrEnum
except ImportError:
    from backports.strenum import StrEnum  # type: ignore [no-redef]

I eventually found an adequate solution that keeps them both happy — but only because I’m writing this blog post and don’t want to look stupid. Normally it would be “stuff is broken, maybe it’s me, maybe it’s them, gotta move on”.

In addition, in some places pyright does not support the assert_never technique that mypy needs, and reports an error. There are other pain points if you try to use both.

There are quite a few places where you find pyright doesn’t do the same thing as mypy because pyright is more correct. Microsoft people tend to know what they are talking about when it comes to type systems. But it means you may find yourself digging through large numbers of issues closed with the “as designed” tag to find answers.

[Update 2024: where I have the choice, I usually use exclusively pyright these days. I use it both as a standalone tool from CLI and in CI etc, and with the lsp-pyright LSP server in Emacs. It now has enough support for assert_never – it sometimes reports a warning for unreachable code, but that’s fine as it’s not an error. pyright seems to understand my Python code a lot better, especially when it comes to type narrowing. When I run mypy over the same bit of code, it reports a large number of spurious errors where pyright is correctly silent, and I don’t think there are many cases where mypy spots errors that pyright fails to see]

Summary

Overall, despite listing more bad things than good, I’m actually happy with the addition of mypy as a required static type checker in this project.

The disappointments I’ve listed may come from my experience and enjoyment of languages like Haskell where you really can lean on the type checker. In those languages, you find both that the rewards of static type checking are massively higher, and that the effort required to use them is massively lower. Haskell type signatures, for instance, are often not needed, and much easier to write and understand than Python’s.

Perhaps the most positive outlook is “static type checking in Python is just an advanced linter, of course it’s not actually reliable”. This can be hard to accept, though, due to the amount of work you have to do to get any real benefit above and beyond linters like flake8 and ruff that, with virtually no changes to your code or workflow, catch a lot of issues with a very low false positive rate.

In terms of tips and advice:

You need to turn up error reporting and spend considerable effort configuring mypy, especially in larger projects.
If you want something approaching reliability, your entire stack of libraries needs to have been designed with static types from the beginning, so you don’t have to use stubs. This means:
- probably not much in stdlib. You’re going to need to wrap everything.
- probably not much that was created more than 5 years ago.

Maybe this works well for MegaCorps with an army of developers and a very large code base that they have to get under control somehow. I think for many projects, you are going to be happy with static type checking in Python only if you can resign yourself to a very low level of reliability, and are mostly leaning on other techniques for correctness, like an extensive test suite.

The different uses of Python type hints

2023-04-05T20:49:38+01:00

When you use type hints/annotations in Python, you could be using them for one or more of at least 5 different things:

Interactive programming help

Many editors will be able to use type hints to give you help with:

autocomplete (e.g. suggesting methods that actually exist on the type of objects you are dealing with)
immediate error checking (e.g. squiggly red lines under mistakes)
code navigation (e.g. jump to definition)
refactoring (e.g. renaming a method and all uses of it)

To be clear, these features can often work without type hints – for example, I’ve used jedi very effectively to provide jump to definition etc. on code bases without any type hints, and many linters like flake8 and ruff also provide a lot of functionality without types. But type hints can help a lot in cases where static analysis wouldn’t otherwise give clear answers.

Static type checking

This is where a tool uses the type annotations to check the correctness of your code. I’m distinguishing this from the “immediate error checking” use case above, even though the same tool such as mypy or pyright might be behind it, because I’m specifically thinking of cases where your code will be rejected by something in your process (like checks in your CI build system) if type checking reports errors. This case is different because help in your editor can be ignored if it is wrong, but static type checks built into your deployment processes etc. either cannot be skipped, or require extra work to ignore. “Friendly assistant” and “opinionated gatekeeper” are quite different personas, and you might not appreciate them equally.

Runtime behaviour determination

Running Python code can use reflection/introspection techniques to inspect type hints and change behaviour on that basis. The most obvious example in my mind is pydantic, which uses type annotations to determine what correct inputs look like, and also serialisation/deserialisation behaviour. Another example would be runtime type checking like beartype or typeguard.

UPDATE: Another use is dependency injection (Lagom) (thanks antoinewdg), and other notable projects leaning on runtime use of type hints include FastAPI and Typer (thanks ubernostrum). There are probably lots more.

Code documentation

A type hint can be used to tell a user what kind of objects a function accepts or produces. This can be extracted in automatically created docs, or shown in your editor (in which case it also falls under “Interactive programming help”).

An interesting application of this is drf-spectacular, which uses type hints as well as other information to extract an OpenAPI spec from a project using DRF. This spec, as well as serving as documentation or input to tools like redoc or Swagger UI, can also be used for type checking or code generation in another language, typically for web frontend code, via tools like OpenAPI generator.

Compiler instructions

I don’t know how many people are doing this, but tools like mypyc will use type hints to compile Python code to something faster, like C extensions. Using type annotations to speed up Python is quite hard in practice.

Conclusion

There isn’t really a point of this post other than to say “be aware of these different use cases”. This awareness can be very important when you are in any discussion about the usefulness or necessity of type hints – which scenarios are you thinking about?

Also, when you are weighing up whether to add type hints, you might decide to do so in order to support some of these but not others – as I did for the parsy library.

Finally, and this is a bit of a gotcha to leave you with, you may need to be very aware of the different use cases when thinking about correctness. If you see count: int, what kind of guarantee do you have that the count name is actually bound to an int object at runtime? Is the type hint invoking some runtime checking, or is it merely docs, or hoping for a static type check that might not actually happen? You probably need to know which it is!

Test factory functions in Django

2022-11-25T16:07:02Z

When writing tests for Django projects, you typically need to create quite a lot of instances of database model objects. This page documents the patterns I recommend, and the ones I don’t.

Before I get going, I should mention that a lot of this can be avoided altogether if you can separate out database independent logic from your models. But you can only go so far without serious contortions, and you’ll probably still need to write a fair number of tests that hit the database.

The aim

We want the following:

Every test should specify each detail about database state it depends on
The test should not specify any detail it doesn’t depend on
We should be able to conveniently and succinctly write “what we mean”, without having to worry about lower level details, especially database schema details that are not intrinsic to the test.

These things are important so that you can understand tests in isolation, and so that changes not relevant to a test should not break that test. Otherwise you will spend a lot of your time fixing broken tests rather than actually doing the changes you need to do.

Using Django ORM create calls directly in your tests is not a great solution, because database constraints often force you to specify fields that you are not interested in.

Custom factory functions

The answer to this is simply to create your own “factory” functions, with optional keyword arguments (preferably keyword only) for almost everything. You can add parameters by hand as and when you need them.

Here are some simple but real examples from the Christian Camps in Wales booking system, which has a BookingAccount model and includes the ability to pay by cheque which is a ManualPayment object:

def create_booking_account(
    *,
    name: str = "A Booker",
    address_line1: str = "",
    address_post_code: str = "XYZ",
    email: str = Auto,
) -> BookingAccount:
    return BookingAccount.objects.create(
        name=name,
        email=email or next(BOOKING_ACCOUNT_EMAIL_SEQUENCE),
        address_line1=address_line1,
        address_post_code=address_post_code,
    )

def create_manual_payment(
    *,
    account: BookingAccount = Auto,
    amount: int = 1,
) -> ManualPayment:
    return ManualPayment.objects.create(
        account=account or create_booking_account(),
        amount=amount,
        payment_type=ManualPaymentType.CHEQUE,
    )

You can find the rest of this project’s test factory functions with this search on GitHub

A few patterns to note:

The Auto sentinel

A number of places here we used a default value of Auto, which is a custom object defined as follows:

class _Auto:
    """
    Sentinel value indicating an automatic default will be used.
    """

    def __bool__(self):
        # Allow `Auto` to be used like `None` or `False` in boolean expressions
        return False


Auto: Any = _Auto()

We use Auto instead of None or something else, because:

Sometimes you need to specify None as an actual value (for nullable DB fields), but not want it as a default.
Often the correct default needs to be defined dynamically:
- you need to create another object at runtime, as in the account: BookingAccount = Auto line above
- a sensible and correct default depends on some other argument, so requires some logic in the body of the function.

We create a singleton value Auto so we can do if foo is Auto checks.

We also give it a type Any so that type checkers don’t complain about using it as a default value. It doesn’t break type checking for the functions calling our factory functions.

Constraints and sequences

Often you have the problem that a unique constraint on a field makes it difficult to provide a static default. As in the example above, I’m using a really simple technique to deal with this – generate a sequence of values that are unlikely to be specified manually in a test. In the above code, you can see BOOKING_ACCOUNT_EMAIL_SEQUENCE which is defined like this at the module level:

BOOKING_ACCOUNT_EMAIL_SEQUENCE = sequence(lambda n: f"booker_{n}@example.com")

Every time we call next() on this object, we get a distinct value, so we avoid issues with constraints.

The sequence utility is actually super simple, but presented here in all it’s type-hinted glory:

import itertools
from typing import Any, Callable, Generator, TypeVar

T = TypeVar("T")


def sequence(func: Callable[[int], T]) -> Generator[T, None, None]:
    """
    Generates a sequence of values from a sequence of integers starting at zero,
    passed through the callable, which must take an integer argument.
    """
    return (func(n) for n in itertools.count())

You could do something even simpler though – just use a generator expression at the top level:

BOOKING_ACCOUNT_EMAIL_SEQUENCE = (f"booker_{n}@example.com" for n in itertools.count())

There can be some cases where you need something more complicated than this (for example to be able to reset sequences) but they are rare in my experience and fairly easy to write [1].

Delegation and sub-objects

Factory functions often delegate to other factory functions, as in the examples above.

It’s also quite common to want to specify something about a sub-object. Rather than build up a tree of objects as the caller, I often add a parameter to the top-level factory itself. This gives you some independence from the actual schema.

Special purpose factories

You aren’t limited to one factory function per model, you can have as many as you like. For example you might have create_staff_user and create_customer which take different parameters, but both happen to return the same User model.

Sensible and minimal defaults

As far as possible, the factory function should pick sensible defaults, based on what parameters were passed in if any. If it can’t because the caller contradicted themselves, it should raise an exception.

I normally take the approach that the defaults should produce minimal and pristine objects, while being complete and usable.

For example, if your model supports soft-delete via deactivation, active=False would be a bad default. On the other hand, creating lots of related objects in order to be “realistic” would not be a good idea.

You should be pragmatic. For example, for a User object, if a brand new, “pristine” user is always forced to go through an on-boarding flow on your website, meaning that every single page but the on-boarding page is blocked until they complete it, then has_onboarded=True is probably a more sensible default – only a few of your tests will want has_onboarded=False.

In many cases, your main business logic may already have functions that initialise database objects into sensible states when creating them, or when changing their states. Test factory functions will often delegate to them, so that things are set up as close as possible to how they would be normally.

Simplified interface

A good factory function will often simplify things for the caller.

For example, in the CCiW project mentioned, the Camp model has a leaders relationship, which is a many-to-many. For several good reasons, the leaders are not User objects, but Person objects, where Person has some metadata and another many-to-many (!) with User objects. However, when I’m writing a test, I might want to be able to say something like:

user = create_user()
camp = create_camp(leader=user)
login(user)

Here, I just care that the user is conceptually the leader of the camp. I don’t care:

that a camp can have more than one leader
that the Camp is actually related to the User object via a Person object.

Sometimes I don’t care about specifying who the leader actually is, just that there is one, so I might want to pass leader=True.

My factory function ends up looking like this:

def create_camp(
    *,
    leader: Person | User | bool = Auto,
    leaders: list[Person | User] = Auto,
) -> Camp:
    ...

It’s redundant, but it’s easy to use, and this approach means you isolate many of your tests from needing changing. Sometimes my factory functions end up having a lot of parameters, and they’re unlikely to win any beauty contests — but who really cares? They are easy to understand and modify.

Type hints

Type hints are great for getting good help in your editor when writing tests. Use them!

Don’t depend on defaults

If a test requires a certain value, and it happens to be the default that the factory will use, the test should still specify it. This makes the test more robust, and allows the factory to change the defaults. If a test doesn’t specify it, it means it doesn’t care, and it should work with any value the factory happens to choose.

Enhancements

If you are using pytest (which I recommend, along with pytest-django), Haki Benita has nice post that explains how to use factory functions as pytest fixtures.

What not to do

Now for the anti-patterns. If you’re happy with the answer above, you don’t need to read this bit.

JSON/YAML fixtures

Django docs used to encourage you to define models in JSON/YAML fixtures for use in tests. Don’t do that! I’ll let Carl Meyer tell you why.

There are some legitimate cases for using these kinds of fixtures in tests – in particular, where you might use the same/similar fixture files for loading data in a production environment. This is typically when you have essentially static data that is defined by some external reality, which happens to be stored in a database table in your app – such as a list of countries and their ISO codes.

`**kwargs`

When writing factory functions, rather than adding loads of parameters, it may be tempting to just let them accept **kwargs and pass those on to the underlying model. I usually prefer not to do that, because:

you get much less help when writing tests
you tend to end up overly tied to the actual schema

django-dynamic-fixture

I used to use django-dynamic-fixture to avoid the tedium of manual factory functions, but have since moved away from that. You are just introducing a layer between yourself and the code that you actually need to write, and have to stop it from doing things you don’t want etc. It also doesn’t understand the “business logic” needed to come up with sensible defaults.

factory_boy

OK, factory_boy, this is like my comments for django-dynamic-fixture, only more so.

Let me put it this way:

You’ve been tasked with providing a procedure for creating model instances, where that procedure will have sensible defaults, but will allow the caller to override them. You have to decide what are the appropriate language features of Python to use. Do you:

Create a function or a method, with parameters for overriding defaults, or,
Define a new class that inherits from Factory, and use the body of the class statement to define a procedure?

If you chose A), congratulations, you got the right answer! You will be rewarded for using the language as it was meant to be used, by things like:

Automatic help inside your editor, both for the parameters and the returned value.
Static type checking if you want it.
Everyone being able to modify your code without looking up some documentation.

If you chose B), you get points for novelty. But you will be punished as follows:

You will have to invent things like:
- nested class Meta for essential configuration of FactoryOptions
- nested class Params
- Trait
- PostGeneration
- @post_generation
- LazyAttribute
- @lazy_attribute
- @lazy_attribute_sequence
- LazyFunction
- SubFactory
- RelatedFactory
- SelfAttribute
- a debug mode (of course)
- and much, much more!
You will have to write thousands of lines of code (1700+), thousands more of tests (5000+), and page after page of documentation (16,000+ words) to support all this.
You will have to get people to read that documentation. Instead of which, they will spend their evenings writing snarky blog posts complaining about all your hard work!
You will have an Open Source side project with hundreds of open issues, fun!
You will get less than zero help from your editor when using these factories – not only will it just display **kwargs for inputs, it will think the output is a Factory instance, which it is not.
For people to find what parameters they can pass to a Factory, they will have to look up the model, and inspect the Factory definition and decipher its “traits” etc.

I don’t want to add any further to the burden of the authors – they have suffered enough already! But I do want to deal with a few objections:

But factory_boy can also create instances without saving them!

This is useful if you want to avoid hitting the DB while being able to test a model method that doesn’t need the DB. In Django, it’s extremely easy to do that without help, because if you aren’t going to save a model instance, you don’t need to worry about any attributes other than the ones you specify – models don’t run validation in the constructor – and so you don’t need factories at all:

def test_address_formatted():
    address = Address(line1="123 Main St", line2="London")
    assert address.formatted() == "123 Main St\nLondon\n")

If you really need it, you could always add a commit: bool = True parameter to your factory functions.

But factory_boy has faker integration!

If you want randomized and realistic looking data, you can use faker directly with almost exactly the same amount of code:

from faker import Faker

faker = Faker()

def create_user():
    return User.objects.create(
        name=faker.name(),
    )

But factory_boy has a create_batch method!

If you need to create a bunch of things, you can just do this:

payments = [create_manual_payment() for i in range(0, 100)]

which really isn’t very hard, and also means you can have arguments that vary depending on the loop variable.

But, because I’m very generous, I will write you a create_batch function for free. Not only that, I’ll add type hints for free, and I’ll leave it right here where you can find it, in the public domain:

from typing import Callable, TypeVar

T = TypeVar("T")


def create_batch(factory: Callable[..., T], count, /, **kwargs) -> list[T]:
    """
    Use `factory` callable to create `count` objects, passing along kwargs
    """
    return [factory(**kwargs) for i in range(0, count)]

Now you can do the following, and your editor and static type checker will know exactly what type of objects payment_1 and payment_2 are:

payment_1, payment_2 = create_batch(create_manual_payment, 2, amount=10)

Conclusion

You don’t need to install anything to create factory functions. Just use built-in language features, and maybe a few tiny helpers like I’ve shown, and you’re good!

The only real issue with my approach is that sometimes it can feel a bit tedious adding another parameter. But slightly tedious code that is extremely easy to understand and modify, and helps you in all the ways I’ve described, is still a big win in my book. There will be many days when you long for slightly tedious code that just works.

Happy testing!

Footnotes

[1]

Advanced sequences:

Sometimes, you might want to reset your sequences, and perhaps automatically between every test case. I would implement that as follows. Replace the previous sequence implementation with:

from __future__ import annotations
import itertools
from typing import Generic, Iterator, TypeVar


T = TypeVar("T")


class sequence(Generic[T]):
    instances: list[sequence] = []

    def __init__(self, func: Callable[[int], T]) -> None:
        self.func = func
        self.reset_sequence()
        self.instances.append(self)

    def reset_sequence(self):
        self.seq: Iterator[T] = (self.func(n) for n in itertools.count())

    def __next__(self) -> T:
        return next(self.seq)

To reset automatically between each test case, assuming use of pytest, add the following autouse fixture to conftest.py:

@pytest.fixture(autouse=True)
def reset_all_sequences():
    from myproject.factory_utils import sequence  # or wherever

    for instance in sequence.instances:
        instance.reset_sequence()

Python Type Hints: case study on parsy

2022-11-21T21:07:02Z

I have been trying to like static type checking in Python. For most of my Django projects, I get annoyed and give up, so I’ve had a go with some smaller projects instead. This blog post documents how it went with Parsy, a parser combinator library I maintain.

Intro to Parsy

I need to explain a few things about Parsy.

In Parsy, you build up Parser objects via a set of primitives and combinators. Each Parser object has a parse method that accepts strings [1] and returns some object – which might be a string for your lowest building blocks, but quickly you build more complex parsers that return different types of objects. A lot of code is written in “fluent” style where you chain methods together.

Here are some basics:

The primitive string just matches and returns the input:

import parsy as P

hello = P.string("hello")
assert hello.parse("hello") == "hello"

But we can change the result to some other type of object:

true_parser = P.string("true").result(True)
assert true_parser.parse("true") == True

We can map the parse result using a callable. This time I’m starting with a regex primitive, which returns strings, but converting to ints:

int_parser = P.regex(r"-?\d+").map(int)
assert int_parser.parse("123") == 123

We can discard things we don’t care about in a number of ways, such as with these “pointy” operators that point to the important bit:

whitespace = P.regex(r"\s*")
assert (whitespace >> int_parser << whitespace).parse(" 123    ") == 123

We can have a sequence of items, here with some separator we don’t care about collecting:

three_ints = P.seq(
  int_parser << P.string("-"),
  int_parser << P.string("-"),
  int_parser
)
assert three_ints.parse("123-45-67") == [123, 45, 67]

If we want something better than a list to store different components in (usually we do), we can use a keyword argument form of seq to give names for the components and collect them in a dict instead of a list, and instead of .map we can do .combine_dict to convert the result into some other object:

from dataclasses import dataclass

@dataclass
class Date:
    year: int
    month: int
    day: int

date_parser = P.seq(
  year=P.regex(r"\d{4}").map(int) << P.string("-"),
  month=P.regex(r"\d{2}").map(int) << P.string("-"),
  day=P.regex(r"\d{2}").map(int)
).combine_dict(Date)

assert date_parser.parse("2022-11-19") == Date(year=2022, month=11, day=19)

We can have alternatives using the | operator:

bool_parser = P.string("true").result(True) | P.string("false").result(False)
assert bool_parser.parse("false") == False

That’s enough to understand the rest of this post, let’s have a look at my 4 different approaches to improving the static type checking story for Parsy.

Simple types

The most obvious thing to do is to add Parser as the return value for a bunch of methods and operators, and other type hints wherever we can for the input arguments. For example, .map is:

def map(self, map_function: Callable) -> Parser:
   ...

What type of object does the parse method return? We don’t know, so we have to do:

def parse(self, input: str) -> Any:
    ...

And this is our first hint that the static type checking isn’t very useful. For example, this faulty code will now type check:

x: str = int_parser.parse("123")

We can see the return value is not going to be the right type, but our static type checker sees the Any and allows it.

The type checker is also not catching TypeError exceptions for .map() that it definitely ought to be able to. For example, suppose we have this faulty parser which is going to attempt to construct a timedelta from strings like "7 days ago":

from datetime import timedelta
days_ago_parser = (P.regex(r"\d+") << P.string(" days ago")).map(timedelta)

This is going to fail every time with:

TypeError: unsupported type for timedelta days component: str

That’s because we forgot a .map(int) after the P.regex parser.

This kind of type error is caught for you by mypy and pyright if you try to pass a string to the timedelta constructor, but here, we’ve got no constraints on the callable that would enable the type checker to pick it up. When we put a bare Callable in a signature, as we did above for the map method, we are really writing Callable[..., Any], so all proper type checking effectively gets disabled.

If you want static type checking, this is not what you expect or want! It’s especially important for Parsy, because almost all the mistakes you are likely to make will be of this nature. Most Parsy code consists of parsers defined at a module level, which means that as soon as you import the module, you’ll know whether you have attempted to use combinator methods that don’t exist, for example, so there is little usefulness in a type checker being able to tell you this. What you want to know is whether you are going to get TypeError or similar when you call the parse method at runtime.

Can we achieve that?

Generics

The answer is to make the Parser type aware of what kind of object it is going to output. This can be achieved by parameterising the type of Parser with a generic type, and is the second main approach.

Very often generics are used for homogeneous containers, to capture the type of the object they contain. Here, we don’t have a container as such. We are instead capturing the type of the object that the parser instance is going to produce when you call .parse() (assuming it succeeds, I’m ignoring all failure cases).

Some of the key type signatures in our Parser code now look like this (lots of details elided):

from typing import TypeVar, Generic

OUT = TypeVar("OUT")
OUT1 = TypeVar("OUT1")
OUT2 = TypeVar("OUT2")


@dataclass
class Result(Generic[OUT]):
    value: OUT
    ...


class Parser(Generic[OUT]):

    def parse(self, stream: str) -> OUT:
        ...

    def map(self: Parser[OUT1], map_function: Callable[[OUT1], OUT2]) -> Parser[OUT2]:
        ...

The main point is that we are capturing the type of output using a type parameter that our static type checker can track, which means that we can indeed catch the type errors that were ignored by the previous approach. I got this to work, and you can see my results in this PR (not merged).

The reason it isn’t merged is that this approach breaks down as soon as you have things like *args or **kwargs where the arguments need to be of different types. We have exactly that, multiple times, once you care about generics. For example, the seq combinator takes sequence of parsers as input, and runs them in order, collecting their results. All of them are Parser instances, so that would work fine with the previous approach, but they could all have different output types. There is no way to specify a type signature for this, as well as for alt, combine and combine_dict.

The best you can do is specify that they return Parser[Any]. This means you are downgrading to no type checking. This problem is going to apply to all but the most trivial cases – it’s difficult to come up with many real world examples where you don’t need sequencing.

Some people would say “well, it works sometimes, so it is better than nothing”. The problem is that you when you start writing your parser, you may well really benefit from the type checking and start to lean on it. Then, as soon as you get beyond the level of your simple (single part) objects and are creating parsers for more complex (multiple part) objects in your language, the type checking silently disappears. Or, if you have strictness turned up high, your type checker will complain about the introduction of Any, but you won’t be able to do anything about it.

Both of these are really bad developer UX in my opinion. If the type checker is going to give up and go home at 2pm on the second day of work, it would be better for it not to show up, which would push developers to lean on other, more reliable methods, like writing tests.

Typed Parsy fork

So we come to my third option, which builds on the second. Can we redesign Parsy so that it doesn’t have any Any? This would be a backwards incompatible fork that removes any API that is impossible to fully type, and attempts to provide some good enough replacements.

This was my most ambitious foray into static type checking in Python, and below are my notes from two perspectives – first implementation, which is important for any potential future contributors and maintainers, and secondly, usage, for the people who actually might want to use this fork.

Implementation perspective

I didn’t complete this for reasons that will become clear. Overall, I’d say working “alongside” mypy and pyright was quite nice at points, and other times really difficult. To keep this article short, I’ve moved most of this section to footnotes. Here are the bullet points:

You can see my results in the typed-parsy repo, especially the single source file.
I dropped support for anything but str as input type, as a simplification.
I discovered that pyright can really shine in various places that mypy is lacking, particularly error messages.
But sometimes they fight each other [2]
I couldn’t work out how Protocols work with respect to operators and dunder methods. [3]
Covariance is tricky, and you have to understand it. [4]
forward_declaration made my head explode. [5]
There are lots of places marked TODO where I just couldn’t solve the new problems I had, even after getting rid of the most problematic code, and I had to give up and do type: ignore quite a few times.

Overall

Too much of this was just too hard, especially given we are only talking about a few hundred lines of code. It seemed much worse than doing the same thing in Haskell for some reason. This might be just because the language wasn’t designed for it. Even just the syntax for types is significantly worse.

I think another issue is that there is no REPL for type level work. Normally when I’m trying to debug something, I jump into a REPL. Working with actual, concrete values is so much easier, and so much closer to the immediate connection that makes programming enjoyable.

An additional problem is that static type checkers have to worry about issues that may not be relevant to my code.

Finally, the type system we have right now for Python is so far behind what Python can actually express. But it isn’t necessarily obvious when this is the case. The answer to “why doesn’t this work” is anywhere between, “I made a dumb mistake”, “I need to learn more”, “there’s a bug in mypy” and “that’s impossible (at the moment)”.

As someone who needs to worry about future contributors and maintainers, these are serious issues. In addition to getting code to work, contributors would also have to get the type checks to pass, and ensure they weren’t breaking type checking for users, which is an extra burden.

Using it

So much for the pains of implementing typed-parsy, what would it look like for a user?

First, it happens that for a lot of typical usage, the user wouldn’t need to worry about types or adding type hints at all, but would still get type checking, which is great.

Second, for the resulting code, mypy and pyright do a very good job of checking almost every type error you would normally make in your parsers. The few places where we lose type safety are limited and don’t result in Any escaping and trashing everything from then on.

However, if you do need to write type signatures, which you probably will if you have mypy settings turned up high and you want to make your own combinator functions (i.e. something that takes a Parser and returns a new Parser), which is fairly common, you’re going to need to understand a lot to create type hints that are both correct and useful.

In addition, to achieve all this, we had to make some big sacrifices:

Sequences

You can’t implement seq, alt, .combine or .combine_dict in a type safe way (without degrading everything to Parser[Any] from then on), and I had to remove them.

The biggest issue is seq, and especially the convenience of the keyword argument version to name things. The alternative I came up with – using & operator for creating a tuple of two results – does work, but turns out to be pretty ugly.

Below are some incomplete extracts from the SQL SELECT example, which illustrate the readability loss fairly well. We have some enum types and dataclasses to hold Abstract Syntax Tree nodes for a SQL parser:

class Operator(enum.Enum):
    EQ = "="
    LT = "<"
    GT = ">"
    LTE = "<="
    GTE = ">="

@dataclass
class Comparison:
    left: ColumnExpression
    operator: Operator
    right: ColumnExpression

# dataclass for Select etc

We then have a bunch of parsers for different components, which we assemble into larger parsers for bigger things, like Comparison or Select. With normal parsy it looks like this:

comparison = P.seq(
    left=column_expr << padding,
    operator=P.from_enum(Operator),
    right=padding >> column_expr,
).combine_dict(Comparison)

SELECT = P.string("SELECT")
FROM = P.string("FROM")
WHERE = P.string("WHERE")

select = P.seq(
    _select=SELECT + space,
    columns=column_expr.sep_by(padding + P.string(",") + padding, min=1),
    _from=space + FROM + space,
    table=table,
    where=(space >> WHERE >> space >> comparison).optional(),
    _end=padding + P.string(";"),
).combine_dict(Select)

There are some things you need to understand: seq runs a sequence of parsers in order, and with its keyword arguments version allows you to give names to each one, to produce a dictionary of results. .combine_dict then passes these to a callable using **kwargs syntax.

.combine_dict also has a neat trick of skipping items whose names start with underscores, to allow you to deal with things that you need to parse but want to discard, like _select, _from and _end above. Notice how easy it is to read the select parser and see what things we are picking out.

With typed-parsy, this was the best I could do, formatted using Black:

comparison = (
    (column_expr << padding) & P.from_enum(Operator) & (padding >> column_expr)
).map(lambda t: Comparison(left=t[0][0], operator=t[0][1], right=t[1]))

SELECT = string("SELECT")
FROM = string("FROM")
WHERE = string("WHERE")

select = (
    (
        (SELECT + space)
        >> (column_expr.sep_by(padding + string(",") + padding, min=1))
        << (space + FROM + space)
    )
    & table
    & ((space >> WHERE >> space >> comparison).optional() << (padding + string(";")))
).map(lambda t: Select(columns=t[0][0], table=t[0][1], where=t[1]))

This is a pretty massive regression. We can’t use commas to separate parts as before, and we can’t use keyword arguments to name components any more. Instead we have to use tuples, and when we end up with nested tuples it’s awful – I literally couldn’t work out how to write the tuple indexing correctly and had to just keep guessing until I got it right. And that’s just with 3 items, some parsers might have many more items in a sequence, each of which adds another level of nesting.

This might have been significantly better if we still had tuple unpacking within function/lambdas signatures (which was removed in Python 3), but still not very nice.

It is kind of impressive that mypy and pyright will handle all this and tell you about type violations very reliably. This is possible because they have support for indexing tuples i.e. in the above statements it can tell you what the types of t[0], t[0][1] etc are. In an IDE, tools like pyright will tell you what you need to supply for .map() – for example for the select statement, inside the final .map call:

(map_fn: (tuple[tuple[list[Field | String | Number], Table], Comparison | None]) -> OUT2@map) -> Parser[OUT2@map]",

But this isn’t my idea of developer friendly. The loss of readability is huge, even for simple cases.

For comparison, I looked at funcparserlib, the Python parsing library closest to Parsy. They claim “fully typed”, but it turns out that their sequencing operator, which returns only tuples and so isn’t as usable as seq, flattens nested tuples. This is much better for usability, but is also impossible to type, so they introduce Any at this point, and so lose type checking, the thing I’ve been trying to avoid.

UPDATE 2022-11-24: I had another idea about how to approach this. Parsy could provide seq2, seq3, seq4 etc. combinators which return 2-tuples, 3-tuples, 4-tuples etc. These functions, which would have to be implemented the long way, would handle the nested tuple unpacking for you, without losing type safety. As an overload, they could also optionally include the functionality of .combine without loss of safety, and in this way you would get close to the usability of the seq(*args) version, with just the annoyance that you have to change the function if you want to add another argument. This still wouldn’t give you the usability of the keyword argument version of seq, but it would probably be a significant improvement.

Error messages

If you are relying on types to fix you up, instead of readable code, then you depend on the error messages your type checker will emit. Below is an example of an error I got when attempting to port the example code in simple_logo_lexer.py, which has a function flatten_list:

Argument 1 to "map" of "Parser" has incompatible type
"Callable[[List[List[T]]], List[T]]"; expected
"Callable[[List[Tuple[Tuple[str, int], str]]], List[T]]"

Here was another one I hit:

Argument 1 to "map" of "Parser" has incompatible type
"Callable[[Tuple[List[List[Tuple[List[List[OUT]], List[OUT]]]],
List[Tuple[List[List[OUT]], List[OUT]]]]], List[List[Tuple[List[List[OUT]],
List[OUT]]]]]";
expected "Callable[[Tuple[List[List[Tuple[List[List[OUT]],
List[OUT]]]], List[Tuple[List[List[OUT]], List[OUT]]]]], List[OUT]]"

I can’t remember what mistake produced that, but I did notice that the code worked perfectly at runtime. Also, pyright didn’t complain, only mypy. This is the kind of thing that makes people hate static typing.

One of the main principles of parsy is that it should be very easy to pull data into appropriate containers where every field is named, as well as typed – rather than a parse that returns lists of nested lists and tuples and dicts. This is why namedtuple is a big improvement over tuple, and dataclasses are a big step up again. But at the level of types and error messages, it seems we are back in the dark ages, with nested tuples and lists everywhere.

@generate decorator

Being able to add conditional logic and control flow into parsers is really important for some cases, and Parsy has an elegant solution in the form of the @generate decorator. Getting this to work in typed-parsy turned out to be only partially possible.

The first issue is that, unlike other ways of building up parsers, the user will need to write a type signature to get type checking, and it’s complex enough that there is no reasonable way for someone to understand what type signature they need without looking up the docs, which is poor usability.

Having done so, they can get type checking on the return type of the parser, and code that uses that parser. However, they get no type checking related to parsers used within the function. The Generator type assumes a homogeneous stream of yield and send types, whereas we have pairs of yield/send types which need to match within the pair, but each pair can be completely different from the next in the stream.

Since you can’t sacrifice @generate without major loss of functionality/usability, you have to live with the fact that you do not have type safety in the body of a @generate function.

Overall

This did not feel like an upgrade for a user, but rather like a pretty big downgrade. typed-parsy was definitely going to be worse than parsy, so I stopped working on it.

Which brings me to my last approach:

Types for documentation

At some point along the way, I noticed that for the original version of parsy, with no type hints at all, my language server (pyright) was able to correctly infer return types of all the methods that returned Parser instances. The types it was inferring were the same as I would have added in the very first section, simple Parser objects, and that meant it could reliably give help with chained method calls, which is pretty nice.

The biggest problems were that the docstrings weren’t helpful (in most cases missing), and that for many parameters it wasn’t entirely obvious what type of object you should be passing in.

So, using a small amount of effort, we could improve usability a lot. We can add those dosctrings, and add type hints that are about the same level of types that pyright was inferring anyway, just a bit more complete.

The one thing I don’t want to do is imply that these types bring Parsy code up to the level of being “type checked”, but there is a simple way I can do that – by not including a py.typed marker file to my package.

So, I’m back at the beginning, but with a different aim. Now, it’s not about helping automated static type checkers – without a py.typed marker in the module they basically ignore the types – it’s about improving usability for developers as they write. I’ve done this work in the master branch now, and will hopefully release it soon.

There is another interesting advantage to this: because I’ve given up on static type checks and I’m not using static type checking for internal use in Parsy at all, I can be a bit looser with the type hints, allowing for greater readability. For example, I can annotate an optional string argument as arg: str = None, even though that’s not strictly compliant with PEP 484. In other places I can use slightly simplified type hints for the sake of having something that doesn’t make your eyes glaze over, even if it doesn’t show all the possibilities – such as saying that input types can be str | bytes | list, when technically I should be writing something much more abstract and complicated using Sequence.

Conclusion

In the end, type hints didn’t work out for use by a static type checker. But as clear and concise documentation for humans that pop up in a code editor, they worked well. Unless or until there are some big improvements in what it is possible to express with static types in Python, this seems to be the best solution for Parsy.

I learnt a lot about the limitations of typing in Python along the way, and hope you found this helpful too!

Footnotes

[3]

Protocols and operators:

I wanted some way of expressing “an object that supports addition”, or actually “an object that supports addition and returns an object of the same type”. I needed this for the + operator, which you can use for both things like Parser[str] and Parser[list[str]].

But I couldn’t get this minimal test case to work:

class Addable(Protocol[T]):
    @abstractmethod
    def __add__(self: T, other: T) -> T:
        pass


def foo(x: Addable[T], y: Addable[T]) -> T:
    return x + y

Mypy reports:

Unsupported operand types for + ("Addable[T]" and "Addable[T]")  [operator]

This is probably my fault, but I couldn’t work it out, I obviously need to understand more about protocols.

[4]

Covariance:

As a replacement for the variadic seq which produces a hetereogeneous list (in its simplest form), I added the & operator which returns a tuple. This can be typed correctly, because unlike list/typing.List which is considered to be a homogeneous container, tuple/typing.Tuple is treated as a product type, like it is in other languages.

This pairs well with the existing | operator for alternatives, which produces a union (or sum type).

def __or__(self: Parser[OUT1], other: Parser[OUT2]) -> Parser[Union[OUT1, OUT2]]:
    ...

def __and__(self: Parser[OUT1], other: Parser[OUT2]) -> Parser[tuple[OUT1, OUT2]]:
    ...

In implementing the first, however, both mypy and pyright complain:

Expression of type "Result[OUT1@__or__]" cannot be assigned to return type
"Result[OUT1@__or__ | OUT2@__or__]"

The issue here is that the type checker needs to know that a Result[A] is a sub-type of a Result[A | B], which is only true if Result is covariant with respect to that type parameter. Since it’s an immutable container of a single item, whose only operation is that you can extract the item, it is covariant.

My first attempt to fix this, however, was to change all the type parameters to covariant=True, which gave me a ton more problems – both mypy and pyright complaining "covariant type variable cannot be used in parameter type" in many places.

It turns out, after a lot of attempts, head scratching, and finally the fresh take involved in writing a blog post, I just needed to use a covariant type parameter only for the definition of Result:

OUT_co = TypeVar("OUT_co", covariant=True)

@dataclass
class Result(Generic[OUT_co]):
    value: OUT_co
    ...

I think this is now correct, and the fact that I’m not using the OUT_co type parameter in all the other places is not a problem, but to be honest I’m not entirely sure. Both mypy and pyright now seem happy, so I think I’m done?

Raising exceptions or returning error objects in Python

2022-06-06T11:29:35+01:00

The other day I got a question about some old code I had written which, instead of raising an exception for an error condition as the reader expected, returned an error object:

With your EmailVerifyTokenGenerator class, why do you return error classes instead of raising custom errors? You could still pass the email to a custom VerifyExpired exception.

https://github.com/cciw-uk/cciw.co.uk/blob/eae8005feb95a5383663e69e92d80e11effe5ee6/cciw/bookings/email.py#L41

I think I'm too eager to raise errors but maybe there's something I'm missing with classes 😁!

The code in question is below (slightly modified and with several uninteresting methods removed). It is part of a system for doing email address verification via magic links in emails.

from dataclasses import dataclass

class VerifyFailed:
    pass


VerifyFailed = VerifyFailed()  # singleton sentinel value


@dataclass
class VerifyExpired:
    email: str


class EmailVerifyTokenGenerator:
    def token_for_email(self, email):
        ...

    def email_from_token(self, token):
        """
        Extracts the verified email address from the token, or a VerifyFailed
        constant if verification failed, or VerifyExpired if the link expired.
        """
        max_age = settings.EMAIL_VERIFY_TIMEOUT
        try:
            unencoded_token = self.url_safe_decode(token)
        except (UnicodeDecodeError, binascii.Error):
            return VerifyFailed
        try:
            return self.signer.unsign(unencoded_token, max_age=max_age)
        except (SignatureExpired,):
            return VerifyExpired(self.signer.unsign(unencoded_token))
        except (BadSignature,):
            return VerifyFailed

To sum up, we have a function that extracts an email address from a token, checking the HMAC signature that it is bundled with. There are 3 possibilities we want to deal with:

The happy case – we’ve got a valid HMAC code, we just need the email address returned.
We’ve got an invalid signature.
We’ve got a valid but expired signature. We want to handle this separately, because we’d like to streamline the user experience for getting a new token generated and sent to them, which means we need to return the email address.

It’s using Django’s signer functions to do the heavy lifting, but that doesn’t matter for our purposes, because we are wrapping it up.

To get going on designing our API for this bit of code, here are some bad options:

We could have a pair of methods or functions: extract_email_from_token and check_signature, which can be used independently. This is bad because you could easily use extract_email_from_token and completely forget to use check_signature.

The principle here is that we want the developer using this API to fall into the pit of success. Either the developer should get their code perfectly correct, or if they don’t, it either will be obviously broken and not work at all, or at least not subtly flawed with some nasty bug, like a security issue.
We could have email_from_token() method or function with a return value of a tuple containing (email_address: str, valid_and_not_expired_signature: bool).

This has a similar issue to above – the calling code could use email_address and forget to check the validity boolean.

Having ruled those out, we’ve got two main contenders for how to design email_from_token():

We could make it raise exceptions for the “invalid” or “expired” cases. We need to pass extra data for the latter, but we can put it inside the exception object – as noted by the original questioner.
We could make it return error objects for the error cases, as coded above.

Both of these satisfy the “pit of success” criterion. If the developer accidentally does not handle the error cases, they won’t have a bug where we verified an email address that should not be verified. We will instead probably have a crasher of some kind, which in the case of a web app, like this one, means a 500 error page being seen, and something in our logs that makes it pretty clear what happened.

If we choose to raise exceptions, naive code which doesn’t check for the exceptions will simply get no further – the exception will propagate up and terminate the handler. With the second option where we return error objects, those objects can’t be accidentally converted into success values – the VerifyExpired object contains the email address, but it is a completely different shape of value from the happy case.

Both of these approaches, to some degree, respect the principle that can be summed up as Parse Don’t Validate. Instead of merely validating a token and extracting an email address as two independent things, we are parsing a token, and encoding the result of the validation in the type of objects that will then flow through the program.

But which is better?

One of the influences on my thinking is the way types work in Haskell and other similar language which make it very easy to create types and constructors. In Haskell, the following is all the code you need to define a return type for this kind of function, and the 3 different data constructors you need, which then do double duty for pattern matching:

data EmailVerificationResult = EmailVerified string
                             | VerifyFailed
                             | VerifyExpired string

Now, Python is not nearly as succinct, but dataclasses were a big improvement for defining things like VerifyExpired.

In Haskell, due to static type checking, this pattern makes it pretty much impossible for the calling code to accidentally fail to handle the return value correctly. But even in Python, which doesn’t have that built in, I think there are some compelling advantages:

We expect the calling code to handle all the different return values at some point, and at the same point. (This is unlike some code where we can raise an exception that we never expect the calling code to specifically handle – it will be handled by more generic methods at a different layer). It therefore makes sense that we treat all 3 values as the same kind of thing — they are just different return values.
If you instead raise exceptions, you are immediately forcing the calling code into a special control flow structure, namely the try/except dance, which can be inconvenient.
In particular, if you want to hand off processing of the value to some other function or code for handling, you can’t do it easily. For example, code like this would be fine with the “return error object” method, but significantly complicated by the “raise exception” method:
```
verify_result = verifier.email_from_token(token)
log_verify_result(request.ip_address, verify_result)
# etc.
```

In the years since I wrote the code, however, some perhaps more compelling arguments have come along for the error object method.

First, with some small changes (specifically, removing the sentinel singleton value), we can now add a type signature for email_from_token:

def email_from_token(self, token, max_age=None) -> str | VerifyFailed | VerifyExpired:
    ...

(You may need typing.Union for older Python versions)

This is a benefit in itself from a documentation point of view, and for better IDE/editor help.

We can go further with mypy. We can structure our calling code as follows to make use of mypy exhaustiveness checking:

from typing_extensions import assert_never

verified_email = EmailVerifyTokenGenerator().email_from_token(token)
if isinstance(verified_email, VerifyFailed):
    ...
elif isinstance(verified_email, VerifyExpired):
    ...
elif isinstance(verified_email, str):
    ...
else:
    assert_never(verified_email)

Now, if we remove one of these blocks, let’s say the VerifyExpired one (or if we added another option to email_from_token), mypy will catch it for us:

error: Argument 1 to "assert_never" has incompatible type "VerifyExpired"; expected "NoReturn"

With the error object method, we could also write our handling code using structural pattern matching. The equivalent code, including our mypy exhaustiveness check, now looks like this:

verified_email = EmailVerifyTokenGenerator().email_from_token(token)
match verified_email:
    case VerifyFailed():
        ...
    case VerifyExpired(expired_token_email):
        ...
    case str():
        ...
    case _:
        assert_never(verified_email)

This has destructuring of the email address in VerifyExpired built in – it is bound to the name expired_token_email in that branch.

Hopefully this gives a good justification for the approach I took with this code. There are times when exceptions are better – generally when the things mentioned above don’t apply, or the opposite applies – but I think error objects also have their place, and sometimes are a much better solution.

Luke Plant's home page (Posts about Python type hints)

Statically checking Python dicts for completeness

That’s not quite it

Python Type Hints: pyastgrep case study

The different uses of Python type hints

Interactive programming help

Static type checking

Runtime behaviour determination

Code documentation

Compiler instructions

Conclusion

Links

Test factory functions in Django

Python Type Hints: case study on parsy

Raising exceptions or returning error objects in Python

Links