ForeignKey

Suppose, to pick one example of many, you are writing django-otp, a third party library that provides a One Time Password implementation for 2FA requirements. You want to create a table of TOTP devices that are linked to user accounts, and so you have something like this:

class TOTPDevice(models.Model):
    user = models.ForeignKey('auth.User', related_name='totp_devices')

Later on, you have code that starts with a User object and retrieves their TOTP devices, and it looks something like this:

devices = user.totp_devices.all()

This is interesting, because my user variable is an instance of a User model that was provided by core Django, which has no knowledge of the third party project that provides the TOTPDevice model.

In fact it goes further: I may not be using Django’s User at all, but my own custom User class, and the TOTPDevice model can easily support that too just by doing this:

user = models.ForeignKey(getattr(settings, "AUTH_USER_MODEL", "auth.User"))

This means that my User model has no knowledge of the TOTPDevice class, nor the other way around, yet instances of these classes both get wired up to refer to each other.

What is actually going on to enable this?

When you import Django and call setup(), it imports all the apps in your project. When it comes to the TOTPDevice class, it sees the "auth.User" reference and finds the class it refers to. It then modifies that class, adding a totp_devices descriptor object to the class attributes.

This is run-time type modification.

The result is that when you do user.totp_devices, you get a Manager instance that does queries against the TOTPDevice table. It is a specific kind of manager, known as a RelatedManager, with the special property that it automatically does the correct filter() calls to limit returned values to those related to the model instance, among other things.

RelatedManager

The RelatedManager class is interesting in a number of ways. First, it is created as a closure – meaning the class itself is created inside a function that is called for each relationship. This is run-time type creation.

Second, there are some additional things it needs to support. In Django, projects often override Manager classes, and the related QuerySet classes, to provide a lot of model layer functionality. This is Django’s answer to the “Repository” pattern – popular in some languages, but laborious to create and painful to use compared to what we have.

Because of this, it’s important that the RelatedManager preserves any custom Manager and QuerySet behaviour defined on the target model. So, the solution is simply that Django makes the created RelatedManager class inherit from your custom Manager. This is dynamic sub-classing – sub-classing of a class that is discovered at run-time. In other OOP languages, you inherit from framework classes. In Python, framework inherits you!

ManyToMany models

One common need in database applications is to have many-to-many relationships between two models. Typically this can be modelled with a separate table that has foreign keys to the two related tables.

To make this easy, Django provides a ManyToManyField. For simple cases, it’s tedious to have to create a model for the intermediate table yourself, so of course Django just creates it for you if you don’t provide your own, using type() with 3 arguments. This is again run-time type creation.

Consequences

These examples of run-time type modification or creation are perhaps not the most extreme or mind-bending. But they are something even better: useful. It’s these features, and things like them, that enable an ecosystem of third party Django libraries that can integrate with your own code without any problems.

Python also always gives us enough flexibility to have a good backwards compatibility story – so that, for example, the swappable User model was introduced with an absolute minimum of fuss for both projects and pluggable Django apps.

I’m interested in functional programming, Haskell in particular – this blog even ran on Haskell for a time – so I always take interest in developments in the Haskell web framework world. I see lots of cool things, but it always seems that the ecosystems around Haskell web frameworks are at least 10 years behind Django. One key issue is that in contrast to Django or other Python frameworks, Haskell web frameworks almost always have some kind of code generation layer. This can be made to work well for the purposes envisaged by the framework authors, but it never seems to enable the ecosystem of external packages that highly dynamic typing supports.

Please note that I’m not claiming here that Python is better than Haskell or anything so grand. I’m simply claiming that Python does enable very useful things to be built, and those things are made possible and easy because of Python’s design, rather than despite it.

I think this is important to say. Python has become massively more popular than it was when I first started to use it, and there are increasing numbers of people who use it only because of network effects, and don’t understand why it got so popular in the first place. These people can sometimes assume that it’s fundamentally a poorly designed language that we are just lumped with – today’s PHP – whose best trajectory is to make it more like Java or C# or Rust etc. I think that would be a big mistake.

Baserow

One project that takes Python’s run-time type creation further is Baserow. In their case, their customers create database applications, and the metadata for those tables is stored in … tables. They like Django and want to use it as much as possible. But they also want their customers’ actual data tables to be normal RDBMS tables, and therefore benefit from all the typical RDBMS features to make their tables fast and compact etc. (I’ve seen and worked on systems that took the opposite approach, where customer schema was relegated to second class storage – essentially a key-value table – and the performance was predictably awful).

And they want plug-in authors to be able to use Django too! Some people are just greedy! They have a nice article describing how they achieved all this: in short, they use type() for run-time type creation and then leverage everything Django gives them.

This has the interesting effect that the metadata tables, along with their own business tables, live at the same level as their customers’ tables which are described by those metadata tables. This “meta and non-meta living at the same level” is a neat illustration of what Python’s type system gives you:

When you first discover the mind-bending relationships around type(type) == type, you might think of an infinitely-recursive relationship. But actually, an infinite relationship has been flattened to being just 3 layers deep – instance, class, metaclass. The last layer just recurses onto itself. The infinity has been tamed, and brought into the same structures that you can already deal with, and without changing language or switching to code generation techniques. This is one reason why many examples of the “hyper-programming” that Hillel talks about can just be dismissed as normal programming – but they are simply hyper-programming that you are now taking for granted.

CCiW data retention policy

CCiW is a small charity I’ve been involved with for a long time. When I came to implement its GDPR and data retention policies, I found another example of how useful it is having access to Django’s meta-layer (generic framework code) on the same level as my normal data layer (business specific classes and tables), in ways that often aren’t the case for statically typed languages that resort to code-generation techniques for some of these things.

I wanted to have a data retention policy that was both human readable and machine readable, so that:

• We don’t have keep two separate documents in sync.

• the CCiW committee and other interested parties would be able to read the policy that actually gets applied, rather than merely what the policy was supposed to be.

• I could have machine level checking of the exhaustiveness of the policy.

My solution was to split the data retention policy into two parts:

• a heavily commented, human-and-machine readable Literate YAML file with a nicely formatted version, that I can genuinely claim is our data retention policy, and that it is automatically applied,

• and a Python implementation that reads this file and applies it, along with some additional logic.

A key part of the neatness of this solution is that the generic, higher level code (which reads in a YAML file, and therefore has to treat field names and table names as strings), and the business/domain specific logic can sit right next to each other. The end result is something that’s both efficient and elegant, with great separation of concerns, and virtually self-maintaining – it complains at me automatically if I fail to update it when adding new fields or tables.

In terms of performance, the daily application for the data retention policy for the entire database requires, at the moment, just 5 UPDATE and 3 DELETE queries, run once a day. This is made possible by:

• using the power of an ORM,

• using generic code to build up **kwargs to pass to QuerySet.update(),

• seamlessly integrating these two with business specific logic.

Here is one of the queries the ORM generates, which is complex enough that I wouldn’t attempt to write this by hand, but it correctly applies business logic like not erasing any data of people who still owe us money, and combines all the erasure that needs to be done into a single query.

Compile-to-Python

In fluent-compiler, the implementation strategy I took was to compile the parsed Fluent AST to Python code, and exec it. I actually use Python AST nodes rather than strings, for various security reasons, but this is basically the same as doing eval, and that same technique is used by various other projects like Jinja and Mako.

If anything qualifies as “programs that create programs on the fly”, then using exec, eval or compile must do so! The main advantage of this technique here is speed. It works particularly well with Fluent, because with a bit of static analysis, we can often completely eliminate the overhead that would otherwise be caused by its more advanced features, like terms and parameterized terms, so that at run-time they cost us nothing.

This works even better when combined with PyPy. For the simple and common cases, under CPython 3.11 my benchmarks show a solution using fluent-compiler is about 15% faster than GNU gettext, while under PyPy it’s more than twice as fast. You should take these numbers with a pinch of salt, but I am confident that the result is not slow, despite having far more advanced capabilities than GNU gettext, which is not true for the first implementation – the compiler is about 5-10x faster than the interpreter for common cases on CPython.

Additionally, there are some neat tricks you can do when implementing a compiler using the same language that you are compiling to, like evaluating some things ahead of time that you know are constants.

Dynamic test methods

While developing the second implementation, I used the first implementation as a reference. I didn’t want to duplicate every test, or really do anything manually to every test, I just wanted a large sub-set of the test suite to automatically test both implementations. I also wanted failures to clearly indicate which implementation had failed, i.e. I wanted them to run as separate test cases, because the reference implementation could potentially be at fault in some corner cases.

As I was using unittest, my solution was this: I added a class decorator that modified the test classes by removing every method that started with test_, replacing it with two methods, one for each implementation.

This provided almost exactly the same functionality as one of Hillel’s wished-for examples:

Add an output assertion to an optimized function in dev/testing, checking that on all invocations it matches the result of an unoptimized function

I just used a slightly different technique that better suited my needs, but also made great use of run-time program manipulation.

morph_into

As part of the Fluent-to-Python compilation process, I have a tree of AST objects that I want to simplify. Simplifications include things like replacing a “string join” operation that has just one string, with that single string – so we need completely different types of objects. Even in a language that has mutation this can be a bit of a pain, because we’ve got to update the parent object and tell it to replace this child with a different child, and there are many different types of parent object with very different shapes. So my solution was morph_into:

def morph_into(item, new_item):
    """
    Change `item` into `new_item` without changing its identity
    """
    item.__class__ = new_item.__class__
    item.__dict__ = new_item.__dict__

With this solution, we leave the identity of the object the same, so none of the pointers to it need to be updated. But its type and all associated data is changed into something else, so that, other than id(), the behaviour of item will now be indistinguishable from new_item. Not many languages allow you to do this!

I spent quite a lot of time wondering if I should be ashamed or proud of this code. But it turned out there was nothing to be ashamed of – it saved me writing a bunch of code and has had really no downsides.

Now, this technique won’t work for some things, like builtin primitives, so it can’t be completely generalised. But it doesn’t need that in order to be useful – all the objects I want to do this on are my own custom AST classes that share an interface, so it works and is “type safe” in its own way.

I’m far from the first person to discover this kind of trick when implementing compilers. In this comparison of several groups of people working on a compiler project, one of the most impressive results came from a single-person team who chose Python. She used way less code, and implemented way more features than the other groups, who all had multiple people on their teams and were using C++/Rust/Haskell etc. Fancy metaprogramming and dynamic typing were a big part of the difference, and by the sounds of it she used exactly the same kinds of things I used:

Another example of the power of metaprogramming and dynamic typing is that we have a 400 line file called visit.rs that is mostly repetitive boilerplate code implementing a visitor on a bunch of AST structures. In Python this could be a short ~10 line function that recursively introspects on the fields of the AST node and visits them (using the __dict__ attribute).

Again, I’m not claiming “dynamic typing is better than static typing” – I don’t think it’s even meaningful to do that comparison. I’m claiming that highly dynamic meta-programming tricks are indeed a significant part of real Python code, and really do make a big difference.

Assert rewriting

The most obvious is perhaps their assert rewriting, which relies on modifying the AST of test modules to inject sub-expression information for when asserts fail. It makes test assertions often much more immediately useful.

Automatic dependency injection of fixtures

Pytest provides one of the few cases of automatic dependency injection in Python where I’ve thought it was a good idea. It also makes use of Python’s dynamism to make this dependency injection extremely low ceremony. All you need to do is add a parameter to your test function, giving the parameter the name of the fixture you want, and pytest will find that fixture in its registry and pass it to your function.