Luke Plant's home page (Posts about Python)

Inverse Sapir-Whorf and programming languages

2026-05-01T09:40:36+01:00

The Sapir-Whorf hypothesis, in its simplest form, is the idea that the language you speak influences the thoughts you think. This post is about a twist on this idea, that I’m calling “Inverse Sapir-Whorf” (for want of a better term), and how we see it in computer programming languages.

Sapir-Whorf is one of those ideas that has been popularised in general culture in a rather misrepresented and exaggerated form. In the field of linguistics, not many people today take seriously the “strong” forms of Sapir-Whorf, such as “linguistic determinism” – the idea that a language controls your thoughts or limits what you can think, or that you even need certain languages to think certain thoughts.

For example, just because a language might lack grammatical tenses, it doesn’t at all follow that the speakers will be more limited in how they think about time – there are always other ways you can express time.

There is a fair amount of evidence that spoken languages can affect perception, skill and attitudes in certain areas, but it’s usually hard to demonstrate a large direct effect.

Inverse Sapir-Whorf is a bit different. I haven’t been able to track down where I first came across the idea, but it goes like this: if classic Sapir-Whorf says your language limits what you can say or think, or makes it hard to say some things, inverse Sapir-Whorf says your language limits what you can’t say, or makes it hard not to say some things, or even hard not to think about some things. Some examples might clear things up.

Examples in natural language

There are many examples to choose from, but they are not always obvious to native speakers of a language. I’ll pick just a few.

English: temporary or permanent present tense

What’s the difference between someone saying “I’m living in London” and “I live in London”? A non-native speaker may not pick this up at all, and a native speaker may pick it up only subconsciously, but “I’m living in London” reveals that the arrangement is temporary.

Now, this might not even be to do with the actual length of time you have been living there, because “temporary” is pretty relative. It might be more about how much you like London. You have to choose a tense, and because you typically do so subconsciously, the language is forcing you to reveal things – either the period of time you’ve been living somewhere, or how you feel about it.

English/Turkish/French: gendered pronouns and nouns

In English, in normal speech you are going to use “he” or “she” when referring to a specific person. “Singular they” does exist, but it’s very unnatural if you are talking about a specific person of known or assumed sex.

You can compare this to another language which doesn’t have gendered pronouns, such as Turkish, which just has “o” for he/she/it. The lack of gendered pronouns in Turkish doesn’t stop you from thinking or talking about a person’s sex, or produce a “less gendered society”, or anywhere close, so it would be difficult to find support for normal Sapir-Whorf here. But the inverse Sapir-Whorf is obvious – English pronouns push you to talk about it whether you want to or not. If you are trying to talk about someone you know, but do so anonymously, it can be very hard to avoid making their identification easier by revealing their sex with an inadvertent “him” or “her”.

Different again is French, in which nouns are gendered, which in some cases can force you to reveal information. If you translate “my friend” into French, you have to choose between “mon ami” (male friend) and “mon amie” (female friend), which are distinct, at least in written form, or “mon copain” vs “ma copine”. Possessive pronouns are also interesting – they are gendered in both English and French (his/her, son/sa), but refer to the gender of the possessor and possessee respectively, and so reveal different information.

Turkish: “mış” tense

With some simplifications, Turkish has two main past tenses: there is the normal one that is similar to “simple past” in English, and then there is the “mış” form (you can pronounce that “mish” if you want).

This has various functions, but when describing a past event, this form is used when you have second hand or unreliable information. If someone asks you “Did Fred come to work on Monday?”, then if you saw him you would use the normal past tense “geldi” (he came), but if you only heard that he came you would instead say “gelmiş” (he came, but second hand information).

The interesting thing to me as a non-native speaker was the effect of having these options, in contrast to English where you can just use simple past tense without any specific indication of reliability or where the information came from. In certain circumstances, Turkish forces you to include information about your level of certainty or whether you witnessed something – the simple past form is not neutral, because the existence of the “mış” form makes it an unnatural choice if it is not the most appropriate of the two.

Interestingly, having learned to think that way, my wife and I have noticed an effect on our English. Often in Turkish the “mış” suffix would come at the end of the last word in a sentence, so now quite frequently we get to the end of an English sentence and notice that we haven’t put in any marker for “this-is-second-hand-info-I-didn’t-actually-witness-it”, and so we tack “mış” on the end.

Of course, you can easily express the same thing in English, using words like “apparently” and other means, but English doesn’t force you to specify, while Turkish pretty much does.

Comments

You often don’t notice these things until you learn another language, or attempt to teach your language to a foreigner. You kind of just understand them subconsciously. The vast majority of times you choose simple present over present continuous, for example, you won’t be consciously thinking about what that implies.

I should also note that when a language forces you express something, it might not be in the form of something included, but in something omitted. For example, I might say “I love cake” or “I love the cake”. In the first case, I’m talking about cake generally, in the second about a specific cake. It is the absence of the word “the” in the first case that makes it unambiguous that I’m referring to all cake, because if I’m referring to a specific cake, I must use the word “the” or some other marker like “this”. In another language, there might not be a direct equivalent to this distinction.

Examples in programming

When it comes to programming languages, I think that the “straight” version of Sapir-Whorf is closer to being true - in some programming languages it is simply hard to express certain concepts. For example, in a language like Python or Haskell it’s hard (though not impossible) to talk about memory allocations. We often talk about the limitations of a language in terms of “things that are hard to express” in that language. Hillel Wayne has some more discussion of this in his post Sapir-Whorf does not apply to Programming Languages.

But I want to talk more about Inverse Sapir-Whorf. What is the language forcing you to talk about, even if you don’t actually care about it?

I think there are actually many, many examples of this, but seeing them can be quite hard, and often requires the “foreigner perspective” that comes from learning multiple languages.

Here are a few:

Most languages force you to express the order in which computation should be done. For example, in Python:
```
x = some_func(y + 1, z + 2)
```
Here you are saying:
- first compute y + 1
- then compute z + 2
- then pass these two values as arguments to some_func
You might not be very conscious of specifying this ordering, but you are doing it, and in Python, there isn’t a way to express the above computation which doesn’t also specify order. Most languages are similar, although in some it gets very complicated.

A few languages are very different, however. In Haskell, in an equivalent expression like some_func (y + 1) (z + 2), due to non-strict semantics you are not specifying an order of evaluation at all. This enables some clever tricks, like referring to values you haven’t defined yet (see Tying the knot).
Function colouring for async is another good example. In languages like Javascript or Python with an explicit async keyword, you have to talk about whether code is sync or async.

In the case of “sync” functions, you do it by omission of the async keyword, but you are still choosing between the two options, and there is no way to write code that is ambivalent on the subject.
Most languages without garbage collection force you to talk about memory allocation and de-allocation.

For languages like C, you normally do this fairly explicitly – or implicitly use stack allocation, but you’ve still got to make that choice. In other languages, it can become more hidden, but doesn’t really go away. In Rust, for example, in gets converted into talk about lifetimes or explicit reference counting. Saying “I just don’t care about when the memory for this gets allocated or de-allocated, please deal with it” is not really one of your options.

Of course, not talking about memory allocation also has a cost. In that case, the language will almost certainly need to put a lot of things on the heap and have a runtime garbage collector. However it may also have significant freedom to choose for you – Haskell will often be able to do this using strictness analysis, for example.
All modern languages I’m aware of force you to think about “scope”. In many cases, scope is expressed by the physical place in which you put a variable, with some additional syntax if you want something different (like global or nonlocal in Python). If you never want to think about scope, you probably have to drop to assembly and live with a single global address space.
Statically typed languages often force you to think about and talk about the type of every variable. This is lessened somewhat by type inference, as the “conversation” involves a more “intelligent” listener who can pick up more things from context, but it’s still there.

Pure dynamically typed languages still allow you to talk about types – for example, using things like isinstance checks in Python, but it is more unnatural (and technically it’s a different thing anyway).

In contrast to both of them, one of the attractions of gradually typed languages is that they genuinely avoid the inverse Sapir-Whorf problem, and allow you the freedom to talk about types, or not, at your preference. I’m not sure how well this works in practice - the existing code base conventions and the linters in use always put pressure in some direction.

I suspect that many of the features of more “approachable” or “readable” programming languages could be analysed in these terms – they have a low inverse Sapir-Whorf barrier, and don’t force you to talk about things you don’t have an opinion on, and may not even understand yet.

Are there more examples of this that you’ve come across? How do they affect the programming languages we use, or how we perceive them?

Announcing fluent-codegen

2026-03-10T12:33:49Z

This post announces fluent-codegen, a Python library for generating Python code. Not everyone needs this, but when you need it, you’ll probably know, and you’ll want probably want a decent solution that is not based on string concatenation.

The history of this project is that it started out as a codegen module in fluent-compiler, which is an implementation of Project Fluent, Mozilla’s internationalisation solution. So the word “fluent” referred to that originally, but now it refers to a set of nice APIs for building up Python expressions.

As I needed a code generation library for another purpose, I pulled out this library and fleshed it out into a relatively complete, standalone project. It has also evolved quite a bit since its earlier form, and is a pretty well rounded library now. You can check out the nice docs, which include a nice little toy example of a “SVG to Python Turtle” compiler. Enjoy!

Help my website is too small

2025-12-19T13:45:33Z

A jobs web site I belong to just emailed me, telling me that some of the links in my public profile on their site are “broken” and “thus have been removed”.

The evidence that these sites are broken? They are too small:

https://www.djangoproject.com/: response body too small (6220 bytes)

https://www.cciw.co.uk/: response body too small (3033 bytes)

The first is the home page of the Django web framework, and is, unsurprisingly, implemented using Django (see the djangoproject.com source code). The second is one of my own projects, and also implemented using Django (source also available for anyone who cares).

Checking in webdev tools on these sites gives very similar numbers to the above for the over-the-wire size of the initial HTML (though I get slightly higher figures), so this wasn’t a blip caused by downtime, as far as I can see.

Apparently, if your HTML is less than 7k, that obviously can’t be a real website, let alone something as ridiculously small as 3k. Even with compression turned up all the way, it’s clearly impossible to return more than an error message with less than at least 4k, right?

So please can Django get it sorted and add some bloat to their home page, and to their framework, and can someone also send me tips on bloating my own sites, so that my profile links can be counted as real websites? Thanks!

Breaking “provably correct” Leftpad

2025-10-03T21:00:00+01:00

I don’t know much about about formal methods, so a while back I read Hillel Wayne’s post Let’s prove Leftpad with interest. However:

I know Donald Knuth’s famous quote: “Beware of bugs in the above code; I have only proved it correct, not tried it”
I also know how it turned out that code that had been proved correct harboured a bug not found for decades.

So I thought I’d take a peek and do some testing on these Leftpad implementations that are all “provably correct”.

Methodology

I’ll pick a few, simple, perfectly ordinary inputs at random, and work out what I think the output should be. This is a pretty trivial problem so I’m expecting that all the implementations will match my output. [narrator: He is is expecting no such thing]

I’m also expecting that, even if for some reason I’ve made a mistake, all the implementations will at least match each other. [narrator: More lies] They’ve all been proved correct, right?

Here are my inputs and expected outputs. I’m going to pad to a length of 10, and use - as padding so it can be seen and counted more easily than spaces.

Item	Input	Length	Expected padding	Expected Output
1	`𝄞`	1	9	`---------𝄞`
2	`Å`	1	9	`---------Å`
3	`𓀾`	1	9	`---------𓀾`
4	`אֳֽ֑`	1	9	`---------אֳֽ֑`
5	`résumé`	6	4	`----résumé`
6	`résumé`	6	4	`----résumé`

[“ordinary”, “random” - I think my work here is done…]

I’ve used a monospace font so that the right hand side of the outputs all line up as you’d expect.

Entry 6 is not a mistake, by the way, it just does “e acute” in a different way to entry 5. Nothing to see here, move along…

Implementations

Not all of the implementations were that easy to run. In fact, many of them didn’t take any kind of “string” as input, but vectors or lists or such things, and it wasn’t obvious to me how to pass strings in. So I discounted them.

For the ones I could run, I attempted to do so by embedding the test inputs directly in the program, if possible.

Liquid Haskell

Embedding the characters directly in Haskell source code kept getting me “lexical error in string/character literal”, so I wrote a small driver program that read from a file.

Java

The leftpad function provided didn’t take a string, but a char[]. Thankfully, it’s easy to convert from String objects, using the .toCharArray() function. So I did that.

Lean4

There is a handy online playground, and the implementation had a helpful #eval block that I could modify to get output. You can play with it here.

Rust

The code here had loads of extra lines regarding specs etc. which I stripped so I could easily run it, which worked fine.

More tricky was that the code didn’t take a string, but some Vec<Thingy>. As I know nothing about Rust, I got ChatGPT to tell me how to convert from a string to that. It gave me two options, I picked the one that looked simpler and less <<angry>>. I didn’t deliberately pick the one which made Rust look even worse than all the others, out of peevish resentment for every time someone has rewritten some Python code (my go-to language) in Rust and made it a million times faster – that’s a ridiculous suggestion.

Some competition!

To make things interesting, let’s compare these provably correct implementations with one vibe-coded by ChatGPT, in some random language, like, say, um, Swift. It gave me this code:

import Foundation

func leftPad(_ string: String, length: Int, pad: Character = " ") -> String {
    let paddingCount = max(0, length - string.count)
    return String(repeating: pad, count: paddingCount) + string
}

You can play with it online here.

Results

Here are the results, green for correct and red for … less correct.

Input	Reference	Java	Haskell	Lean	Rust	Swift
𝄞	---------𝄞	--------𝄞	---------𝄞	---------𝄞	------𝄞	---------𝄞
Å	---------Å	--------Å	--------Å	--------Å	-------Å	---------Å
𓀾	---------𓀾	--------𓀾	---------𓀾	---------𓀾	------𓀾	---------𓀾
אֳֽ֑	---------אֳֽ֑	------אֳֽ֑	------אֳֽ֑	------אֳֽ֑	--אֳֽ֑	---------אֳֽ֑
résumé	----résumé	----résumé	----résumé	----résumé	--résumé	----résumé
résumé	----résumé	--résumé	--résumé	--résumé	résumé	----résumé

And pivoted the other way around so you can compare individual inputs more easily:

Language	𝄞	Å	𓀾	אֳֽ֑	résumé	résumé
Reference	---------𝄞	---------Å	---------𓀾	---------אֳֽ֑	----résumé	----résumé
Java	--------𝄞	--------Å	--------𓀾	------אֳֽ֑	----résumé	--résumé
Haskell	---------𝄞	--------Å	---------𓀾	------אֳֽ֑	----résumé	--résumé
Lean	---------𝄞	--------Å	---------𓀾	------אֳֽ֑	----résumé	--résumé
Rust	------𝄞	-------Å	------𓀾	--אֳֽ֑	--résumé	résumé
Swift	---------𝄞	---------Å	---------𓀾	---------אֳֽ֑	----résumé	----résumé

Comments

Rust, as expected, gets nul points. What can I say?

Vibe-coding with Swift: 💯

Other that, we can see:

The only item that all implementations (apart from Rust) get correct is entry 5, the first of the two résumé options.
Java is mostly consistent with the others, but it appears it doesn’t like musical notation, or Egyptian hieroglyphics (item 3 is “standing mummy”), which seems a little rude.

The score so far:

Fancy-pants languages and formal verification: 0
Vibe-coding it with ChatGPT: 1

Explanation

OK, I’ve had my fun now :-)

(The original “Let’s Prove Leftpad” project was done “because it is funny”, and this post is in the same spirit. I want to be especially clear that I’m not actually a fan of vibe-coding).

What’s actually going on here? There are two main issues, both tied up with the concept of “the length of a string”.

(If you already know enough about Unicode, or don’t care about the details, you can skip to the “What went wrong?” section to continue discussion regarding formal verification).

First:

What is a character?

Strings are composed of “characters”, but what are they?

Most modern computer languages, and all the ones I included above, use Unicode as the basis for answering this. Unicode, at its heart, is a list of “code points” (although it is bit more than this). A code point, however, is not exactly a character.

Many of the code points you use most often, like Latin Capital Letter A U+0041, are exactly what you think of as a character. But many are not. These include:

combining characters, which are used to add accents and marks to other characters. These have some complexity:
- First, whether you regard the accent as part of the character or not can be a matter of debate. For example, in é, “e acute”, you might think of this as a different letter to e, or an e plus an acute accent. In some languages, this distinction is critical e.g. in Turkish ç is not just “c with some decoration”, it’s an entirely distinct letter.
- Second, Unicode often has multiple ways of generating these accented characters, either “pre-composed” as a single code point, or as a combination of multiple code points. So, there is both:
  - é: Latin Small Letter E With Acute U+00E9
  - é: Latin Small Letter E U+0065 + Combining Acute Accent U+0301
various spacing characters
control characters such as bidi isolation characters.
and probably more

So, Unicode has another concept, called the “extended grapheme cluster”, or “user-perceived character”, which more closely maps to what you might think of as a “character”. That’s the concept I’m implicitly using for my claim of what leftpad should output.

Secondly, there is the question of:

How does a programming language handle strings?

Different languages have different fundamental answers to the question of “what is a string?”, different internal representations of them (how the data is actually stored), and different ways of exposing strings to the programmer.

Some languages, especially performance oriented ones, provide little to zero insulation from the internal representation, while others provide a fairly strong abstraction. Some languages, like Haskell, provide multiple string types (which can be used with string literals in your code with the OverloadedStrings extension).

At this point, as well as “code points”, we’ve got to consider “encodings”. If you have a code point, even a “simple” one like U+0041 (A), you have said nothing about how you are going to store or transmit that data. An encoding is a system for doing that. Two relevant ones here:

UTF-8 is probably the most common/popular one. It uses anything from 1 to 4 bytes to express code points. It has lots of useful properties, but an important one is backwards compatibility with ASCII - if you happen to be limited to the 127 characters found in ASCII, then UTF-8 encoded Unicode text is one-byte-per-codepoint, and is byte-for-byte the the same as ASCII.
UTF-16 is an encoding where most code points (specifically those in the Basic Multilingual Plane or BMP) take up 2 bytes, and the remainder can be specified using 4 bytes and a system called “surrogate pairs”.

UTF-16 exists because originally Unicode had less than 65,536 code points, meaning you could represent all points with just two bytes, and it was thought we would never need more.

In terms of languages today, with some simplification we can say the following:

In Haskell, Lean, Python, and many other languages, strings are sequences of Unicode code points.

There is little attempt to hide the idea of code points from you, although in some, like Python, the internal representation itself might be hidden (see PEP 393).
In Java and Javascript, strings are sequences of UTF-16 items.

Originally strings were intended to be sequences of code points, but when Unicode grew, to avoid breaking all existing Java code, it morphed into the UTF-16 compromise.
In Rust, strings are UTF-8 encoded Unicode code points (see String).

Rust does also have an easily accessible chars method/concept, which corresponds to a Unicode code point. I didn’t use this above - Rust would have behaved the same as Haskell/Lean if I had.
In Swift, strings are sequences of user-perceived characters.

I’m not a user of Swift, but from the docs it appears to do a pretty good job of abstracting away from the “sequence of code point” paradigm. For example, iterating over a string gets you the “characters” (i.e. extended grapheme clusters) one by one, and the .count property gives you the number of such characters. It does also have a .length property that gives you the number of code points.

Putting them together

These differences, between them, explain the differences in output above. In more detail:

the treble clef 𝄞 is a single code point, but it is outside the BMP and requires 2 UTF-16 items. So Java considers the length to be 2, and only 8 padding characters were added, in contrast to other languages.
I created the Å using two code points (although there is a pre-composed version of this character).
The אֳֽ֑ is Hebrew, and is composed an Aleph followed by multiple vowel marks and a cantillation mark, bringing it up to 4 code points.
résumé was spelled in two different ways, one with precomposed é which is one code point, the other with combining characters where each é requires two code points.
None of the inputs was wholly in the ASCII range, so encoding them as UTF-8 requires more bytes, which is why Rust (as I used it) behaved as it did.

What went wrong?

For me, the biggest issue is not the “code points” vs “characters” debate, which is responsible for most of the variation shown, but the issue that resulted in the difference in the Java output i.e. UTF-16. All of the others (if I hadn’t stitched up Rust) would have resulted in the same output at least.

Apparently, nothing in the process of doing the formal verification forced the implementations to converge, and I think it is pretty fair to conclude that that at least one of the implementations must be faulty, given that they produce different output.

So what went wrong?

Lies, damned lies and natural language

English (or any natural language) is at the heart of the problem here. We should start with the phrase “provably correct”. It has a technical definition, but I’m not convinced those English words help us. The post accompanying the LiquidHaskell entry for Leftpad puts it this way:

My eyes roll whenever I read the phrase “proved X (a function, a program) correct”.

There is no such thing as “correct”.

There are only “specifications” or “properties”, and proofs that ensures that your code matches those specifications or properties.

For these reasons, I think I’d prefer talking about “formally verified” functions – it at least prompts you to ask “what does that mean”, and maybe suggests that you should be thinking about what, specifically, has been verified.

The next bit of English to trip us up is “the length of a string”. It’s extremely easy to imagine this is an easy concept, but in the presence of Unicode it really isn’t.

Hillel’s original informal requirements don’t actually use that phrase, instead they use the pseudo-code max(n, len(str)). Looking at the implementations, it appears people have subconsciously interpreted this as “the length of the string”, and then assumed that the functions like length or size that their language provides do “the right thing”.

We could conclude this is in fact a problem with informal requirements – it was at the level of interpreting those requirements that this went wrong. Therefore, we need more formal specifications and verification, not less. But I don’t think this gets round the fact that we’ve got to translate at some point, and at that point you’ve got trouble.

What is correct?

The issue I haven’t addressed so far is whether my reference output and the Swift implementation are actually “correct”. The reality is that you can make arguments for different things.

Implicitly, I’m arguing that left pad should be used for visual alignment in a fixed width context, and the implementation that does the best at that is the best one. I think that is a pretty reasonable case. But I’m sure you could make a case for other output – there isn’t actually anything that says what left pad should be used for. It’s possible that there are use cases where “the language’s underlying concept of the length of a string, whatever that may be” is the most important thing.

In addition, I was hiding the fact that “fixed width” is yet another lie:

I was originally going to use a flag character like 🏴󠁧󠁢󠁥󠁮󠁧󠁿 as one of my inputs, which is a single “extended grapheme cluster” that uses no less then 7 code points. It also results in 14 UTF-16 units in Java. The problem was that this character, like most emojis and many other characters from wide scripts like Chinese, takes up a double width even with a monospace font.

To maintain the subterfuge of “look how these all line up neatly in the correct output”, I was forced to use other examples. In other words, the example use case I was relying on to prove that these leftpad implementations were broken, is itself a broken concept. But I would still maintain that my reference output is closer to what you would expect leftpad to do.

A big point is this: even if we argue that a give implementation is “correct” (in that it does what its specifications say it does), that doesn’t mean you are using it correctly. Are you using it for its intended purpose and context? That seems like a really hard question to answer even for leftpad, and many other real world functions are similar.

So, I’m not sure what my final conclusion is, other than “programming is hard, let’s go shopping let’s eat chocolate” (alternative suggested by my wife, that’s my plan for the evening then).

Confessions and corrections

The Swift implementation was indeed written by ChatGPT, and it got it right first time, with just the prompt “Implement leftpad in Swift”. However:
- Swift is the only language I know where an implementation that does what I wanted it to do is that simple.
- I followed up by getting ChatGPT to produce a Python version, and it had all the same problems as Haskell/Lean and similar.
- I noticed that Swift doesn’t calculate strings lengths the way I would need for my use case for some characters, such as Zero Width Space U+200B and Right-To-Left Isolate U+2067, which I would need to count for zero length.
As mentioned, the other way to use the Rust version has the same behaviour as the Haskell/Lean/etc versions. ChatGPT did actually point out to me that this way was better, and the other way (the one I used) was adequate only if the input was limited to ASCII.

I do feel slightly justified in my treatment of this solution however - the provided code didn’t give a function that actually took a string, nor did it force you to use it in a specific way. It dodged the hard part, so I punished it.

Response

Hillel was kind enough to look at this post, and had this response to add:

In general, formally verified code can “go wrong” in two ways: the proven properties that don’t match what we need, or the proof depends on assumptions that are not true in practice. This is a good example of the former. An example of the latter would be the assumption that the output string is storable in memory. None of the formally verified functions will correctly render leftpad(“-“, 1e300, “foo”). This is why we always need to be careful when talking about “proving correctness”. In formal methods, “correct” always means “conforms to a certain specification under certain assumptions”, which is very different from the colloquial use of “correct” (does what you want and doesn’t have bugs).

He also pointed out that padding/alignment functionality available in standard libraries, like Python’s Format Specification Mini-Language and Javascript’s padStart, have similar issues.

Why I’m not letting the juniors use GenAI for coding

2025-07-26T21:30:00+01:00

In my current project, I am training some junior developers — some of them pretty much brand new developers — and one of the first rules I gave them was “ensure that Copilot (or any other Generative AI assistant that will write code for you in your editor) is turned off”. This post explains why. The long and short of it is this: it’s because I want the junior developers to become senior developers, and I want them to enjoy programming as well.

Other people might also be interested in my reasons, so I’m writing this as a blog post.

I’ve read many, many other posts that are for and against, or just plain depressed, and some of this is about personal preference, but as I’m making rules for other people, I feel I ought to justify those rules just a little.

I’m also attempting to write this up in a way that hopefully non-programmers can understand. I don’t want to write a whole load of posts about this increasingly tedious subject, so I’m making one slightly broader one that I can just link to if anyone asks my opinion.

Rather than talk generalities, I’ll build my case using a single very concrete example of real output from an LLM on a real task.

The problem

This example comes from a one-off problem I created when I accidentally ended up with data in two separate SQLite databases, when I wanted it one. The data wasn’t that important – it was all just test data for the project I’m currently working on – so I could have just thrown one of these databases away. But the data had some value to me, so I decided to see if I could use an LLM to quickly create a one-off script to merge the two databases.

Merging databases sounds like something so common there would be a generic tool to do it, but in reality the devil is in the details, and every database schema has specifics that mean you need a custom solution.

The specific databases in question were pretty small, with a pretty simple schema. The only big problem with the schema, common to many databases schemas, is how you deal with ID fields:

This database schema has multiple tables, and records in one table are related to records in another table using an ID column, which in my case was just a unique number starting at one: 1, 2, 3 etc. To make it concrete, let’s say my database stored a list of houses, and a list of rooms in each house (it didn’t, but it works as an example).

So you have a houses table:

id	address
1	45 Main St
2	67 Mayfair
3	2 Bag End
…	…

And a rooms table:

id	house_id	name
1	1	Dining room
2	1	Living room
3	2	The Library
…	…	…

The house_id column above links each room with a specific house from the houses table. In the example above, the rooms with ids 1 and 2 both belong to house with ID 1, aka “45 Main St”.

Due to the database merge, we’ve got multiple instances of each of these tables, and they could very easily be using the same ID values to refer to different things. When I come to merge the tables:

I’ve got to assign new values in the id columns for each table
for the rooms table, I’ve got to remember the mapping of old-to-new IDs from the houses table and apply the same mapping to the house_id column.

If I get this wrong, the data will be horribly confused and corrupted.

What I did

I used aider.chat, which is a pretty good project with a good reputation, and one that I’m used to, to the point of being reasonably competent (although I can’t claim I use any of these tools a massive amount). This was a while back, and I can’t remember which LLM model I was using, but I think it was one of the best ones from Claude, or maybe DeepSeek-R1, both of which are (or were) well regarded.

I fed it the database schema as a SQL file, then prompted it similar to below:

Write a script that takes a list of db files as command line args, and merges the contents. The output will have a single ’metadata’ table copied from one of the input files. The ’rooms’ and ’houses’ tables will be merged, being careful to update ’id’ and ’house_id’ columns correctly, so that they refer to new values.

I’m not going to spend time arguing about whether this was the best possible tool/model/prompt, that’s an endless time sink – I’m happy that I did a decent enough job on all fronts for my comments to be fair.

How it went

The results were basically great in most ways that most people would care about: the LLM created a mostly working script of a few hundred lines of code in a few minutes. If I remember correctly it was pretty much there on the first attempt.

I did actually care about the data, so I carefully checked the script, especially the code that mapped the IDs.

There was one bit of code that created the mapping, and a second bit of code that then used it. For this second part, correct code would have looked something like this:

new_id = id_mapping[old_id]

Here:

id_mapping is a mapping (dictionary) that contains the “old ID” as keys, and the “new ID” as values
old_id is a variable containing an old ID value.
the brackets syntax [old_id] does a dictionary lookup and returns the value found – in other words, given the old ID it returns the new ID.

There is a crucial question with mappings like this: what happens if, for some reason, the mapping doesn’t contain the old_id value? With the code as written above, the dictionary lookup will raise an exception, which will cause the code to abort at this point. This is a good thing – since somehow we are missing a value, there is nothing we can do with the current data, and obviously there is some serious problem with our code which means that aborting loudly is the best of our options, or at least the most sensible default.

However, what the LLM actually wrote was like this:

new_id = id_mapping.get(old_id, old_id)

This code also does a dictionary lookup, but it uses the get method to supply a default value. If the lookup fails because the value is missing, the default is returned instead of raising an exception. The default given here is old_id, which is the original value. This is, of course, disastrously wrong. If, for whatever reason, the new ID value is missing, the old ID value is not a good enough fallback – that’s just going to create horribly corrupted data. Worst of all, it will do so absolutely silently – it could just fill the output data with essentially garbage, with no warning that something had gone wrong.

We might ask where this idea came from — the LLM has written extra code to produce a much worse result, why? The answer is most likely found, in part, in the way the LLM was trained – that is, on mediocre code.

A better answer to “why” the AI wrote this is much more troubling, but I’ll come back to that.

We might also ask, “does it matter?”

Part of the answer to that is context. For the project I’m currently working on, “silently wrong output” is one of the very worst things we can do. There are projects with different priorities, and there are even a few where quality barely matters at all, for which we really wouldn’t care about things like this. There are also lots of projects where you might expect people would care about this, but they actually don’t, which is fairly sad. Anyway, I’m glad not to be currently working on any projects like that – correct output matters, and I like that.

In this case, there is a second reason you might say it doesn’t matter: the rest of the code was actually correct. This meant that the mapping dictionary was complete, so the incorrect behaviour would never get triggered – the problem I’ve found is actually hypothetical. So what am I worrying about?

The problem is that in real code, the hypothetical could become reality very quickly. For example:

Some change to the code could introduce a bug which means that the mapping is now missing entries.
A refactoring means the code gets used in a slightly different situation.
There is some change to the context in which the code is used. For example, if other processes were writing to one of these databases while the merge operation was happening, it would be entirely possible for the mapping dictionary to be missing entries.

So we find ourselves in an interesting situation:

The code, as written by the LLM, appears on the one hand to be perfectly adequate, if measured according to the metric of “does it work right now”.
On the other hand, the code is disastrously inadequate if measured by the standard of letting it go anywhere near important data, or anything you would want to live long term in your code base.

The main point

Writing correct code is hard. The difference between correct and disastrously bad can be extremely subtle. These differences are easily missed by the untrained eye, and you might not even be able to come up with an example where the bad code fails, because in its initial context it does not.

There are many, many other examples of this in programming, and there are many examples of LLMs tripping up like this. Often it is not what is present that is even the problem, but what is absent.

So what?

OK, so LLMs sometimes write horribly flawed code that appears to work. Couldn’t we say the same about junior programmers?

Yes, we could. I think the big difference comes when you think about what happens next, after this bad code is written. So, I’ll think about this under 3 scenarios.

Scenario 1

In this scenario, I, as a senior developer, am the person who got the LLM to write the code, and I’m now tasked with code review in order to find potential flaws and time-bombs like the above.

First, in this scenario, the temptation to not check carefully is very strong. The whole reason I’m tempted to use an LLM in the first place is that I don’t want to devote much time to the task. For me this happens when:

I can’t justify much time, because I consider it’s not that important - it’s just something I need to do to get back to the main thing I’m doing.
I don’t think I will enjoy spending that time.

For both of these cases, the “must go faster” mindset means it’s psychologically very hard to slow down and do the careful review needed.

So, I’m unlikely to review this code as carefully as I should. For me, assuming that the code matters at all, this is a killer problem.

Maybe someone else would review it and catch this? That’s not good enough for me – I don’t rely on other people reviewing my code. I’m a professional software developer who works on important projects. Sometimes I work alone, with no-one else doing effective review. My clients expect and deserve that I write code that actually works.

Of course I also know that I’m far from perfect, and welcome any code review I can get. But even when I think there is review going on, I treat it as a backup, an extra safety measure, and not a reason to justify being sloppy and careless.

Scenario 2

In this scenario, the code was written by a junior developer. In contrast to the previous section, I don’t expect junior developers to produce code that works. But I do expect them to learn to do so. And I hope that that I will be rightly suspicious and do a thorough review.

Like Glyph, I actually quite enjoy doing code review for junior developers, as long as they actually have both the willingness and capacity to improve (which they usually do, but not always). Code review can be a great opportunity to actually train someone.

So what happens in this scenario when I, hopefully, after careful review spot the flaw and point it out?

If I have time to properly mentor a developer, the process would be a conversation (preferably in person or a video call) that starts something like this:

When you wrote new_id = old_id_to_new_id.get(old_id, old_id), can you explain what was going through your mind? Why did you choose to write .get(old_id, old_id), rather than the simpler and shorter [old_id]?

It’s possible the reason was them thinking “doing something is better than crashing”. We can then address that disastrously wrong (but quite common) mindset. This will hopefully be a very fruitful discussion, because it will actually correct the issue at the root, and so stop it happening again, or at least make it much less likely.

In some cases, there isn’t a reason “why” they wrote the code they did – sometimes junior developers just don’t understand a lot of the code they are writing, they are just guessing until something seems to work. In that case, we can address that problem:

The developer needs to go back to basics of things like assignments and function calls etc. The developer must reach the point where they can explain and justify every last jot and tittle of code they produce. It seems pretty obvious to me that the best way to achieve that is to make them write pretty much all their code, at least at the level of choosing each “token” that they add (e.g. choosing from a list of methods with auto-complete is OK, but not more than that). If I want them to be able to justify each token choice, it is essential to make them engage their brain and choose each token.

Scenario 3

The third scenario is again that the junior developer “produced” the code, and I’m now reviewing it, but it turns out they just prompted some LLM assistant.

At this point, the first step in the root cause analysis – the question “what was your thought process in producing this code” – fails immediately.

There is no real answer to the question “why” the LLM wrote the bad code in the first place. You can’t ask it “what were you were thinking” and get a meaningful answer – it’s pointless to ask so don’t even try. It lacks the meta-cognition needed for self-improvement, and much of the time when it looks like it is reasoning, it is actually just pattern matching.

The next problem is that the LLM basically can’t learn, at least not deeply enough to make a significant difference. You can tell it “don’t do that again”, and it might partly work, but you can’t really change how it thinks, just what the current instructions are.

So we can’t address the root cause.

What about the junior developer learning from this, in this scenario – is there something they could take away which would prevent the mistake in the future?

If their own mind wasn’t engaged in making the mistake, I think it is quite unlikely that they will be able to effectively learn from the LLM’s mistakes. For the junior dev, the possible take-aways about changes to their own behaviour are:

Check the LLM’s output more carefully. But I’m not convinced they are equipped at this point to really do checking – they first need practice in the careful thinking about what correct code looks like, to gain the trained eye that can spot subtle issues.
Don’t use LLMs to write code (which is what I’m arguing here)

What about the mid-level programmer?

At what point do I suggest the junior developers should start using LLMs for some of the more boring tasks? Once they are “mid-level” (whatever that means), is it appropriate?

There is obviously a point at which you let people make their own decisions.

For myself, I’m pretty firmly convinced that the software I’ve just recently created with 25+ years’ professional experience, I simply couldn’t have created with only 15 years experience. Not because I’m a poor programmer – I think I’ve got good reason to believe I’m significantly above average (for example, I taught myself machine code and assembly as a young teenager, and I’ve been part of the core team of top open-source projects). There is just a lot to learn, and always a lot further you could go.

In this most recent project, we’ve seen a lot of success because of certain key architectural decisions that I made right at the beginning. Some of these used advanced patterns that 10 years I would not have instinctively reached for, or wouldn’t even have known well enough to attempt them.

For example, due to difficulties with manual testing of the output in my current project, we depend critically on regression tests that make heavy use of a version of the Command pattern or plan-execute pattern. In fact we use multiple levels of it, one level being essentially an embedded DSL that has both an evaluator and compiler. This code is not trivial, and it’s also quite far from the simplest thing that you would think of, but it has proven critical to success.

How did I know I needed these? Well, I can tell you one thing for sure: I would never have got the experience and judgement necessary if I hadn’t been doing a lot of the coding myself over the past 25 years.

In his video A tale of two problem solvers, youtuber 3Blue1Brown has a section that is hugely relevant here, especially from 36 minutes onwards. In it, he describes how practising mathematicians will often do hundreds of concrete examples in order to build up intuition and sharpen their skills so that they can tackle more general and harder problems. What particularly struck me was that famous mathematicians throughout history have done this – “they all have this seemingly infinite patience for doing tedious calculations”.

Computer programming may seem different, in that we deliberately don’t do the tedious calculations, but teach the computer to do that. However, there are huge similarities. Programming, like mathematics, involves a formal notation and problem solving. In the case pf programming, the formal notation is aimed at a machine that will mechanically interpret it to produce a desired behaviour.

Obviously we do avoid doing lots of long multiplication once we’ve taught the computer to do that. However, given the similarities of the mental processes involved in maths and programming, when it comes to any of the higher level things about how to structure programs, I think we are absolutely fooling ourselves if we think we can avoid doing all the “grunt work” of writing code, organising code bases, slowly improving code structure, etc. and still end up magically knowing all the patterns that we need to use, understanding their trade-offs, and all the reasons why certain architectures or patterns would or wouldn’t be appropriate.

If you ever want to progress beyond mid-level, I strongly suspect that offloading significant parts of programming to an LLM will greatly reduce your growth. You may be much faster at outputting things of a similar level to what you can currently handle, but I doubt you’ll be able to tackle fundamentally harder projects in the future.

While I’m talking about youtubers, the video The Expert Myth by Veritasium is also really helpful here. He describes how expertise requires the following ingredients:

Valid environment
Many repetitions
Timely feedback
Deliberate practice

As far as I can see, heavy use of LLMs to write code for you will destroy points 2 and 3 – neither the repetitions nor the feedback are really happening if you don’t actually write the code yourself. I doubt that the “deliberate practice and study, in an uncomfortable zone” is going to happen either if you never get happy with the manual bits of coding.

And what about the senior programmer?

To complete this post, we of course want to ask, should the “senior programmer” use LLMs to do a lot of the grunt programming? By senior, I do mean significantly more than the 5 years that many “seniors” have. Does there come a point where you have “arrived” and the arguments in the previous section no longer apply? A time when you basically don’t need to do much more learning and sharpening of skills?

My answer to that is “I hope not”. Like others, I’m still hoping that by the end of my career I could be at least 10x faster than I am now (without an LLM writing the code for me), maybe even more. Computers are mental levers, so I don’t think that’s ridiculous. I’m hoping not just to be 10x faster, but 10x better in other ways – in terms of reliability, or in terms of being able to tackle problems that would just stump me today, or to which my solutions today would be massively inferior.

Everything I know about learning says that outsourcing my actual thinking to LLMs is unlikely to produce that result.

In addition, there are at least some people who, after actively using LLMs integrated into their editor (like Copilot and Cursor), have now stopped doing so because they noticed their skills were rusting.

These arguments, plus the small amount of evidence I have, are enough that I don’t feel the need to make a guinea pig out of myself to collect more data.

So, for myself, I choose to use LLMs very rarely for actually writing code. Rarely enough that if they were to disappear completely, it would make little difference to me.

But, aside from the practical arguments regarding becoming a better programmer, one of the big reasons for this is that I simply enjoy programming. I enjoy the challenge and the process of getting the computer to do exactly what I want it to do, in a reasonable amount of time, expressing myself both precisely and concisely.

When you are programming at the optimal level of abstraction, it is actually much nicer to express yourself in code compared to English, and often much faster too. Natural language is truly horrible for times when you want precision. And in computer programming, you usually are able to create the abstractions you need, so you can often get close to that optimal level of abstraction.

Obviously there are times when there is a bad fit between the abstractions you want and the abstractions you have, resulting in a lot of redundancy. You can’t always rewrite everything to make it all as ideal as you want. But if I’m writing large amounts of code at the wrong abstraction level, that’s bad, and I don’t really want a system that helps me to write more and more code like that.

The point of this section is really for the benefit of the junior developers that I’m forcing to do things “the hard way”. What I’m saying is this: I’m not withholding some special treat that I reserve just for myself. I willingly code in exactly the same way as you, and I really enjoy it. I believe that I’m sparing you the miserable existence of never becoming good at programming, while you keep trying to cajole something that doesn’t understand what it’s doing into producing something you don’t understand either. That just doesn’t sound like my idea of fun.

Updates

March 2026:

It looks like Carson Gross, of whom I’m a big fan, agrees with this stance when it comes to his students.
Since writing the above, I have successfully used coding agents to produce significant, high quality projects where I have done extremely little of the actual coding. However, I stand by what I’ve written above: while the agent was coding away, I never relegated myself to merely managing it. I was coding away just as hard, but on a different, related bit of work. I still don’t use AI integrated into my editor.

I’m not letting my skills stagnate, and if I get an agent to work outside my area of competency, I do it consciously. For example, in the planegcs library linked above, I was quite happy to not improve my understanding of C++ and its build tools. On the other hand, I think I’m going to need to do a Rust project, and I’m intending to do all the coding myself, even though I know an agent could make much more rapid progress, because I actually want to get good at Rust.

Statically checking Python dicts for completeness

2025-06-27T11:09:03+01:00

In Python, I often have the situation where I create a dictionary, and want to ensure that it is complete – it has an entry for every valid key.

Let’s say for my (currently hypothetical) automatic squirrel-deterring water gun system, I have a number of different states the water tank can be in, defined using an enum:

from enum import StrEnum

class TankState(StrEnum):
    FULL = "FULL"
    HALF_FULL = "HALF_FULL"
    NEARLY_EMPTY = "NEARLY_EMPTY"
    EMPTY = "EMPTY"

In a separate bit of code, I define an RGB colour for each of these states, using a simple dict.

TANK_STATE_COLORS = {
    TankState.FULL: 0x00FF00,
    TankState.HALF_FULL: 0x28D728,
    TankState.NEARLY_EMPTY: 0xFF9900,
    TankState.EMPTY: 0xFF0000,
}

This is deliberately distinct from my TankState code and related definitions, because it relates to a different part of the project - the user interface. The UI concerns shouldn’t be mixed up with the core logic.

This dict is fine, and currently complete. But I’d like to ensure that if I add a new item to TankState, I don’t forget to update the TANK_STATE_COLORS dict.

With a growing ability to do static type checks in Python, some people have asked how we can ensure this using static type checks. The short answer is, we can’t (at least at the moment).

But the better question is “how can we (somehow) ensure we don’t forget?” It doesn’t have to be a static type check, as long as it’s very hard to forget, and if it preferably runs as early as possible.

Instead of shoe-horning everything into static type checks, let’s just make use of the fact that this is Python and we can write any code we want at module level. All we need to do is this:

TANK_STATE_COLORS = {
    # …
}
for val in TankState:
    assert val in TANK_STATE_COLORS, f"TANK_STATE_COLORS is missing an entry for {val}"

That’s it, that’s the whole technique. I’d argue that this is a pretty much optimal, Pythonic solution to the problem. No clever type tricks to debug later, just 2 lines of plain simple code, and it’s impossible to import your code until you fix the problem, which means you get the early checking you want. Plus you get exactly the error message you want, not some obscure compiler output, which is also really important.

It can also be extended if you want to do something more fancy (e.g. allow some values of the enum to be missing), and if it does get in your way, you can turn it off temporarily by just commenting out a couple of lines.

That’s not quite it

OK, in a project where I’m using this a lot, I did eventually get bored of this small bit of boilerplate. So, as a Pythonic extension of this Pythonic solution, I now do this:

TANK_STATE_COLORS: dict[TankState, int] = {
    TankState.FULL: 0x00FF00,
    TankState.HALF_FULL: 0x28D728,
    TankState.NEARLY_EMPTY: 0xFF9900,
    TankState.EMPTY: 0xFF0000,
}
assert_complete_enumerations_dict(TANK_STATE_COLORS)

Specifically, I’m adding:

a type hint on the constant
a call to a clever utility function that does just the right amount of Python magic.

This function needs to be “magical” because we want it to produce good error messages, like we had before. This means it needs to get hold of the name of the dict in the calling module, but functions don’t usually have access to that.

In addition, it wants to get hold of the type hint (although there would be other ways to infer it without a type hint, there are advantages this way), for which we also need the name.

The specific magic we need is:

the clever function needs to get hold of the module that called it
it then looks through the module dictionary to get the name of the object that has been passed in
then it can find type hints, and do the checking.

So, because you don’t want to write all that yourself, the code is below. It also supports:

having a tuple of Enum types as the key
allowing some items to be missing

using Literal as the key. So you can do things like this:

X: dict[typing.Literal[-1, 0, 1], str] = {
    -1: "negative",
    0: "zero",
    1: "positive",
}
assert_complete_enumerations_dict(X)

It’s got a ton of error checking, because once you get magical then you really don’t want to be debugging obscure messages.

Enjoy!

I hereby place the following code into the public domain - CC0 1.0 Universal.

import inspect
import itertools
import sys
import typing
from collections.abc import Mapping, Sequence
from enum import Enum

from frozendict import frozendict


def assert_complete_enumerations_dict[T](the_dict: Mapping[T, object], *, allowed_missing: Sequence[T] = ()):
    """
    Magically assert that the dict in the calling module has a
    value for every item in an enumeration.
    The dict object must be bound to a name in the module.
    It must be type hinted, with the key being an Enum subclass, or Literal.
    The key may also be a tuple of Enum subclasses

    If you expect some values to be missing, pass them in `allowed_missing`
    """
    assert isinstance(the_dict, Mapping), f"{the_dict!r} is not a dict or mapping, it is a {type(the_dict)}"

    frame_up = sys._getframe(1)  # type: ignore[reportPrivateUsage]
    assert frame_up is not None
    module = inspect.getmodule(frame_up)
    assert module is not None, f"Couldn't get module for frame {frame_up}"
    msg_prefix = f"In module `{module.__name__}`,"

    module_dict = frame_up.f_locals
    name: str | None = None
    # Find the object:
    names = [k for k, val in module_dict.items() if val is the_dict]
    assert names, f"{msg_prefix} there is no name for {the_dict}, please check"

    # Any name that has a type hint will do, there will usually be one.
    hints = typing.get_type_hints(module)
    hinted_names = [name for name in names if name in hints]
    assert (
        hinted_names
    ), f"{msg_prefix} no type hints were found for {', '.join(names)}, they are needed to use assert_complete_enumerations_dict"
    name = hinted_names[0]

    hint = hints[name]
    origin = typing.get_origin(hint)
    assert origin is not None, f"{msg_prefix} type hint for {name} must supply arguments"
    assert origin in (
        dict,
        typing.Mapping,
        Mapping,
        frozendict,
    ), f"{msg_prefix} type hint for {name} must be dict/frozendict/Mapping with arguments to use assert_complete_enumerations_dict, not {origin}"

    args = typing.get_args(hint)
    assert len(args) == 2, f"{msg_prefix} type hint for {name} must have two args"

    arg0, _ = args
    arg0_origin = typing.get_origin(arg0)
    if arg0_origin is tuple:
        # tuple of Enums
        enum_list = typing.get_args(arg0)
        for enum_cls in enum_list:
            assert issubclass(
                enum_cls, Enum
            ), f"{msg_prefix} type hint must be an Enum to use assert_complete_enumerations_dict, not {enum_cls}"

        items = list(itertools.product(*(list(enum_cls) for enum_cls in enum_list)))
    elif arg0_origin is typing.Literal:
        items = typing.get_args(arg0)
    else:
        assert issubclass(
            arg0, Enum
        ), f"{msg_prefix} type hint must be an Enum to use assert_complete_enumerations_dict, not {arg0}"
        items = list(arg0)

    for item in items:
        if item in allowed_missing:
            continue

        # This is the assert we actually want to do, everything else is just error checking:
        assert item in the_dict, f"{msg_prefix} {name} needs an entry for {item}"

Knowledge creates technical debt

2025-05-13T09:08:50+01:00

The term technical debt, now used widely in software circles, was coined to explain a deliberate process where you write software quickly to gain knowledge, and then you have to use that knowledge gained to improve your software.

This perspective is still helpful today when people speak of technical debt as only a negative, or only as a result of bad decisions. Martin Fowler’s Tech Debt Quadrant is a useful antidote to that.

A consequence of this perspective is that technical debt can appear at any time, apparently from nowhere, if you are unfortunate enough to gain some knowledge.

If you discover a better way to do things, the old way of doing it that is embedded in your code base is now “debt”:

you can either live with the debt, “paying interest” in the form of all the ways that it makes your code harder to work with;
or you can “pay down” the debt by fixing all the code in light of your new knowledge, which takes up front resources which could have been spent on something else, but hopefully will make sense in the long term.

This “better way” might be a different language, library, tool or pattern. In some cases, the better way has only recently been invented. It might be your own personal discovery, or something industry wide. It might be knowledge gained through the actual work of doing the current project (which was Ward Cunningham’s usage of the tem), or from somewhere else. But the end result is the same – you know more than you did, and now you have a debt.

The problem is that this doesn’t sound like a good thing. You learn something, and now you have a problem you didn’t have before, and it’s difficult to put a good spin on “I discovered a debt”.

But from another angle, maybe this perspective gives us different language to use when communicating with others and explaining why we need to address technical debt. Rather than say “we have a liability”, the knowledge we have gained can be framed as an opportunity. Failure to take the opportunity is an opportunity cost.

The “pile of technical debt” is essentially a pile of knowledge – everything we now think is bad about the code represents what we’ve learned about how to do software better. The gap between what it is and what it should be is the gap between what we used to know and what we now know.

And fixing that code is not “a debt we have to pay off”, but an investment opportunity that will reap rewards. You can refuse to take that opportunity if you want, but it’s a tragic waste of your hard-earned knowledge – a waste of the investment you previously made in learning – and eventually you’ll be losing money, and losing out to competitors who will be making the most of their knowledge.

Finally, I think phrasing it in terms of knowledge can help tame some of our more rash instincts to call everything we don’t like “tech debt”. Can I really say “we now know” that the existing code is inferior? Is it true that fixing the code is “investing my knowledge”? If it’s just a hunch, or a personal preference, or the latest fashion, maybe I can both resist the urge for unnecessary rewrites, and feel happier about it at the same time.

Recursive project search in Emacs

2024-12-30T19:34:33Z

Before reading this, you might want to check it out in video form on Youtube, or with the embed below:

Video is probably a more helpful format for demonstrating the workflow I’m talking about. Otherwise read on. If you’re coming from the video to look at the Elisp code, it is found towards the bottom of this post.

“Recursive project search” is the name I’m giving to the flow where you do some kind of search to identify things that need to be done, but each of those tasks may leads you to do another search, etc. You need to complete all the sub-searches, but without losing your place in the parent searches.

This is extremely common in software development and maintenance, whether you are just trying to scope out a set of changes, or whether actually doing them. In fact just about any task can end being some form of this, and you never know when it will turn out that way.

This post is about how I use Emacs to do this, which is not rocket science but includes some tips and Elisp tweaks that can help a lot. When it comes to other editors or IDEs I’ve tried, I’ve never come close to finding a decent workflow for this, so I’ll have to leave it to other people to describe their approaches with other editors.

Example task - pyastgrep

I’m going to take as an example my pyastgrep project and a fairly simple refactoring I needed to do recently.

For background, pyastgrep is a command line program and library that allows you to grep Python code at the level of Abstract Syntax Trees rather than just string. At the heart of this is a function that takes a Python path, and converts it to AST and also XML.

The refactoring I want to do is make this function swappable, mostly so that users can apply different caching strategies to it. This is going to be a straightforward example of turning it into a parameter, or “dependency injection” if you want a fancy term. But that may involve modifying a number of functions in several layers of function calls.

The function in question is process_python_file.

Example workflow

The first step is a search, which in this case I will doing using lsp-mode. I happen to use lsp-pyright for Python, but there are other options.

So I’ll kick off by opening the file, putting my cursor on the function python_process_file, and calling M-x lsp-find-references. This returns a bunch of references. I can then step through them using M-x next-error and M-x previous-error — for which there are shortcuts defined, and I also do this so much that I have an F key for it — F2 and shift-F2 respectively.

Notice that in addition to the normal cursor, there is also a little triangle in the search results which shows the current result you are on.

In this case, after the function definition itself, and an import, there is just one real usage – that last item in the search results.

The details of doing this refactoring aren’t that important, but I’ll include some of the steps for completeness. The last result brings me to code like this:

def search_python_file(
    path: Path | BinaryIO,
    query_func: XMLQueryFunc,
    expression: str,
) -> Iterable[Match | ReadError | NonElementReturned]:

    ...

    processed_python = process_python_file(path)

So I make process_python_file a parameter:

def search_python_file(
    path: Path | BinaryIO,
    query_func: XMLQueryFunc,
    expression: str,
    *,
    python_file_processor: Callable[[Path], ProcessedPython | ReadError] = process_python_file,
) -> Iterable[Match | ReadError | NonElementReturned]:

    ...

    processed_python = python_file_processor(path)

Having done this, I now need to search for all usages of the function I just modified, search_python_file, so that I can pass the new parameter — another M-x lsp-find-references. I won’t go into the details this time, in this case it involves the following:

Wherever search_python_file is used, either:

don’t pass python_file_processor, because the default is what we want.
or do pass it, usually by similarly adding a python_file_processor parameter to the calling function, and passing that parameter into search_python_file.

This quickly gets me to search_python_files (note the s), and I find that it is imported in pyastgrep.api. There are no usages to be fixed here, but it is exported in __all__. This reminds me that the new parameter to this search_python_files function is actually intended to be a part of the publicly documented API — in fact this is the whole reason I’m doing this change. This means I now need to fix the docs. Another search is needed, but this time a string-based grep in the docs folder. For this, I use ripgrep and M-x rg-project-all-files.

So now I have another buffer of results to get through – I’m about 4 levels deep at this point.

Now comes one of the critical points in this workflow. I’ve completed the docs fix, and I’ve reached the end of that ripgrep buffer of search results:

So I’ve come to a “leaf” of my search. But there were a whole load of other searches that I only got half way through. What happens now?

All I do is kill these finished buffers – both the file I’m done with, and the the search buffer. And that puts me back to the previous search buffer, at exactly the point I left off, with the cursor in the search buffer in the expected place.

So I just continue. This process repeats itself, with any number of additional “side quests”, such as adding tests etc., until I get to the end of last search buffer, at which point I’m done.

Explanation

What I’ve basically done here is a depth-first recursive search over “everything that needs to be done to complete the task”. I started working on one thing, which lead to another and another, and for each one I went off and completed them as they came up.

Doing a search like that requires keeping track of a fair amount of state, and if I were doing that in my head, I would get lost very quickly and forget what I was doing. So what I do instead is to use Emacs buffers to maintain all of that state.

The buffers themselves form a stack, and each buffer has a cursor within it which tells me how far through the list of results I am. (This is equivalent to how recursive function calls typically work in a program - there will be a stack of function calls, each with local variables stored in a frame somehow).

In the above case I only went about 4 or 5 levels deep, and each level was fairly short. But you can go much deeper and not get lost, because the buffers are maintaining all the state you need. You can be 12 levels down, deep in the woods, and then just put it to one side and come back after lunch, or the next day, and just carry on, because the buffers are remembering everything for you, and the buffer that is in front of you tells you what you were doing last.

It doesn’t even matter if you switch between buffers and get them a bit out of order – you just have to ensure that you get to the bottom of each one.

Another important feature is that you can use different kinds of search, and mix and match them as you need, such as lsp-find-references and the ripgrep searches above. In addition to these two, you can insert other “search-like” things, like linters, static type checkers, compilers and build processes – anything that will return a list of items to check. So this is not a feature of just one mode, it’s a feature of how buffers work together.

At each step, you can also apply different kinds of fixes – e.g. instead of manually editing, you might be using M-x lsp-rename on each instance, and you might be using keyboard macros etc.

When I’ve attempted to use editors other than Emacs, this is one of the things I’ve missed most of all. Just getting them to do the most basic requirement of “do a search, without overwriting the previous search results” has been a headache or impossible – although I may have given up too soon.

I’m guessing people manage somehow – or perhaps not: I’ve sometimes noticed that I’ve been happy to take on tasks that involved this kind of workflow which appeared to be daunting to other people, and completed them without problem, which was apparently impressive to others. I also wonder whether difficulties in using search in an editor drive a reluctance to take on basic refactoring tasks, such as manual renames, or an inability to complete them correctly. If so, it would help to explain why codebases often have many basic problems (like bad names).

In any case, I don’t think I can take for granted that people can do this, so that’s why I’m bothering to post about it!

Requirements

What do you need in Emacs for this to work? Or what equivalent functionality is needed if you want to reproduce this elsewhere?

First, the search buffer must have the ability to remember your position. All the search modes I’ve seen in Emacs do this.
Second, you need an easy way to step through results, and this is provided by the really convenient M-x next-error function which does exactly what you want (see the Emacs docs for it, which describe how it works). This function is part of Compilation Mode or Compilation Minor Mode, and all the search modes I’ve seen use this correctly.
Thirdly, the search modes mustn’t re-use buffers for different searches, otherwise you’ll clobber earlier search results that you hadn’t finished processing. Some modes do this automatically, others have a habit of re-using buffers — but we can fix that:

Unique buffers for searches

If search commands re-use search buffers by default, I’ve found it’s usually pretty easy to override the behaviour so that you automatically always get a unique buffer for each new search you do.

The approach you need often varies slightly for each mode, but the basic principle is similar - different searches should get different buffer names, and you can use some Elisp Advice to insert the behaviour you want.

So here are the main ones that I override:

rg.el for ripgrep searching:

(defadvice rg-run (before rg-run-before activate)
  (rg-save-search))

This is just using the save feature built in to rg.el.

This goes in your init.el. Since I use use-package I usually put it inside the relevant (use-package :config) block.

For lsp-find-references I use the following which makes a unique buffer name based on the symbol being searched for:

(advice-add 'lsp-find-references
            :around #'my/lsp-find-references-unique-buffer)

(defun my/lsp-find-references-unique-buffer (orig-func &rest args)
  "Gives lsp-find-references a unique buffer name, to help with recursive search."
  (let
      ((xref-buffer-name (format "%s %s" xref-buffer-name (symbol-at-point))))
    (apply orig-func args)))

Then for general M-x compile or projectile-compile-project commands I use the following, which gives a unique buffer name based on the compilation command used:

(advice-add 'compilation-start
            :around #'my/compilation-unique-buffer)

(defun my/compilation-unique-buffer (orig-func &rest args)
  "Give compilation buffers a unique name so that new compilations get new
buffers. This helps with recursive search.

If a compile command is run starting from the compilation buffer,
the buffer will be re-used, but if it is started from a different
buffer a new compilation buffer will be created."
  (let* ((command (car args))
         (compilation-buffer (apply orig-func args))
         (new-buffer-name (concat (buffer-name compilation-buffer) " " command)))
      (with-current-buffer compilation-buffer
        (rename-buffer new-buffer-name t))))

I use this mode quite a lot via M-x compile or M-x projectile-compile-project to do things other than compilation – for custom search commands, like pyastgrep, for linters and static checkers.

Other tips

External tools

Many of externals tools that you might run via M-x compile already have output that is in exactly the format that Emacs compilation-mode expects, so that search results or error messages become hyperlinked as expected inside Emacs. For example, mypy has the right format by default, and most older compilers like gcc etc.

For those tools that don’t, sometimes you can tweak the options of how they print. For example, if you run ripgrep as rg --no-heading (just using normal M-x compile, without a dedicated ripgrep mode), it produces the necessary format.

Alternatively, you can make custom wrappers that fix the format. For example, I’ve got this one-liner bash script to wrap pyright:

#!/bin/sh
# Adjust ANSI colour and codes and whitespace output from pyright to make it work nicer in Emacs
pyright "$@" | sed 's/\x1b\[[0-9;]*[mGK]//g' | awk '{$1=$1};1' |  sed 's/\(.*:[0-9]*:[0-9]*\) -/\1: -/g'

Buffer order and keeping things tidy

Emacs typically manages your buffers as a stack. However, it’s very easy for things to get out of order as you are jumping around files or search buffers. It doesn’t matter too much if you go through the search buffers in the “wrong” order – as long as you get to the bottom of all of them. But to be sure I have got to the bottom, I do two things:

Before starting anything that I anticipate will be more than a few levels deep, I tidy up by closing all buffers but the one I’m working on. You can do this with crux-kill-other-buffers from crux. I have my own version that is more customised for my needs — crux-kill-other-buffers only kills buffers that are files.
At the end, when I think I’m done, I check the buffer list (C-x C-b) for anything else I missed.

Project context

In order to limit the scope of searches to my project, I’m typically leaning on projectile.el, but there are other options.

Conclusion

I hope this post has been helpful, and if you’ve got additional tips for this kind of workflow, please leave a comment!

Check if a point is in a cylinder - geometry and code

2024-12-05T16:40:24Z

In my current project I’m doing a fair amount of geometry, and one small problem I needed to solve a while back was finding whether a point is inside a cylinder.

The accepted answer for this on math.stackexchange.com wasn’t ideal — part of it was very over-complicated, and also didn’t work under some circumstances. So I contributed my own answer. In this post, in addition to the maths, I’ll give an implementation in Python.

Method

We can solve this problem by constructing the cylinder negatively:

Start with an infinite space
Throw out everything that isn't within the cylinder.

This is a classic mathematician’s approach, but it works great here, and it also works pretty well for any simply-connected solid object with only straight or concave surfaces, depending on how complex those surfaces are.

First some definitions:

Our cylinder is defined by two points, A and B, at the start/end of the cylinder centre line respectively, and a radius R.
The point we want to test is P.
The vectors from the origin to points A, B and P are \(\boldsymbol{r}_A\), \(\boldsymbol{r}_B\) and \(\boldsymbol{r}_P\) respectively.

We start with the infinite space, that is we assume all points are within the cylinder until we show they aren’t.

Then we construct 3 cuts to exclude certain values of \(\boldsymbol{r}_P\).

First, a cylindrical cut of radius R about an infinite line that goes through A and B. (This was taken from John Alexiou's answer in the link above):

The vector from point A to point B is:

\begin{equation*} \boldsymbol{e} = \boldsymbol{r}_B-\boldsymbol{r}_A \end{equation*}

This defines the direction of the line through A and B.
The distance from any point P at vector \(\boldsymbol{r}_P\) to the line is:

\begin{equation*} d = \frac{\| \boldsymbol{e}\times\left(\boldsymbol{r}_{P}-\boldsymbol{r}_{A}\right) \|}{\|\boldsymbol{e}\|} \end{equation*}

This is based on finding the distance of a point to a line, and using point A as an arbitrary point on the line. We could equally have used B.

(You may notice that this fails if \({\|\boldsymbol{e}\|} = 0\), which will happen if A and B are exactly coincident. In this case you have a cylinder of zero volume, which contains no points. For robustness you should check for this case and return “False”)
We then simply exclude all points with \(d > R\).

This can be optimised slightly by squaring both sides of the comparison to avoid two square root operations.

Second, a planar cut that throws away the space above the "top" of the cylinder, which I'm calling A.

The plane is defined by any point on it (A will do), and any normal pointing out of the cylinder, \(-\boldsymbol{e}\) will do (i.e. in the opposite direction to \(\boldsymbol{e}\) as defined above).
We can see which side a point is of this plane as per Relation between a point and a plane:

The point is "above" the plane (in the direction of the normal) if:

\begin{equation*} (\boldsymbol{r}_P - \boldsymbol{r}_A) \cdot -\boldsymbol{e} > 0 \end{equation*}
We exclude points which match the above.

Third, a planar cut which throws away the space below the bottom of the cylinder B.

This is the same as the previous step, but with the other end of the cylinder and the normal vector in the other direction, so the condition is:

\begin{equation*} (\boldsymbol{r}_P - \boldsymbol{r}_B) \cdot \boldsymbol{e} > 0 \end{equation*}

Python implementation

Below is a minimal implementation with zero dependencies outside the standard lib, in which there are just enough classes, with just enough methods, to express the algorithm neatly, following the above steps exactly.

In a real implementation:

You might separate out a Point class as being semantically different from Vec
You could move some functions to be methods (in my real implementation, I have a Cylinder.contains_point() method, for example)
You should probably use @dataclass(frozen=True) – immutable objects are a good default, I didn’t use them here because there aren’t needed and I’m focusing on clarity of the code.
Conversely, if performance is more of a consideration, and you don’t have a more general need for classes like Vec and Cylinder:
- you might use a more efficient representation, such as a tuple or list, or a numpy array especially if you wanted bulk operations.
- you could use more generic dot product functions etc. from numpy.
- you might inline more of the algorithm into a single function.

For clarity, I also have not implemented the optimisation mentioned in which you can avoid doing some square root operations.

from __future__ import annotations

import math
from dataclasses import dataclass

# Implementation of https://math.stackexchange.com/questions/3518495/check-if-a-general-point-is-inside-a-given-cylinder

# See the accompanying blog post http://lukeplant.me.uk/blog/posts/check-if-a-point-is-in-a-cylinder-geometry-and-code/


# -- Main algorithm --


def cylinder_contains_point(cylinder: Cylinder, point: Vec) -> bool:
    # First condition: distance from axis
    cylinder_direction: Vec = cylinder.end - cylinder.start
    abs_cylinder_direction = abs(cylinder_direction)
    if abs_cylinder_direction == 0:
        # Empty cylinder. We also have to avoid a divide-by-zero below
        return False
    point_distance_from_axis: float = abs(
        cross_product(
            cylinder_direction,
            (point - cylinder.start),
        )
    ) / abs(cylinder_direction)
    if point_distance_from_axis > cylinder.radius:
        return False

    # Second condition: point must lie below the top plane.
    # Third condition: point must lie above the bottom plane

    # We construct planes with normals pointing out of the cylinder at both
    # ends, and exclude points that are outside ("above") either plane.

    start_plane = Plane(cylinder.start, -cylinder_direction)
    if point_is_above_plane(point, start_plane):
        return False

    end_plane = Plane(cylinder.end, cylinder_direction)
    if point_is_above_plane(point, end_plane):
        return False

    return True


# -- Supporting classes and functions --


@dataclass
class Vec:
    """
    A Vector in 3 dimensions, also used to represent points in space
    """

    x: float
    y: float
    z: float

    def __add__(self, other: Vec) -> Vec:
        return Vec(self.x + other.x, self.y + other.y, self.z + other.z)

    def __sub__(self, other: Vec) -> Vec:
        return self + (-other)

    def __neg__(self) -> Vec:
        return -1 * self

    def __mul__(self, scalar: float) -> Vec:
        return Vec(self.x * scalar, self.y * scalar, self.z * scalar)

    def __rmul__(self, scalar: float) -> Vec:
        return self * scalar

    def __abs__(self) -> float:
        return math.sqrt(self.x**2 + self.y**2 + self.z**2)


@dataclass
class Plane:
    """
    A plane defined by a point on the plane, `origin`, and
    a `normal` vector to the plane.
    """

    origin: Vec
    normal: Vec


@dataclass
class Cylinder:
    """
    A closed cylinder defined by start and end points along the center
    line and a radius
    """

    start: Vec
    end: Vec
    radius: float


def cross_product(a: Vec, b: Vec) -> Vec:
    return Vec(
        a.y * b.z - a.z * b.y,
        a.z * b.x - a.x * b.z,
        a.x * b.y - a.y * b.x,
    )


def dot_product(a: Vec, b: Vec) -> float:
    return a.x * b.x + a.y * b.y + a.z * b.z


def point_is_above_plane(point: Vec, plane: Plane) -> bool:
    """
    Returns True if `point` is above the plane — that is on the side
    of the plane which is in the direction of the plane `normal`.
    """
    # See https://math.stackexchange.com/a/2998886/78071
    return dot_product((point - plane.origin), plane.normal) > 0


# -- Tests --


def test_cylinder_contains_point():
    # Test cases constructed with help of Geogebra - https://www.geogebra.org/calculator/tnc3arfm
    cylinder = Cylinder(start=Vec(1, 0, 0), end=Vec(6.196, 3, 0), radius=0.5)

    # In the Z plane:
    assert cylinder_contains_point(cylinder, Vec(1.02, 0, 0))
    assert not cylinder_contains_point(cylinder, Vec(0.98, 0, 0))  # outside bottom plane

    assert cylinder_contains_point(cylinder, Vec(0.8, 0.4, 0))
    assert not cylinder_contains_point(cylinder, Vec(0.8, 0.5, 0))  # too far from center
    assert not cylinder_contains_point(cylinder, Vec(0.8, 0.3, 0))  # outside bottom plane

    assert cylinder_contains_point(cylinder, Vec(1.4, -0.3, 0))
    assert not cylinder_contains_point(cylinder, Vec(1.4, -0.4, 0))  # too far from center

    assert cylinder_contains_point(cylinder, Vec(6.2, 2.8, 0))
    assert not cylinder_contains_point(cylinder, Vec(6.2, 2.2, 0))  # too far from center
    assert not cylinder_contains_point(cylinder, Vec(6.2, 3.2, 0))  # outside top plane

    # Away from Z plane
    assert cylinder_contains_point(cylinder, Vec(1.02, 0, 0.2))
    assert not cylinder_contains_point(cylinder, Vec(1.02, 0, 1))  # too far from center
    assert not cylinder_contains_point(cylinder, Vec(0.8, 0.3, 2))  # too far from center, and outside bottom plane

    zero_cylinder = Cylinder(start=Vec(1, 0, 0), end=Vec(1, 0, 0), radius=0.5)
    assert not cylinder_contains_point(zero_cylinder, Vec(1, 0, 0))

Keeping things in sync: derive vs test

2024-06-28T10:15:00+01:00

An extremely common problem in programming is that multiple parts of a program need to be kept in sync – they need to do exactly the same thing or behave in a consistent way. It is in response to this problem that we have mantras like “DRY” (Don’t Repeat Yourself), or, as I prefer it, OAOO, “Each and every declaration of behaviour should appear Once And Only Once”.

For both of these mantras, if you are faced with possible duplication of any kind, the answer is simply “just say no”. However, since programming mantras are to be understood as proverbs, not absolute laws, there are times that obeying this mantra can hurt more than it helps, so in this post I’m going to discuss other approaches.

Most of what I say is fairly language agnostic I think, but I’ve got specific tips for Python and web development.

The essential problem

To step back for a second, the essential problem that we are addressing here is that if making a change to a certain behaviour requires changing more than one place in the code, we have the risk that one will be forgotten. This results in bugs, which can be of various degrees of seriousness depending on the code in question.

To pick a concrete example, suppose we have a rule that says that items in a deleted folder get stored for 30 days, then expunged. We’re going to need some code that does the actual expunging after 30 days, but we’re also going to need to tell the user about the limit somewhere in the user interface. “Once And Only Once” says that the 30 days limit needs to be defined in a single place somewhere, and then reused.

There is a second kind of motivating example, which I think often crops up when people quote “Don’t Repeat Yourself”, and it’s really about avoiding tedious things from a developer perspective. Suppose you need to add an item to a menu, and you find out that first you’ve got to edit the MENU_ITEMS file to add an entry, then you’ve got to edit the MAIN_MENU constant to refer to the new entry, then you’ve got to define a keyboard shortcut in the MENU_SHORTCUTS file, then a menu icon somewhere else etc. All of these different places are in some way repeating things about how menus work. I think this is less important in general, but it is certainly life-draining as a developer if code is structured in this way, especially if it is difficult to discover or remember all the things that have to be done.

The ideal solution: derive

OAOO and DRY say that we aim to have a single place that defines the rule or logic, and any other place should be derived from this.

Regarding the simple example of a time limit displayed in the UI and used in the backend, this might be as simple as defining a constant e.g. in Python:

from datetime import timedelta

EXPUNGE_TIME_LIMIT = timedelta(days=30)

We then import and use this constant in both our UI and backend.

An important part of this approach is that the “deriving” process should be entirely automatic, not something that you can forget to do. In the case of a Python import statement, that is very easy to achieve, and relatively hard to get wrong – if you change the constant where it is defined in one module, any other code that uses it will pick up the change the next time the Python process is restarted.

Alternative solution: test

By “test”, I mean ideally an automated test, but manual tests may also work if they are properly scripted. The idea is that you write a test that checks the behaviour of code is synced. Often, it may be that for one (or more) instances that need the behaviour will define it using some constant as above, let’s say the “backend” code. Then, for one instance, e.g. the UI, you would hard code “30 days” without using the constant, but have a test that uses the backend constant to build a string, and checks the UI for that string.

Examples

In the example above, it might be hard to see why you want to use the fundamentally less reliable, less automatic method I’m suggesting. So I now have to show some motivating examples where the “derive” method ends up losing to the cruder, simpler alternative of “test”.

Example 1 - external data sources

My first example comes from the project I’m currently working on, which involves creating CAM files from input data. Most of the logic for that is driven using code, but there are some dimensions that are specified as data tables by the engineers of the physical product.

These data tables look something like below. The details here aren’t important, and I’ve changed them – it’s enough to know that we’ve are creating some physical “widgets” which need to have specific dimensions specified:

Widgets have length 150mm unless specified below
Widget id	Location	Length (mm)
A	start	100
A	end	120
F	start	105
F	end	110

These tables are supplied at design-time rather than run-time i.e. they are bundled with the software and can’t be changed after the code is shipped. But it is still convenient to read them in automatically rather than simply duplicate the tables in my code by some process. So, for the body of the table, that’s exactly what my code does on startup – it reads the bundled XLSX/CSV files.

So we are obeying “derive” here — there is a single, canonical source of data, and anywhere that needs it derives it by an entirely automatic process.

But what about that “150mm” default value specified in the header of that table?

It would be possible to “derive” it by having a parser. Writing such a parser is not hard to do – for this kind of thing in Python I like parsy, and it is as simple as:

import parsy as P

default_length_parser = (
  P.string("Widgets have length ") >>
  P.regex(r"\d+").map(int)
  << P.string("mm unless specified below")
)

In fact I do something similar in some cases. But in reality, the “parser” here is pretty simplistic – it can’t deal with the real variety of English text that might be put into the sentence, and to claim I’m “deriving” it from the table is a bit of a stretch – I’m just matching a specific, known pattern. In addition, it’s probably not the case that any value for the default length would work – most likely if it was 10 times larger, there would be some other problem, and I’d want to do some manual checking.

So, let’s admit that we are really just checking for something expected, using the “test” approach. You can still define a constant that you use in most of the code:

DEFAULT_LENGTH_MM = 150

And then you test it is what you expect when you load the data file:

assert worksheets[0].cell(1, 1).value == f"Widgets have length {DEFAULT_LENGTH_MM}mm unless specified below"

So, I’ve achieved my aim: a guard against the original problem of having multiple sources of information that could potentially be out of sync. But I’ve done it using a simple test, rather than a more complex and fragile “derive” that wouldn’t have worked well anyway.

By the way, for this specific project – we’re looking for another contract developer! It’s a very worthwhile project, and one I’m really enjoying – a small flexible team, with plenty of problem solving and fun challenges, so if you’re a talented developer and interested give me a shout.

Example 2 - defining UI behaviour for domain objects

Suppose you have a database that stores information about some kind of entity, like customers say, and you have different types of customer, represented using an enum of some kind, perhaps a string enum like this in Python:

from enum import StrEnum


class CustomerType(StrEnum):
    ENTERPRISE = "Enterprise"
    SMALL_FRY = "Small fry"  # Let’s be honest! Try not to let the name leak…
    LEGACY = "Legacy"

We need to a way edit the different customer types, and they are sufficiently different that we want quite different interfaces. So, we might have a dictionary mapping the customer type to a function or class that defines the UI. If this were a Django project, it might be a different Form class for each type:

CUSTOMER_EDIT_FORMS = {
    CustomerType.ENTERPRISE: EnterpriseCustomerForm,
    CustomerType.SMALL_FRY: SmallFryCustomerForm,
    CustomerType.LEGACY: LegacyCustomerForm,
}

Now, the DRY instinct kicks in and we notice that we now have two things we have to remember to keep in sync — any addition to the customer enum requires a corresponding addition to the UI definition dictionary. Maybe there are multiple dictionaries like this.

We could attempt to solve this by “deriving”, or some “correct by construction” mechanism that puts the creation of a new customer type all in one place.

For example, maybe we’ll have a base Customer class with get_edit_form_class() as an abstractmethod, which means it is required to be implemented. If I fail to implement it in a subclass, I can’t even construct an instance of the new customer subclass – it will throw an error.

from abc import abstractmethod

class Customer:
    @abstractmethod
    def get_edit_form_class(self):
        pass


class EnterpriseCustomer(Customer):
    def get_edit_form_class(self):
        return EnterpriseCustomerForm

class LegacyCustomer(Customer):
    ...  # etc.

I still need my enum value, or at least a list of valid values that I can use for my database field. Maybe I could derive that automatically by looking at all the sublclasses?

CUSTOMER_TYPES = [
    cls.__name__.upper().replace("CUSTOMER", "")
    for cls in Customer.__subclasses__()
]

Or maybe an __init_subclass__ trick, and I can perhaps also set up the various mappings I’ll need that way?

It’s at this point you should stop and think. In addition to requiring you to mix UI concerns into the Customer class definitions, it’s getting complex and magical.

The alternative I’m suggesting is this: require manual syncing of the two parts of the code base, but add a test to ensure that you did it. All you need is a few lines after your CUSTOMER_EDIT_FORMS definition:

CUSTOMER_EDIT_FORMS = {
    # etc as before
}

for c_type in CustomerType:
    assert (
        c_type in CUSTOMER_EDIT_FORMS
    ), f"You've defined a new customer type {c_type}, you need to add an entry in CUSTOMER_EDIT_FORMS"

You could do this as a more traditional unit test in a separate file, but for simple things like this, I think an assertion right next to the code works much better. It really helps local reasoning to be able to look and immediately conclude “yes, I can see that this dictionary must be exhaustive because the assertion tells me so.” Plus you get really early failure – as soon as you import the code.

This kind of thing crops up a lot – if you create a class here, you’ve got to create another one over there, or add a dictionary entry etc. In these cases, I’m finding simple tests and assertions have a ton of advantages when compared to clever architectural contortions (or other things like advanced static typing gymnastics):

they are massively simpler to create and understand.
you can write your own error message in the assertion. If you make a habit of using really clear error messages, like the one above, your code base will literally tell you how to maintain it.
you can easily add things like exceptions. “Every Customer type needs an edit UI defined, except Legacy because they are read only” is an easy, small change to the above.
- This contrasts with cleverer mechanisms, which might require relaxing other constraints to the point where you defeat the whole point of the mechanism, or create more difficulties for yourself.
the rule about how the code works is very explicit, rather than implicit in some complicated code structure, and typically needs no comment other than what you write in the assertion message.
you express and enforce the rule, with any complexities it gains, in just one place. Ironically, if you try to enforce this kind of constraint using type systems or hierarchies to eliminate repetition or the need for any kind of code syncing, you may find that when you come to change the constraint it actually requires touching far more places.
temporarily silencing the assertion while developing is easy and doesn’t have far reaching consequences.

Of course, there are many times when being able to automatically derive things at the code level, including some complex relationships between parts of the code, can be a win, and it’s the kind of thing you can do in Python with its many powerful techniques.

But my point is that you should remember the alternative: “synchronise manually, and have a test to check you did it.” Being able to add any kind of executable code at module level – the same level as class/function/constant definitions – is a Python super-power that you should use.

Example 3 - external polymorphism and static typing

A variant of the above problem is when, instead of an enum defining different types, I’ve got a set of classes that all need some behaviour defined.

Often we just use polymorphism where a base class defines the methods or interfaces needed and sub-classes provide the implementation. However, as in the previous case, this can involve mixing concerns e.g. user interface code, possibly of several types, is mixed up with the base domain objects. It also imposes constraints on class hierarchies.

Recently for these kind of cases, I’m more likely to prefer external polymorphism to avoid these problems. To give an example, in my current project I’m using the Command pattern or plan-execute pattern extensively, and it involves manipulating CAM objects using a series of command objects that look something like this:

@dataclass
class DeleteFeature:
    feature_name: str


@dataclass
class SetParameter:
    param_name: str
    value: float


@dataclass
class SetTextSegment:
    text_name: str
    segment: int
    value: str


Command: TypeAlias = DeleteFeature | SetParameter | SetTextSegment

Note that none of them share a base class, but I do have a union type that gives me the complete set.

It’s much more convenient to define the behaviour associated with these separately from these definitions, and so I have multiple other places that deal with Command, such as the place that executes these commands and several others. One example that requires very little code to show is where I’m generating user-presentable tables that show groups of commands. I convert each of these Command objects into key-value pairs that are used for column headings and values:

def get_command_display(command: Command) -> tuple[str, str | float | bool]:
    match command:
        case DeleteFeature(feature_name=feature_name):
            return (f"Delete {feature_name}", True)
        case SetParameter(param_name=param_name, value=value):
            return (param_name, value)
        case SetTextSegment(text_name=text_name, segment=segment, value=value):
            return (f"{text_name}[{segment}]", value)

This is giving me a similar problem to the one I had before I had before: if I add a new Command, I have to remember to add the new branch to get_command_display.

I could split out get_command_display into a dictionary of functions, and apply the same technique as in the previous example, but it’s more work, a less natural fit for the problem and potentially less flexible.

Instead, all I need to do is add exhaustiveness checking with one more branch:

match command:
    ...  # etc
    case _:
        assert_never(command)

Now, pyright will check that I didn’t forget to add branches here for any new Command. The error message is not controllable, in contrast to hand-written asserts, but it is clear enough.

The theme here is that additions in one part of the code require synchronised additions in other parts of the code, rather than being automatically correct “by construction”, but you have something that tests you didn’t forget.

Example 4 - generated code

In web development, ensuring consistent design and keeping different things in sync is a significant problem. There are many approaches, but let’s start with the simple case of using a single CSS stylesheet to define all the styles.

We may want a bunch of components to have a consistent border colour, and a first attempt might look like this (ignoring the many issues of naming conventions here):

.card-component, .bordered-heading {
   border-color: #800;
}

This often becomes impractical when we want to organise by component, rather than by property, which introduces duplication:

.card-component {
   border-color: #800;
}

/* somewhere far away … */

.bordered-heading {
   border-color: #800;
}

Thankfully, CSS has variables, so the first application of “derive” is straightforward – we define a variable which we can use in multiple places:

:root {
    --primary-border-color: #800;
}

/* elsewhere */

.bordered-heading {
    border-bottom: 1px solid var(--primary-border-color);
}

However, as the project grows, we may find that we want to use the same variables in different contexts where CSS isn’t applicable. So the next step at this point is typically to move to Design Tokens.

Practically speaking, this might mean that we now have our variables defined in a separate JSON file. Maybe something like this (using a W3C draft spec):

{
  "primary-border-color": {
    "$value": "#800000",
    "$type": "color"
  }
  "primary-hightlight-color": {
    "$value": "#FBC100",
    "$type": "color"
  }
}

From this, we can automatically generate CSS fragments that contain the same variables quite easily – for simple cases, this isn’t more than a 50 line Python script.

However, we’ve got some choices when it comes to how we put everything together. I think the general assumption in web development world is that a fully automatic “derive” is the only acceptable answer. This typically means you have to put your own CSS in a separate file, and then you have a build tool that watches for changes, and compiles your CSS plus the generated CSS into the final output that gets sent to the browser.

In addition, once you’ve bought into these kind of tools you’ll find they want to do extensive changes to the output, and define more and more extensions to the underlying languages. For example, postcss-design-tokens wants you to write things like:

.foo {
     color: design-token('color.background.primary');
 }

And instead of using CSS variables in the output, it puts the value of the token right in to every place in your code that uses it.

This approach has various problems, in particular that you become more and more dependent on the build process, and the output gets further from your input. You can no longer use the Dev Tools built in to your browser to do editing – the flow of using Dev Tools to experiment with changing a single spacing or colour CSS variable for global changes is broken, you need your build tool. And you can’t easily copy changes from Dev Tools back into the source, because of the transformation step, and debugging can be similarly difficult. And then, you’ll probably want special IDE support for the special CSS extensions, rather than being able to lean on your editor simply understanding CSS, and any other tools that want to look at your CSS now need support etc.

It’s also a lot of extra infrastructure and complexity to solve this one problem, especially when our design tokens JSON file is probably not going to change that often, or is going to have long periods of high stability. There are good reasons to want to be essentially build free. The current state of the art in this space is that to get your build tool to compile your CSS you add import './styles.css' in your entry point Javascript file! What if I don’t even have a Javascript file? I think I understand how this sort of thing came about, but don’t try to tell me that it’s anything less than completely bonkers.

Do we have an alternative to the fully automatic derive?

Using the “test” approach, we do. We can even stick with our single CSS file – we just write it like this:

/* DESIGN TOKENS START */
/* auto-created block - do not edit */
:root {
    --primary-border-color: #800000;
    --primary-highlight-color: #FBC100;
}
/* DESIGN TOKENS END */

/* the rest of our CSS here */

The contents of this block will be almost certainly auto-generated. We won’t have a process that fully automatically updates it, however, because this is the same file where we are putting our custom CSS, and we don’t want any possibility of lost work due to the file being overwritten as we are editing it.

On the other hand we don’t want things to get out of sync, so we’ll add a test that checks whether the current styles.css contains the block of design tokens that we expect to be there, based on the JSON. For actually updating the block, we’ll need some kind of manual step – maybe a script that can find and update the DESIGN TOKEN START block, maybe cog – which is a perfect little tool for this use case — or we could just copy-paste.

There are also slightly simpler solutions in this case, like using a CSS import if you don’t mind having multiple CSS files.

Conclusion

For all the examples above, the solutions I’ve presented might not work perfectly for your context. You might also want to draw the line at different place to me. But my main point is that we don’t have to go all the way with a fully automatic derive solution to eliminate any manual syncing. Having some manual work plus a mechanism to test that two things are in sync is a perfectly legitimate solution, and it can avoid some of the large costs that come with structuring everything around “derive”.

Luke Plant's home page (Posts about Python)

Inverse Sapir-Whorf and programming languages

Examples in natural language

English: temporary or permanent present tense

English/Turkish/French: gendered pronouns and nouns

Turkish: “mış” tense

Comments

Examples in programming

Links

Announcing fluent-codegen

Help my website is too small

Links

Breaking “provably correct” Leftpad

Why I’m not letting the juniors use GenAI for coding

Statically checking Python dicts for completeness

That’s not quite it

Knowledge creates technical debt

Links

Recursive project search in Emacs

Check if a point is in a cylinder - geometry and code

Method

Python implementation

Keeping things in sync: derive vs test