Luke Plant's home page (Posts about Software development)

Inverse Sapir-Whorf and programming languages

2026-05-01T09:40:36+01:00

The Sapir-Whorf hypothesis, in its simplest form, is the idea that the language you speak influences the thoughts you think. This post is about a twist on this idea, that I’m calling “Inverse Sapir-Whorf” (for want of a better term), and how we see it in computer programming languages.

Sapir-Whorf is one of those ideas that has been popularised in general culture in a rather misrepresented and exaggerated form. In the field of linguistics, not many people today take seriously the “strong” forms of Sapir-Whorf, such as “linguistic determinism” – the idea that a language controls your thoughts or limits what you can think, or that you even need certain languages to think certain thoughts.

For example, just because a language might lack grammatical tenses, it doesn’t at all follow that the speakers will be more limited in how they think about time – there are always other ways you can express time.

There is a fair amount of evidence that spoken languages can affect perception, skill and attitudes in certain areas, but it’s usually hard to demonstrate a large direct effect.

Inverse Sapir-Whorf is a bit different. I haven’t been able to track down where I first came across the idea, but it goes like this: if classic Sapir-Whorf says your language limits what you can say or think, or makes it hard to say some things, inverse Sapir-Whorf says your language limits what you can’t say, or makes it hard not to say some things, or even hard not to think about some things. Some examples might clear things up.

Examples in natural language

There are many examples to choose from, but they are not always obvious to native speakers of a language. I’ll pick just a few.

English: temporary or permanent present tense

What’s the difference between someone saying “I’m living in London” and “I live in London”? A non-native speaker may not pick this up at all, and a native speaker may pick it up only subconsciously, but “I’m living in London” reveals that the arrangement is temporary.

Now, this might not even be to do with the actual length of time you have been living there, because “temporary” is pretty relative. It might be more about how much you like London. You have to choose a tense, and because you typically do so subconsciously, the language is forcing you to reveal things – either the period of time you’ve been living somewhere, or how you feel about it.

English/Turkish/French: gendered pronouns and nouns

In English, in normal speech you are going to use “he” or “she” when referring to a specific person. “Singular they” does exist, but it’s very unnatural if you are talking about a specific person of known or assumed sex.

You can compare this to another language which doesn’t have gendered pronouns, such as Turkish, which just has “o” for he/she/it. The lack of gendered pronouns in Turkish doesn’t stop you from thinking or talking about a person’s sex, or produce a “less gendered society”, or anywhere close, so it would be difficult to find support for normal Sapir-Whorf here. But the inverse Sapir-Whorf is obvious – English pronouns push you to talk about it whether you want to or not. If you are trying to talk about someone you know, but do so anonymously, it can be very hard to avoid making their identification easier by revealing their sex with an inadvertent “him” or “her”.

Different again is French, in which nouns are gendered, which in some cases can force you to reveal information. If you translate “my friend” into French, you have to choose between “mon ami” (male friend) and “mon amie” (female friend), which are distinct, at least in written form, or “mon copain” vs “ma copine”. Possessive pronouns are also interesting – they are gendered in both English and French (his/her, son/sa), but refer to the gender of the possessor and possessee respectively, and so reveal different information.

Turkish: “mış” tense

With some simplifications, Turkish has two main past tenses: there is the normal one that is similar to “simple past” in English, and then there is the “mış” form (you can pronounce that “mish” if you want).

This has various functions, but when describing a past event, this form is used when you have second hand or unreliable information. If someone asks you “Did Fred come to work on Monday?”, then if you saw him you would use the normal past tense “geldi” (he came), but if you only heard that he came you would instead say “gelmiş” (he came, but second hand information).

The interesting thing to me as a non-native speaker was the effect of having these options, in contrast to English where you can just use simple past tense without any specific indication of reliability or where the information came from. In certain circumstances, Turkish forces you to include information about your level of certainty or whether you witnessed something – the simple past form is not neutral, because the existence of the “mış” form makes it an unnatural choice if it is not the most appropriate of the two.

Interestingly, having learned to think that way, my wife and I have noticed an effect on our English. Often in Turkish the “mış” suffix would come at the end of the last word in a sentence, so now quite frequently we get to the end of an English sentence and notice that we haven’t put in any marker for “this-is-second-hand-info-I-didn’t-actually-witness-it”, and so we tack “mış” on the end.

Of course, you can easily express the same thing in English, using words like “apparently” and other means, but English doesn’t force you to specify, while Turkish pretty much does.

Comments

You often don’t notice these things until you learn another language, or attempt to teach your language to a foreigner. You kind of just understand them subconsciously. The vast majority of times you choose simple present over present continuous, for example, you won’t be consciously thinking about what that implies.

I should also note that when a language forces you express something, it might not be in the form of something included, but in something omitted. For example, I might say “I love cake” or “I love the cake”. In the first case, I’m talking about cake generally, in the second about a specific cake. It is the absence of the word “the” in the first case that makes it unambiguous that I’m referring to all cake, because if I’m referring to a specific cake, I must use the word “the” or some other marker like “this”. In another language, there might not be a direct equivalent to this distinction.

Examples in programming

When it comes to programming languages, I think that the “straight” version of Sapir-Whorf is closer to being true - in some programming languages it is simply hard to express certain concepts. For example, in a language like Python or Haskell it’s hard (though not impossible) to talk about memory allocations. We often talk about the limitations of a language in terms of “things that are hard to express” in that language. Hillel Wayne has some more discussion of this in his post Sapir-Whorf does not apply to Programming Languages.

But I want to talk more about Inverse Sapir-Whorf. What is the language forcing you to talk about, even if you don’t actually care about it?

I think there are actually many, many examples of this, but seeing them can be quite hard, and often requires the “foreigner perspective” that comes from learning multiple languages.

Here are a few:

Most languages force you to express the order in which computation should be done. For example, in Python:
```
x = some_func(y + 1, z + 2)
```
Here you are saying:
- first compute y + 1
- then compute z + 2
- then pass these two values as arguments to some_func
You might not be very conscious of specifying this ordering, but you are doing it, and in Python, there isn’t a way to express the above computation which doesn’t also specify order. Most languages are similar, although in some it gets very complicated.

A few languages are very different, however. In Haskell, in an equivalent expression like some_func (y + 1) (z + 2), due to non-strict semantics you are not specifying an order of evaluation at all. This enables some clever tricks, like referring to values you haven’t defined yet (see Tying the knot).
Function colouring for async is another good example. In languages like Javascript or Python with an explicit async keyword, you have to talk about whether code is sync or async.

In the case of “sync” functions, you do it by omission of the async keyword, but you are still choosing between the two options, and there is no way to write code that is ambivalent on the subject.
Most languages without garbage collection force you to talk about memory allocation and de-allocation.

For languages like C, you normally do this fairly explicitly – or implicitly use stack allocation, but you’ve still got to make that choice. In other languages, it can become more hidden, but doesn’t really go away. In Rust, for example, in gets converted into talk about lifetimes or explicit reference counting. Saying “I just don’t care about when the memory for this gets allocated or de-allocated, please deal with it” is not really one of your options.

Of course, not talking about memory allocation also has a cost. In that case, the language will almost certainly need to put a lot of things on the heap and have a runtime garbage collector. However it may also have significant freedom to choose for you – Haskell will often be able to do this using strictness analysis, for example.
All modern languages I’m aware of force you to think about “scope”. In many cases, scope is expressed by the physical place in which you put a variable, with some additional syntax if you want something different (like global or nonlocal in Python). If you never want to think about scope, you probably have to drop to assembly and live with a single global address space.
Statically typed languages often force you to think about and talk about the type of every variable. This is lessened somewhat by type inference, as the “conversation” involves a more “intelligent” listener who can pick up more things from context, but it’s still there.

Pure dynamically typed languages still allow you to talk about types – for example, using things like isinstance checks in Python, but it is more unnatural (and technically it’s a different thing anyway).

In contrast to both of them, one of the attractions of gradually typed languages is that they genuinely avoid the inverse Sapir-Whorf problem, and allow you the freedom to talk about types, or not, at your preference. I’m not sure how well this works in practice - the existing code base conventions and the linters in use always put pressure in some direction.

I suspect that many of the features of more “approachable” or “readable” programming languages could be analysed in these terms – they have a low inverse Sapir-Whorf barrier, and don’t force you to talk about things you don’t have an opinion on, and may not even understand yet.

Are there more examples of this that you’ve come across? How do they affect the programming languages we use, or how we perceive them?

When perfection is table stakes

2026-03-16T15:37:13Z

For a rewrite or a replacement of existing software, very often you will find that the minimum you must achieve is something that works at least as good as the previous system in every respect, for all your users all of the time. Perfection is table stakes.

This is quite annoying if you happen to have the job of doing the rewrite, and not understanding this is a source of agro.

Here are a couple of examples from the Open Source world:

Pulseaudio is a “sound server” for Linux. A sound server is the part of your audio system that manages the fact that multiple applications might all want to be playing at once, and the multiple sound input/output devices you might be using.

Before Pulseaudio there was a different system, and the first time my system was “upgraded” to Pulseaudio, it had poor sound quality for some apps. I’m not an audiophile, but the cracks and pops and hissing were very obvious to me.

Now, compared to the previous system, Pulseaudio also came with a bunch of new features, like the ability to have a per-application volume control, and do sound over Bluetooth more easily. But I didn’t care about those features I had been given (I didn’t need them), while I certainly cared about music sounding bad. Perfect sound quality, all the time, is the minimum you have to achieve as a sound server.

I also found that if I uninstalled Pulseaudio, then I got back to the previous state where I had ALSA which just worked, and everything sounded fine. So I was pretty annoyed at this “upgrade”.

That was some years ago, and these days Pulseaudio has now apparently been replaced by Pipewire in my current Linux system. This happened without me knowing it, and in fact this is one of the highest compliments you can give to a rewrite or replacement system: the users didn’t notice.
More ambitiously in Linux world, there is Wayland, a project started in 2008 which is attempting to replace the X Window System as the protocol for how to do a windowing system, graphics output and keyboard/mouse input (so, almost everything you consciously interact with on a computer). Unsurprisingly, this is a lot of work, and there are still lots of things that don’t work for some people, and lots of problems that people didn’t have before.

Such reports are going to be pretty frustrating for the creators of the new system, whose only reward for decades of work, usually done for free, is a long list of complaints.

But for anyone who is forced to switch, if they find any regression in anything at any level, they are going to say “this sucks”. And fair enough - why should they care about all the work you did when the end result for them is worse? Why should they care about all the new features, which they don’t yet need (or don’t know they need), when some of the existing features they definitely do need are now broken?

So, what is the lesson? Don’t bother?

Rather, I think it is this: go ahead and do the rewrite, just buckle up, and set your expectations properly.

Until it has reached 100% of what it is supposed to replace, you will mostly get complaints. Don’t expect to be thanked for all your work, whether it is paid and commercial or free Open Source development. Nobody cares about the fantastic architectural improvements (which mainly help you, not them), nor about the new features they aren’t using yet. Don’t complain about the complainers and the naysayers, or ask them to shut up, or pretend that their problems don’t exist - why shouldn’t they complain, when you made their life worse?

(Do the complainers actually have a right to complain? As a side note, when talking about this with my wife, her view was that they really don’t, and maybe they should think about paying for this free software before complaining. This is coming from the perspective of “a wife of an Open Source project maintainer who spends a lot of hours on projects for which he is not thanked (or paid)”. But however much truth there is in that perspective, people will complain, and I don’t think it is sensible to assume otherwise).

I’m writing this down partly for myself, as I start a project rewriting a working piece of software for a client. This rewrite will include a large interface overhaul as well as a lot of internal architecture changes, and I’m hoping that the interface changes will be welcome.

However, I have to prepare for the fact that initial reactions will likely be negative, or not nearly as positive as we might expect, because just by changing anything I’ve already made someone’s life worse: having to relearn how to do something they used to know is not most people’s idea of fun. To make up for that, if I want them to be positively enthusiastic, not only do I have to cater for every need and fulfil every previous expectation, the improvements have also got to be both large and obvious — it’s got to be an absolute slam dunk, from the very first time they try it.

We’ve got a tiny number of current users for this application, so compared to the other projects I mentioned, the stakes are much lower and we should be able to accommodate everyone’s needs. But getting my expectations right, having empathy with the user, is essential.

Breaking “provably correct” Leftpad

2025-10-03T21:00:00+01:00

I don’t know much about about formal methods, so a while back I read Hillel Wayne’s post Let’s prove Leftpad with interest. However:

I know Donald Knuth’s famous quote: “Beware of bugs in the above code; I have only proved it correct, not tried it”
I also know how it turned out that code that had been proved correct harboured a bug not found for decades.

So I thought I’d take a peek and do some testing on these Leftpad implementations that are all “provably correct”.

Methodology

I’ll pick a few, simple, perfectly ordinary inputs at random, and work out what I think the output should be. This is a pretty trivial problem so I’m expecting that all the implementations will match my output. [narrator: He is is expecting no such thing]

I’m also expecting that, even if for some reason I’ve made a mistake, all the implementations will at least match each other. [narrator: More lies] They’ve all been proved correct, right?

Here are my inputs and expected outputs. I’m going to pad to a length of 10, and use - as padding so it can be seen and counted more easily than spaces.

Item	Input	Length	Expected padding	Expected Output
1	`𝄞`	1	9	`---------𝄞`
2	`Å`	1	9	`---------Å`
3	`𓀾`	1	9	`---------𓀾`
4	`אֳֽ֑`	1	9	`---------אֳֽ֑`
5	`résumé`	6	4	`----résumé`
6	`résumé`	6	4	`----résumé`

[“ordinary”, “random” - I think my work here is done…]

I’ve used a monospace font so that the right hand side of the outputs all line up as you’d expect.

Entry 6 is not a mistake, by the way, it just does “e acute” in a different way to entry 5. Nothing to see here, move along…

Implementations

Not all of the implementations were that easy to run. In fact, many of them didn’t take any kind of “string” as input, but vectors or lists or such things, and it wasn’t obvious to me how to pass strings in. So I discounted them.

For the ones I could run, I attempted to do so by embedding the test inputs directly in the program, if possible.

Liquid Haskell

Embedding the characters directly in Haskell source code kept getting me “lexical error in string/character literal”, so I wrote a small driver program that read from a file.

Java

The leftpad function provided didn’t take a string, but a char[]. Thankfully, it’s easy to convert from String objects, using the .toCharArray() function. So I did that.

Lean4

There is a handy online playground, and the implementation had a helpful #eval block that I could modify to get output. You can play with it here.

Rust

The code here had loads of extra lines regarding specs etc. which I stripped so I could easily run it, which worked fine.

More tricky was that the code didn’t take a string, but some Vec<Thingy>. As I know nothing about Rust, I got ChatGPT to tell me how to convert from a string to that. It gave me two options, I picked the one that looked simpler and less <<angry>>. I didn’t deliberately pick the one which made Rust look even worse than all the others, out of peevish resentment for every time someone has rewritten some Python code (my go-to language) in Rust and made it a million times faster – that’s a ridiculous suggestion.

Some competition!

To make things interesting, let’s compare these provably correct implementations with one vibe-coded by ChatGPT, in some random language, like, say, um, Swift. It gave me this code:

import Foundation

func leftPad(_ string: String, length: Int, pad: Character = " ") -> String {
    let paddingCount = max(0, length - string.count)
    return String(repeating: pad, count: paddingCount) + string
}

You can play with it online here.

Results

Here are the results, green for correct and red for … less correct.

Input	Reference	Java	Haskell	Lean	Rust	Swift
𝄞	---------𝄞	--------𝄞	---------𝄞	---------𝄞	------𝄞	---------𝄞
Å	---------Å	--------Å	--------Å	--------Å	-------Å	---------Å
𓀾	---------𓀾	--------𓀾	---------𓀾	---------𓀾	------𓀾	---------𓀾
אֳֽ֑	---------אֳֽ֑	------אֳֽ֑	------אֳֽ֑	------אֳֽ֑	--אֳֽ֑	---------אֳֽ֑
résumé	----résumé	----résumé	----résumé	----résumé	--résumé	----résumé
résumé	----résumé	--résumé	--résumé	--résumé	résumé	----résumé

And pivoted the other way around so you can compare individual inputs more easily:

Language	𝄞	Å	𓀾	אֳֽ֑	résumé	résumé
Reference	---------𝄞	---------Å	---------𓀾	---------אֳֽ֑	----résumé	----résumé
Java	--------𝄞	--------Å	--------𓀾	------אֳֽ֑	----résumé	--résumé
Haskell	---------𝄞	--------Å	---------𓀾	------אֳֽ֑	----résumé	--résumé
Lean	---------𝄞	--------Å	---------𓀾	------אֳֽ֑	----résumé	--résumé
Rust	------𝄞	-------Å	------𓀾	--אֳֽ֑	--résumé	résumé
Swift	---------𝄞	---------Å	---------𓀾	---------אֳֽ֑	----résumé	----résumé

Comments

Rust, as expected, gets nul points. What can I say?

Vibe-coding with Swift: 💯

Other that, we can see:

The only item that all implementations (apart from Rust) get correct is entry 5, the first of the two résumé options.
Java is mostly consistent with the others, but it appears it doesn’t like musical notation, or Egyptian hieroglyphics (item 3 is “standing mummy”), which seems a little rude.

The score so far:

Fancy-pants languages and formal verification: 0
Vibe-coding it with ChatGPT: 1

Explanation

OK, I’ve had my fun now :-)

(The original “Let’s Prove Leftpad” project was done “because it is funny”, and this post is in the same spirit. I want to be especially clear that I’m not actually a fan of vibe-coding).

What’s actually going on here? There are two main issues, both tied up with the concept of “the length of a string”.

(If you already know enough about Unicode, or don’t care about the details, you can skip to the “What went wrong?” section to continue discussion regarding formal verification).

First:

What is a character?

Strings are composed of “characters”, but what are they?

Most modern computer languages, and all the ones I included above, use Unicode as the basis for answering this. Unicode, at its heart, is a list of “code points” (although it is bit more than this). A code point, however, is not exactly a character.

Many of the code points you use most often, like Latin Capital Letter A U+0041, are exactly what you think of as a character. But many are not. These include:

combining characters, which are used to add accents and marks to other characters. These have some complexity:
- First, whether you regard the accent as part of the character or not can be a matter of debate. For example, in é, “e acute”, you might think of this as a different letter to e, or an e plus an acute accent. In some languages, this distinction is critical e.g. in Turkish ç is not just “c with some decoration”, it’s an entirely distinct letter.
- Second, Unicode often has multiple ways of generating these accented characters, either “pre-composed” as a single code point, or as a combination of multiple code points. So, there is both:
  - é: Latin Small Letter E With Acute U+00E9
  - é: Latin Small Letter E U+0065 + Combining Acute Accent U+0301
various spacing characters
control characters such as bidi isolation characters.
and probably more

So, Unicode has another concept, called the “extended grapheme cluster”, or “user-perceived character”, which more closely maps to what you might think of as a “character”. That’s the concept I’m implicitly using for my claim of what leftpad should output.

Secondly, there is the question of:

How does a programming language handle strings?

Different languages have different fundamental answers to the question of “what is a string?”, different internal representations of them (how the data is actually stored), and different ways of exposing strings to the programmer.

Some languages, especially performance oriented ones, provide little to zero insulation from the internal representation, while others provide a fairly strong abstraction. Some languages, like Haskell, provide multiple string types (which can be used with string literals in your code with the OverloadedStrings extension).

At this point, as well as “code points”, we’ve got to consider “encodings”. If you have a code point, even a “simple” one like U+0041 (A), you have said nothing about how you are going to store or transmit that data. An encoding is a system for doing that. Two relevant ones here:

UTF-8 is probably the most common/popular one. It uses anything from 1 to 4 bytes to express code points. It has lots of useful properties, but an important one is backwards compatibility with ASCII - if you happen to be limited to the 127 characters found in ASCII, then UTF-8 encoded Unicode text is one-byte-per-codepoint, and is byte-for-byte the the same as ASCII.
UTF-16 is an encoding where most code points (specifically those in the Basic Multilingual Plane or BMP) take up 2 bytes, and the remainder can be specified using 4 bytes and a system called “surrogate pairs”.

UTF-16 exists because originally Unicode had less than 65,536 code points, meaning you could represent all points with just two bytes, and it was thought we would never need more.

In terms of languages today, with some simplification we can say the following:

In Haskell, Lean, Python, and many other languages, strings are sequences of Unicode code points.

There is little attempt to hide the idea of code points from you, although in some, like Python, the internal representation itself might be hidden (see PEP 393).
In Java and Javascript, strings are sequences of UTF-16 items.

Originally strings were intended to be sequences of code points, but when Unicode grew, to avoid breaking all existing Java code, it morphed into the UTF-16 compromise.
In Rust, strings are UTF-8 encoded Unicode code points (see String).

Rust does also have an easily accessible chars method/concept, which corresponds to a Unicode code point. I didn’t use this above - Rust would have behaved the same as Haskell/Lean if I had.
In Swift, strings are sequences of user-perceived characters.

I’m not a user of Swift, but from the docs it appears to do a pretty good job of abstracting away from the “sequence of code point” paradigm. For example, iterating over a string gets you the “characters” (i.e. extended grapheme clusters) one by one, and the .count property gives you the number of such characters. It does also have a .length property that gives you the number of code points.

Putting them together

These differences, between them, explain the differences in output above. In more detail:

the treble clef 𝄞 is a single code point, but it is outside the BMP and requires 2 UTF-16 items. So Java considers the length to be 2, and only 8 padding characters were added, in contrast to other languages.
I created the Å using two code points (although there is a pre-composed version of this character).
The אֳֽ֑ is Hebrew, and is composed an Aleph followed by multiple vowel marks and a cantillation mark, bringing it up to 4 code points.
résumé was spelled in two different ways, one with precomposed é which is one code point, the other with combining characters where each é requires two code points.
None of the inputs was wholly in the ASCII range, so encoding them as UTF-8 requires more bytes, which is why Rust (as I used it) behaved as it did.

What went wrong?

For me, the biggest issue is not the “code points” vs “characters” debate, which is responsible for most of the variation shown, but the issue that resulted in the difference in the Java output i.e. UTF-16. All of the others (if I hadn’t stitched up Rust) would have resulted in the same output at least.

Apparently, nothing in the process of doing the formal verification forced the implementations to converge, and I think it is pretty fair to conclude that that at least one of the implementations must be faulty, given that they produce different output.

So what went wrong?

Lies, damned lies and natural language

English (or any natural language) is at the heart of the problem here. We should start with the phrase “provably correct”. It has a technical definition, but I’m not convinced those English words help us. The post accompanying the LiquidHaskell entry for Leftpad puts it this way:

My eyes roll whenever I read the phrase “proved X (a function, a program) correct”.

There is no such thing as “correct”.

There are only “specifications” or “properties”, and proofs that ensures that your code matches those specifications or properties.

For these reasons, I think I’d prefer talking about “formally verified” functions – it at least prompts you to ask “what does that mean”, and maybe suggests that you should be thinking about what, specifically, has been verified.

The next bit of English to trip us up is “the length of a string”. It’s extremely easy to imagine this is an easy concept, but in the presence of Unicode it really isn’t.

Hillel’s original informal requirements don’t actually use that phrase, instead they use the pseudo-code max(n, len(str)). Looking at the implementations, it appears people have subconsciously interpreted this as “the length of the string”, and then assumed that the functions like length or size that their language provides do “the right thing”.

We could conclude this is in fact a problem with informal requirements – it was at the level of interpreting those requirements that this went wrong. Therefore, we need more formal specifications and verification, not less. But I don’t think this gets round the fact that we’ve got to translate at some point, and at that point you’ve got trouble.

What is correct?

The issue I haven’t addressed so far is whether my reference output and the Swift implementation are actually “correct”. The reality is that you can make arguments for different things.

Implicitly, I’m arguing that left pad should be used for visual alignment in a fixed width context, and the implementation that does the best at that is the best one. I think that is a pretty reasonable case. But I’m sure you could make a case for other output – there isn’t actually anything that says what left pad should be used for. It’s possible that there are use cases where “the language’s underlying concept of the length of a string, whatever that may be” is the most important thing.

In addition, I was hiding the fact that “fixed width” is yet another lie:

I was originally going to use a flag character like 🏴󠁧󠁢󠁥󠁮󠁧󠁿 as one of my inputs, which is a single “extended grapheme cluster” that uses no less then 7 code points. It also results in 14 UTF-16 units in Java. The problem was that this character, like most emojis and many other characters from wide scripts like Chinese, takes up a double width even with a monospace font.

To maintain the subterfuge of “look how these all line up neatly in the correct output”, I was forced to use other examples. In other words, the example use case I was relying on to prove that these leftpad implementations were broken, is itself a broken concept. But I would still maintain that my reference output is closer to what you would expect leftpad to do.

A big point is this: even if we argue that a give implementation is “correct” (in that it does what its specifications say it does), that doesn’t mean you are using it correctly. Are you using it for its intended purpose and context? That seems like a really hard question to answer even for leftpad, and many other real world functions are similar.

So, I’m not sure what my final conclusion is, other than “programming is hard, let’s go shopping let’s eat chocolate” (alternative suggested by my wife, that’s my plan for the evening then).

Confessions and corrections

The Swift implementation was indeed written by ChatGPT, and it got it right first time, with just the prompt “Implement leftpad in Swift”. However:
- Swift is the only language I know where an implementation that does what I wanted it to do is that simple.
- I followed up by getting ChatGPT to produce a Python version, and it had all the same problems as Haskell/Lean and similar.
- I noticed that Swift doesn’t calculate strings lengths the way I would need for my use case for some characters, such as Zero Width Space U+200B and Right-To-Left Isolate U+2067, which I would need to count for zero length.
As mentioned, the other way to use the Rust version has the same behaviour as the Haskell/Lean/etc versions. ChatGPT did actually point out to me that this way was better, and the other way (the one I used) was adequate only if the input was limited to ASCII.

I do feel slightly justified in my treatment of this solution however - the provided code didn’t give a function that actually took a string, nor did it force you to use it in a specific way. It dodged the hard part, so I punished it.

Response

Hillel was kind enough to look at this post, and had this response to add:

In general, formally verified code can “go wrong” in two ways: the proven properties that don’t match what we need, or the proof depends on assumptions that are not true in practice. This is a good example of the former. An example of the latter would be the assumption that the output string is storable in memory. None of the formally verified functions will correctly render leftpad(“-“, 1e300, “foo”). This is why we always need to be careful when talking about “proving correctness”. In formal methods, “correct” always means “conforms to a certain specification under certain assumptions”, which is very different from the colloquial use of “correct” (does what you want and doesn’t have bugs).

He also pointed out that padding/alignment functionality available in standard libraries, like Python’s Format Specification Mini-Language and Javascript’s padStart, have similar issues.

Why I’m not letting the juniors use GenAI for coding

2025-07-26T21:30:00+01:00

In my current project, I am training some junior developers — some of them pretty much brand new developers — and one of the first rules I gave them was “ensure that Copilot (or any other Generative AI assistant that will write code for you in your editor) is turned off”. This post explains why. The long and short of it is this: it’s because I want the junior developers to become senior developers, and I want them to enjoy programming as well.

Other people might also be interested in my reasons, so I’m writing this as a blog post.

I’ve read many, many other posts that are for and against, or just plain depressed, and some of this is about personal preference, but as I’m making rules for other people, I feel I ought to justify those rules just a little.

I’m also attempting to write this up in a way that hopefully non-programmers can understand. I don’t want to write a whole load of posts about this increasingly tedious subject, so I’m making one slightly broader one that I can just link to if anyone asks my opinion.

Rather than talk generalities, I’ll build my case using a single very concrete example of real output from an LLM on a real task.

The problem

This example comes from a one-off problem I created when I accidentally ended up with data in two separate SQLite databases, when I wanted it one. The data wasn’t that important – it was all just test data for the project I’m currently working on – so I could have just thrown one of these databases away. But the data had some value to me, so I decided to see if I could use an LLM to quickly create a one-off script to merge the two databases.

Merging databases sounds like something so common there would be a generic tool to do it, but in reality the devil is in the details, and every database schema has specifics that mean you need a custom solution.

The specific databases in question were pretty small, with a pretty simple schema. The only big problem with the schema, common to many databases schemas, is how you deal with ID fields:

This database schema has multiple tables, and records in one table are related to records in another table using an ID column, which in my case was just a unique number starting at one: 1, 2, 3 etc. To make it concrete, let’s say my database stored a list of houses, and a list of rooms in each house (it didn’t, but it works as an example).

So you have a houses table:

id	address
1	45 Main St
2	67 Mayfair
3	2 Bag End
…	…

And a rooms table:

id	house_id	name
1	1	Dining room
2	1	Living room
3	2	The Library
…	…	…

The house_id column above links each room with a specific house from the houses table. In the example above, the rooms with ids 1 and 2 both belong to house with ID 1, aka “45 Main St”.

Due to the database merge, we’ve got multiple instances of each of these tables, and they could very easily be using the same ID values to refer to different things. When I come to merge the tables:

I’ve got to assign new values in the id columns for each table
for the rooms table, I’ve got to remember the mapping of old-to-new IDs from the houses table and apply the same mapping to the house_id column.

If I get this wrong, the data will be horribly confused and corrupted.

What I did

I used aider.chat, which is a pretty good project with a good reputation, and one that I’m used to, to the point of being reasonably competent (although I can’t claim I use any of these tools a massive amount). This was a while back, and I can’t remember which LLM model I was using, but I think it was one of the best ones from Claude, or maybe DeepSeek-R1, both of which are (or were) well regarded.

I fed it the database schema as a SQL file, then prompted it similar to below:

Write a script that takes a list of db files as command line args, and merges the contents. The output will have a single ’metadata’ table copied from one of the input files. The ’rooms’ and ’houses’ tables will be merged, being careful to update ’id’ and ’house_id’ columns correctly, so that they refer to new values.

I’m not going to spend time arguing about whether this was the best possible tool/model/prompt, that’s an endless time sink – I’m happy that I did a decent enough job on all fronts for my comments to be fair.

How it went

The results were basically great in most ways that most people would care about: the LLM created a mostly working script of a few hundred lines of code in a few minutes. If I remember correctly it was pretty much there on the first attempt.

I did actually care about the data, so I carefully checked the script, especially the code that mapped the IDs.

There was one bit of code that created the mapping, and a second bit of code that then used it. For this second part, correct code would have looked something like this:

new_id = id_mapping[old_id]

Here:

id_mapping is a mapping (dictionary) that contains the “old ID” as keys, and the “new ID” as values
old_id is a variable containing an old ID value.
the brackets syntax [old_id] does a dictionary lookup and returns the value found – in other words, given the old ID it returns the new ID.

There is a crucial question with mappings like this: what happens if, for some reason, the mapping doesn’t contain the old_id value? With the code as written above, the dictionary lookup will raise an exception, which will cause the code to abort at this point. This is a good thing – since somehow we are missing a value, there is nothing we can do with the current data, and obviously there is some serious problem with our code which means that aborting loudly is the best of our options, or at least the most sensible default.

However, what the LLM actually wrote was like this:

new_id = id_mapping.get(old_id, old_id)

This code also does a dictionary lookup, but it uses the get method to supply a default value. If the lookup fails because the value is missing, the default is returned instead of raising an exception. The default given here is old_id, which is the original value. This is, of course, disastrously wrong. If, for whatever reason, the new ID value is missing, the old ID value is not a good enough fallback – that’s just going to create horribly corrupted data. Worst of all, it will do so absolutely silently – it could just fill the output data with essentially garbage, with no warning that something had gone wrong.

We might ask where this idea came from — the LLM has written extra code to produce a much worse result, why? The answer is most likely found, in part, in the way the LLM was trained – that is, on mediocre code.

A better answer to “why” the AI wrote this is much more troubling, but I’ll come back to that.

We might also ask, “does it matter?”

Part of the answer to that is context. For the project I’m currently working on, “silently wrong output” is one of the very worst things we can do. There are projects with different priorities, and there are even a few where quality barely matters at all, for which we really wouldn’t care about things like this. There are also lots of projects where you might expect people would care about this, but they actually don’t, which is fairly sad. Anyway, I’m glad not to be currently working on any projects like that – correct output matters, and I like that.

In this case, there is a second reason you might say it doesn’t matter: the rest of the code was actually correct. This meant that the mapping dictionary was complete, so the incorrect behaviour would never get triggered – the problem I’ve found is actually hypothetical. So what am I worrying about?

The problem is that in real code, the hypothetical could become reality very quickly. For example:

Some change to the code could introduce a bug which means that the mapping is now missing entries.
A refactoring means the code gets used in a slightly different situation.
There is some change to the context in which the code is used. For example, if other processes were writing to one of these databases while the merge operation was happening, it would be entirely possible for the mapping dictionary to be missing entries.

So we find ourselves in an interesting situation:

The code, as written by the LLM, appears on the one hand to be perfectly adequate, if measured according to the metric of “does it work right now”.
On the other hand, the code is disastrously inadequate if measured by the standard of letting it go anywhere near important data, or anything you would want to live long term in your code base.

The main point

Writing correct code is hard. The difference between correct and disastrously bad can be extremely subtle. These differences are easily missed by the untrained eye, and you might not even be able to come up with an example where the bad code fails, because in its initial context it does not.

There are many, many other examples of this in programming, and there are many examples of LLMs tripping up like this. Often it is not what is present that is even the problem, but what is absent.

So what?

OK, so LLMs sometimes write horribly flawed code that appears to work. Couldn’t we say the same about junior programmers?

Yes, we could. I think the big difference comes when you think about what happens next, after this bad code is written. So, I’ll think about this under 3 scenarios.

Scenario 1

In this scenario, I, as a senior developer, am the person who got the LLM to write the code, and I’m now tasked with code review in order to find potential flaws and time-bombs like the above.

First, in this scenario, the temptation to not check carefully is very strong. The whole reason I’m tempted to use an LLM in the first place is that I don’t want to devote much time to the task. For me this happens when:

I can’t justify much time, because I consider it’s not that important - it’s just something I need to do to get back to the main thing I’m doing.
I don’t think I will enjoy spending that time.

For both of these cases, the “must go faster” mindset means it’s psychologically very hard to slow down and do the careful review needed.

So, I’m unlikely to review this code as carefully as I should. For me, assuming that the code matters at all, this is a killer problem.

Maybe someone else would review it and catch this? That’s not good enough for me – I don’t rely on other people reviewing my code. I’m a professional software developer who works on important projects. Sometimes I work alone, with no-one else doing effective review. My clients expect and deserve that I write code that actually works.

Of course I also know that I’m far from perfect, and welcome any code review I can get. But even when I think there is review going on, I treat it as a backup, an extra safety measure, and not a reason to justify being sloppy and careless.

Scenario 2

In this scenario, the code was written by a junior developer. In contrast to the previous section, I don’t expect junior developers to produce code that works. But I do expect them to learn to do so. And I hope that that I will be rightly suspicious and do a thorough review.

Like Glyph, I actually quite enjoy doing code review for junior developers, as long as they actually have both the willingness and capacity to improve (which they usually do, but not always). Code review can be a great opportunity to actually train someone.

So what happens in this scenario when I, hopefully, after careful review spot the flaw and point it out?

If I have time to properly mentor a developer, the process would be a conversation (preferably in person or a video call) that starts something like this:

When you wrote new_id = old_id_to_new_id.get(old_id, old_id), can you explain what was going through your mind? Why did you choose to write .get(old_id, old_id), rather than the simpler and shorter [old_id]?

It’s possible the reason was them thinking “doing something is better than crashing”. We can then address that disastrously wrong (but quite common) mindset. This will hopefully be a very fruitful discussion, because it will actually correct the issue at the root, and so stop it happening again, or at least make it much less likely.

In some cases, there isn’t a reason “why” they wrote the code they did – sometimes junior developers just don’t understand a lot of the code they are writing, they are just guessing until something seems to work. In that case, we can address that problem:

The developer needs to go back to basics of things like assignments and function calls etc. The developer must reach the point where they can explain and justify every last jot and tittle of code they produce. It seems pretty obvious to me that the best way to achieve that is to make them write pretty much all their code, at least at the level of choosing each “token” that they add (e.g. choosing from a list of methods with auto-complete is OK, but not more than that). If I want them to be able to justify each token choice, it is essential to make them engage their brain and choose each token.

Scenario 3

The third scenario is again that the junior developer “produced” the code, and I’m now reviewing it, but it turns out they just prompted some LLM assistant.

At this point, the first step in the root cause analysis – the question “what was your thought process in producing this code” – fails immediately.

There is no real answer to the question “why” the LLM wrote the bad code in the first place. You can’t ask it “what were you were thinking” and get a meaningful answer – it’s pointless to ask so don’t even try. It lacks the meta-cognition needed for self-improvement, and much of the time when it looks like it is reasoning, it is actually just pattern matching.

The next problem is that the LLM basically can’t learn, at least not deeply enough to make a significant difference. You can tell it “don’t do that again”, and it might partly work, but you can’t really change how it thinks, just what the current instructions are.

So we can’t address the root cause.

What about the junior developer learning from this, in this scenario – is there something they could take away which would prevent the mistake in the future?

If their own mind wasn’t engaged in making the mistake, I think it is quite unlikely that they will be able to effectively learn from the LLM’s mistakes. For the junior dev, the possible take-aways about changes to their own behaviour are:

Check the LLM’s output more carefully. But I’m not convinced they are equipped at this point to really do checking – they first need practice in the careful thinking about what correct code looks like, to gain the trained eye that can spot subtle issues.
Don’t use LLMs to write code (which is what I’m arguing here)

What about the mid-level programmer?

At what point do I suggest the junior developers should start using LLMs for some of the more boring tasks? Once they are “mid-level” (whatever that means), is it appropriate?

There is obviously a point at which you let people make their own decisions.

For myself, I’m pretty firmly convinced that the software I’ve just recently created with 25+ years’ professional experience, I simply couldn’t have created with only 15 years experience. Not because I’m a poor programmer – I think I’ve got good reason to believe I’m significantly above average (for example, I taught myself machine code and assembly as a young teenager, and I’ve been part of the core team of top open-source projects). There is just a lot to learn, and always a lot further you could go.

In this most recent project, we’ve seen a lot of success because of certain key architectural decisions that I made right at the beginning. Some of these used advanced patterns that 10 years I would not have instinctively reached for, or wouldn’t even have known well enough to attempt them.

For example, due to difficulties with manual testing of the output in my current project, we depend critically on regression tests that make heavy use of a version of the Command pattern or plan-execute pattern. In fact we use multiple levels of it, one level being essentially an embedded DSL that has both an evaluator and compiler. This code is not trivial, and it’s also quite far from the simplest thing that you would think of, but it has proven critical to success.

How did I know I needed these? Well, I can tell you one thing for sure: I would never have got the experience and judgement necessary if I hadn’t been doing a lot of the coding myself over the past 25 years.

In his video A tale of two problem solvers, youtuber 3Blue1Brown has a section that is hugely relevant here, especially from 36 minutes onwards. In it, he describes how practising mathematicians will often do hundreds of concrete examples in order to build up intuition and sharpen their skills so that they can tackle more general and harder problems. What particularly struck me was that famous mathematicians throughout history have done this – “they all have this seemingly infinite patience for doing tedious calculations”.

Computer programming may seem different, in that we deliberately don’t do the tedious calculations, but teach the computer to do that. However, there are huge similarities. Programming, like mathematics, involves a formal notation and problem solving. In the case pf programming, the formal notation is aimed at a machine that will mechanically interpret it to produce a desired behaviour.

Obviously we do avoid doing lots of long multiplication once we’ve taught the computer to do that. However, given the similarities of the mental processes involved in maths and programming, when it comes to any of the higher level things about how to structure programs, I think we are absolutely fooling ourselves if we think we can avoid doing all the “grunt work” of writing code, organising code bases, slowly improving code structure, etc. and still end up magically knowing all the patterns that we need to use, understanding their trade-offs, and all the reasons why certain architectures or patterns would or wouldn’t be appropriate.

If you ever want to progress beyond mid-level, I strongly suspect that offloading significant parts of programming to an LLM will greatly reduce your growth. You may be much faster at outputting things of a similar level to what you can currently handle, but I doubt you’ll be able to tackle fundamentally harder projects in the future.

While I’m talking about youtubers, the video The Expert Myth by Veritasium is also really helpful here. He describes how expertise requires the following ingredients:

Valid environment
Many repetitions
Timely feedback
Deliberate practice

As far as I can see, heavy use of LLMs to write code for you will destroy points 2 and 3 – neither the repetitions nor the feedback are really happening if you don’t actually write the code yourself. I doubt that the “deliberate practice and study, in an uncomfortable zone” is going to happen either if you never get happy with the manual bits of coding.

And what about the senior programmer?

To complete this post, we of course want to ask, should the “senior programmer” use LLMs to do a lot of the grunt programming? By senior, I do mean significantly more than the 5 years that many “seniors” have. Does there come a point where you have “arrived” and the arguments in the previous section no longer apply? A time when you basically don’t need to do much more learning and sharpening of skills?

My answer to that is “I hope not”. Like others, I’m still hoping that by the end of my career I could be at least 10x faster than I am now (without an LLM writing the code for me), maybe even more. Computers are mental levers, so I don’t think that’s ridiculous. I’m hoping not just to be 10x faster, but 10x better in other ways – in terms of reliability, or in terms of being able to tackle problems that would just stump me today, or to which my solutions today would be massively inferior.

Everything I know about learning says that outsourcing my actual thinking to LLMs is unlikely to produce that result.

In addition, there are at least some people who, after actively using LLMs integrated into their editor (like Copilot and Cursor), have now stopped doing so because they noticed their skills were rusting.

These arguments, plus the small amount of evidence I have, are enough that I don’t feel the need to make a guinea pig out of myself to collect more data.

So, for myself, I choose to use LLMs very rarely for actually writing code. Rarely enough that if they were to disappear completely, it would make little difference to me.

But, aside from the practical arguments regarding becoming a better programmer, one of the big reasons for this is that I simply enjoy programming. I enjoy the challenge and the process of getting the computer to do exactly what I want it to do, in a reasonable amount of time, expressing myself both precisely and concisely.

When you are programming at the optimal level of abstraction, it is actually much nicer to express yourself in code compared to English, and often much faster too. Natural language is truly horrible for times when you want precision. And in computer programming, you usually are able to create the abstractions you need, so you can often get close to that optimal level of abstraction.

Obviously there are times when there is a bad fit between the abstractions you want and the abstractions you have, resulting in a lot of redundancy. You can’t always rewrite everything to make it all as ideal as you want. But if I’m writing large amounts of code at the wrong abstraction level, that’s bad, and I don’t really want a system that helps me to write more and more code like that.

The point of this section is really for the benefit of the junior developers that I’m forcing to do things “the hard way”. What I’m saying is this: I’m not withholding some special treat that I reserve just for myself. I willingly code in exactly the same way as you, and I really enjoy it. I believe that I’m sparing you the miserable existence of never becoming good at programming, while you keep trying to cajole something that doesn’t understand what it’s doing into producing something you don’t understand either. That just doesn’t sound like my idea of fun.

Updates

March 2026:

It looks like Carson Gross, of whom I’m a big fan, agrees with this stance when it comes to his students.
Since writing the above, I have successfully used coding agents to produce significant, high quality projects where I have done extremely little of the actual coding. However, I stand by what I’ve written above: while the agent was coding away, I never relegated myself to merely managing it. I was coding away just as hard, but on a different, related bit of work. I still don’t use AI integrated into my editor.

I’m not letting my skills stagnate, and if I get an agent to work outside my area of competency, I do it consciously. For example, in the planegcs library linked above, I was quite happy to not improve my understanding of C++ and its build tools. On the other hand, I think I’m going to need to do a Rust project, and I’m intending to do all the coding myself, even though I know an agent could make much more rapid progress, because I actually want to get good at Rust.

Knowledge creates technical debt

2025-05-13T09:08:50+01:00

The term technical debt, now used widely in software circles, was coined to explain a deliberate process where you write software quickly to gain knowledge, and then you have to use that knowledge gained to improve your software.

This perspective is still helpful today when people speak of technical debt as only a negative, or only as a result of bad decisions. Martin Fowler’s Tech Debt Quadrant is a useful antidote to that.

A consequence of this perspective is that technical debt can appear at any time, apparently from nowhere, if you are unfortunate enough to gain some knowledge.

If you discover a better way to do things, the old way of doing it that is embedded in your code base is now “debt”:

you can either live with the debt, “paying interest” in the form of all the ways that it makes your code harder to work with;
or you can “pay down” the debt by fixing all the code in light of your new knowledge, which takes up front resources which could have been spent on something else, but hopefully will make sense in the long term.

This “better way” might be a different language, library, tool or pattern. In some cases, the better way has only recently been invented. It might be your own personal discovery, or something industry wide. It might be knowledge gained through the actual work of doing the current project (which was Ward Cunningham’s usage of the tem), or from somewhere else. But the end result is the same – you know more than you did, and now you have a debt.

The problem is that this doesn’t sound like a good thing. You learn something, and now you have a problem you didn’t have before, and it’s difficult to put a good spin on “I discovered a debt”.

But from another angle, maybe this perspective gives us different language to use when communicating with others and explaining why we need to address technical debt. Rather than say “we have a liability”, the knowledge we have gained can be framed as an opportunity. Failure to take the opportunity is an opportunity cost.

The “pile of technical debt” is essentially a pile of knowledge – everything we now think is bad about the code represents what we’ve learned about how to do software better. The gap between what it is and what it should be is the gap between what we used to know and what we now know.

And fixing that code is not “a debt we have to pay off”, but an investment opportunity that will reap rewards. You can refuse to take that opportunity if you want, but it’s a tragic waste of your hard-earned knowledge – a waste of the investment you previously made in learning – and eventually you’ll be losing money, and losing out to competitors who will be making the most of their knowledge.

Finally, I think phrasing it in terms of knowledge can help tame some of our more rash instincts to call everything we don’t like “tech debt”. Can I really say “we now know” that the existing code is inferior? Is it true that fixing the code is “investing my knowledge”? If it’s just a hunch, or a personal preference, or the latest fashion, maybe I can both resist the urge for unnecessary rewrites, and feel happier about it at the same time.

Recursive project search in Emacs

2024-12-30T19:34:33Z

Before reading this, you might want to check it out in video form on Youtube, or with the embed below:

Video is probably a more helpful format for demonstrating the workflow I’m talking about. Otherwise read on. If you’re coming from the video to look at the Elisp code, it is found towards the bottom of this post.

“Recursive project search” is the name I’m giving to the flow where you do some kind of search to identify things that need to be done, but each of those tasks may leads you to do another search, etc. You need to complete all the sub-searches, but without losing your place in the parent searches.

This is extremely common in software development and maintenance, whether you are just trying to scope out a set of changes, or whether actually doing them. In fact just about any task can end being some form of this, and you never know when it will turn out that way.

This post is about how I use Emacs to do this, which is not rocket science but includes some tips and Elisp tweaks that can help a lot. When it comes to other editors or IDEs I’ve tried, I’ve never come close to finding a decent workflow for this, so I’ll have to leave it to other people to describe their approaches with other editors.

Example task - pyastgrep

I’m going to take as an example my pyastgrep project and a fairly simple refactoring I needed to do recently.

For background, pyastgrep is a command line program and library that allows you to grep Python code at the level of Abstract Syntax Trees rather than just string. At the heart of this is a function that takes a Python path, and converts it to AST and also XML.

The refactoring I want to do is make this function swappable, mostly so that users can apply different caching strategies to it. This is going to be a straightforward example of turning it into a parameter, or “dependency injection” if you want a fancy term. But that may involve modifying a number of functions in several layers of function calls.

The function in question is process_python_file.

Example workflow

The first step is a search, which in this case I will doing using lsp-mode. I happen to use lsp-pyright for Python, but there are other options.

So I’ll kick off by opening the file, putting my cursor on the function python_process_file, and calling M-x lsp-find-references. This returns a bunch of references. I can then step through them using M-x next-error and M-x previous-error — for which there are shortcuts defined, and I also do this so much that I have an F key for it — F2 and shift-F2 respectively.

Notice that in addition to the normal cursor, there is also a little triangle in the search results which shows the current result you are on.

In this case, after the function definition itself, and an import, there is just one real usage – that last item in the search results.

The details of doing this refactoring aren’t that important, but I’ll include some of the steps for completeness. The last result brings me to code like this:

def search_python_file(
    path: Path | BinaryIO,
    query_func: XMLQueryFunc,
    expression: str,
) -> Iterable[Match | ReadError | NonElementReturned]:

    ...

    processed_python = process_python_file(path)

So I make process_python_file a parameter:

def search_python_file(
    path: Path | BinaryIO,
    query_func: XMLQueryFunc,
    expression: str,
    *,
    python_file_processor: Callable[[Path], ProcessedPython | ReadError] = process_python_file,
) -> Iterable[Match | ReadError | NonElementReturned]:

    ...

    processed_python = python_file_processor(path)

Having done this, I now need to search for all usages of the function I just modified, search_python_file, so that I can pass the new parameter — another M-x lsp-find-references. I won’t go into the details this time, in this case it involves the following:

Wherever search_python_file is used, either:

don’t pass python_file_processor, because the default is what we want.
or do pass it, usually by similarly adding a python_file_processor parameter to the calling function, and passing that parameter into search_python_file.

This quickly gets me to search_python_files (note the s), and I find that it is imported in pyastgrep.api. There are no usages to be fixed here, but it is exported in __all__. This reminds me that the new parameter to this search_python_files function is actually intended to be a part of the publicly documented API — in fact this is the whole reason I’m doing this change. This means I now need to fix the docs. Another search is needed, but this time a string-based grep in the docs folder. For this, I use ripgrep and M-x rg-project-all-files.

So now I have another buffer of results to get through – I’m about 4 levels deep at this point.

Now comes one of the critical points in this workflow. I’ve completed the docs fix, and I’ve reached the end of that ripgrep buffer of search results:

So I’ve come to a “leaf” of my search. But there were a whole load of other searches that I only got half way through. What happens now?

All I do is kill these finished buffers – both the file I’m done with, and the the search buffer. And that puts me back to the previous search buffer, at exactly the point I left off, with the cursor in the search buffer in the expected place.

So I just continue. This process repeats itself, with any number of additional “side quests”, such as adding tests etc., until I get to the end of last search buffer, at which point I’m done.

Explanation

What I’ve basically done here is a depth-first recursive search over “everything that needs to be done to complete the task”. I started working on one thing, which lead to another and another, and for each one I went off and completed them as they came up.

Doing a search like that requires keeping track of a fair amount of state, and if I were doing that in my head, I would get lost very quickly and forget what I was doing. So what I do instead is to use Emacs buffers to maintain all of that state.

The buffers themselves form a stack, and each buffer has a cursor within it which tells me how far through the list of results I am. (This is equivalent to how recursive function calls typically work in a program - there will be a stack of function calls, each with local variables stored in a frame somehow).

In the above case I only went about 4 or 5 levels deep, and each level was fairly short. But you can go much deeper and not get lost, because the buffers are maintaining all the state you need. You can be 12 levels down, deep in the woods, and then just put it to one side and come back after lunch, or the next day, and just carry on, because the buffers are remembering everything for you, and the buffer that is in front of you tells you what you were doing last.

It doesn’t even matter if you switch between buffers and get them a bit out of order – you just have to ensure that you get to the bottom of each one.

Another important feature is that you can use different kinds of search, and mix and match them as you need, such as lsp-find-references and the ripgrep searches above. In addition to these two, you can insert other “search-like” things, like linters, static type checkers, compilers and build processes – anything that will return a list of items to check. So this is not a feature of just one mode, it’s a feature of how buffers work together.

At each step, you can also apply different kinds of fixes – e.g. instead of manually editing, you might be using M-x lsp-rename on each instance, and you might be using keyboard macros etc.

When I’ve attempted to use editors other than Emacs, this is one of the things I’ve missed most of all. Just getting them to do the most basic requirement of “do a search, without overwriting the previous search results” has been a headache or impossible – although I may have given up too soon.

I’m guessing people manage somehow – or perhaps not: I’ve sometimes noticed that I’ve been happy to take on tasks that involved this kind of workflow which appeared to be daunting to other people, and completed them without problem, which was apparently impressive to others. I also wonder whether difficulties in using search in an editor drive a reluctance to take on basic refactoring tasks, such as manual renames, or an inability to complete them correctly. If so, it would help to explain why codebases often have many basic problems (like bad names).

In any case, I don’t think I can take for granted that people can do this, so that’s why I’m bothering to post about it!

Requirements

What do you need in Emacs for this to work? Or what equivalent functionality is needed if you want to reproduce this elsewhere?

First, the search buffer must have the ability to remember your position. All the search modes I’ve seen in Emacs do this.
Second, you need an easy way to step through results, and this is provided by the really convenient M-x next-error function which does exactly what you want (see the Emacs docs for it, which describe how it works). This function is part of Compilation Mode or Compilation Minor Mode, and all the search modes I’ve seen use this correctly.
Thirdly, the search modes mustn’t re-use buffers for different searches, otherwise you’ll clobber earlier search results that you hadn’t finished processing. Some modes do this automatically, others have a habit of re-using buffers — but we can fix that:

Unique buffers for searches

If search commands re-use search buffers by default, I’ve found it’s usually pretty easy to override the behaviour so that you automatically always get a unique buffer for each new search you do.

The approach you need often varies slightly for each mode, but the basic principle is similar - different searches should get different buffer names, and you can use some Elisp Advice to insert the behaviour you want.

So here are the main ones that I override:

rg.el for ripgrep searching:

(defadvice rg-run (before rg-run-before activate)
  (rg-save-search))

This is just using the save feature built in to rg.el.

This goes in your init.el. Since I use use-package I usually put it inside the relevant (use-package :config) block.

For lsp-find-references I use the following which makes a unique buffer name based on the symbol being searched for:

(advice-add 'lsp-find-references
            :around #'my/lsp-find-references-unique-buffer)

(defun my/lsp-find-references-unique-buffer (orig-func &rest args)
  "Gives lsp-find-references a unique buffer name, to help with recursive search."
  (let
      ((xref-buffer-name (format "%s %s" xref-buffer-name (symbol-at-point))))
    (apply orig-func args)))

Then for general M-x compile or projectile-compile-project commands I use the following, which gives a unique buffer name based on the compilation command used:

(advice-add 'compilation-start
            :around #'my/compilation-unique-buffer)

(defun my/compilation-unique-buffer (orig-func &rest args)
  "Give compilation buffers a unique name so that new compilations get new
buffers. This helps with recursive search.

If a compile command is run starting from the compilation buffer,
the buffer will be re-used, but if it is started from a different
buffer a new compilation buffer will be created."
  (let* ((command (car args))
         (compilation-buffer (apply orig-func args))
         (new-buffer-name (concat (buffer-name compilation-buffer) " " command)))
      (with-current-buffer compilation-buffer
        (rename-buffer new-buffer-name t))))

I use this mode quite a lot via M-x compile or M-x projectile-compile-project to do things other than compilation – for custom search commands, like pyastgrep, for linters and static checkers.

Other tips

External tools

Many of externals tools that you might run via M-x compile already have output that is in exactly the format that Emacs compilation-mode expects, so that search results or error messages become hyperlinked as expected inside Emacs. For example, mypy has the right format by default, and most older compilers like gcc etc.

For those tools that don’t, sometimes you can tweak the options of how they print. For example, if you run ripgrep as rg --no-heading (just using normal M-x compile, without a dedicated ripgrep mode), it produces the necessary format.

Alternatively, you can make custom wrappers that fix the format. For example, I’ve got this one-liner bash script to wrap pyright:

#!/bin/sh
# Adjust ANSI colour and codes and whitespace output from pyright to make it work nicer in Emacs
pyright "$@" | sed 's/\x1b\[[0-9;]*[mGK]//g' | awk '{$1=$1};1' |  sed 's/\(.*:[0-9]*:[0-9]*\) -/\1: -/g'

Buffer order and keeping things tidy

Emacs typically manages your buffers as a stack. However, it’s very easy for things to get out of order as you are jumping around files or search buffers. It doesn’t matter too much if you go through the search buffers in the “wrong” order – as long as you get to the bottom of all of them. But to be sure I have got to the bottom, I do two things:

Before starting anything that I anticipate will be more than a few levels deep, I tidy up by closing all buffers but the one I’m working on. You can do this with crux-kill-other-buffers from crux. I have my own version that is more customised for my needs — crux-kill-other-buffers only kills buffers that are files.
At the end, when I think I’m done, I check the buffer list (C-x C-b) for anything else I missed.

Project context

In order to limit the scope of searches to my project, I’m typically leaning on projectile.el, but there are other options.

Conclusion

I hope this post has been helpful, and if you’ve got additional tips for this kind of workflow, please leave a comment!

Keeping things in sync: derive vs test

2024-06-28T10:15:00+01:00

An extremely common problem in programming is that multiple parts of a program need to be kept in sync – they need to do exactly the same thing or behave in a consistent way. It is in response to this problem that we have mantras like “DRY” (Don’t Repeat Yourself), or, as I prefer it, OAOO, “Each and every declaration of behaviour should appear Once And Only Once”.

For both of these mantras, if you are faced with possible duplication of any kind, the answer is simply “just say no”. However, since programming mantras are to be understood as proverbs, not absolute laws, there are times that obeying this mantra can hurt more than it helps, so in this post I’m going to discuss other approaches.

Most of what I say is fairly language agnostic I think, but I’ve got specific tips for Python and web development.

The essential problem

To step back for a second, the essential problem that we are addressing here is that if making a change to a certain behaviour requires changing more than one place in the code, we have the risk that one will be forgotten. This results in bugs, which can be of various degrees of seriousness depending on the code in question.

To pick a concrete example, suppose we have a rule that says that items in a deleted folder get stored for 30 days, then expunged. We’re going to need some code that does the actual expunging after 30 days, but we’re also going to need to tell the user about the limit somewhere in the user interface. “Once And Only Once” says that the 30 days limit needs to be defined in a single place somewhere, and then reused.

There is a second kind of motivating example, which I think often crops up when people quote “Don’t Repeat Yourself”, and it’s really about avoiding tedious things from a developer perspective. Suppose you need to add an item to a menu, and you find out that first you’ve got to edit the MENU_ITEMS file to add an entry, then you’ve got to edit the MAIN_MENU constant to refer to the new entry, then you’ve got to define a keyboard shortcut in the MENU_SHORTCUTS file, then a menu icon somewhere else etc. All of these different places are in some way repeating things about how menus work. I think this is less important in general, but it is certainly life-draining as a developer if code is structured in this way, especially if it is difficult to discover or remember all the things that have to be done.

The ideal solution: derive

OAOO and DRY say that we aim to have a single place that defines the rule or logic, and any other place should be derived from this.

Regarding the simple example of a time limit displayed in the UI and used in the backend, this might be as simple as defining a constant e.g. in Python:

from datetime import timedelta

EXPUNGE_TIME_LIMIT = timedelta(days=30)

We then import and use this constant in both our UI and backend.

An important part of this approach is that the “deriving” process should be entirely automatic, not something that you can forget to do. In the case of a Python import statement, that is very easy to achieve, and relatively hard to get wrong – if you change the constant where it is defined in one module, any other code that uses it will pick up the change the next time the Python process is restarted.

Alternative solution: test

By “test”, I mean ideally an automated test, but manual tests may also work if they are properly scripted. The idea is that you write a test that checks the behaviour of code is synced. Often, it may be that for one (or more) instances that need the behaviour will define it using some constant as above, let’s say the “backend” code. Then, for one instance, e.g. the UI, you would hard code “30 days” without using the constant, but have a test that uses the backend constant to build a string, and checks the UI for that string.

Examples

In the example above, it might be hard to see why you want to use the fundamentally less reliable, less automatic method I’m suggesting. So I now have to show some motivating examples where the “derive” method ends up losing to the cruder, simpler alternative of “test”.

Example 1 - external data sources

My first example comes from the project I’m currently working on, which involves creating CAM files from input data. Most of the logic for that is driven using code, but there are some dimensions that are specified as data tables by the engineers of the physical product.

These data tables look something like below. The details here aren’t important, and I’ve changed them – it’s enough to know that we’ve are creating some physical “widgets” which need to have specific dimensions specified:

Widgets have length 150mm unless specified below
Widget id	Location	Length (mm)
A	start	100
A	end	120
F	start	105
F	end	110

These tables are supplied at design-time rather than run-time i.e. they are bundled with the software and can’t be changed after the code is shipped. But it is still convenient to read them in automatically rather than simply duplicate the tables in my code by some process. So, for the body of the table, that’s exactly what my code does on startup – it reads the bundled XLSX/CSV files.

So we are obeying “derive” here — there is a single, canonical source of data, and anywhere that needs it derives it by an entirely automatic process.

But what about that “150mm” default value specified in the header of that table?

It would be possible to “derive” it by having a parser. Writing such a parser is not hard to do – for this kind of thing in Python I like parsy, and it is as simple as:

import parsy as P

default_length_parser = (
  P.string("Widgets have length ") >>
  P.regex(r"\d+").map(int)
  << P.string("mm unless specified below")
)

In fact I do something similar in some cases. But in reality, the “parser” here is pretty simplistic – it can’t deal with the real variety of English text that might be put into the sentence, and to claim I’m “deriving” it from the table is a bit of a stretch – I’m just matching a specific, known pattern. In addition, it’s probably not the case that any value for the default length would work – most likely if it was 10 times larger, there would be some other problem, and I’d want to do some manual checking.

So, let’s admit that we are really just checking for something expected, using the “test” approach. You can still define a constant that you use in most of the code:

DEFAULT_LENGTH_MM = 150

And then you test it is what you expect when you load the data file:

assert worksheets[0].cell(1, 1).value == f"Widgets have length {DEFAULT_LENGTH_MM}mm unless specified below"

So, I’ve achieved my aim: a guard against the original problem of having multiple sources of information that could potentially be out of sync. But I’ve done it using a simple test, rather than a more complex and fragile “derive” that wouldn’t have worked well anyway.

By the way, for this specific project – we’re looking for another contract developer! It’s a very worthwhile project, and one I’m really enjoying – a small flexible team, with plenty of problem solving and fun challenges, so if you’re a talented developer and interested give me a shout.

Example 2 - defining UI behaviour for domain objects

Suppose you have a database that stores information about some kind of entity, like customers say, and you have different types of customer, represented using an enum of some kind, perhaps a string enum like this in Python:

from enum import StrEnum


class CustomerType(StrEnum):
    ENTERPRISE = "Enterprise"
    SMALL_FRY = "Small fry"  # Let’s be honest! Try not to let the name leak…
    LEGACY = "Legacy"

We need to a way edit the different customer types, and they are sufficiently different that we want quite different interfaces. So, we might have a dictionary mapping the customer type to a function or class that defines the UI. If this were a Django project, it might be a different Form class for each type:

CUSTOMER_EDIT_FORMS = {
    CustomerType.ENTERPRISE: EnterpriseCustomerForm,
    CustomerType.SMALL_FRY: SmallFryCustomerForm,
    CustomerType.LEGACY: LegacyCustomerForm,
}

Now, the DRY instinct kicks in and we notice that we now have two things we have to remember to keep in sync — any addition to the customer enum requires a corresponding addition to the UI definition dictionary. Maybe there are multiple dictionaries like this.

We could attempt to solve this by “deriving”, or some “correct by construction” mechanism that puts the creation of a new customer type all in one place.

For example, maybe we’ll have a base Customer class with get_edit_form_class() as an abstractmethod, which means it is required to be implemented. If I fail to implement it in a subclass, I can’t even construct an instance of the new customer subclass – it will throw an error.

from abc import abstractmethod

class Customer:
    @abstractmethod
    def get_edit_form_class(self):
        pass


class EnterpriseCustomer(Customer):
    def get_edit_form_class(self):
        return EnterpriseCustomerForm

class LegacyCustomer(Customer):
    ...  # etc.

I still need my enum value, or at least a list of valid values that I can use for my database field. Maybe I could derive that automatically by looking at all the sublclasses?

CUSTOMER_TYPES = [
    cls.__name__.upper().replace("CUSTOMER", "")
    for cls in Customer.__subclasses__()
]

Or maybe an __init_subclass__ trick, and I can perhaps also set up the various mappings I’ll need that way?

It’s at this point you should stop and think. In addition to requiring you to mix UI concerns into the Customer class definitions, it’s getting complex and magical.

The alternative I’m suggesting is this: require manual syncing of the two parts of the code base, but add a test to ensure that you did it. All you need is a few lines after your CUSTOMER_EDIT_FORMS definition:

CUSTOMER_EDIT_FORMS = {
    # etc as before
}

for c_type in CustomerType:
    assert (
        c_type in CUSTOMER_EDIT_FORMS
    ), f"You've defined a new customer type {c_type}, you need to add an entry in CUSTOMER_EDIT_FORMS"

You could do this as a more traditional unit test in a separate file, but for simple things like this, I think an assertion right next to the code works much better. It really helps local reasoning to be able to look and immediately conclude “yes, I can see that this dictionary must be exhaustive because the assertion tells me so.” Plus you get really early failure – as soon as you import the code.

This kind of thing crops up a lot – if you create a class here, you’ve got to create another one over there, or add a dictionary entry etc. In these cases, I’m finding simple tests and assertions have a ton of advantages when compared to clever architectural contortions (or other things like advanced static typing gymnastics):

they are massively simpler to create and understand.
you can write your own error message in the assertion. If you make a habit of using really clear error messages, like the one above, your code base will literally tell you how to maintain it.
you can easily add things like exceptions. “Every Customer type needs an edit UI defined, except Legacy because they are read only” is an easy, small change to the above.
- This contrasts with cleverer mechanisms, which might require relaxing other constraints to the point where you defeat the whole point of the mechanism, or create more difficulties for yourself.
the rule about how the code works is very explicit, rather than implicit in some complicated code structure, and typically needs no comment other than what you write in the assertion message.
you express and enforce the rule, with any complexities it gains, in just one place. Ironically, if you try to enforce this kind of constraint using type systems or hierarchies to eliminate repetition or the need for any kind of code syncing, you may find that when you come to change the constraint it actually requires touching far more places.
temporarily silencing the assertion while developing is easy and doesn’t have far reaching consequences.

Of course, there are many times when being able to automatically derive things at the code level, including some complex relationships between parts of the code, can be a win, and it’s the kind of thing you can do in Python with its many powerful techniques.

But my point is that you should remember the alternative: “synchronise manually, and have a test to check you did it.” Being able to add any kind of executable code at module level – the same level as class/function/constant definitions – is a Python super-power that you should use.

Example 3 - external polymorphism and static typing

A variant of the above problem is when, instead of an enum defining different types, I’ve got a set of classes that all need some behaviour defined.

Often we just use polymorphism where a base class defines the methods or interfaces needed and sub-classes provide the implementation. However, as in the previous case, this can involve mixing concerns e.g. user interface code, possibly of several types, is mixed up with the base domain objects. It also imposes constraints on class hierarchies.

Recently for these kind of cases, I’m more likely to prefer external polymorphism to avoid these problems. To give an example, in my current project I’m using the Command pattern or plan-execute pattern extensively, and it involves manipulating CAM objects using a series of command objects that look something like this:

@dataclass
class DeleteFeature:
    feature_name: str


@dataclass
class SetParameter:
    param_name: str
    value: float


@dataclass
class SetTextSegment:
    text_name: str
    segment: int
    value: str


Command: TypeAlias = DeleteFeature | SetParameter | SetTextSegment

Note that none of them share a base class, but I do have a union type that gives me the complete set.

It’s much more convenient to define the behaviour associated with these separately from these definitions, and so I have multiple other places that deal with Command, such as the place that executes these commands and several others. One example that requires very little code to show is where I’m generating user-presentable tables that show groups of commands. I convert each of these Command objects into key-value pairs that are used for column headings and values:

def get_command_display(command: Command) -> tuple[str, str | float | bool]:
    match command:
        case DeleteFeature(feature_name=feature_name):
            return (f"Delete {feature_name}", True)
        case SetParameter(param_name=param_name, value=value):
            return (param_name, value)
        case SetTextSegment(text_name=text_name, segment=segment, value=value):
            return (f"{text_name}[{segment}]", value)

This is giving me a similar problem to the one I had before I had before: if I add a new Command, I have to remember to add the new branch to get_command_display.

I could split out get_command_display into a dictionary of functions, and apply the same technique as in the previous example, but it’s more work, a less natural fit for the problem and potentially less flexible.

Instead, all I need to do is add exhaustiveness checking with one more branch:

match command:
    ...  # etc
    case _:
        assert_never(command)

Now, pyright will check that I didn’t forget to add branches here for any new Command. The error message is not controllable, in contrast to hand-written asserts, but it is clear enough.

The theme here is that additions in one part of the code require synchronised additions in other parts of the code, rather than being automatically correct “by construction”, but you have something that tests you didn’t forget.

Example 4 - generated code

In web development, ensuring consistent design and keeping different things in sync is a significant problem. There are many approaches, but let’s start with the simple case of using a single CSS stylesheet to define all the styles.

We may want a bunch of components to have a consistent border colour, and a first attempt might look like this (ignoring the many issues of naming conventions here):

.card-component, .bordered-heading {
   border-color: #800;
}

This often becomes impractical when we want to organise by component, rather than by property, which introduces duplication:

.card-component {
   border-color: #800;
}

/* somewhere far away … */

.bordered-heading {
   border-color: #800;
}

Thankfully, CSS has variables, so the first application of “derive” is straightforward – we define a variable which we can use in multiple places:

:root {
    --primary-border-color: #800;
}

/* elsewhere */

.bordered-heading {
    border-bottom: 1px solid var(--primary-border-color);
}

However, as the project grows, we may find that we want to use the same variables in different contexts where CSS isn’t applicable. So the next step at this point is typically to move to Design Tokens.

Practically speaking, this might mean that we now have our variables defined in a separate JSON file. Maybe something like this (using a W3C draft spec):

{
  "primary-border-color": {
    "$value": "#800000",
    "$type": "color"
  }
  "primary-hightlight-color": {
    "$value": "#FBC100",
    "$type": "color"
  }
}

From this, we can automatically generate CSS fragments that contain the same variables quite easily – for simple cases, this isn’t more than a 50 line Python script.

However, we’ve got some choices when it comes to how we put everything together. I think the general assumption in web development world is that a fully automatic “derive” is the only acceptable answer. This typically means you have to put your own CSS in a separate file, and then you have a build tool that watches for changes, and compiles your CSS plus the generated CSS into the final output that gets sent to the browser.

In addition, once you’ve bought into these kind of tools you’ll find they want to do extensive changes to the output, and define more and more extensions to the underlying languages. For example, postcss-design-tokens wants you to write things like:

.foo {
     color: design-token('color.background.primary');
 }

And instead of using CSS variables in the output, it puts the value of the token right in to every place in your code that uses it.

This approach has various problems, in particular that you become more and more dependent on the build process, and the output gets further from your input. You can no longer use the Dev Tools built in to your browser to do editing – the flow of using Dev Tools to experiment with changing a single spacing or colour CSS variable for global changes is broken, you need your build tool. And you can’t easily copy changes from Dev Tools back into the source, because of the transformation step, and debugging can be similarly difficult. And then, you’ll probably want special IDE support for the special CSS extensions, rather than being able to lean on your editor simply understanding CSS, and any other tools that want to look at your CSS now need support etc.

It’s also a lot of extra infrastructure and complexity to solve this one problem, especially when our design tokens JSON file is probably not going to change that often, or is going to have long periods of high stability. There are good reasons to want to be essentially build free. The current state of the art in this space is that to get your build tool to compile your CSS you add import './styles.css' in your entry point Javascript file! What if I don’t even have a Javascript file? I think I understand how this sort of thing came about, but don’t try to tell me that it’s anything less than completely bonkers.

Do we have an alternative to the fully automatic derive?

Using the “test” approach, we do. We can even stick with our single CSS file – we just write it like this:

/* DESIGN TOKENS START */
/* auto-created block - do not edit */
:root {
    --primary-border-color: #800000;
    --primary-highlight-color: #FBC100;
}
/* DESIGN TOKENS END */

/* the rest of our CSS here */

The contents of this block will be almost certainly auto-generated. We won’t have a process that fully automatically updates it, however, because this is the same file where we are putting our custom CSS, and we don’t want any possibility of lost work due to the file being overwritten as we are editing it.

On the other hand we don’t want things to get out of sync, so we’ll add a test that checks whether the current styles.css contains the block of design tokens that we expect to be there, based on the JSON. For actually updating the block, we’ll need some kind of manual step – maybe a script that can find and update the DESIGN TOKEN START block, maybe cog – which is a perfect little tool for this use case — or we could just copy-paste.

There are also slightly simpler solutions in this case, like using a CSS import if you don’t mind having multiple CSS files.

Conclusion

For all the examples above, the solutions I’ve presented might not work perfectly for your context. You might also want to draw the line at different place to me. But my main point is that we don’t have to go all the way with a fully automatic derive solution to eliminate any manual syncing. Having some manual work plus a mechanism to test that two things are in sync is a perfectly legitimate solution, and it can avoid some of the large costs that come with structuring everything around “derive”.

Never, Sometimes, Always

2024-06-14T10:00:11+01:00

In software development, we often use the Zero one infinity rule (or “zero one many”) to decide how many instances of things we should allow.

For example, a customer record in your database might have zero email addresses associated with it, one email address, or many email addresses. If you currently support one and someone says they need two, you should skip two and go straight to infinity. Adding in support for just a second email address is a bad idea, because you will still have to cope with a variable number (one or two), so it is actually simpler to cope with any number, plus you are future proof.

There is a parallel to frequencies of events: just as programmers only care about the numbers 0, 1 and ∞, the only three frequencies we care about are Never, Sometimes, and Always.

I’ve often had conversations with a client that go like this:

Me: is it possible that situation X will arise?
Client: No, no, no. That doesn’t happen. Hardly ever.

The problem is that Hardly Ever is still Sometimes, so the client’s response has gone from being a definite “no” to a definite “yes” in less than a second. It doesn’t matter to me that it rarely happens – the situation still happens, and I’ve got to write code to cope with it. The fact that the code won’t be used very often doesn’t make it cheaper to write – it’s not like engineering where a machine that is used less will require less maintenance.

The Hardly Ever cases can in fact be significantly more costly to cope with, because, for example, it might be harder to find or construct examples to use for testing. If a special branch of code is constructed to handle the rare situation, such code is likely to be less well tested and more buggy, and cost more in the long run than the code which handles the common case.

This kind of thing often comes up with a client when converting from a manual system in which all input is being handled by an intelligent agent, so Hardly Ever doesn’t cause too many problems, because the person just does something sensible. So to the client it might feel like I’m being overly pedantic and focusing all my energy on the weird cases – but in converting to the computerised system, we don’t have an intelligent agent behind the wheel, so every case has got to be covered off. Understanding the weird cases as early as possible actually helps me come up with a design in which those cases are not exceptions at all, they are just normal operation and require zero additional code – and this makes the design a lot more robust.

In fact, Hardly Ever means that the frequency is relatively high – it probably means that you’ve witnessed at least one case of it in the past, so that’s a definite Sometimes. More slippery cases are things like It’s Never Happened Yet, which just means you haven’t personally seen an instance of it, but there’s no theoretical reason why it couldn’t happen. In other words, Sometimes. And then there is It’ll Never Happen, at which point my spidey sense is saying “it’s going to happen, and probably sooner than we expect”. So, again, it’s a Sometimes.

Caveats

There are, of course, times when the programmer does care about relative frequency – where 1% is very different from 10% or 90% – and most of them fall under the term “optimisation”.

If you are a programmer wearing a product manager’s hat, you are of course allowed to produce something that is an “80% solution”, or even a “20%” solution, knowing that your product simply won’t cope at all with some cases, for which people will need to find other solutions. You are optimising for a common case at the level of business needs, and you’ll want to know what that common case is.

You may also care about relative frequency for other kinds of optimisation, such as performance or user interface design. You are allowed to have an interface which works great for the typical case but is more clunky for the exceptional one, for example.

Even here there are dangers though. Suppose you optimise an algorithm for what happens 99% of the time, so that you get, let’s say linear time complexity for normal workloads. Unfortunately, for the 1% case you drop into a worse time complexity, like quadratic time. You still have to deal with the 1%, at which point your system may be brought to its knees. It may be bad enough that the code effectively does not work at all for the user. Or if we are talking about services you run, dealing with the 1% case ends up taking up so much CPU or memory that in terms of resources, your 1% case is actually 99% of the problem.

In addition, if you are running in a hostile environment like the internet, an attacker may be able to force the worst case performance, and now you have a Denial of service vulnerability.

You may also have qualitative as well as quantitative differences in the way that the code copes with the “rarely” cases. At the very bottom end it might be to add an assertion that will fail and crash the process if the unlikely thing happens, assuming that this will not be a critical problem and you’ll get notified. Then, once you start getting those notifications, you can assess whether it is worth devoting more resources to.

Or, you might have slightly more graceful handling, and perhaps a manual override to deal with the exceptional case – if such a thing is possible, and isn’t itself more work than just dealing with it in the code.

A final caveat I can think of is that there is always some cut-off at which you count extremely unlikely events as Never. For example, the odds of a collision in a 256-bit hash function like SHA-256 are approximately one in a gazillion. That’s technically Sometimes, but it’s pretty reasonable to count as Never.

Of course, that’s what they probably said about MD5 as well…

pyastgrep and custom linting

2024-05-23T20:07:34+01:00

A while back I released pyastgrep, which is a rewrite of astpath. It’s a tool that allows you to search for specific Python syntax elements using XPath as a query language.

As part of the rewrite, I separated out the layers of code so that it can now be used as a library as well as a command line tool. I haven’t committed to very much API surface area for library usage, but there is enough.

My main personal use of this has been for linting tasks or enforcing of conventions that might be difficult to do otherwise. I don’t always use this – quite often I’d reach for custom Semgrep rules, and at other times I use introspection to enforce conventions. However, there are times when both of these fail or are rather difficult.

Examples

Some examples of the kinds of rules I’m thinking of include:

Boolean arguments to functions/methods should always be “keyword only”.

Keyword-only arguments are a big win in many cases, and especially when it comes to boolean values. For example, forcing delete_thing(True, False) to be something like delete_thing(permanent=True, force=False) is an easy win, and this is common enough that applying this as a default policy across the code base will probably be a good idea.

The pattern can be distinguished easily at syntax level. Good:
```
def foo(*, my_bool_arg: bool):
    ...
```
Bad:
```
def foo(my_bool_arg: bool):
    ...
```
Simple coding conventions like “Don’t use single letter variables like i or j as a loop variables, use index or idx instead”.

This can be found by looking for code like:
```
for i, val in enumerate(...):
    ...
```
You might not care about this, but if you do, you really want the rule to be applied as an automated test, not a nit-picky code review.
A Django-specific one: for inclusion tags, the tag names should match the template file name. This is nice for consistency and code navigation, plus I actually have some custom “jump to definition” code in my editor that relies on it for fast navigation.

The pattern can again be seen quite easily at the syntax level. Good:
```
@inclusion_tag("something/foo.html")
def foo():
    ...
```
Bad:
```
@inclusion_tag("something/bar.html")
def foo():
    ...
```
Any ’task’ (something decorated with @task) should be named foo_task in order to give a clue that it works as an asynchronous call, and its return value is just a promise object.

There are many more examples you’ll come up with once you start thinking like this.

Method

Having identified the bad patterns we want to find and fix, my method for doing so looks as follows. It contains a number of tips and refinements I’ve made over the past few years.

First, I open a test file, e.g. tests/test_conventions.py, and start by inserting some example code – at least one bad example (the kind we are trying to fix), and one good example.

There are a few reasons for this:

First, I need to make sure I can prove life exists on earth, as John D. Cook puts it. I’ll say more about this later on.
Second, it gives me a deliberately simplified bit of code that I can pass to pyastdump.
Third, it provides some explanation for the test I’m going to write, and a potentially rather hairy XPath expression.

I’ll use my first example above, keyword-only boolean args. I start by inserting the following text into my test file:

def bad_boolean_arg(foo: bool):
    pass


def good_boolean_arg(*, foo: bool):
    pass

Then, I copy both of these in turn to the clipboard (or both together if there isn’t much code, like in this case), and pass them through pyastdump. From a terminal, I do:

$ xsel | pyastdump -

I’m using the xsel Linux utility, you can also use xclip -out, or pbpaste on MacOS, or Get-Clipboard in Powershell.

This gives me some AST to look at, structured as XML:

<Module>
  <body>
    <FunctionDef lineno="1" col_offset="0" type="str" name="bad_boolean_arg">
      <args>
        <arguments>
          <posonlyargs/>
          <args>
            <arg lineno="1" col_offset="20" type="str" arg="foo">
              <annotation>
                <Name lineno="1" col_offset="25" type="str" id="bool">
                  <ctx>
                    <Load/>
                  </ctx>
                </Name>
              </annotation>
            </arg>
          </args>
          <kwonlyargs/>
          <kw_defaults/>
          <defaults/>
        </arguments>
      </args>
      <body>
        <Pass lineno="2" col_offset="4"/>
      </body>
      <decorator_list/>
    </FunctionDef>
  </body>
  <type_ignores/>
</Module>

In this case, the current structure of Python’s AST has helped us out a lot – it has separated out posonlyargs (positional only arguments), args (positional or keyword), and kwonlyargs (keyword only args). We can see the offending annotation containing a Name with id="bool" inside the args, when we want it only to be allowed as a keyword-only argument.

(Do we want to disallow boolean-annotated arguments as positional only? I’m leaning towards “no” here, as positional only is quite rare and usually a very deliberate choice).

I now have to construct an XPath expression that will find the offending XML nodes, but not match good examples. It’s pretty straightforward in this case, once you know the basics of XPath. I test it out straight away at the CLI:

pyastgrep './/FunctionDef/args/arguments/args/arg/annotation/Name[@id="bool"]' tests/test_conventions.py

If I’ve done it correctly, it should print my bad example, and not my good example.

Then I widen the net, omitting tests/test_conventions.py to search everywhere in my current directory.

At this point, I’ve probably got some real results that I want to address, but I might also notice there are other variants of the same thing I need to be able to match, and so I iterate, adding more bad/good examples as necessary.

Now I need to write a test. It’s going to look like this:

def test_boolean_arguments_are_keyword_only():
    assert_expected_pyastgrep_matches(
        """
        .//FunctionDef/args/arguments/args/arg/annotation/Name[@id="bool"]
        """,
        message="Function arguments with type `bool` should be keyword-only",
        expected_count=1,
    )

Of course, the real work is being done inside my assert_expected_pyastgrep_matches utility, which looks like this:

from pathlib import Path
from boltons import iterutils
from pyastgrep.api import Match, search_python_files

SRC_ROOT = Path(__file__).parent.parent.resolve()  # depends on project structure


def assert_expected_pyastgrep_matches(xpath_expr: str, *, expected_count: int, message: str):
    """
    Asserts that the pyastgrep XPath expression matches only `expected_count` times,
    each of which must be marked with `pyastgrep_exception`

    `message` is a message to be printed on failure.

    """
    xpath_expr = xpath_expr.strip()
    matches: list[Match] = [item for item in search_python_files([SRC_ROOT], xpath_expr) if isinstance(item, Match)]

    expected_matches, other_matches = iterutils.partition(
        matches, key=lambda match: "pyastgrep: expected" in match.matching_line
    )

    if len(expected_matches) < expected_count:
        assert False, f"Expected {expected_count} matches but found {len(expected_matches)} for {xpath_expr}"

    assert not other_matches, (
        message
        + "\n Failing examples:\n"
        + "\n".join(
            f"  {match.path}:{match.position.lineno}:{match.position.col_offset}:{match.matching_line}"
            for match in other_matches
        )
    )

There is a bit of explaining to do now.

Being sure that you can “find life on earth” is especially important for a negative test like this. It would be very easy to have an XPath query that you thought worked but didn’t, as it might just silently return zero results. In addition, Python’s AST is not stable – so a query that works now might stop working in the future.

It’s like you have a machine that claims to be able to find needles in haystacks – when it comes back and says “no needles found”, do you believe it? To increase your confidence that everything works and continues to work, you place a few needles at locations that you know, then check that the machine is able to find those needles. When it claims “found exactly 2 needles”, and you can account for those, you’ve got much more confidence that it has indeed found the only needles.

So, it’s important to leave my bad examples in there.

But, I obviously don’t want the bad examples to cause the test to fail! In addition, I want a mechanism for exceptions. A simple mechanism I’ve chosen is to add the text pyastgrep: expected as a comment.

So, I need to change my bad example like this:

def bad_boolean_arg(foo: bool):  # pyastgrep: expected
    pass

I also pass expected_count=1 to indicate that I expect to find at least one bad example (or more, if I’ve added more bad examples).

Hopefully that explains everything assert_expected_pyastgrep_matches does. A couple more notes:

it uses boltons, a pretty useful set of Python utilities
it requires a SRC_ROOT folder to be defined, which will depend on your project, and might be different depending on which folder(s) you want to apply the convention too.

Now, everything is set up, and I run the test for real, hopefully locating all the bad usages. I work through them and fix, then leave the test in.

Tips

pyastgrep works strictly at the syntax level, so unlike Semgrep you might get caught out by aliases if you try match on specific names:

from foo import bar
from foo import bar as foo_bar
import foo

# These all call the same function but look different in AST:
foo.bar()
bar()
foo_bar()

There is however, an advantage to this – you don’t need a real import to construct your bad examples, you can just use a Mock. e.g. for my inclusion_tag example above, I have code like:
```
from unittest.mock import Mock

register = Mock()

@register.inclusion_tag(filename="something/not_bad_tag.html")
def bad_tag():  # pyastgrep: expected
    pass
```
You can see the full code on GitHub.
You might be able to use a mixture of techniques:
- A Semgrep rule avoids one set of bad patterns using some thirdparty.func, and requiring everyone to use your own wrapper, which is then constructed in such a way to make it easier to apply a pyastgrep rule
- Some introspection that produces a list of classes or functions to which some rule applies, then dynamically generates XPath expression to pass to pyastgrep.

Conclusion

Syntax level searching isn’t right for every job, but it can be a powerful addition to your toolkit, and with a decent query language like XPath, you can do a surprising amount. Have a look at the pyastgrep examples for inspiration!

Programming mantras are proverbs

2024-05-07T20:27:26+01:00

I believe that many of the arguments we have around software development practices could be avoided by the simple understanding that all of our mantras need to be understood as proverbs and not laws.

If you understand proverbs, then you’ll know that every proverb has an equal and opposite proverb.

So, that means it’s fine to say DRY - Don’t Repeat Yourself, and also WET - Write Everything Twice.

It means it’s fine that we quote Knuth saying premature optimization is the root of all evil, while also believing that Performance is a Feature.

We can understand and apply separation of concerns, while also seeing the value of locality of behaviour.

There is a time to YAGNI, and there is a time to refrain from YAGNI.

None of the above involve logical contradictions. Proverbs encapsulate a small bit of wisdom in a pithy phrase, but you still need to learn to apply that wisdom correctly.

Good proverbs are also supposed to make you think. They slow you down before you leap into something that is likely to be bad, or they help you reflect on what did go wrong.

One pair of Biblical proverbs that illustrate both of these things well is found in Proverbs 26:4-5:

4 Do not answer a fool according to his folly,

or you yourself will be just like him.

5 Answer a fool according to his folly,

or he will be wise in his own eyes.

The first step is you are supposed to think: is it worth arguing in this situation?

The second step, when you’ve had multiple failed interactions with a person, is to think: is it ever worth arguing with them? If the person is a fool, then you just can’t win.

The third step is when you think like this: I notice that no-one ever argues with me for long. Is that because I’m always right, or because I’m a fool who refuses to lose an argument?

So when it comes to programming, you’re not just supposed to pick whichever of the conflicting proverbs suits what you wanted to do anyway, you’re supposed to recognise you might be about to do something silly, and reflect before you continue.

If it looks like you’re about to go against accumulated wisdom, you need to think about where the proverb came from, and the consequences of ignoring it.

For example, with “DRY”, you need to understand that it’s not about repetitive or boring code, but about embedding the same knowledge into a system in multiple places, and how this is a problem when you come to change something. WET, on the other hand, is about premature abstractions being more costly than duplication.

Once you’ve slowed down enough to think about it, you can work out which applies in your situation, and what you can do to mitigate the difficulties you might encounter if you choose to ignore the proverb – in other words, you can apply Chesterton’s Fence.

Luke Plant's home page (Posts about Software development)

Inverse Sapir-Whorf and programming languages

Examples in natural language

English: temporary or permanent present tense

English/Turkish/French: gendered pronouns and nouns

Turkish: “mış” tense

Comments

Examples in programming

Links

When perfection is table stakes

Breaking “provably correct” Leftpad

Why I’m not letting the juniors use GenAI for coding

Knowledge creates technical debt

Links

Recursive project search in Emacs

Keeping things in sync: derive vs test

Never, Sometimes, Always

Caveats

Links

pyastgrep and custom linting

Examples

Method

Tips

Conclusion

Links

Programming mantras are proverbs

Links