About pyastgrep

This tool is a utility to allow you to grep Python code for specific syntax elements using XPath as a powerful query language. The main functions are:

• intelligently work out which files to search (respecting .gitignore files etc.)

• parse the Python files as AST and convert to XML

• apply a user-supplied XPath expression to search for specific AST elements

• print the results (with the complexity of handling different context strategies and colouring)

Here is an example showing pyastgrep searching its own code base for all usages of names (variables etc) that contain the substring idx:

$ pyastgrep './/Name[contains(@id, "idx")]'
src/pyastgrep/files.py:60:5:    current_idx = 0
src/pyastgrep/files.py:64:9:        linebreak_idx = python_file_bytes.find(b"\n", current_idx)
src/pyastgrep/files.py:64:55:        linebreak_idx = python_file_bytes.find(b"\n", current_idx)
src/pyastgrep/files.py:65:12:        if linebreak_idx < 0:
src/pyastgrep/files.py:66:38:            line = python_file_bytes[current_idx:]
src/pyastgrep/files.py:68:38:            line = python_file_bytes[current_idx:linebreak_idx]
src/pyastgrep/files.py:68:50:            line = python_file_bytes[current_idx:linebreak_idx]
src/pyastgrep/files.py:72:12:        if linebreak_idx < 0:
src/pyastgrep/files.py:75:13:            current_idx = linebreak_idx + 1
src/pyastgrep/files.py:75:27:            current_idx = linebreak_idx + 1
src/pyastgrep/printer.py:244:9:        start_line_idx = line_index - before_context
src/pyastgrep/printer.py:245:9:        end_line_idx = line_index + after_context
src/pyastgrep/printer.py:246:9:        stop_line_idx = end_line_idx + 1
src/pyastgrep/printer.py:246:25:        stop_line_idx = end_line_idx + 1
src/pyastgrep/printer.py:248:19:        if (path, end_line_idx) in self.printed_context_lines:
src/pyastgrep/printer.py:252:19:        if (path, start_line_idx - 1) not in self.printed_context_lines:
src/pyastgrep/printer.py:253:57:            header = self.formatter.format_header(path, start_line_idx)
src/pyastgrep/printer.py:257:44:        code = "\n".join(result.file_lines[start_line_idx:stop_line_idx])
src/pyastgrep/printer.py:257:59:        code = "\n".join(result.file_lines[start_line_idx:stop_line_idx])
src/pyastgrep/printer.py:260:13:        for idx in range(start_line_idx, stop_line_idx):
src/pyastgrep/printer.py:260:26:        for idx in range(start_line_idx, stop_line_idx):
src/pyastgrep/printer.py:260:42:        for idx in range(start_line_idx, stop_line_idx):
src/pyastgrep/printer.py:261:51:            self.printed_context_lines.add((path, idx))

Things I liked

There were some things I really liked about the flow of using a type checker, and I was able to lean on the checker pretty heavily for some things.

One example of using type-driven programming was going between the layer of my code that found matches and the layer that printed them. I found that the search function needed to go from returning a simple Iterable[Match] eventually all the way to Iterable[Match | MissingPath | ReadError | NonElementReturned | FileFinished]. In each case, I could do something like:

• add the new return value, something like yield MissingPath(...), in the body of the function.

• fix up the function type signature, in response to the type error that mypy would now report.

• then respond to the type error that mypy would report in the places that consumed this iterable, usually by implementing handling of the new result type. isinstance checks can be used to drive type narrowing and satisfy mypy that everything is fine.

It was nice when this went through multiple layers, knowing that something was checking it for me and driving me to the next bit of code that needed fixing. Having the types there explicitly also helps to make you conscious of decisions you are making about how the layers are working, which was helpful in keeping the layers straight, so that I can use this code both as a library and a command line tool.

Being able to convert various exceptions, errors and corner cases to sum types in this way was also great, and I leaned on this heavily — probably more than if I hadn’t been using a type checker. I quite like these kind of ad-hoc sum types – in some ways they often work nicer than sum types in Haskell etc.

Use of mypy has encouraged me to use lots of small custom classes, because I can use “Find references” to search the code base for everything related to a particular type, or a particular attribute or method. For custom classes I create and use within the code base, this works perfectly, which is really nice. The same goes for renaming things using IDE tools (I’m using Emacs and the pyright LSP server).

I have a high degree of confidence that I’ll be able to come back to this code base after years without touching and be able to navigate it and make changes very easily.

Issues

However, I have a long list of complaints about issues I found too!

check_untyped_defs = true
disallow_untyped_calls = true
disallow_untyped_defs = true
disallow_incomplete_defs = true

path: Path = Path(...)

if path == "-":
    ...

class AST:
    ...
    # TODO: Not all nodes have all of the following attributes

if args.color == UseColor.AUTO:
    colorer = make_default_colorer()
elif args.colors == UseColor.NEVER:
    colorer = NullColorer()

def foo() -> None:
    if 1 + 1 == 3:
        x = "hello"
    print(x)

def print_greeting(username: str, type: Greeting) -> None:
    if type == Greeting.HELLO:
        greeting = "hello"
    elif type == Greeting.GOODBYE:
        greeting = "goodbye"

    print(f"{greeting} {username}")

pyright

More recently I’ve tried pyright as an alternative to mypy. Generally I’ve found it to be significantly less buggy. However, mypy has a lot going for it in terms of features and extensions, and I don’t really want to have two different type checkers. At the moment I’m experimenting with mainly using pyright for interactive checks in my editor, and using mypy for pre-commit/CI checks.

The overlapping feature sets can be kind of annoying though. For example, for the potential unbound variable error above, the latest version of pyright does warn you. It also has built-in exhaustiveness checking without needing the assert_never technique. However, in one case it wasn’t working for me, until I finally tracked down the issue — mypy was able to handle this code and correctly deduce the base class of my enum, but pyright wasn’t:

try:
    from enum import StrEnum
except ImportError:
    from backports.strenum import StrEnum  # type: ignore [no-redef]

I eventually found an adequate solution that keeps them both happy — but only because I’m writing this blog post and don’t want to look stupid. Normally it would be “stuff is broken, maybe it’s me, maybe it’s them, gotta move on”.

In addition, in some places pyright does not support the assert_never technique that mypy needs, and reports an error. There are other pain points if you try to use both.

There are quite a few places where you find pyright doesn’t do the same thing as mypy because pyright is more correct. Microsoft people tend to know what they are talking about when it comes to type systems. But it means you may find yourself digging through large numbers of issues closed with the “as designed” tag to find answers.

Summary

Overall, despite listing more bad things than good, I’m actually happy with the addition of mypy as a required static type checker in this project.

The disappointments I’ve listed may come from my experience and enjoyment of languages like Haskell where you really can lean on the type checker. In those languages, you find both that the rewards of static type checking are massively higher, and that the effort required to use them is massively lower. Haskell type signatures, for instance, are often not needed, and much easier to write and understand than Python’s.

Perhaps the most positive outlook is “static type checking in Python is just an advanced linter, of course it’s not actually reliable”. This can be hard to accept, though, due to the amount of work you have to do to get any real benefit above and beyond linters like flake8 and ruff that, with virtually no changes to your code or workflow, catch a lot of issues with a very low false positive rate.

In terms of tips and advice:

• You need to turn up error reporting and spend considerable effort configuring mypy, especially in larger projects.

• If you want something approaching reliability, your entire stack of libraries needs to have been designed with static types from the beginning, so you don’t have to use stubs. This means:

  • probably not much in stdlib. You’re going to need to wrap everything.

  • probably not much that was created more than 5 years ago.

Maybe this works well for MegaCorps with an army of developers and a very large code base that they have to get under control somehow. I think for many projects, you are going to be happy with static type checking in Python only if you can resign yourself to a very low level of reliability, and are mostly leaning on other techniques for correctness, like an extensive test suite.

Links

• Discussion of this post on Lobsters