Tools for rewriting Python code

Posted in:

When writing (or reviewing) code, you have better things to do than concern yourself with low-level details about coding style or other changes that are essentially mechanical in nature. Thankfully, the tooling ecosystem for doing these kind of boring changes to Python code has become much stronger in the past few years.

Below is my collection, with some alternatives and recommendations. These all go beyond being linters, which only report problems, to being able to fix your code automatically. Most of these work really well with tools like pre-commit so that by the time you come to code review, all the boring stuff is already fixed.

Formatting and coding style

  • Black is probably the most populate Python code formatter today.

    YAPF is another with a similar ethos to Black, but less popular AFAIK, and I don’t use it.

  • autopep8 doesn’t go as far as Black or YAPF - it fixes PEP8 violations but otherwise leaves your code alone. This is useful for cases where people aren’t quite ready for Black yet.

  • isort and reorder_python_imports will sort your Python imports for you.

    I personally prefer the former, isort. reorder_python_imports has a much more verbose style, resulting in many lines for imports. This is useful for reducing merge conflicts, but with the other tools listed here, I don’t find those much of a problem – if you aren’t sure which imports are still needed, include them all and let isort remove the duplicates, and autoflake remove the unneeded ones.

  • table-format makes it easy to have aligned columns in your Python source code.

  • ruff is a more recent tool that combines quite a few tools into one — isort, flake8, Black, flynt and others — and is very fast as well.

Upgrades

The following tools will do upgrades on your code:

  • pyupgrade – moves code to the most modern Python idioms.

  • flynt – rewrites older string formatting code using % to use .format and/or f-strings where possible.

  • django-upgrade and django-codemod – include various fixes for breaking changes or new features in Django.

  • setup-py-upgrade – upgrades your setup.py to a setup.cfg file.

Type hints

  • Monkeytype and pyannotate – add type hints based on instrumented test suite runs.

  • pytype – this does type checking and produces .pyi files based on inference, and also includes a merge-pyi tool that can merge .pyi files into .py files.

  • autotyping – a tool to add type hints for various cases where this can be done automatically.

    (As a comment, I’m not wild about some of these automated changes. Annotating __str__ with -> str, when it is required to be a str, seems like a failure of our static typing tools, and it adds a lot of noise.)

  • no_implicit_optional – a small tool to make some type hints more compliant with PEP 484.

Refactoring

Many IDEs/editors provide a bunch of tools to rewrite Python code (for example doing renames), often by integrating with language servers.

In VSCode, the default is Pylance which is proprietary and can only be used with VSCode. However, pyright powers most of its functionality, and is Open Source. As well as being a command line static type checker, it also functions as a language server, and it’s the one I use from Emacs at the moment.

One of the issues I find is that these is that they can be hard to use from the command line, to be able to do more automated refactoring – in fact I haven’t found a good way to do so, other than scripting things using elisp.

So here are some other tools that are designed for more stand-alone use and have some refactoring features:

Other

  • autoflake – remove unused imports.

  • Fixit – custom linting rules with automatic fixes.

  • shed – bundles together a few of the above.

Write your own

Finally, there are great libraries like libCST that will help you to manipulate Python code but without losing comments etc., so that writing your own tool to do this is no longer a massive task.

Also looking for packages that depend on LibCST, on GitHub or on libraries.io, is a great way to find more tools like this.

Have fun writing code to fix your code!

Comments §

Comments should load when you scroll to here...