REPL Python programming and debugging with IPython

Posted in:

When programming in Python, I spend a large amount of time using IPython and its powerful interactive prompt, not just for some one-off calculations, but for significant chunks of actual programming and debugging. I use it especially for exploratory programming where I’m unsure of the APIs available to me, or what the state of the system will be at a particular point in the code.

While it looks like I’ve been doing this for 12 years now, I’m not sure how widespread this method of working is, as I rarely hear other people talk about it. So I thought it would be worth sharing in some detail.

If you like videos and want to see this method in action for writing a test, you could have a look at the django-functest video about writing tests interactively, or skip to the bit where I start using the REPL.

Setup

You normally need IPython installed into your current virtualenv for it to work properly:

pip install ipython

Methods

There are basically two ways I open an IPython prompt. The first is by running it directly from a terminal:

$ ipython
Python 3.9.5 (default, Jul  1 2021, 11:45:58)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

In a Django project, ./manage.py shell can also be used if you have IPython installed, with the advantage that it will properly initialise Django for you.

This works fine if you want to explore writing some “top level” code — for example, a new bit of functionality where the entry points have not been created yet. However, most code I write is not like that. Most of the time I find myself wanting to write code when I am already 10 levels of function calls down — for example:

  • I’m writing some view code in a Django application, which has a request object — an object you could not easily recreate if you started from scratch at an IPython prompt.

  • or, model layer code such as inside a save() method that is itself being called by some other code you have not written, like the Django admin or some signal.

  • or, inside a test, where the setup code has already created a whole bunch of things that are not available to you when you open IPython.

For these cases, I use the second method:

  • Find the bit of code I want to modify, explore or debug. This will often be my own code, but could equally be a 3rd party library. I’m always working in a virtualenv, so even with 3rd party libraries ,“go to definition” in my editor will take me straight to a writable copy of the code (apart from code not written in Python).

  • Insert the code for an IPython prompt and save the file:

    import IPython; IPython.embed()
    

    I have this bound to a function key in my editor.

    So the code might end up looking like this, if it was a Django view for example:

    def contact_us(request):
        if request.method == "POST":
            form = ContactUsForm(request.POST)
            if form.is_valid():
                import IPython; IPython.embed()
    
            # …
    

    I sometimes also might put the snippet inside a new if clause that I add to catch a particular condition, especially when using this for debugging.

  • Trigger the code in the appropriate way. For the above case, it would involve first running the Django development server in a terminal, then opening the web page, filling out the form and pressing submit. For a test, it would be running the specific test from a terminal. For command line apps it would be running the app directly.

  • In the terminal, I will now find myself in the IPython REPL, and I can go ahead and:

    • work out what code I need to write

    • or debug the code that I’m confused about.

Note that you can write and edit multi-line code at this REPL — it’s not quite as comfortable as an editor, but it’s OK, and has good history support. There’s much more to say about IPython and its features that I won’t write here, you can learn about it in the docs.

For those with a background in other languages, it might also be worth pointing out that a Python REPL is not a different thing from normal Python. Everything you can do in normal Python, like defining functions and classes, is possible right there in the REPL.

Once I’m done with my exploring, I can copy any useful snippets back from the REPL into my real code, using the history to scan back through what I typed.

Advantages

The advantages of this method are:

  1. You can explore APIs and objects much more easily when you actually have the object, rather than docs about the object, or what your editor’s auto-complete tools believe to be true about the object. For example, what attributes and methods are available on Django’s HttpRequest? You don’t have to ensure you’ve got correct type annotations, and hope they are complete, or make assumptions about what the values are - you’ve got the object right there, you can inspect it, with extensive and correct tab completion. You can actually call functions and see what they do.

    For example, Django’s request object typically has a user attribute which is not part of the HttpRequest definition, because of how it is added later. It’s visible in a REPL though.

  2. You can directly explore the state of the system. This can be a huge advantage for both exploratory programming and debugging.

    For debugging, pdb and similar debugging tools and environments will often provide you with “the state of the system”, and they are much better at being able to step through multiple layers of code. But I often find that the power and comfort of an IPython prompt is much nicer for exploring and finding solutions.

The feel of this kind of environment is not quite a smooth as REPL-driven programming in Lisp, but I still find it hugely enjoyable and productive. Compared to many other methods, like iterating on your code followed by manual or automated testing, it cuts the latency of the feedback loop from seconds or minutes to milliseconds, and that is huge.

Tips and gotchas

  • IPython has tons of cool features that will help you in a REPL environment, like %autoreload (thanks haki), and many other cool magics. You should spend the time getting to know them!

  • In a multi-threaded (or multi-process) environment, IPython prompts won’t play nice. Turn off multi-threading if possible, or otherwise ensure that you don’t hit that gotcha.

  • If you do get messed up in a terminal, you may need to manually find the processes to kill and do reset in your terminal.

  • With the Django development server:

    • It’s multi-threaded by default, so either ensure that you don’t hit the view code multiple times, or use --nothreading.

    • Beware of auto-reloading, which will mess you up if you are still in an IPython prompt when it kicks in. Either use --noreload or just ensure you exit IPython cleanly before doing anything that will trigger a reload.

  • Beware of environments that capture standard input/output, that will break this technique.

  • pytest captures standard input and breaks things by default. You can turn it off using -s. Also if you are using pytest-xdist you should remember to do -n0 to turn off multiple processes.

  • When using IPython.embed() there’s an annoying bug involving closures and undefined names due to Python limitations. It often shows itself when using generator expressions, but at other times too. It can often be worked around by doing:

    globals().update(locals())
    

End

That’s it, I hope you found it useful. Do you have any other tips for using this technique?

Comments §

Comments should load when you scroll to here...