Koan 11: The Flowing River (Part 2)

Understanding how Python's list comprehensions work under the hood

Last week we discovered a new tool to inspect Python programs. This week, we shall use it to understand what is happening inside list comprehensions between Python 3.10 and Python 3.12.

A Gentle Reminder

The CPython virtual machine is a stack-based machine. This means it performs most of its operations by pushing and popping values onto a data structure called the stack.

dis.dis disassembles Python code into the individual bytecode instructions used by the interpreter to operate on the stack. For example:

• Instructions like LOAD_FAST or LOAD_CONST push values onto the stack.

• Instructions like BINARY_ADD or CALL_FUNCTION pop values from the stack to perform an operation and then push the result back.

This allows us to infer the state of the stack at each step.

Part 3: Python 3.10: The Old Way

Let’s start with a simple example: [x for x in range(5)]

When we observe the output, we can see two sections. The first refers to the outer code (or main program), but what about the second? This is actually a new hidden function that Python creates for the list comprehension. Let’s break down exactly what it does:

The Outer Code (Main Program)

This section shows what the main program does.

• 0 LOAD_CONST 0 (<code object <listcomp>...>): Python first loads the compiled code for the list comprehension itself. It's a separate, self-contained piece of code.

• 2 LOAD_CONST 1 ('<listcomp>'): Loads the name <listcomp> to identify the function.

• 4 MAKE_FUNCTION 0: This is the crucial step. This instruction creates a new “hidden” function object from the code object loaded in the first step.

• 6 LOAD_NAME 0 (range): Loads the built-in range function.

• 8 LOAD_CONST 2 (5): Loads the constant value 5.

• 10 CALL_FUNCTION 1: Calls the range function with the argument 5 to create a range object.

• 12 GET_ITER: Gets an iterator from the range(5) object.

• 14 CALL_FUNCTION 1: This calls the list comprehension function that was created earlier. The iterator from range(5) is passed to it as an argument.

• 16 RETURN_VALUE: The function returns None to indicate completion.

The Inner Code (The List Comprehension Function)

This is the actual "hidden" function that was created by Python to build the list. It's a separate, compiled block of code.

• 0 BUILD_LIST 0: Creates an empty list on the stack to hold the results of the comprehension.

• 2 LOAD_FAST 0 (.0): Loads the iterator that was passed into this function from the outer code. It's stored in a local variable named .0.

• >> 4 FOR_ITER 4 (to 14): This is the start of the loop. It iterates over the loaded iterator. If there are no more items, it jumps to instruction 14 to exit the loop.

• 6 STORE_FAST 1 (x): The FOR_ITER instruction gets the next item from the iterator and stores it in a local variable named x.

• 8 LOAD_FAST 1 (x): Loads the value of x back onto the stack.

• 10 LIST_APPEND 2: Appends the value on the top of the stack (which is x) to the list that was created in the first step. The 2 refers to the number of items it has to inspect on the stack.

• 12 JUMP_ABSOLUTE 2 (to 4): Jumps back to the top of the loop (FOR_ITER) to get the next item.

• >> 14 RETURN_VALUE: Once the loop is finished, this instruction returns the final list back to the outer code.

Now we can begin to understand why the code in the original Koan might have failed. We’re getting a NameError because the list comprehension is being run within a hidden function, and creating it’s own scope which doesn’t know about the river variable.

Part 4: Python 3.11: The New Way

But why don’t we get the NameError in Python 3.12?

Because the entire process happens within the main code block, without creating a hidden function. The process is:

• 2 PUSH_NULL: A NULL value is pushed to the stack. This is part of the new calling convention for functions in Python 3.11+.

• 4 LOAD_NAME 0 (range) and 6 LOAD_CONST 0 (5): The range function and the integer 3 are loaded onto the stack.

• 8 CALL 1: The range function is called with one argument, 5, returning a range(5) object.

• 16 GET_ITER: An iterator is created from the range(5) object.

• 18 LOAD_FAST_AND_CLEAR 0 (x): This is a key instruction for the new approach. It efficiently handles the "isolation" of the loop variable x. If a variable named x already exists in this scope, its value is saved to the stack, and the local variable x is set to NULL. This "pushes" the clashing local variable out of the way.

• 20 SWAP 2 and 22 BUILD_LIST 0: The BUILD_LIST instruction creates a new empty list. The SWAP instructions are used to organize the stack, ensuring the empty list and the iterator are in the correct positions for the loop to begin.

• 26 FOR_ITER 4 (to 38): The start of the loop. It gets the next item from the iterator and pushes it to the stack. If there are no more items, the loop ends, and execution jumps to line 38.

• 30 STORE_FAST 0 (x): The value from the iterator is stored into the local variable x. This is the loop variable.

• 32 LOAD_FAST 0 (x) and 34 LIST_APPEND 2: The value of x is loaded and then appended directly to the list.

• 36 JUMP_BACKWARD 6 (to 26): Jumps back to the top of the loop to get the next item.

• 38 END_FOR: The loop is finished.

• 40 SWAP 2 and 42 STORE_FAST 0 (x): This is the "pop" part of the stack manipulation. The SWAP and STORE_FAST instructions restore the original value of x that was saved in step 18.

• The remaining instructions handle printing the result and returning from the function.

This disassembly clearly shows that there is no separate code block for the list comprehension. The entire process, from creating the iterator to building the list and appending items, happens in one continuous flow of instructions.

The use of LOAD_FAST_AND_CLEAR and SWAP instructions handles the necessary variable isolation without the overhead of creating an entirely new function frame. This change, known as Inlined Comprehensions was introduced in Python 3.12 with PEP 709.

Part 5: The Two List Comprehensions

Now we can see exactly what happened in our original Koan. In Python 3.10:

• exec() does not have access to the local scope of the surrounding code.

• In the traceback, you can see a separate <listcomp> frame, which acts as a barrier.

• The exec("river*single_drop") statement tries to find the variable river in its own scope (which is the <listcomp> function's scope) and then the global scope. It cannot access river from the lambda's scope, leading to the NameError.

In Python 3.12 and later, where the list comprehension is inlined:

• The list comprehension runs in the same frame as the surrounding lambda function.

• exec() can now access the local variables of the surrounding scope, including the variable river.

• The code runs successfully, and since exec() returns None and the list comprehension produces a list containing None.

Part 6: Making it work on Python 3.10

There is in fact a way to stop the error from occurring in Python 3.10. the exec function accepts an optional dictionary of local and or global variables. So we can explicitly provide the variables missing in the list comprehension scope. And we can use the trick we learnt in Koan 5 to define and run the lambda in the one statement:

Drinking the water

Just as the water in one cup flowed through a sieve, and the other was sourced directly from the river; the difference between the two Python outputs is due to a change in how list comprehensions are implemented under the hood by CPython.

• In Python 3.10 list comprehension act as a separate, nested function.

• In Python 3.12 and later, the list comprehension is inlined.

While the two cups of water may appear identical, the methods used to obtain them are profoundly different.