Python List Comprehension Is Not Just Syntactic Sugar

I guess you must find out there are too many articles telling us to use the list comprehension rather than a for-loop in Python. I have seen too many. However, I'm kind of surprised that there was almost no article explaining why.

People like me won't be simply convinced by the reasons like "Pythonic" or "readability". Instead, these "reasons" actually give Python newbies a wrong impression. That is, the Python list comprehension is just a syntactic sugar.

In fact, the list comprehension is one of the great optimisations in Python. In this article, let's have a deep dive into the mechanisms under the hood. You will get answers to the following questions.

• What is list comprehension in Python?
• Why its performance is better than a for-loop generally?
• When we should NOT use list comprehension?

1. A Simple Performance Comparison

Let's start from the basic part, which is writing a simple program with for-loop and list comprehension respectively. Then, we can compare the performance.

factor = 2
results = []

for i in range(100):
    results.append(i * factor)

The code above defines a for-loop that will be executed 100 times. The range(100) function generates 100 integers and each of them will be multiplied with a factor. The factor that was defined earlier, is 2.

Now, let's have a look at the list comprehension version.

factor = 2
results = [i * factor for i in range(100)]

In this example, it's absolutely much easier and more readable. Let's run these two equivalent code snippets and compare the performance.

The results showed that the performance of the list comprehension is almost 2x faster than the normal for-loop.

No surprise, this is so far many articles told you. However, this article will focus on discussing why list comprehension is faster, and what are the major differences under the hood.

In order to obtain the evidence and baseline of all the explanations for the rest of this article, let's use the Python built-in dis module to get the bytecode of the above two implementations for the for-loop and list comprehension respectively.

import dis

dis.dis("""
factor = 2
results = []

for i in range(100):
    results.append(i * factor)
""")

dis.dis("""
factor = 2
results = [i * 2 for i in range(100)]
""")

The execution results are as follows. You don't have to understand the bytecode. Just need to know that the "actions" in the bytecode are called "operational code" or "opcode" for short. I'll reference them in the later section.

2. Global Variable v.s. Local Variable

The first major difference will be the scope of the variables. From the above bytecode, this refers to

• LOAD_NAME in the for-loop v.s. LOAD_FAST in the list comprehension
• STORE_NAME in the for-loop v.s. STORE_FAST in the list comprehension

To be clear, the variable i in the for-loop version is in the global scope, whereas the variable i in the list comprehension is in its local scope and only exists while it is running.

How do we verify the scopes of the variable?

It will be easier to verify this in a notebook environment such as Python, Jupyter or Google Colab. Make sure you start a fresh new session, then just run the for-loop version followed by the magic command %whos which will list all the user-defined variables in the current session.

factor = 2
results = []

for i in range(100):
    results.append(i * factor)

-------------------

%whos

It shows that the variable i is still there even after the for-loop was finished. If you run global() method now, you will be able to find the variable i in there, too.

However, if we restart the session and run the list comprehension, the variable i will not show up.

This has proven that the variable i is a local variable in the list comprehension implementation.

Why is the performance different?

When an operation has to access a variable in the global namespace, it needs to go through the list of all the objects in the global namespace. If you're curious about what is in the global namespace, just run globals() method in your session.

In my case, I started a brand new session in Google Colab. After running the for-loop, there are already 26 objects in the global namespace dictionary. Imagine that we are running a more complex Python application, the performance might be even worse.

On the other hand, what are the "local variables" like?

Unfortunately, it's not easy to show the local variables in the list comprehension while it is running. So, let me explain this using a simple function because we can add a print() method inside the function.

def my_function(x, y):
    z = x + y
    print("local variables:", locals())
    return z

my_function(1, 2)

In this example, the x, y and z are local variables. At the compile time, an array will be created to hold all the local variables, and each of the variables will be given a fixed index. Conceptually, the local variable array is like the following:

• x is at the index 0
• y is at the index 1
• z is at the index 2

Therefore, when the function (list comprehension) needs to access the local variables, it will use the index. Therefore, it is much more effective compared to searching in the global namespace.

Variable v.s. Constant

Apart from the variable i, another optimisation because of the local variables is happening, too.

Please pay attention to the variable factor. Indeed, we defined the factor variable outside the list comprehension, but it will be loaded into the list comprehension as a local constant.

In the above bytecode. The variable factor needs to be loaded as a global variable in every single loop.

However, in the list comprehension, it is safe enough for the compiler to load the value of factor as an immutable constant and keep it in the local scope. Therefore, it doesn't need to search the variable in the global namespace.

Of course, this is another factor contributing to the performance of list comprehension.

3. Generic Method v.s. Optimised Method

Another major difference comes from the append() method. Here are the steps for the two implementations.

In the for-loop version:

1. LOAD_METHOD load the append() method from the list object results.
2. LOAD_NAME load the variable i.
3. LOAD_NAME load the variable factor.
4. BINARY_MULTIPLY perform the mathematics calculation
5. CALL_METHOD call the append() method to execute it, that is, appending the result to the results list.

In the list comprehension version:

1. LOAD_FAST load the variable i from local variables.
2. LOAD_CONST load constant 2, which is the value of the variable factor.
3. BINARY_MULTIPLY perform the mathematics calculation.
4. LIST_APPEND append the result to the list that is also from the local variable.

The better performance of the list comprehension version is not simply because there is one step less, but also the mechanism under the hood.

Why LOAD_METHOD is different from LIST_APPEND?

We call the append() method from the results list. That is, results.append(...). When this is being executed, the Python runtime needs to look up the method from the List object space. This is almost equivalent to calling the dir() method for the object.

print(dir(results))

Obviously, this "searching" action will happen in every single loop.

However, in the list comprehension version, the results list is also a local variable. The mechanism is that the list will be created, and it is being constructed along with the list comprehension execution.

Let me give an analogy. Suppose we are organising small items into a big container.

The for-loop version uses the append() method is like every time we need to give the item and the box to another person, and this person will give the container back to us after the item is put into it.

The list comprehension version enables ourselves to be able to put the items into the container without help from others. Therefore, the performance is much better.

4. When We Should Not Use List Comprehension?

Of course, list comprehension should not be abused. In other words, it shouldn't be used in some scenarios even though it can be.

One of the typical scenarios is that the filter conditions are too complicated. Considering the following example, we have a list of tuples with students' names and exam marks.

students = [("Alice", 85), ("Bob", 95), ("Cindy", 100), ("David", 65), ("Eva", 70)]

Then, we want to filter certain names based on some conditions. The marks should be more than 80, and the names start with "A" or "C". Below is the for-loop implementation.

filtered_names = []

for name, score in students:
    if score > 80 and name.startswith(("A", "C")):
        filtered_names.append(name)

And here is the list comprehension implementations.

filtered_names = [name for name, score in students if score > 80 and name.startswith(("A", "C"))]

Well, the performance of list comprehension is still a little bit better than the for-loop one. However, I have to say that the list comprehension starts to have much worse readability because of the complex conditions.

Should we sacrifice the readability and debugging flexibility to get a little improvement in performance, it can be very subjective. I'll leave the decision to yourself

Tags: Data Science Machine Learning Programming Python Technology

Comment