Having only started really delving deeply into Python and regularly using it in the last 12 months, I am constantly discovering the wealth of language features available that make Python such a joy to use. One of these little gems is list comprehensions, this post serves as a quick primer on list comprehensions, what they are, how to use them, benefits, and of course when not to use them.
What are list comprehensions?
List comprehensions in Python are an alternate way to build a list from another collection, they are most commonly used where some form of an operation on each element of the existing collection is required. This provides a much more clean and concise solution than a traditional Python for loop.
This feature was added in Python version 2 all the back in July 2000 as part of PEP202.
So…How do I write one?
List comprehensions consist of expression, followed by the loop syntax and optionally an if or if/else condition all wrapped up inside a set of square brackets. Assigning the list comprehension to a variable will give the variable the value of the list resulting from the list comprehension.
One thing to take note of is that there is a subtle difference with regards to using if and if_else conditions in a list comprehension when just an if condition is going to be used it goes at the end of the loop. However when it is an if_else then it goes between the expression and the loop.
Common uses & examples
List comprehensions are most often used to replace a simple for loop, where the only purpose of the loop is to iterate over a collection of items, perform some operation or filtering condition on each element and add it to another (resulting) collection.
Perhaps we are performing some mathematical function on a list of numeric elements, some manipulation to string elements, or filtering out some elements based on a particular condition.
Below we have an example of a Pig Latin function firstly using a traditional loop and then converting it to use a list comprehension.
Another example is implementing a function to filter only the even numbers in a list. Again firstly using a traditional loop, then refactored to use a list comprehension.
Loop vs List Comprehension
List comprehensions are marginally more performant than their equivalent for loop counterparts. As can be seen in the byte code dumps below, this is due to the overhead of the
for loop setting up the loop and the searching of the resulting list when calling the append method on it. Additionally, the function using the traditional for loop has some overhead of adding the loop into the stack and popping it when exiting from the loop.
Given the only minor difference in runtime performance and resource utilisation, I would not replace a traditional loop with a list comprehension for these reasons alone. The real benefit that is provided comes from a readability and brevity perspective.
Bytecode of a list comprehension:
Here we can see that when compiled to bytecode, the list comprehension is undertaking the following actions.
- Build a list (for the resulting list).
- Create’s a for loop for byte code offsets 12 through to 18 to use.
- Create and store our variable x.
- Load the constant 2 which is for our pow operation.
- Loop through the list argument and perform a BINARY_POWER operation of x ** 2.
- Return the result list created at offset 0.
Bytecode of traditional loop:
The below is the bytecode output by a traditional for loop used to manipulate the elements in an existing list, manually creating an output list and appending to it.
- Build the output list and store it.
- Setup the for loop.
- Load the lst argument into the loop.
- Create and store our x variable.
- Load our output list.
- Load the append method.
- Load our x variable and constant 2.
- Perform binary power operation on x and 2.
- Call the append method and add the result of x ** 2 to the output list.
- Pop top item from the stack.
- Pop the for loop from the stack.
- Load our output list and return it.
Note steps 5–10 are being executed inside the for loop so will be repeated for every element in the lst argument.https://gist.github.com/lachlan-eagling/cbd93a7b23f3b134dd6e220e736b3f5b
When performing simple operations on an existing lists elements, a list comprehension provides greater readability and brevity over a traditional for loop. Caution should be given however to cases where any complex expressions or conditions are involved and a traditional loop should be considered instead.
If there is any ambiguity as to the intended logic or purpose of a list comprehension, refactoring to improve the clarity should be considered and if potentially a comment added to clarify the intention and/or logic.
When should I not use a list comprehension?
List comprehensions should really only be used when we need to transform the elements from one list to another and if the logic of the operation is simple. In the event of complex logic or conditions on the transformation or if we don’t need to output another list based on the operation performed a normal loop should be used.
An example of this (perhaps a somewhat contrived one) below is a Fizz buzz function.
In this example, it is clear that a traditional loop and if/elif condition is the preferable option from a readability and clarity perspective.
As with almost anything in the programming world, list comprehensions have distinct benefits and drawbacks depending on the required solution.
List comprehensions are NOT a one size fits all replacement for traditional loops in Python and careful thought should be given to whether or not they would be a good fit for the given job.
Please if there is any feedback, suggested improvements or corrections please reach out to me in the comments or on Twitter.