Understanding iteration in Python

Iteration is a fundamental concept in programming where an operation or set of instructions is repeated multiple times. Many solutions to programming problems involve this repetition, either performing the same operation repeatedly or applying it to each element in a sequence. For example, if you wanted to get the square of each number in a list of numbers, you would use iteration to go through the list and multiply each number by itself. Another example is the ls command available on Linux and MacOS terminals, which lists all the files in a given directory.

Prerequisites

A beginner+ level of understanding of the Python language. I will be using f-strings. If not familiar with them, acquaint yourself here.

Iterables and iterators

The main idea of the Iterator pattern is to extract the traversal behavior of a collection into a separate object called an iterator - refactoring.guru

The iterable stores the items, the iterator provides the mechanics of getting items from the iterable. This allows us to fetch data without putting the entire collection in memory. This separation of concerns (storage vs. access) enables lazy evaluation, which can be crucial for memory efficiency when dealing with large datasets or infinite sequences. Another benefit of this separation is that we can perform multiple independent traversals of the same collection simultaneously. An iterable can also have different iterators, implementing different traversal methods for the same iterable (e.g., forward, backward, filtered).

Iterable: An object that implements the __iter__ method. The __iter__ method instantiates an iterator. When Python wants to iterate on an iterable, it calls iter(iterable), which in turn calls the iterable's __iter__ method to obtain an iterator. Although there is a fall-back mechanism where Python checks if an object implements __getitem__ and creates an iterator that fetches items by index. However, if you want your object to be iterated over, implement __iter__ since explicit is better than implicit. Examples of iterables in the standard library are list, set, dict, str.
Iterator: Objects that have an __iter__ method that returns self and a __next__ method that returns the next item in the series. If there is no next item, raise StopIteration exception. Examples of iterators in Python are enumerate, zip.

Ways to perform iterations in Python

Typically, we perform iteration until a specific condition is met. The condition could be exhausting the elements of a sequence, or a change in some state.

1. `for` loop

To iterate over items in an iterable, we can use the for..in syntax which gives us access to items of an iterable.

Consider this example where we want to print the list of names and greetings.

names = ['John', 'Johnte', 'Kamum', 'Kamau', 'Jane']
 
# greet each of them
for name in names:
    print(f'Hi {name}')

If we would also like to access elements by index, we can incorporate the range function like so for ... in range(start, stop, step). Python's range() function generates a sequence of numbers from start to stop-1, incremented by step. If start is not provided, Python defaults this to 0. If step is not provided, Python defaults to incrementing numbers by 1.

Let's print the names together with the indices

names = ['John', 'Johnte', 'Kamum', 'Kamau', 'Jane']
 
for index in range(len(names)):
    # adding 1 to index so as not to offend John by calling them number 0
    print(f'Hi {names[index]} you are number {index + 1}')

We can also use enumerate to get a tuple of index and element. Let's reimplement the previous example.

names = ['John', 'Johnte', 'Kamum', 'Kamau', 'Jane']
 
for index, name in enumerate(names):
    print(f'Hi {name} you are number {index + 1}')

2. `while` loop

while condition:
    # do something

While condition evaluates to True or truthy, execute the indented code.

Let's count up to 5.

count = 1
 
while count <=5:
    print(f'Count is {count}')
    count += 1

3. `for...in...else`

We can also add else to the end of a for-loop, and this is run after the loop gets to the end (did not error out or encounter a break or a return statement).

Look at this example that searches for a name in a list of names. If we find the name, we print a message saying we found the name. If we traverse the entire list of names and don't find our search term, we print a message informing the user that the given query was not found. 'Barrack' is not in the list of names, so let's use so as to ensure the code in the else clause runs.

def search(query):
    names = ['John', 'Johnte', 'Kamum', 'Kamau', 'Jane']
 
    for name in names:
        if name == query:
            print('Found the name')
            break
    else:
        print(f'Name {query} was not found')
 
search('Barrack')

4. Comprehensions

List

Introduced by PEP 202, A list comprehension creates a new list and can be used in place of a for-loop.

The syntax of a list comprehension follows the pattern:

expression: defines each element in the new list
item: variable representing each element in the iterable
iterable: source sequence
if condition: optional filter

[expression for item in iterable if condition]

Consider this for-loop that counts the number of letters in each word and stores the result in a new list, counts:

words = ['The', 'quick', 'brown', 'fox', 'is', 'dead', 'bro']
counts = []
 
for word in words:
    counts.append(len(word))

We can rewrite the example as a short and sweet list comprehension:

words = ['The', 'quick', 'brown', 'fox', 'is', 'dead', 'bro']
counts = [len(word) for word in words]

Set

The syntax resembles that of a list comprehension although it creates a new set (unique elements) and uses curly braces { } instead of square brackets [ ]

Let's say we have a list storing people's favorite snacks and we would like to remove duplicates

snacks = ['chocolate', 'ice cream', 'cake', 'cake', 'ice cream']
 
unique_snacks = {snack for snack in snacks}

Dict

{ key: value  for  item  in  iterable  if condition }

Dict comprehension creates a new dictionary by applying the key and value expressions to each item in the iterable, optionally filtered by the condition.

key: Defines each key in the new dict
value: Defines each value in the new dict

Let's say we want to create a dict of word counts where key is the word and the value is the number of letters.

words = ['The', 'quick', 'brown', 'fox', 'is', 'dead', 'bro']
 
count_map = {word: len(word) for word in words}

5. Generator expressions

Introduced by PEP 289, Generator expressions resemble list comprehensions syntax-wise but use brackets ( ) instead of square brackets[ ]. Generator expressions create a generator, which can be iterated on, but are more memory-efficient than lists.

(expression for item in iterable if condition)

6. map, reduce, filter

These functions operate on iterables and help us perform an operation on elements of an iterable efficiently without writing a for-loop. map allows us to transform each element in an iterable and return the result, reduce combines all elements in an iterable into a single result by applying a given operation repeatedly to the elements, accumulating the result as it goes, filter only "picks" elements that meet a certain condition from an iterable.

map

map accepts a function, and one or more iterables, then creates an iterator that applies the provided function on each element in the iterable(s) and returns the resulting element. We can even pass inbuilt functions as we will see below.

map(function, iterable)

We do not need to import map to use it.

Let's say you have a list of words and you want to capitalize them.

words = ['The', 'quick', 'brown', 'fox', 'is', 'dead', 'bro']
capitalized = map(str.upper, words)
 
# confirm the result by printing
for word in capitalized:
    print(capitalized)

filter

filter accepts a function and an iterable and makes an iterator that returns only the items in iterable for which function returns True.

filter(function, iterable)

Let's use filter to "filter" even numbers from a list of numbers ranging from 1 to 10

numbers = range(1, 11)
 
def is_even(num):
    return num % 2 == 0
 
even_numbers = filter(is_even, numbers)

We can rewrite is_even as a lambda function

even_numbers = filter(lambda num: num % 2 ==0, numbers)

We do not need to import filter to use it.

We use the lambda keyword to create anonymous functions in python. lambda parameter: return_value; lambda being the keyword used to create an anonymous function, parameter denotes the input to the function,return_value is the expression that defines the return value.

reduce

reduce is a function available in the functools module and has to be imported.

reduce(function, iterable)

reduce Applies a function to members of an iterable cummulatively from left to right and returns a single value. function should accept two argument; an acummulated value and the next value from the iterable.

Conside this example that calculates the sum of a list of numbers. Note that Python has a builtin function called sum that does what we are trying to do. We're just having fun.

def calc_total(nums):
    total = 0
    for num in nums:
        total += num
    return total
 
numbers = range(1, 6)
total = calc_total(numbers)  # total is 15

we can rewrite calc_total using reduce

from functools import reduce
 
 
def calc_total(nums):
    return reduce(lambda x, y: x + y, nums)
 
numbers = range(1, 6)
total = calc_total(numbers)  # total is 15

Conclusion

Hopefully, we now understand python iterators, iterables, iteration, loops. We have also seen various ways we can rewrite loops or process iterables in a succint and efficient way. Happy iteration!