Iterate Like a Pro

Introduction

In this lab, you will learn about the fundamental concept of iteration in Python programming. Iteration enables you to efficiently process elements in sequences such as lists, tuples, and dictionaries. Mastering iteration techniques can significantly enhance your Python coding skills.

You will explore several powerful Python iteration techniques, including basic for loop iteration, sequence unpacking, using built - in functions like enumerate() and zip(), and leveraging generator expressions for better memory efficiency.

Basic Iteration and Sequence Unpacking

In this step, we'll explore basic iteration using for loops and sequence unpacking in Python. Iteration is a fundamental concept in programming, allowing you to go through each item in a sequence one by one. Sequence unpacking, on the other hand, lets you assign individual elements of a sequence to variables in a convenient way.

Loading Data from a CSV File

Let's start by loading some data from a CSV file. CSV (Comma-Separated Values) is a common file format used to store tabular data. To begin, we need to open a terminal in the WebIDE and start the Python interpreter. This will allow us to run Python code interactively.

cd ~/project
python3

Now that we're in the Python interpreter, we can execute the following Python code to read data from the portfolio.csv file. First, we import the csv module, which provides functionality for working with CSV files. Then, we open the file and create a csv.reader object to read the data. We use the next function to get the column headers, and convert the remaining data to a list. Finally, we use the pprint function from the pprint module to print the rows in a more readable format.

import csv

f = open('portfolio.csv')
f_csv = csv.reader(f)
headers = next(f_csv)    ## Get the column headers
rows = list(f_csv)       ## Convert the remaining data to a list
from pprint import pprint
pprint(rows)             ## Pretty print the rows

You should see output similar to this:

[['AA', '100', '32.20'],
 ['IBM', '50', '91.10'],
 ['CAT', '150', '83.44'],
 ['MSFT', '200', '51.23'],
 ['GE', '95', '40.37'],
 ['MSFT', '50', '65.10'],
 ['IBM', '100', '70.44']]

Basic Iteration with for Loops

The for statement in Python is used to iterate over any sequence of data, such as a list, tuple, or string. In our case, we'll use it to iterate over the rows of data we loaded from the CSV file.

for row in rows:
    print(row)

This code will go through each row in the rows list and print it. You'll see each row of data from our CSV file printed one by one.

['AA', '100', '32.20']
['IBM', '50', '91.10']
['CAT', '150', '83.44']
['MSFT', '200', '51.23']
['GE', '95', '40.37']
['MSFT', '50', '65.10']
['IBM', '100', '70.44']

Sequence Unpacking in Loops

Python allows you to unpack sequences directly in a for loop. This is very useful when you know the structure of each item in the sequence. In our case, each row in the rows list contains three elements: a name, the number of shares, and the price. We can unpack these elements directly in the for loop.

for name, shares, price in rows:
    print(name, shares, price)

This code will unpack each row into the variables name, shares, and price, and then print them. You'll see the data printed in a more readable format.

AA 100 32.20
IBM 50 91.10
CAT 150 83.44
MSFT 200 51.23
GE 95 40.37
MSFT 50 65.10
IBM 100 70.44

If you don't need some values, you can use _ as a placeholder to indicate that you don't care about those values. For example, if you only want to print the name and the price, you can use the following code:

for name, _, price in rows:
    print(name, price)

This code will ignore the second element in each row and print only the name and the price.

AA 32.20
IBM 91.10
CAT 83.44
MSFT 51.23
GE 40.37
MSFT 65.10
IBM 70.44

Extended Unpacking with the * Operator

For more advanced unpacking, you can use the * operator as a wildcard. This allows you to collect multiple elements into a list. Let's group our data by name using this technique.

from collections import defaultdict

byname = defaultdict(list)
for name, *data in rows:
    byname[name].append(data)

## Print the data for IBM
print(byname['IBM'])

## Iterate through IBM's data
for shares, price in byname['IBM']:
    print(shares, price)

In this code, we first import the defaultdict class from the collections module. A defaultdict is a dictionary that automatically creates a new value (in this case, an empty list) if the key doesn't exist. Then, we use the * operator to collect all elements except the first one into a list called data. We store this list in the byname dictionary, grouped by the name. Finally, we print the data for IBM and iterate through it to print the shares and price.

Output:

[['50', '91.10'], ['100', '70.44']]
50 91.10
100 70.44

In this example, *data collects all items except the first one into a list, which we then store in a dictionary grouped by name. This is a powerful technique for handling data with variable-length sequences.

Using enumerate() and zip() Functions

In this step, we're going to explore two incredibly useful built - in functions in Python that are essential for iteration: enumerate() and zip(). These functions can significantly simplify your code when you're working with sequences.

Counting with enumerate()

When you're iterating over a sequence, you might often need to keep track of the index or position of each item. That's where the enumerate() function comes in handy. The enumerate() function takes a sequence as input and returns pairs of (index, value) for each item in that sequence.

If you've been following along in the Python interpreter from the previous step, you can continue using it. If not, start a new session. Here's how you can set up the data if you're starting fresh:

## If you're starting a new session, reload the data first:
## import csv
## f = open('portfolio.csv')
## f_csv = csv.reader(f)
## headers = next(f_csv)
## rows = list(f_csv)

## Use enumerate to get row numbers
for rowno, row in enumerate(rows):
    print(rowno, row)

When you run the above code, the enumerate(rows) function will generate pairs of an index (starting from 0) and the corresponding row from the rows sequence. The for loop then unpacks these pairs into the variables rowno and row, and we print them out.

Output:

0 ['AA', '100', '32.20']
1 ['IBM', '50', '91.10']
2 ['CAT', '150', '83.44']
3 ['MSFT', '200', '51.23']
4 ['GE', '95', '40.37']
5 ['MSFT', '50', '65.10']
6 ['IBM', '100', '70.44']

We can make the code even more readable by combining enumerate() with unpacking. Unpacking allows us to directly assign the elements of a sequence to individual variables.

for rowno, (name, shares, price) in enumerate(rows):
    print(rowno, name, shares, price)

In this code, we're using an extra pair of parentheses around (name, shares, price) to properly unpack the row data. The enumerate(rows) still gives us the index and the row, but now we're unpacking the row into name, shares, and price variables.

Output:

0 AA 100 32.20
1 IBM 50 91.10
2 CAT 150 83.44
3 MSFT 200 51.23
4 GE 95 40.37
5 MSFT 50 65.10
6 IBM 100 70.44

Pairing Data with zip()

The zip() function is another powerful tool in Python. It's used to combine corresponding elements from multiple sequences. When you pass multiple sequences to zip(), it creates an iterator that produces tuples, where each tuple contains elements from each of the input sequences at the same position.

Let's see how we can use zip() with the headers and row data we've been working with.

## Recall the headers variable from earlier
print(headers)  ## Should show ['name', 'shares', 'price']

## Get the first row
row = rows[0]
print(row)      ## Should show ['AA', '100', '32.20']

## Use zip to pair column names with values
for col, val in zip(headers, row):
    print(col, val)

In this code, zip(headers, row) takes the headers sequence and the row sequence and pairs up their corresponding elements. The for loop then unpacks these pairs into col (for the column name from headers) and val (for the value from row), and we print them out.

Output:

['name', 'shares', 'price']
['AA', '100', '32.20']
name AA
shares 100
price 32.20

One very common use of zip() is to create dictionaries from key - value pairs. In Python, a dictionary is a collection of key - value pairs.

## Create a dictionary from headers and row values
record = dict(zip(headers, row))
print(record)

Here, zip(headers, row) creates pairs of column names and values, and the dict() function takes these pairs and turns them into a dictionary.

Output:

{'name': 'AA', 'shares': '100', 'price': '32.20'}

We can extend this idea to convert all rows in our rows sequence to dictionaries.

## Convert all rows to dictionaries
for row in rows:
    record = dict(zip(headers, row))
    print(record)

In this loop, for each row in rows, we use zip(headers, row) to create key - value pairs and then dict() to turn those pairs into a dictionary. This technique is very common in data processing applications, especially when working with CSV files where the first row contains headers.

Output:

{'name': 'AA', 'shares': '100', 'price': '32.20'}
{'name': 'IBM', 'shares': '50', 'price': '91.10'}
{'name': 'CAT', 'shares': '150', 'price': '83.44'}
{'name': 'MSFT', 'shares': '200', 'price': '51.23'}
{'name': 'GE', 'shares': '95', 'price': '40.37'}
{'name': 'MSFT', 'shares': '50', 'price': '65.10'}
{'name': 'IBM', 'shares': '100', 'price': '70.44'}

Generator Expressions and Memory Efficiency

In this step, we're going to explore generator expressions. These are incredibly useful when you're dealing with large datasets in Python. They can make your code much more memory-efficient, which is crucial when you're working with a large amount of data.

Understanding Generator Expressions

A generator expression is similar to a list comprehension, but there's a key difference. When you use a list comprehension, Python creates a list with all the results at once. This can take up a lot of memory, especially if you're working with a large dataset. On the other hand, a generator expression produces results one at a time as they're needed. This means it doesn't need to store all the results in memory at once, which can save a significant amount of memory.

Let's look at a simple example to see how this works:

## Start a new Python session if needed
## python3

## List comprehension (creates a list in memory)
nums = [1, 2, 3, 4, 5]
squares_list = [x*x for x in nums]
print(squares_list)

## Generator expression (creates a generator object)
squares_gen = (x*x for x in nums)
print(squares_gen)  ## This doesn't print the values, just the generator object

## Iterate through the generator to get values
for n in squares_gen:
    print(n)

When you run this code, you'll see the following output:

[1, 4, 9, 16, 25]
<generator object <genexpr> at 0x7f...>
1
4
9
16
25

One important thing to note about generators is that they can only be iterated over once. Once you've gone through all the values in a generator, it's exhausted, and you can't get the values again.

## Try to iterate again over the same generator
for n in squares_gen:
    print(n)  ## Nothing will be printed, as the generator is already exhausted

You can also manually get values from a generator one at a time using the next() function.

## Create a fresh generator
squares_gen = (x*x for x in nums)

## Get values one by one
print(next(squares_gen))  ## 1
print(next(squares_gen))  ## 4
print(next(squares_gen))  ## 9

When there are no more values in the generator, calling next() will raise a StopIteration exception.

Generator Functions with yield

For more complex generator logic, you can write generator functions using the yield statement. A generator function is a special type of function that uses yield to return values one at a time instead of returning a single result all at once.

def squares(nums):
    for x in nums:
        yield x*x

## Use the generator function
for n in squares(nums):
    print(n)

When you run this code, you'll see the following output:

1
4
9
16
25

Reducing Memory Usage with Generator Expressions

Now, let's see how generator expressions can save memory when working with large datasets. We'll use the CTA bus data file, which is quite large.

cd /home/labex/project
unzip ctabus.csv.zip && rm ctabus.csv.zip

First, let's try a memory-intensive approach:

import tracemalloc
tracemalloc.start()

import readrides
rows = readrides.read_rides_as_dicts('ctabus.csv')
rt22 = [row for row in rows if row['route'] == '22']
max_row = max(rt22, key=lambda row: int(row['rides']))
print(max_row)

## Check memory usage
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory usage: {current / 1024 / 1024:.2f} MB")
print(f"Peak memory usage: {peak / 1024 / 1024:.2f} MB")

Now, exit Python and restart it to compare with a generator-based approach:

exit() python3

import tracemalloc
tracemalloc.start()

import csv
f = open('ctabus.csv')
f_csv = csv.reader(f)
headers = next(f_csv)

## Use generator expressions for all operations
rows = (dict(zip(headers, row)) for row in f_csv)
rt22 = (row for row in rows if row['route'] == '22')
max_row = max(rt22, key=lambda row: int(row['rides']))
print(max_row)

## Check memory usage
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory usage: {current / 1024 / 1024:.2f} MB")
print(f"Peak memory usage: {peak / 1024 / 1024:.2f} MB")

You should notice a significant difference in memory usage between these two approaches. The generator-based approach processes the data incrementally without loading everything into memory at once, which is much more memory-efficient.

Generator Expressions with Reduction Functions

Generator expressions are particularly useful when combined with functions like sum(), min(), max(), any(), and all() that process an entire sequence and produce a single result.

Let's look at some examples:

from readport import read_portfolio
portfolio = read_portfolio('portfolio.csv')

## Calculate the total value of the portfolio
total_value = sum(s['shares']*s['price'] for s in portfolio)
print(f"Total portfolio value: {total_value}")

## Find the minimum number of shares in any holding
min_shares = min(s['shares'] for s in portfolio)
print(f"Minimum shares in any holding: {min_shares}")

## Check if any stock is IBM
has_ibm = any(s['name'] == 'IBM' for s in portfolio)
print(f"Portfolio contains IBM: {has_ibm}")

## Check if all stocks are IBM
all_ibm = all(s['name'] == 'IBM' for s in portfolio)
print(f"All stocks are IBM: {all_ibm}")

## Count IBM shares
ibm_shares = sum(s['shares'] for s in portfolio if s['name'] == 'IBM')
print(f"Total IBM shares: {ibm_shares}")

Generator expressions are also useful for string operations. Here's how to join values in a tuple:

s = ('GOOG', 100, 490.10)
## This would fail: ','.join(s)
## Use a generator expression to convert all items to strings
joined = ','.join(str(x) for x in s)
print(joined)  ## Output: 'GOOG,100,490.1'

The key advantage of using generator expressions in these examples is that no intermediate lists are created, resulting in more memory-efficient code.

Summary

In this lab, you have learned several powerful Python iteration techniques. First, you mastered basic iteration and sequence unpacking, using for loops to iterate over sequences and unpack them into individual variables. Second, you explored built - in functions like enumerate() to track indices during iteration and zip() to pair elements from different sequences.

These techniques are fundamental for efficient Python programming. They enable you to write more concise, readable, and memory - efficient code. By mastering these iteration patterns, you can handle data processing tasks more effectively in your Python projects.