Customize Iteration Using Generators

Introduction

In this lab, you will learn how to customize iteration using generators in Python. You'll also implement iterator functionality in custom classes and create generators for streaming data sources.

The structure.py file will be modified, and a new file named follow.py will be created during the experiment.

Skills Graph

Understanding Python Generators

Generators are a powerful feature in Python. They offer a simple and elegant way to create iterators. In Python, when you deal with data sequences, iterators are very useful as they allow you to loop through a series of values one by one. Regular functions typically return a single value and then stop executing. However, generators are different. They can yield a sequence of values over time, which means they can produce multiple values in a step - by - step manner.

What is a Generator?

A generator function has a similar appearance to a regular function. But the key difference lies in how it returns values. Instead of using the return statement to provide a single result, a generator function uses the yield statement. The yield statement is special. Each time it is executed, the function's state is paused, and the value that follows the yield keyword is returned to the caller. When the generator function is called again, it resumes execution right where it left off.

Let's start by creating a simple generator function. The built - in range() function in Python doesn't support fractional steps. So, we'll create a generator function that can produce a range of numbers with a fractional step.

1. First, you need to open a new Python terminal in the WebIDE. To do this, click on the "Terminal" menu and then select "New Terminal".
2. Once the terminal is open, type the following code in the terminal. This code defines a generator function and then tests it.

def frange(start, stop, step):
    current = start
    while current < stop:
        yield current
        current += step

## Test the generator with a for loop
for x in frange(0, 2, 0.25):
    print(x, end=' ')

In this code, the frange function is a generator function. It initializes a variable current with the start value. Then, as long as current is less than the stop value, it yields the current value and then increments current by the step value. The for loop then iterates over the values produced by the frange generator function and prints them.

You should see the following output:

0 0.25 0.5 0.75 1.0 1.25 1.5 1.75

The One - Time Nature of Generators

An important characteristic of generators is that they are exhaustible. This means that once you have iterated through all the values produced by a generator, it can't be used again to produce the same sequence of values. Let's demonstrate this with the following code:

## Create a generator object
f = frange(0, 2, 0.25)

## First iteration works fine
print("First iteration:")
for x in f:
    print(x, end=' ')
print("\n")

## Second iteration produces nothing
print("Second iteration:")
for x in f:
    print(x, end=' ')
print("\n")

In this code, we first create a generator object f using the frange function. The first for loop iterates over all the values produced by the generator and prints them. After the first iteration, the generator has been exhausted, which means it has already produced all the values it can. So, when we try to iterate over it again in the second for loop, it doesn't produce any new values.

Output:

First iteration:
0 0.25 0.5 0.75 1.0 1.25 1.5 1.75

Second iteration:

Notice that the second iteration didn't produce any output because the generator was already exhausted.

Creating Reusable Generators with Classes

If you need to iterate multiple times over the same sequence of values, you can wrap the generator in a class. By doing this, each time you start a new iteration, a fresh generator will be created.

class FRange:
    def __init__(self, start, stop, step):
        self.start = start
        self.stop = stop
        self.step = step

    def __iter__(self):
        n = self.start
        while n < self.stop:
            yield n
            n += self.step

## Create an instance
f = FRange(0, 2, 0.25)

## We can iterate multiple times
print("First iteration:")
for x in f:
    print(x, end=' ')
print("\n")

print("Second iteration:")
for x in f:
    print(x, end=' ')
print("\n")

In this code, we define a class FRange. The __init__ method initializes the start, stop, and step values. The __iter__ method is a special method in Python classes. It is used to create an iterator. Inside the __iter__ method, we have a generator that produces values in a similar way to the frange function we defined earlier.

When we create an instance f of the FRange class and iterate over it multiple times, each iteration calls the __iter__ method, which creates a fresh generator. So, we can get the same sequence of values multiple times.

Output:

First iteration:
0 0.25 0.5 0.75 1.0 1.25 1.5 1.75

Second iteration:
0 0.25 0.5 0.75 1.0 1.25 1.5 1.75

This time, we can iterate multiple times because the __iter__() method creates a fresh generator each time it's called.

Adding Iteration to Custom Classes

Now that you have grasped the basics of generators, we're going to use them to add iteration capabilities to custom classes. In Python, if you want to make a class iterable, you need to implement the __iter__() special method. An iterable class allows you to loop through its elements, just like you can loop through a list or a tuple. This is a powerful feature that makes your custom classes more flexible and easier to work with.

Understanding the __iter__() Method

The __iter__() method is a crucial part of making a class iterable. It should return an iterator object. An iterator is an object that can be iterated (looped) over. A simple and effective way to achieve this is by defining __iter__() as a generator function. A generator function uses the yield keyword to produce a sequence of values one at a time. Each time the yield statement is encountered, the function pauses and returns the value. The next time the iterator is called, the function resumes from where it left off.

Modifying the Structure Class

In the setup for this lab, we provided a base Structure class. Other classes, like Stock, can inherit from this Structure class. Inheritance is a way to create a new class that inherits the properties and methods of an existing class. By adding an __iter__() method to the Structure class, we can make all its subclasses iterable. This means that any class that inherits from Structure will automatically have the ability to be looped over.

1. Open the file structure.py in the WebIDE:

cd ~/project

This command changes the current working directory to the project directory where the structure.py file is located. You need to be in the correct directory to access and modify the file.

2. Look at the current implementation of the Structure class:

class Structure(metaclass=StructureMeta):
    _fields = []
    def __init__(self, *args):
        if len(args) != len(self._fields):
            raise TypeError(f'Expected {len(self._fields)} arguments')
        for name, val in zip(self._fields, args):
            setattr(self, '_'+name, val)

The Structure class has a _fields list that stores the names of the attributes. The __init__() method is the constructor of the class. It initializes the object's attributes by checking if the number of arguments passed is equal to the number of fields. If not, it raises a TypeError. Otherwise, it sets the attributes using the setattr() function.

3. Add an __iter__() method that yields each attribute value in order:

def __iter__(self):
    for name in self._fields:
        yield getattr(self, name)

This __iter__() method is a generator function. It loops through the _fields list and uses the getattr() function to get the value of each attribute. The yield keyword then returns the value one by one.

The complete structure.py file should now look like this:

class StructureMeta(type):
    def __new__(cls, name, bases, clsdict):
        fields = clsdict.get('_fields', [])
        for name in fields:
            clsdict[name] = property(lambda self, name=name: getattr(self, '_'+name))
        return super().__new__(cls, name, bases, clsdict)

class Structure(metaclass=StructureMeta):
    _fields = []
    def __init__(self, *args):
        if len(args) != len(self._fields):
            raise TypeError(f'Expected {len(self._fields)} arguments')
        for name, val in zip(self._fields, args):
            setattr(self, '_'+name, val)

    def __iter__(self):
        for name in self._fields:
            yield getattr(self, name)

This updated Structure class now has the __iter__() method, which makes it and its subclasses iterable.

4. Save the file.
   After making changes to the structure.py file, you need to save it so that the changes are applied.

5. Now let's test the iteration capability by creating a Stock instance and iterating over it:

python3 -c "from stock import Stock; s = Stock('GOOG', 100, 490.1); print('Iterating over Stock:'); [print(val) for val in s]"

This command creates an instance of the Stock class, which inherits from the Structure class. It then iterates over the instance using a list comprehension and prints each value.

You should see output like this:

Iterating over Stock:
GOOG
100
490.1

Now any class that inherits from Structure will automatically be iterable, and iteration will yield the attribute values in the order defined by the _fields list. This means that you can easily loop through the attributes of any subclass of Structure without having to write additional code for iteration.

Enhancing Classes with Iteration Capabilities

Now, we've made our Structure class and its subclasses support iteration. Iteration is a powerful concept in Python that allows you to loop through a collection of items one by one. When a class supports iteration, it becomes more flexible and can work with many built - in Python features. Let's explore how this support for iteration enables many powerful features in Python.

Leveraging Iteration for Sequence Conversions

In Python, there are built - in functions like list() and tuple(). These functions are very useful because they can take any iterable object as an input. An iterable object is something that you can loop over, like a list, a tuple, or now, our Structure class instances. Since our Structure class now supports iteration, we can easily convert instances of it to lists or tuples.

1. Let's try these operations with a Stock instance. The Stock class is a subclass of Structure. Run the following command in your terminal:

python3 -c "from stock import Stock; s = Stock('GOOG', 100, 490.1); print('As list:', list(s)); print('As tuple:', tuple(s))"

This command first imports the Stock class, creates an instance of it, and then converts this instance to a list and a tuple using the list() and tuple() functions respectively. The output will show you the instance represented as a list and a tuple:

As list: ['GOOG', 100, 490.1]
As tuple: ('GOOG', 100, 490.1)

Unpacking

Python has a very useful feature called unpacking. Unpacking allows you to take an iterable object and assign its elements to individual variables in one go. Since our Stock instance is iterable, we can use this unpacking feature on it.

python3 -c "from stock import Stock; s = Stock('GOOG', 100, 490.1); name, shares, price = s; print(f'Name: {name}, Shares: {shares}, Price: {price}')"

In this code, we create a Stock instance and then unpack its elements into three variables: name, shares, and price. Then we print these variables. The output will show the values of these variables:

Name: GOOG, Shares: 100, Price: 490.1

Adding Comparison Capabilities

When a class supports iteration, it becomes easier to implement comparison operations. Comparison operations are used to check if two objects are equal or not. Let's add an __eq__() method to our Structure class to compare instances.

1. Open the structure.py file again. The __eq__() method is a special method in Python that is called when you use the == operator to compare two objects. Add the following code to the Structure class in the structure.py file:

def __eq__(self, other):
    return isinstance(other, type(self)) and tuple(self) == tuple(other)

This method first checks if the other object is an instance of the same class as self using the isinstance() function. Then it converts both self and other to tuples and checks if these tuples are equal.

The complete structure.py file should now look like this:

class StructureMeta(type):
    def __new__(cls, name, bases, clsdict):
        fields = clsdict.get('_fields', [])
        for name in fields:
            clsdict[name] = property(lambda self, name=name: getattr(self, '_'+name))
        return super().__new__(cls, name, bases, clsdict)

class Structure(metaclass=StructureMeta):
    _fields = []
    def __init__(self, *args):
        if len(args) != len(self._fields):
            raise TypeError(f'Expected {len(self._fields)} arguments')
        for name, val in zip(self._fields, args):
            setattr(self, '_'+name, val)

    def __iter__(self):
        for name in self._fields:
            yield getattr(self, name)

    def __eq__(self, other):
        return isinstance(other, type(self)) and tuple(self) == tuple(other)

2. After adding the __eq__() method, save the structure.py file.

3. Let's test the comparison capability. Run the following command in your terminal:

python3 -c "from stock import Stock; a = Stock('GOOG', 100, 490.1); b = Stock('GOOG', 100, 490.1); c = Stock('AAPL', 200, 123.4); print(f'a == b: {a == b}'); print(f'a == c: {a == c}')"

This code creates three Stock instances: a, b, and c. Then it compares a with b and a with c using the == operator. The output will show the results of these comparisons:

a == b: True
a == c: False

4. Now, to make sure everything is working correctly, we need to run the unit tests. Unit tests are a set of code that checks if different parts of your program are working as expected. Run the following command in your terminal:

python3 teststock.py

If everything is working correctly, you should see output indicating that the tests have passed:

..
----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

By adding just two simple methods (__iter__() and __eq__()), we've significantly enhanced our Structure class with capabilities that make it more Pythonic and easier to use.

Creating a Generator for Streaming Data

In programming, generators are a powerful tool, especially when dealing with real - world problems like monitoring a streaming data source. In this section, we'll learn how to apply what we've learned about generators to such a practical scenario. We're going to create a generator that keeps an eye on a log file and gives us new lines as they're added to the file.

Setting Up the Data Source

Before we start creating the generator, we need to set up a data source. In this case, we'll use a simulation program that generates stock market data.

First, you need to open a new terminal in the WebIDE. This is where you'll run commands to start the simulation.

After opening the terminal, you'll run the stock simulation program. Here are the commands you need to enter:

cd ~/project
python3 stocksim.py

The first command cd ~/project changes the current directory to the project directory in your home directory. The second command python3 stocksim.py runs the stock simulation program. This program will generate stock market data and write it to a file named stocklog.csv in the current directory. Let this program run in the background while we work on the monitoring code.

Creating a Simple File Monitor

Now that we have our data source set up, let's create a program that monitors the stocklog.csv file. This program will display any price changes that are negative.

1. First, create a new file called follow.py in the WebIDE. To do this, you need to change the directory to the project directory using the following command in the terminal:

cd ~/project

2. Next, add the following code to the follow.py file. This code opens the stocklog.csv file, moves the file pointer to the end of the file, and then continuously checks for new lines. If a new line is found and it represents a negative price change, it prints the stock name, price, and change.

## follow.py
import os
import time

f = open('stocklog.csv')
f.seek(0, os.SEEK_END)   ## Move file pointer 0 bytes from end of file

while True:
    line = f.readline()
    if line == '':
        time.sleep(0.1)   ## Sleep briefly and retry
        continue
    fields = line.split(',')
    name = fields[0].strip('"')
    price = float(fields[1])
    change = float(fields[4])
    if change < 0:
        print('%10s %10.2f %10.2f' % (name, price, change))

3. After adding the code, save the file. Then, run the program using the following command in the terminal:

python3 follow.py

You should see output that shows stocks with negative price changes. It might look something like this:

      AAPL     148.24      -1.76
      GOOG    2498.45      -1.55

If you want to stop the program, press Ctrl+C in the terminal.

Converting to a Generator Function

While the previous code works, we can make it more reusable and modular by converting it to a generator function. A generator function is a special type of function that can be paused and resumed, and it yields values one at a time.

1. Open the follow.py file again and modify it to use a generator function. Here's the updated code:

## follow.py
import os
import time

def follow(filename):
    """
    Generator function that yields new lines in a file as they are added.
    Similar to the 'tail -f' Unix command.
    """
    f = open(filename)
    f.seek(0, os.SEEK_END)   ## Move to the end of the file

    while True:
        line = f.readline()
        if line == '':
            time.sleep(0.1)   ## Sleep briefly and retry
            continue
        yield line

## Example usage - monitor stocks with negative price changes
if __name__ == '__main__':
    for line in follow('stocklog.csv'):
        fields = line.split(',')
        name = fields[0].strip('"')
        price = float(fields[1])
        change = float(fields[4])
        if change < 0:
            print('%10s %10.2f %10.2f' % (name, price, change))

The follow function is now a generator function. It opens the file, moves to the end, and then continuously checks for new lines. When a new line is found, it yields that line.

2. Save the file and run it again using the command:

python3 follow.py

The output should be the same as before. But now, the file monitoring logic is neatly encapsulated in the follow generator function. This means we can reuse this function in other programs that need to monitor a file.

Understanding the Power of Generators

By converting our file - reading code into a generator function, we've made it much more flexible and reusable. The follow() function can be used in any program that needs to monitor a file, not just for stock data.

For example, you could use it to monitor server logs, application logs, or any other file that gets updated over time. This shows how generators are a great way to handle streaming data sources in a clean and modular way.

Summary

In this lab, you have learned how to customize iteration in Python using generators. You created simple generators with the yield statement to generate value sequences, added iteration support to custom classes by implementing the __iter__() method, leveraged iteration for sequence conversions, unpacking, and comparison, and built a practical generator for monitoring a streaming data source.

Generators are a powerful Python feature that enables you to create iterators with minimal code. They are especially useful for processing large data sets, working with streaming data, creating data pipelines, and implementing custom iteration patterns. Using generators allows you to write cleaner, more memory - efficient code that clearly conveys your intent.