Learn About Managed Generators

Introduction

Objectives:

• Learn about managed generators

Files Created: multitask.py, server.py

A generator or coroutine function can never execute without being driven by some other code. For example, a generator used for iteration doesn't do anything unless iteration is actually carried out using a for-loop. Similarly, a collection of coroutines won't run unless their send() method is invoked somehow.

In advanced applications of generators, it is possible to drive generators in various unusual ways. In this exercise, we look at a few examples.

Generators as tasks

If a file multitask.py, define the following code:

## multitask.py

from collections import deque

tasks = deque()
def run():
    while tasks:
        task = tasks.popleft()
        try:
            task.send(None)
            tasks.append(task)
        except StopIteration:
            print('Task done')

This code implements a tiny task scheduler that runs generator functions. Try it by running it on the following functions.

## multitask.py
...

def countdown(n):
    while n > 0:
        print('T-minus', n)
        yield
        n -= 1

def countup(n):
    x = 0
    while x < n:
        print('Up we go', x)
        yield
        x += 1

if __name__ == '__main__':
    tasks.append(countdown(10))
    tasks.append(countdown(5))
    tasks.append(countup(20))
    run()

When you run this, you should see output from all of the generators interleaved together. For example:

T-minus 10
T-minus 5
Up we go 0
T-minus 9
T-minus 4
Up we go 1
T-minus 8
T-minus 3
Up we go 2
T-minus 7
T-minus 2
Up we go 3
T-minus 6
T-minus 1
Up we go 4
T-minus 5
Task done
Up we go 5
T-minus 4
Up we go 6
T-minus 3
Up we go 7
T-minus 2
Up we go 8
T-minus 1
Up we go 9
Task done
Up we go 10
Up we go 11
Up we go 12
Up we go 13
Up we go 14
Up we go 15
Up we go 16
Up we go 17
Up we go 18
Up we go 19
Task done

That's interesting, but not especially compelling. Move on to the next example.

Generators as Tasks Serving Network Connections

In the file server.py and put the following code into it:

## server.py

from socket import *
from select import select
from collections import deque

tasks = deque()
recv_wait = {}   ##  sock -> task
send_wait = {}   ##  sock -> task

def run():
    while any([tasks, recv_wait, send_wait]):
        while not tasks:
            can_recv, can_send, _ = select(recv_wait, send_wait, [])
            for s in can_recv:
                tasks.append(recv_wait.pop(s))
            for s in can_send:
                tasks.append(send_wait.pop(s))
        task = tasks.popleft()
        try:
            reason, resource = task.send(None)
            if reason == 'recv':
                recv_wait[resource] = task
            elif reason == 'send':
                send_wait[resource] = task
            else:
                raise RuntimeError('Unknown reason %r' % reason)
        except StopIteration:
            print('Task done')

This code is a slightly more complicated version of the task scheduler in part (a). It will require a bit of study, but the idea is that not only will each task yield, it will indicate a reason for doing so (receiving or sending). Depending on the reason, the task will move over to a waiting area. The scheduler then runs any available tasks or waits for I/O events to occur when nothing is left to do.

It's all a bit tricky perhaps, but add the following code which implements a simple echo server:

## server.py
...

def tcp_server(address, handler):
    sock = socket(AF_INET, SOCK_STREAM)
    sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
    sock.bind(address)
    sock.listen(5)
    while True:
        yield 'recv', sock
        client, addr = sock.accept()
        tasks.append(handler(client, addr))

def echo_handler(client, address):
    print('Connection from', address)
    while True:
        yield 'recv', client
        data = client.recv(1000)
        if not data:
            break
        yield 'send', client
        client.send(b'GOT:' + data)
    print('Connection closed')

if __name__ == '__main__':
    tasks.append(tcp_server(('',25000), echo_handler))
    run()

Run this server in its own terminal window. In another terminal, connect to it using a command such as telnet or nc. For example:

nc localhost 25000
Hello
Got: Hello
World
Got: World

If you don't have access to nc or telnet you can also use Python itself:

python3 -m telnetlib localhost 25000
Hello
Got: Hello
World
Got: World

If it's working, you should see output being echoed back to you. Not only that, if you connect multiple clients, they'll all operate concurrently.

This tricky use of generators is not something that you would likely have to code directly. However, they are used in certain advanced packages such as asyncio that was added to the standard library in Python 3.4.

Summary

Congratulations! You have completed the Learn About Managed Generators lab. You can practice more labs in LabEx to improve your skills.