Python Multiprocessing for Parallel Execution

Introduction

Python multiprocessing is a powerful tool that can significantly speed up the execution of Python programs that require high processing power. In this lab, you will learn about Python multiprocessing and how to use it to run processes in parallel. We will start with simple examples and gradually move towards more complex ones.

Skills Graph

Creating a Simple Multiprocessing Program

The first step in learning about multiprocessing in Python is to create a simple program that demonstrates how it works. In this program, we will create a function that takes a single argument and returns the square of that number. We will then use multiprocessing to run this function on multiple processes.

Please complete square.py.

import multiprocessing

def square(x):
    return x * x

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=4)
    results = pool.map(square, range(10))
    print(results)

This code creates a pool of four processes and uses the map() function to apply the square() function to each of the numbers in the range from 0 to 9. The results are then printed to the console.

Using Multiprocessing to Speed Up Processing

Now that you understand how multiprocessing works in Python, we can move on to a more complex example. In this example, we will use multiprocessing to speed up the processing of a large list of numbers.

Please complete complex_square.py.

import multiprocessing

def square(x):
    return x * x

if __name__ == '__main__':
    with multiprocessing.Pool(processes=4) as pool:
        results = pool.map(square, range(1000000))
    print(results[:10])

This code creates a pool of four processes and uses the map() function to apply the square() function to each of the numbers in the range from 0 to 999999. The with statement is used to ensure that the pool of processes is properly closed when the processing is complete. The first ten results are then printed to the console.

Inter-Process Communication

In some cases, you may need to communicate between processes when using multiprocessing in Python. In this example, we will use the multiprocessing.Queue() function to create a shared queue between processes.

Please complete multiprocessing_queue.py.

import multiprocessing

def producer(queue):
    for i in range(10):
        queue.put(i)

def consumer(queue):
    while True:
        item = queue.get()
        if item is None:
            break
        print(item)

if __name__ == '__main__':
    queue = multiprocessing.Queue()
    process_producer = multiprocessing.Process(target=producer, args=(queue,))
    process_consumer = multiprocessing.Process(target=consumer, args=(queue,))
    process_producer.start()
    process_consumer.start()
    process_producer.join()
    queue.put(None)
    process_consumer.join()

This code creates a shared queue queue using the multiprocessing.Queue() function. We then create two processes that each call a function producer() and consumer(). The producer() function puts the numbers 0 to 9 into the queue, while the consumer() function gets each item from the queue and prints it to the console.

Asynchronous Tasks with Pool.apply_async()

In addition to map(), the multiprocessing.Pool() class provides another method for running processes in parallel called apply_async(). This method allows you to submit a function call to a pool of processes and immediately continue execution without waiting for the result. Instead, you can use a callback function to retrieve the result of the function call when it is ready.

Please completeslow_square.py.

import multiprocessing
import time

def slow_square(x):
    time.sleep(1)
    return x * x

def callback(result):
    print(result)

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=4)
    results = [pool.apply_async(slow_square, (x,), callback=callback) for x in range(10)]
    for result in results:
        result.wait()
    print("All tasks completed.")

In this example, we define a function slow_square() that takes a number x, waits for one second, and then returns the square of x. We also define a callback function callback() that simply prints the result when it is available.

We then create a pool of four processes and use apply_async() to submit a function call to the pool for each number in the range from 0 to 9. We pass in the slow_square() function and the argument x using the args parameter, and also specify the callback function to be called when the result is ready.

We then loop through the list of results and call result.wait() to block the main thread until the result is ready. Finally, we print a message to indicate that all tasks are completed.

Summary

In this lab, you learned how to use Python multiprocessing to run processes in parallel. You started with a simple program that demonstrated how multiprocessing works, and then moved on to more complex examples that used multiprocessing to speed up processing and share memory between processesand perform asynchronous tasks using apply_async(). By the end of this lab, you should have a good understanding of how to use multiprocessing in your own Python programs to achieve greater processing power and efficiency,and to communicate between processes when necessary.