Organizing Larger Python Programs

Introduction

If writing a larger program, you don't really want to organize it as a large of collection of standalone files at the top level. This section introduces the concept of a package.

Skills Graph

Modules

Any Python source file is a module.

## foo.py
def grok(a):
    ...
def spam(b):
    ...

An import statement loads and executes a module.

## program.py
import foo

a = foo.grok(2)
b = foo.spam('Hello')
...

Packages vs Modules

For larger collections of code, it is common to organize modules into a package.

## From this
pcost.py
report.py
fileparse.py

## To this
porty/
    __init__.py
    pcost.py
    report.py
    fileparse.py

You pick a name and make a top-level directory. porty in the example above (clearly picking this name is the most important first step).

Add an __init__.py file to the directory. It may be empty.

Put your source files into the directory.

Using a Package

A package serves as a namespace for imports.

This means that there are now multilevel imports.

import porty.report
port = porty.report.read_portfolio('portfolio.csv')

There are other variations of import statements.

from porty import report
port = report.read_portfolio('portfolio.csv')

from porty.report import read_portfolio
port = read_portfolio('portfolio.csv')

Two problems

There are two main problems with this approach.

• imports between files in the same package break.
• Main scripts placed inside the package break.

So, basically everything breaks. But, other than that, it works.

Problem: Imports

Imports between files in the same package must now include the package name in the import. Remember the structure.

porty/
    __init__.py
    pcost.py
    report.py
    fileparse.py

Modified import example.

from porty import fileparse

def read_portfolio(filename):
    return fileparse.parse_csv(...)

All imports are absolute, not relative.

import fileparse    ## BREAKS. fileparse not found

...

Relative Imports

Instead of directly using the package name, you can use . to refer to the current package.

from . import fileparse

def read_portfolio(filename):
    return fileparse.parse_csv(...)

Syntax:

from . import modname

This makes it easy to rename the package.

Problem: Main Scripts

Running a package submodule as a main script breaks.

$ python porty/pcost.py ## BREAKS
...

Reason: You are running Python on a single file and Python doesn't see the rest of the package structure correctly (sys.path is wrong).

All imports break. To fix this, you need to run your program in a different way, using the -m option.

$ python -m porty.pcost ## WORKS
...

__init__.py files

The primary purpose of these files is to stitch modules together.

Example: consolidating functions

## porty/__init__.py
from .pcost import portfolio_cost
from .report import portfolio_report

This makes names appear at the top-level when importing.

from porty import portfolio_cost
portfolio_cost('portfolio.csv')

Instead of using the multilevel imports.

from porty import pcost
pcost.portfolio_cost('portfolio.csv')

Another solution for scripts

As noted, you now need to use -m package.module to run scripts within your package.

$ python3 -m porty.pcost portfolio.csv

There is another alternative: Write a new top-level script.

#!/usr/bin/env python3
## pcost.py
import porty.pcost
import sys
porty.pcost.main(sys.argv)

This script lives outside the package. For example, looking at the directory structure:

pcost.py       ## top-level-script
porty/         ## package directory
    __init__.py
    pcost.py
    ...

Application Structure

Code organization and file structure is key to the maintainability of an application.

There is no "one-size fits all" approach for Python. However, one structure that works for a lot of problems is something like this.

porty-app/
  README.txt
  script.py         ## SCRIPT
  porty/
    ## LIBRARY CODE
    __init__.py
    pcost.py
    report.py
    fileparse.py

The top-level porty-app is a container for everything else--documentation, top-level scripts, examples, etc.

Again, top-level scripts (if any) need to exist outside the code package. One level up.

#!/usr/bin/env python3
## porty-app/script.py
import sys
import porty

porty.report.main(sys.argv)

At this point, you have a directory with several programs:

pcost.py          ## computes portfolio cost
report.py         ## Makes a report
ticker.py         ## Produce a real-time stock ticker

There are a variety of supporting modules with other functionality:

stock.py          ## Stock class
portfolio.py      ## Portfolio class
fileparse.py      ## CSV parsing
tableformat.py    ## Formatted tables
follow.py         ## Follow a log file
typedproperty.py  ## Typed class properties

In this exercise, we're going to clean up the code and put it into a common package.

Exercise 9.1: Making a simple package

Make a directory called porty/ and put all of the above Python files into it. Additionally create an empty __init__.py file and put it in the directory. You should have a directory of files like this:

porty/
    __init__.py
    fileparse.py
    follow.py
    pcost.py
    portfolio.py
    report.py
    stock.py
    tableformat.py
    ticker.py
    typedproperty.py

Remove the file __pycache__ that's sitting in your directory. This contains pre-compiled Python modules from before. We want to start fresh.

Try importing some of package modules:

>>> import porty.report
>>> import porty.pcost
>>> import porty.ticker

If these imports fail, go into the appropriate file and fix the module imports to include a package-relative import. For example, a statement such as import fileparse might change to the following:

## report.py
from . import fileparse
...

If you have a statement such as from fileparse import parse_csv, change the code to the following:

## report.py
from .fileparse import parse_csv
...

Exercise 9.2: Making an application directory

Putting all of your code into a "package" isn't often enough for an application. Sometimes there are supporting files, documentation, scripts, and other things. These files need to exist OUTSIDE of the porty/ directory you made above.

Create a new directory called porty-app. Move the porty directory you created in Exercise 9.1 into that directory. Copy the portfolio.csv and prices.csv test files into this directory. Additionally create a README.txt file with some information about yourself. Your code should now be organized as follows:

porty-app/
    portfolio.csv
    prices.csv
    README.txt
    porty/
        __init__.py
        fileparse.py
        follow.py
        pcost.py
        portfolio.py
        report.py
        stock.py
        tableformat.py
        ticker.py
        typedproperty.py

To run your code, you need to make sure you are working in the top-level porty-app/ directory. For example, from the terminal:

$ cd porty-app
$ python3
>>> import porty.report
>>>

Try running some of your prior scripts as a main program:

$ cd porty-app
$ python3 -m porty.report portfolio.csv prices.csv txt
      Name     Shares      Price     Change
---------- ---------- ---------- ----------
        AA        100       9.22     -22.98
       IBM         50     106.28      15.18
       CAT        150      35.46     -47.98
      MSFT        200      20.89     -30.34
        GE         95      13.48     -26.89
      MSFT         50      20.89     -44.21
       IBM        100     106.28      35.84

$

Exercise 9.3: Top-level Scripts

Using the python -m command is often a bit weird. You may want to write a top level script that simply deals with the oddities of packages. Create a script print-report.py that produces the above report:

#!/usr/bin/env python3
## print-report.py
import sys
from porty.report import main
main(sys.argv)

Put this script in the top-level porty-app/ directory. Make sure you can run it in that location:

$ cd porty-app
$ python3 print-report.py portfolio.csv prices.csv txt
      Name     Shares      Price     Change
---------- ---------- ---------- ----------
        AA        100       9.22     -22.98
       IBM         50     106.28      15.18
       CAT        150      35.46     -47.98
      MSFT        200      20.89     -30.34
        GE         95      13.48     -26.89
      MSFT         50      20.89     -44.21
       IBM        100     106.28      35.84

$

Your final code should now be structured something like this:

porty-app/
    portfolio.csv
    prices.csv
    print-report.py
    README.txt
    porty/
        __init__.py
        fileparse.py
        follow.py
        pcost.py
        portfolio.py
        report.py
        stock.py
        tableformat.py
        ticker.py
        typedproperty.py

Summary

Congratulations! You have completed the Packages lab. You can practice more labs in LabEx to improve your skills.