Exception Handling and Logging

PythonPythonBeginner
Practice Now

This tutorial is from open-source community. Access the source code

Introduction

In this lab, you will learn how to implement exception handling in Python. You'll understand how to use the logging module for better error reporting, which is crucial for debugging and maintaining your code.

You will also practice modifying the reader.py file to handle errors gracefully instead of crashing. This hands - on experience will enhance your ability to write robust Python programs.


Skills Graph

%%%%{init: {'theme':'neutral'}}%%%% flowchart RL python(("Python")) -.-> python/ModulesandPackagesGroup(["Modules and Packages"]) python(("Python")) -.-> python/ErrorandExceptionHandlingGroup(["Error and Exception Handling"]) python(("Python")) -.-> python/ControlFlowGroup(["Control Flow"]) python/ControlFlowGroup -.-> python/conditional_statements("Conditional Statements") python/ControlFlowGroup -.-> python/for_loops("For Loops") python/ModulesandPackagesGroup -.-> python/standard_libraries("Common Standard Libraries") python/ErrorandExceptionHandlingGroup -.-> python/catching_exceptions("Catching Exceptions") python/ErrorandExceptionHandlingGroup -.-> python/raising_exceptions("Raising Exceptions") subgraph Lab Skills python/conditional_statements -.-> lab-132507{{"Exception Handling and Logging"}} python/for_loops -.-> lab-132507{{"Exception Handling and Logging"}} python/standard_libraries -.-> lab-132507{{"Exception Handling and Logging"}} python/catching_exceptions -.-> lab-132507{{"Exception Handling and Logging"}} python/raising_exceptions -.-> lab-132507{{"Exception Handling and Logging"}} end

Understanding Exceptions in Python

In this step, we're going to learn about exceptions in Python. Exceptions are an important concept in programming. They help us deal with unexpected situations that can occur while a program is running. We'll also figure out why the current code crashes when it tries to process invalid data. Understanding this will help you write more robust and reliable Python programs.

What are Exceptions?

In Python, exceptions are events that happen during the execution of a program and disrupt the normal flow of instructions. Think of it like a roadblock on a highway. When everything goes smoothly, your program follows a set path, just like a car on a clear road. But when an error occurs, Python creates an exception object. This object is like a report that contains information about what went wrong, such as the type of error and where it happened in the code.

If these exceptions are not properly handled, they cause the program to crash. When a crash occurs, Python shows a traceback message. This message is like a map that shows you the exact location in the code where the error occurred. It's very useful for debugging.

Examining the Current Code

Let's first take a look at the reader.py file structure. This file contains functions that are used to read and convert CSV data. To open the file in the editor, we need to navigate to the correct directory. We'll use the cd command in the terminal.

cd /home/labex/project

Now that we're in the right directory, let's look at the content of reader.py. This file has several important functions:

  1. convert_csv(): This function takes rows of data and uses a provided converter function to convert them. It's like a machine that takes raw materials (data rows) and turns them into a different form according to a specific recipe (the converter function).
  2. csv_as_dicts(): This function reads CSV data and turns it into a list of dictionaries. It also performs type conversion, which means it makes sure that each piece of data in the dictionary is of the correct type, like a string, an integer, or a float.
  3. read_csv_as_dicts(): This is a wrapper function. It's like a manager that calls the csv_as_dicts() function to get the job done.

Demonstrating the Problem

Let's see what happens when the code tries to process invalid data. We'll open a Python interpreter, which is like a playground where we can test our Python code interactively. To open the Python interpreter, we'll use the following command in the terminal:

python3

Once the Python interpreter is open, we'll try to read the missing.csv file. This file contains some missing or invalid data. We'll use the read_csv_as_dicts() function from the reader.py file to read the data.

from reader import read_csv_as_dicts
port = read_csv_as_dicts('missing.csv', types=[str, int, float])

When you run this code, you should see an error message like this:

Traceback (most recent call last):
  ...
ValueError: invalid literal for int() with base 10: ''

This error occurs because the code tries to convert an empty string to an integer. An empty string doesn't represent a valid integer, so Python can't do the conversion. The function crashes at the first error it encounters, and it stops processing the rest of the valid data in the file.

To exit the Python interpreter, type the following command:

exit()

Understanding the Error Flow

The error happens in the convert_csv() function, specifically in the following line:

return list(map(lambda row: converter(headers, row), rows))

The map() function applies the converter function to each row in the rows list. The converter function tries to apply the types (str, int, float) to each row. But when it encounters a row with missing data, it fails. The map() function doesn't have a built - in way to handle exceptions. So when an exception occurs, the entire process crashes.

In the next step, you'll modify the code to handle these exceptions gracefully. This means that instead of crashing, the program will be able to deal with the errors and continue processing the rest of the data.

Implementing Exception Handling

In this step, we'll focus on making your code more robust. When a program encounters bad data, it often crashes. But we can use a technique called exception handling to deal with these issues gracefully. You'll modify the reader.py file to implement this. Exception handling allows your program to continue running even when it faces unexpected data, instead of abruptly stopping.

Understanding Try-Except Blocks

Python provides a powerful way to handle exceptions using try-except blocks. Let's break down how they work.

try:
    ## Code that might cause an exception
    result = risky_operation()
except SomeExceptionType as e:
    ## Code that runs if the exception occurs
    handle_exception(e)

In the try block, you put the code that might raise an exception. An exception is an error that occurs during the execution of a program. For example, if you try to divide a number by zero, Python will raise a ZeroDivisionError exception. When an exception occurs in the try block, Python stops executing the code in the try block and jumps to the matching except block. The except block contains the code that will handle the exception. The SomeExceptionType is the type of exception you want to catch. You can catch specific types of exceptions or use a general Exception to catch all types of exceptions. The as e part allows you to access the exception object, which contains information about the error.

Modifying the Code

Now, let's apply what we've learned about try-except blocks to the convert_csv() function. Open the reader.py file in your editor.

  1. Replace the current convert_csv() function with the following code:
def convert_csv(rows, converter, header=True):
    """
    Convert a sequence of rows to an output sequence according to a conversion function.
    """
    if header:
        headers = next(rows)
    else:
        headers = []

    result = []
    for row_idx, row in enumerate(rows, start=1):
        try:
            ## Try to convert the row
            result.append(converter(headers, row))
        except Exception as e:
            ## Print a warning message for bad rows
            print(f"Row {row_idx}: Bad row: {row}")
            continue

    return result

In this new implementation:

  • We use a for loop instead of map() to process each row. This gives us more control over the processing of each row.
  • We wrap the conversion code in a try-except block. This means that if an exception occurs during the conversion of a row, the program won't crash. Instead, it will jump to the except block.
  • In the except block, we print an error message for invalid rows. This helps us identify which rows have issues.
  • After printing the error message, we use the continue statement to skip the current row and continue processing the remaining rows.

Save the file after making these changes.

Testing Your Changes

Let's test your modified code with the missing.csv file. First, open the Python interpreter by running the following command in your terminal:

python3

Once you're in the Python interpreter, run the following code:

from reader import read_csv_as_dicts
port = read_csv_as_dicts('missing.csv', types=[str, int, float])
print(f"Number of valid rows processed: {len(port)}")

When you run this code, you should see error messages for each problematic row. But the program will continue processing and return the valid rows. Here's an example of what you might see:

Row 4: Bad row: ['C', '', '53.08']
Row 7: Bad row: ['DIS', '50', 'N/A']
Row 8: Bad row: ['GE', '', '37.23']
Row 13: Bad row: ['INTC', '', '21.84']
Row 17: Bad row: ['MCD', '', '51.11']
Row 19: Bad row: ['MO', '', '70.09']
Row 22: Bad row: ['PFE', '', '26.40']
Row 26: Bad row: ['VZ', '', '42.92']
Number of valid rows processed: 20

Let's also verify that the program works correctly with valid data. Run the following code in the Python interpreter:

valid_port = read_csv_as_dicts('valid.csv', types=[str, int, float])
print(f"Number of valid rows processed: {len(valid_port)}")

You should see that all rows are processed without errors. Here's an example of the output:

Number of valid rows processed: 17

To exit the Python interpreter, run the following command:

exit()

Now your code is more robust. It can handle invalid data gracefully by skipping bad rows instead of crashing. This makes your program more reliable and user-friendly.

โœจ Check Solution and Practice

Implementing Logging

In this step, we're going to make your code better. Instead of using simple print messages, we'll use Python's logging module for proper logging. Logging is a great way to keep track of what your program is doing, especially when it comes to handling errors and understanding the flow of your code.

Understanding the Logging Module

The logging module in Python gives us a flexible way to send out log messages from our applications. It's much more powerful than just using simple print statements. Here's what it can do:

  1. Different log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL): These levels help us categorize the importance of the messages. For example, DEBUG is for detailed information that's useful during development, while CRITICAL is for serious errors that might stop the program.
  2. Configurable output format: We can decide how the log messages will look, like adding timestamps or other useful information.
  3. Messages can be directed to different outputs (console, files, etc.): We can choose to show the log messages on the console, save them to a file, or even send them to a remote server.
  4. Log filtering based on severity: We can control which messages we see based on their log level.

Adding Logging to reader.py

Now, let's change your code to use the logging module. Open the reader.py file.

First, we need to import the logging module and set up a logger for this module. Add the following code at the top of the file:

import logging

## Set up a logger for this module
logger = logging.getLogger(__name__)

The import logging statement brings in the logging module so we can use its functions. The logging.getLogger(__name__) creates a logger for this specific module. Using __name__ ensures that the logger has a unique name related to the module.

Next, we'll modify the convert_csv() function to use logging instead of print statements. Here's the updated code:

def convert_csv(rows, converter, header=True):
    """
    Convert a sequence of rows to an output sequence according to a conversion function.
    """
    if header:
        headers = next(rows)
    else:
        headers = []

    result = []
    for row_idx, row in enumerate(rows, start=1):
        try:
            ## Try to convert the row
            result.append(converter(headers, row))
        except Exception as e:
            ## Log a warning message for bad rows
            logger.warning(f"Row {row_idx}: Bad row: {row}")
            ## Log the reason at debug level
            logger.debug(f"Row {row_idx}: Reason: {str(e)}")
            continue

    return result

The main changes here are:

  • We replaced print() with logger.warning() for the error message. This way, the message is logged with the appropriate warning level, and we can control its visibility later.
  • We added a new logger.debug() message with details about the exception. This gives us more information about what went wrong, but it's only shown if the logging level is set to DEBUG or lower.
  • The str(e) converts the exception to a string, so we can display the error reason in the log message.

After making these changes, save the file.

Testing Logging

Let's test your code with logging enabled. Open the Python interpreter by running the following command in your terminal:

python3

Once you're in the Python interpreter, execute the following code:

import logging
import reader

## Configure logging level to see all messages
logging.basicConfig(level=logging.DEBUG)

port = reader.read_csv_as_dicts('missing.csv', types=[str, int, float])
print(f"Number of valid rows processed: {len(port)}")

Here, we first import the logging module and our reader module. Then, we set the logging level to DEBUG using logging.basicConfig(level=logging.DEBUG). This means we'll see all log messages, including DEBUG, INFO, WARNING, ERROR, and CRITICAL. We then call the read_csv_as_dicts function from the reader module and print the number of valid rows processed.

You should see output like this:

WARNING:reader:Row 4: Bad row: ['C', '', '53.08']
DEBUG:reader:Row 4: Reason: invalid literal for int() with base 10: ''
WARNING:reader:Row 7: Bad row: ['DIS', '50', 'N/A']
DEBUG:reader:Row 7: Reason: could not convert string to float: 'N/A'
...
Number of valid rows processed: 20

Notice that the logging module adds a prefix to each message, showing the log level (WARNING/DEBUG) and the module name.

Now, let's see what happens if we change the log level to only show warnings. Run the following code in the Python interpreter:

## Reset the logging configuration
import logging
logging.basicConfig(level=logging.WARNING)

port = reader.read_csv_as_dicts('missing.csv', types=[str, int, float])

This time, we set the logging level to WARNING using logging.basicConfig(level=logging.WARNING). Now you'll only see the WARNING messages, and the DEBUG messages will be hidden:

WARNING:reader:Row 4: Bad row: ['C', '', '53.08']
WARNING:reader:Row 7: Bad row: ['DIS', '50', 'N/A']
...

This shows the advantage of using different logging levels. We can control how much detail is shown in the logs without changing our code.

To exit the Python interpreter, run the following command:

exit()

Congratulations! You've now implemented proper exception handling and logging in your Python program. This makes your code more reliable and gives you better information when errors occur.

Summary

In this lab, you have learned several key concepts about exception handling and logging in Python. First, you grasped how exceptions occur during data processing and implemented try-except blocks to handle them gracefully. You also modified code to keep processing valid data when errors arise.

Secondly, you learned about Python's logging module and its benefits over print statements. You implemented different log levels like WARNING and DEBUG and saw how to configure logging for different levels of detail. These skills are crucial for writing robust Python applications, especially when dealing with error - prone external data, building unattended applications, or developing systems that need diagnostic information. You can now apply these techniques to your Python projects for better reliability and maintainability.