Early Stopping for Machine Learning

Introduction

In this project, you will learn how to implement the early stopping technique in machine learning models. Early stopping is a powerful method to prevent overfitting and improve the performance of your models.

🎯 Tasks

In this project, you will learn:

• Understand the concept of early stopping and its main steps
• Implement the early stopping function to determine the optimal stopping epoch
• Test the early stopping function on a sample dataset

🏆 Achievements

After completing this project, you will be able to:

• Split a dataset into training and validation sets
• Monitor the model's performance on the validation set during training
• Define a stopping criterion based on the validation set loss
• Use the early stopping function to optimize your model's training process

Skills Graph

Understand the Early Stopping Concept and Implement the Function

In this step, you will first learn about the concept of early stopping and its main steps.

The basic idea behind early stopping is to compute the model's performance on a validation set during training. When the model's performance on the validation set starts to decrease, training is stopped to avoid overfitting. The main steps are as follows:

1. Split the original training dataset into a training set and a validation set.
2. Train the model only on the training set and compute the model's error on the validation set at the end of each epoch.
3. Compare the model's error on the validation set with the training history. Stop training when the comparison meets the stopping criterion.
4. Use the parameters from the last iteration as the final parameters for the model.

There are many different stopping criteria, and they can be quite flexible. One commonly used criterion is to monitor the loss value on the validation set. When the loss value has not been further optimized for n consecutive epochs (always greater than min loss), training is stopped.

Now, you will implement the early_stop function in the early_stop.py file.

The function checks the loss values epoch by epoch. If the loss doesn't improve (decrease) for a number of epochs equal to patience, the training is recommended to be stopped.

Here's the code for the early_stop function:

def early_stop(loss: List[float], patience: int) -> Tuple[int, float]:
    """
    Determines the epoch at which training should stop based on the provided loss values and patience.

    The function checks the loss values epoch by epoch. If the loss doesn't improve (decrease) for a
    number of epochs equal to `patience`, the training is recommended to be stopped.

    Parameters:
    - loss (List[float]): A list of loss values, typically in the order they were recorded during training.
    - patience (int): The number of epochs with no improvement on loss after which training should be stopped.

    Returns:
    - Tuple[int, float]: A tuple containing two values:
        1. The epoch number at which training should be stopped (1-indexed).
        2. The minimum loss value recorded up to that point.
    """

    min_loss = np.Inf
    max_patience = 0
    stop_epoch = 0
    for epoch, current_loss in enumerate(loss):
        if current_loss < min_loss:
            min_loss = current_loss
            stop_epoch = epoch
            max_patience = 0
        else:
            max_patience += 1
        if max_patience == patience:
            break
    stop_epoch += 1
    return stop_epoch, min_loss

In the early_stop function, you implement the logic to determine the epoch at which training should be stopped based on the provided loss values and the patience parameter.

The function should return a tuple containing two values:

1. The epoch number at which training should be stopped (1-indexed).
2. The minimum loss value recorded up to that point.

Test the Early Stopping Function

In this step, you will test the early_stop function by running the early_stop.py file.

Add the following code in the early_stop.py file:

if __name__ == "__main__":
    loss = [
        1.11,
        1.01,
        0.99,
        0.89,
        0.77,
        0.69,
        0.57,
        0.44,
        0.51,
        0.43,
        0.55,
        0.61,
        0.77,
        0.89,
        0.78,
    ]
    patience = 3
    stop_epoch, min_loss = early_stop(loss, patience)
    print(f"{stop_epoch=}, {min_loss=}")

Then, run the script from the terminal:

python early_stop.py

The output should be:

stop_epoch = 10, loss = 0.43

This means that the training should be stopped at epoch 10, and the minimum loss value recorded up to that point is 0.43.

Congratulations! You have successfully implemented the early stopping function. You can now use this function in your machine learning projects to prevent overfitting and improve the performance of your models.

Summary

Congratulations! You have completed this project. You can practice more labs in LabEx to improve your skills.