Validation Curves: Plotting Scores to Evaluate Models

Introduction

In machine learning, every estimator has its advantages and drawbacks. The generalization error of an estimator can be decomposed into bias, variance, and noise. The bias of an estimator is the average error for different training sets, while the variance indicates its sensitivity to varying training sets. Noise is a property of the data.

In this lab, we will explore how to use validation curves to evaluate the performance of machine learning models. Validation curves allow us to plot the influence of a single hyperparameter on the training score and the validation score, helping us determine if the model is overfitting or underfitting for different hyperparameter values.

VM Tips

After the VM startup is done, click the top left corner to switch to the Notebook tab to access Jupyter Notebook for practice.

Sometimes, you may need to wait a few seconds for Jupyter Notebook to finish loading. The validation of operations cannot be automated because of limitations in Jupyter Notebook.

If you face issues during learning, feel free to ask Labby. Provide feedback after the session, and we will promptly resolve the problem for you.

Skills Graph

%%%%{init: {'theme':'neutral'}}%%%% flowchart RL sklearn(("`Sklearn`")) -.-> sklearn/UtilitiesandDatasetsGroup(["`Utilities and Datasets`"]) sklearn(("`Sklearn`")) -.-> sklearn/CoreModelsandAlgorithmsGroup(["`Core Models and Algorithms`"]) sklearn(("`Sklearn`")) -.-> sklearn/ModelSelectionandEvaluationGroup(["`Model Selection and Evaluation`"]) ml(("`Machine Learning`")) -.-> ml/FrameworkandSoftwareGroup(["`Framework and Software`"]) sklearn/UtilitiesandDatasetsGroup -.-> sklearn/datasets("`Datasets`") sklearn/CoreModelsandAlgorithmsGroup -.-> sklearn/linear_model("`Linear Models`") sklearn/ModelSelectionandEvaluationGroup -.-> sklearn/model_selection("`Model Selection`") ml/FrameworkandSoftwareGroup -.-> ml/sklearn("`scikit-learn`") subgraph Lab Skills sklearn/datasets -.-> lab-71125{{"`Validation Curves: Plotting Scores to Evaluate Models`"}} sklearn/linear_model -.-> lab-71125{{"`Validation Curves: Plotting Scores to Evaluate Models`"}} sklearn/model_selection -.-> lab-71125{{"`Validation Curves: Plotting Scores to Evaluate Models`"}} ml/sklearn -.-> lab-71125{{"`Validation Curves: Plotting Scores to Evaluate Models`"}} end

Import the Required Libraries and Load the Data

Let's start by importing the necessary libraries and loading a dataset. In this example, we will use the Iris dataset.

import numpy as np
from sklearn.model_selection import validation_curve
from sklearn.datasets import load_iris
from sklearn.linear_model import Ridge

np.random.seed(0)
X, y = load_iris(return_X_y=True)

Shuffle the Data

To ensure randomness in our analysis, let's shuffle the order of the samples in our dataset.

indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

Plot the Validation Curve

Now, let's plot the validation curve using the validation_curve function. We will use the Ridge estimator and vary the alpha hyperparameter over a range of values.

param_range = np.logspace(-7, 3, 3)
train_scores, valid_scores = validation_curve(
    Ridge(), X, y, param_name="alpha", param_range=param_range, cv=5)

Summary

In this lab, we explored the concept of validation curves and how they can be used to evaluate machine learning models. By plotting the training score and the validation score for different hyperparameter values, we can determine if a model is overfitting or underfitting. This information helps us select the best hyperparameters for our models and improve their performance.