Nonparametric Isotonic Regression with Scikit-Learn

Introduction

In this tutorial, we will learn about isotonic regression, which is a non-parametric regression technique that finds a non-decreasing approximation of a function while minimizing the mean squared error on the training data. We will use scikit-learn, a popular machine learning library in Python, to implement isotonic regression and compare it with linear regression.

VM Tips

After the VM startup is done, click the top left corner to switch to the Notebook tab to access Jupyter Notebook for practice.

Sometimes, you may need to wait a few seconds for Jupyter Notebook to finish loading. The validation of operations cannot be automated because of limitations in Jupyter Notebook.

If you face issues during learning, feel free to ask Labby. Provide feedback after the session, and we will promptly resolve the problem for you.

Skills Graph

Import Required Libraries

We will start by importing the required libraries for this tutorial, which are NumPy, Matplotlib, and scikit-learn.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.collections import LineCollection

from sklearn.linear_model import LinearRegression
from sklearn.isotonic import IsotonicRegression
from sklearn.utils import check_random_state

Generate Data

Next, we will generate some data to use for our regression. We will create a non-linear monotonic trend with homoscedastic uniform noise.

n = 100
x = np.arange(n)
rs = check_random_state(0)
y = rs.randint(-50, 50, size=(n,)) + 50.0 * np.log1p(np.arange(n))

Fit Isotonic and Linear Regression Models

We will now fit both the isotonic and linear regression models to the generated data.

ir = IsotonicRegression(out_of_bounds="clip")
y_ = ir.fit_transform(x, y)

lr = LinearRegression()
lr.fit(x[:, np.newaxis], y)  ## x needs to be 2d for LinearRegression

Plot Results

Finally, we will plot the results of both regression models to visualize how well they fit the data.

segments = [[[i, y[i]], [i, y_[i]]] for i in range(n)]
lc = LineCollection(segments, zorder=0)
lc.set_array(np.ones(len(y)))
lc.set_linewidths(np.full(n, 0.5))

fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(12, 6))

ax0.plot(x, y, "C0.", markersize=12)
ax0.plot(x, y_, "C1.-", markersize=12)
ax0.plot(x, lr.predict(x[:, np.newaxis]), "C2-")
ax0.add_collection(lc)
ax0.legend(("Training data", "Isotonic fit", "Linear fit"), loc="lower right")
ax0.set_title("Isotonic regression fit on noisy data (n=%d)" % n)

x_test = np.linspace(-10, 110, 1000)
ax1.plot(x_test, ir.predict(x_test), "C1-")
ax1.plot(ir.X_thresholds_, ir.y_thresholds_, "C1.", markersize=12)
ax1.set_title("Prediction function (%d thresholds)" % len(ir.X_thresholds_))

plt.show()

Summary

In this tutorial, we learned about isotonic regression, a non-parametric regression technique that finds a non-decreasing approximation of a function while minimizing the mean squared error on the training data. We also implemented isotonic regression using scikit-learn and compared it with linear regression.