Mastering Time Series Data with Python

Introduction

This lab will guide you through handling time series data using the Python package, Pandas. We will be working with air quality data for this tutorial. You will learn how to convert strings into datetime objects, perform operations on these datetime objects, resample time series to another frequency, and more.

VM Tips

After the VM startup is done, click the top left corner to switch to the Notebook tab to access Jupyter Notebook for practice.

Sometimes, you may need to wait a few seconds for Jupyter Notebook to finish loading. The validation of operations cannot be automated because of limitations in Jupyter Notebook.

If you face issues during learning, feel free to ask Labby. Provide feedback after the session, and we will promptly resolve the problem for you.

Skills Graph

%%%%{init: {'theme':'neutral'}}%%%% flowchart RL python(("`Python`")) -.-> python/BasicConceptsGroup(["`Basic Concepts`"]) pandas(("`Pandas`")) -.-> pandas/ReadingDataGroup(["`Reading Data`"]) pandas(("`Pandas`")) -.-> pandas/DataSelectionGroup(["`Data Selection`"]) pandas(("`Pandas`")) -.-> pandas/DataAnalysisGroup(["`Data Analysis`"]) pandas(("`Pandas`")) -.-> pandas/DataVisualizationGroup(["`Data Visualization`"]) pandas(("`Pandas`")) -.-> pandas/AdvancedOperationsGroup(["`Advanced Operations`"]) python(("`Python`")) -.-> python/ControlFlowGroup(["`Control Flow`"]) python(("`Python`")) -.-> python/DataStructuresGroup(["`Data Structures`"]) python(("`Python`")) -.-> python/ModulesandPackagesGroup(["`Modules and Packages`"]) python(("`Python`")) -.-> python/PythonStandardLibraryGroup(["`Python Standard Library`"]) python(("`Python`")) -.-> python/DataScienceandMachineLearningGroup(["`Data Science and Machine Learning`"]) python(("`Python`")) -.-> python/FunctionsGroup(["`Functions`"]) python/BasicConceptsGroup -.-> python/comments("`Comments`") pandas/ReadingDataGroup -.-> pandas/read_csv("`Read CSV`") pandas/DataSelectionGroup -.-> pandas/select_columns("`Select Columns`") pandas/DataSelectionGroup -.-> pandas/conditional_selection("`Conditional Selection`") pandas/DataAnalysisGroup -.-> pandas/basic_statistics("`Basic Statistics`") pandas/DataAnalysisGroup -.-> pandas/groupby_operations("`GroupBy Operations`") pandas/DataVisualizationGroup -.-> pandas/bar_plots("`Bar Plots`") pandas/DataVisualizationGroup -.-> pandas/line_plots("`Line Plots`") pandas/AdvancedOperationsGroup -.-> pandas/time_series_analysis("`Time Series Analysis`") pandas/AdvancedOperationsGroup -.-> pandas/reshape_data("`Reshaping Data`") python/ControlFlowGroup -.-> python/for_loops("`For Loops`") python/DataStructuresGroup -.-> python/lists("`Lists`") python/DataStructuresGroup -.-> python/tuples("`Tuples`") python/DataStructuresGroup -.-> python/dictionaries("`Dictionaries`") python/ModulesandPackagesGroup -.-> python/importing_modules("`Importing Modules`") python/ModulesandPackagesGroup -.-> python/standard_libraries("`Common Standard Libraries`") python/PythonStandardLibraryGroup -.-> python/date_time("`Date and Time`") python/DataScienceandMachineLearningGroup -.-> python/numerical_computing("`Numerical Computing`") python/DataScienceandMachineLearningGroup -.-> python/data_analysis("`Data Analysis`") python/DataScienceandMachineLearningGroup -.-> python/data_visualization("`Data Visualization`") python/FunctionsGroup -.-> python/build_in_functions("`Build-in Functions`") subgraph Lab Skills python/comments -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/read_csv -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/select_columns -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/conditional_selection -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/basic_statistics -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/groupby_operations -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/bar_plots -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/line_plots -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/time_series_analysis -.-> lab-65438{{"`Handling Time Series Data`"}} pandas/reshape_data -.-> lab-65438{{"`Handling Time Series Data`"}} python/for_loops -.-> lab-65438{{"`Handling Time Series Data`"}} python/lists -.-> lab-65438{{"`Handling Time Series Data`"}} python/tuples -.-> lab-65438{{"`Handling Time Series Data`"}} python/dictionaries -.-> lab-65438{{"`Handling Time Series Data`"}} python/importing_modules -.-> lab-65438{{"`Handling Time Series Data`"}} python/standard_libraries -.-> lab-65438{{"`Handling Time Series Data`"}} python/date_time -.-> lab-65438{{"`Handling Time Series Data`"}} python/numerical_computing -.-> lab-65438{{"`Handling Time Series Data`"}} python/data_analysis -.-> lab-65438{{"`Handling Time Series Data`"}} python/data_visualization -.-> lab-65438{{"`Handling Time Series Data`"}} python/build_in_functions -.-> lab-65438{{"`Handling Time Series Data`"}} end

Import the necessary libraries and load the data

First, we need to import the required Python libraries and load the air quality data. The data will be read into a pandas DataFrame, which is a 2-dimensional labeled data structure.

## import necessary libraries
import pandas as pd
import matplotlib.pyplot as plt

## load the air quality data
air_quality = pd.read_csv("data/air_quality_no2_long.csv")

## rename the "date.utc" column to "datetime"
air_quality = air_quality.rename(columns={"date.utc": "datetime"})

Convert strings to datetime objects

The dates in the "datetime" column are currently strings. We want to convert these to datetime objects for easier manipulation.

## convert "datetime" column to datetime objects
air_quality["datetime"] = pd.to_datetime(air_quality["datetime"])

Add a new column for the month of the measurement

Now, we want to add a new column to our DataFrame that contains only the month of each measurement. This can be achieved using the dt accessor.

## add a new column for the month of each measurement
air_quality["month"] = air_quality["datetime"].dt.month

Calculate the average NO2 concentration for each day of the week

We can now calculate the average NO2 concentration for each day of the week at each measurement location. This can be done using the groupby method.

## calculate the average NO2 concentration for each day of the week
average_NO2 = air_quality.groupby([air_quality["datetime"].dt.weekday, "location"])["value"].mean()

Plot the average NO2 values for each hour of the day

We can also plot the average NO2 values for each hour of the day. This can be done using the plot method.

## plot the average NO2 values for each hour of the day
fig, axs = plt.subplots(figsize=(12, 4))
air_quality.groupby(air_quality["datetime"].dt.hour)["value"].mean().plot(kind='bar', rot=0, ax=axs)
plt.xlabel("Hour of the day")
plt.ylabel("$NO_2 (µg/m^3)$")

Resample time series data

The resample method is a powerful way to change the frequency of time series data. Here, we will aggregate the current hourly time series data to the monthly maximum value at each measurement station.

## By pivoting the data, the datetime information became the index of the table.
no_2 = air_quality.pivot(index="datetime", columns="location", values="value")
no_2.head()

## Create a plot of the values in the different stations from the 20th of May till the end of 21st of May
no_2["2019-05-20":"2019-05-21"].plot()

## resample time series data
monthly_max = no_2.resample("M").max()
monthly_max

Summary

In this lab, we learned how to handle time series data in Python using the pandas library. We loaded air quality data, converted date strings to datetime objects, calculated the average NO2 concentration for each day of the week, plotted the average NO2 values for each hour of the day, and resampled the time series data to a different frequency.

Handling Time Series Data