Enhance Pandas with PyArrow

Introduction

This lab will guide you through the process of utilizing PyArrow in pandas to extend functionality and improve the performance of various APIs. PyArrow enhances pandas with more extensive data types, missing data support for all data types, IO reader integration, and interoperability with other data frame libraries.

VM Tips

After the VM startup is done, click the top left corner to switch to the Notebook tab to access Jupyter Notebook for practice.

Sometimes, you may need to wait a few seconds for Jupyter Notebook to finish loading. The validation of operations cannot be automated because of limitations in Jupyter Notebook.

If you face issues during learning, feel free to ask Labby. Provide feedback after the session, and we will promptly resolve the problem for you.

Skills Graph

%%%%{init: {'theme':'neutral'}}%%%% flowchart RL python(("`Python`")) -.-> python/BasicConceptsGroup(["`Basic Concepts`"]) python(("`Python`")) -.-> python/FileHandlingGroup(["`File Handling`"]) pandas(("`Pandas`")) -.-> pandas/ReadingDataGroup(["`Reading Data`"]) pandas(("`Pandas`")) -.-> pandas/DataSelectionGroup(["`Data Selection`"]) pandas(("`Pandas`")) -.-> pandas/DataCleaningGroup(["`Data Cleaning`"]) pandas(("`Pandas`")) -.-> pandas/DataAnalysisGroup(["`Data Analysis`"]) python(("`Python`")) -.-> python/DataStructuresGroup(["`Data Structures`"]) python(("`Python`")) -.-> python/ModulesandPackagesGroup(["`Modules and Packages`"]) python(("`Python`")) -.-> python/PythonStandardLibraryGroup(["`Python Standard Library`"]) python(("`Python`")) -.-> python/DataScienceandMachineLearningGroup(["`Data Science and Machine Learning`"]) python/BasicConceptsGroup -.-> python/comments("`Comments`") python/FileHandlingGroup -.-> python/with_statement("`Using with Statement`") pandas/ReadingDataGroup -.-> pandas/read_csv("`Read CSV`") pandas/DataSelectionGroup -.-> pandas/select_columns("`Select Columns`") pandas/DataSelectionGroup -.-> pandas/conditional_selection("`Conditional Selection`") pandas/DataCleaningGroup -.-> pandas/handle_missing_values("`Handling Missing Values`") pandas/DataAnalysisGroup -.-> pandas/basic_statistics("`Basic Statistics`") python/BasicConceptsGroup -.-> python/variables_data_types("`Variables and Data Types`") python/BasicConceptsGroup -.-> python/booleans("`Booleans`") python/DataStructuresGroup -.-> python/lists("`Lists`") python/DataStructuresGroup -.-> python/tuples("`Tuples`") python/DataStructuresGroup -.-> python/dictionaries("`Dictionaries`") python/ModulesandPackagesGroup -.-> python/importing_modules("`Importing Modules`") python/ModulesandPackagesGroup -.-> python/standard_libraries("`Common Standard Libraries`") python/PythonStandardLibraryGroup -.-> python/data_collections("`Data Collections`") python/DataScienceandMachineLearningGroup -.-> python/numerical_computing("`Numerical Computing`") python/DataScienceandMachineLearningGroup -.-> python/data_analysis("`Data Analysis`") subgraph Lab Skills python/comments -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/with_statement -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} pandas/read_csv -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} pandas/select_columns -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} pandas/conditional_selection -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} pandas/handle_missing_values -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} pandas/basic_statistics -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/variables_data_types -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/booleans -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/lists -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/tuples -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/dictionaries -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/importing_modules -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/standard_libraries -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/data_collections -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/numerical_computing -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} python/data_analysis -.-> lab-65451{{"`Enhance Pandas with PyArrow`"}} end

Installing PyArrow

Before starting, make sure you have installed the minimum supported PyArrow version. You can do this by running the following command in your Python environment:

## This is a comment
## Use pip to install PyArrow
pip install pyarrow

Data Structure Integration

PyArrow allows pandas data structures to be directly backed by a PyArrow ChunkedArray, similar to a NumPy array. Here is how to do this:

## Import pandas
import pandas as pd

## Create a pandas Series, Index and DataFrame with PyArrow data type
ser = pd.Series([-1.5, 0.2, None], dtype="float32[pyarrow]")
idx = pd.Index([True, None], dtype="bool[pyarrow]")
df = pd.DataFrame([[1, 2], [3, 4]], dtype="uint64[pyarrow]")

Using PyArrow Types with Parameters

For PyArrow types that accept parameters, you can pass in a PyArrow type with those parameters into ArrowDtype to use in the dtype parameter.

## Import PyArrow
import pyarrow as pa

## Create a pandas Series with PyArrow list type
list_str_type = pa.list_(pa.string())
ser = pd.Series([["hello"], ["there"]], dtype=pd.ArrowDtype(list_str_type))

Converting PyArrow Array to pandas Data Structures

If you have a PyArrow Array or ChunkedArray, you can convert it into pandas data structures like Series, Index or DataFrame.

## Create a PyArrow array
pa_array = pa.array([{"1": "2"}, {"10": "20"}, None], type=pa.map_(pa.string(), pa.string()))

## Convert the PyArrow array to a pandas Series
ser = pd.Series(pd.arrays.ArrowExtensionArray(pa_array))

PyArrow Operations

PyArrow data structure integration is implemented through pandas' ExtensionArray interface. Supported functionality exists where this interface is integrated within the pandas API.

## Create a pandas Series with PyArrow data type
ser = pd.Series([-1.545, 0.2, None], dtype="float32[pyarrow]")

## Perform various operations
ser.mean()
ser + ser
ser > (ser + 1)
ser.dropna()
ser.isna()
ser.fillna(0)

Reading Data with PyArrow

PyArrow provides IO reading functionality that has been integrated into several pandas IO readers.

## Import IO module
import io

## Create a StringIO object
data = io.StringIO("""a,b,c\n1,2.5,True\n3,4.5,False""")

## Read the data into a pandas DataFrame using PyArrow as the engine
df = pd.read_csv(data, engine="pyarrow")

Summary

In this lab, we explored how PyArrow can be used with pandas to extend its functionality and improve performance. We learned how to convert pandas data structures into PyArrow data types and perform various operations. We also saw how to read data using PyArrow's IO reading functionality.