Pandas Basics: DataFrame Memory and Operations

Introduction

Welcome to the Pandas Basics Lab! In this lab, we will explore some fundamental aspects of the Pandas library: DataFrame memory usage, handling if/truth statements, using User Defined Function (UDF) methods, dealing with NA values, differences with NumPy, and thread-safety considerations.

VM Tips

After the VM startup is done, click the top left corner to switch to the Notebook tab to access Jupyter Notebook for practice.

Sometimes, you may need to wait a few seconds for Jupyter Notebook to finish loading. The validation of operations cannot be automated because of limitations in Jupyter Notebook.

If you face issues during learning, feel free to ask Labby. Provide feedback after the session, and we will promptly resolve the problem for you.

Skills Graph

%%%%{init: {'theme':'neutral'}}%%%% flowchart RL python(("`Python`")) -.-> python/BasicConceptsGroup(["`Basic Concepts`"]) pandas(("`Pandas`")) -.-> pandas/DataSelectionGroup(["`Data Selection`"]) pandas(("`Pandas`")) -.-> pandas/DataManipulationGroup(["`Data Manipulation`"]) pandas(("`Pandas`")) -.-> pandas/DataCleaningGroup(["`Data Cleaning`"]) python(("`Python`")) -.-> python/ControlFlowGroup(["`Control Flow`"]) python(("`Python`")) -.-> python/DataStructuresGroup(["`Data Structures`"]) python(("`Python`")) -.-> python/FunctionsGroup(["`Functions`"]) python(("`Python`")) -.-> python/ModulesandPackagesGroup(["`Modules and Packages`"]) python(("`Python`")) -.-> python/PythonStandardLibraryGroup(["`Python Standard Library`"]) python(("`Python`")) -.-> python/DataScienceandMachineLearningGroup(["`Data Science and Machine Learning`"]) python/BasicConceptsGroup -.-> python/comments("`Comments`") pandas/DataSelectionGroup -.-> pandas/select_columns("`Select Columns`") pandas/DataSelectionGroup -.-> pandas/conditional_selection("`Conditional Selection`") pandas/DataManipulationGroup -.-> pandas/change_data_types("`Changing Data Types`") pandas/DataCleaningGroup -.-> pandas/data_mapping("`Data Mapping`") python/BasicConceptsGroup -.-> python/variables_data_types("`Variables and Data Types`") python/BasicConceptsGroup -.-> python/booleans("`Booleans`") python/ControlFlowGroup -.-> python/conditional_statements("`Conditional Statements`") python/ControlFlowGroup -.-> python/for_loops("`For Loops`") python/DataStructuresGroup -.-> python/lists("`Lists`") python/DataStructuresGroup -.-> python/tuples("`Tuples`") python/DataStructuresGroup -.-> python/dictionaries("`Dictionaries`") python/FunctionsGroup -.-> python/function_definition("`Function Definition`") python/ModulesandPackagesGroup -.-> python/importing_modules("`Importing Modules`") python/ModulesandPackagesGroup -.-> python/standard_libraries("`Common Standard Libraries`") python/PythonStandardLibraryGroup -.-> python/math_random("`Math and Random`") python/PythonStandardLibraryGroup -.-> python/data_collections("`Data Collections`") python/DataScienceandMachineLearningGroup -.-> python/numerical_computing("`Numerical Computing`") python/DataScienceandMachineLearningGroup -.-> python/data_analysis("`Data Analysis`") python/FunctionsGroup -.-> python/build_in_functions("`Build-in Functions`") subgraph Lab Skills python/comments -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} pandas/select_columns -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} pandas/conditional_selection -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} pandas/change_data_types -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} pandas/data_mapping -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/variables_data_types -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/booleans -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/conditional_statements -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/for_loops -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/lists -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/tuples -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/dictionaries -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/function_definition -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/importing_modules -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/standard_libraries -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/math_random -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/data_collections -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/numerical_computing -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/data_analysis -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} python/build_in_functions -.-> lab-65446{{"`Pandas Basics: DataFrame Memory and Operations`"}} end

Understanding DataFrame Memory Usage

Pandas provides several methods to understand the memory usage of a DataFrame. The .info() method can be used to see a summary, including memory usage.

import pandas as pd
import numpy as np

## Create a DataFrame
dtypes = ["int64", "float64", "datetime64[ns]", "timedelta64[ns]", "complex128", "object", "bool"]
n = 5000
data = {t: np.random.randint(100, size=n).astype(t) for t in dtypes}
df = pd.DataFrame(data)
df["categorical"] = df["object"].astype("category")

## Display DataFrame info
df.info()

Using if/truth Statements with Pandas

Pandas does not support using if/truth statements directly due to ambiguity. Instead, use methods like .any(), .all(), or .empty().

## Check if any value in the Series is True
if pd.Series([False, True, False]).any():
    print("At least one True value in the Series")

Mutating with User Defined Function (UDF) Methods

When using a pandas method that takes a UDF, avoid changing the DataFrame inside the UDF. Instead, make a copy before making changes.

def f(s):
    s = s.copy()
    s.pop("a")
    return s

df = pd.DataFrame({"a": [1, 2, 3], 'b': [4, 5, 6]})
df.apply(f, axis="columns")

Dealing with NA Values

Pandas provides nullable-integer extension dtypes to represent integers with possibly missing values.

s_int = pd.Series([1, 2, 3, 4, 5], index=list("abcde"), dtype=pd.Int64Dtype())
s2_int = s_int.reindex(["a", "b", "c", "f", "u"])

Understanding Differences with NumPy

Pandas and NumPy have slight differences in how they calculate variance. This is important to consider when switching between the two libraries.

## Variance in pandas
var_pandas = df.var()

## Variance in NumPy
var_numpy = np.var(df.values)

Considering Thread-safety in Pandas

Pandas is not 100% thread safe. Be cautious when sharing pandas objects among multiple threads.

Handling Byte-ordering Issues

You may encounter byte-ordering issues when dealing with data created on a machine with a different byte order. Convert the data to the native system byte order before passing it to Pandas.

x = np.array(list(range(10)), ">i4")  ## big endian
newx = x.byteswap().newbyteorder()  ## force native byteorder
s = pd.Series(newx)

Summary

In this lab, we explored some crucial aspects of the Pandas library. Understanding these aspects will help you utilize Pandas more effectively and avoid common pitfalls.