Python Pandas | DataFrame Memory Usage | Data Analysis Tutorial

Introduction

In this lab, we will learn how to use the DataFrame.memory_usage() method in Python Pandas. This method allows us to calculate the memory usage of each column in a DataFrame. We will go through step-by-step instructions on how to use this method with examples.

VM Tips

After the VM startup is done, click the top left corner to switch to the Notebook tab to access Jupyter Notebook for practice.

Sometimes, you may need to wait a few seconds for Jupyter Notebook to finish loading. The validation of operations cannot be automated because of limitations in Jupyter Notebook.

If you face issues during learning, feel free to ask Labby. Provide feedback after the session, and we will promptly resolve the problem for you.

Skills Graph

%%%%{init: {'theme':'neutral'}}%%%% flowchart RL python(("`Python`")) -.-> python/BasicConceptsGroup(["`Basic Concepts`"]) pandas(("`Pandas`")) -.-> pandas/DataSelectionGroup(["`Data Selection`"]) pandas(("`Pandas`")) -.-> pandas/DataAnalysisGroup(["`Data Analysis`"]) python(("`Python`")) -.-> python/DataStructuresGroup(["`Data Structures`"]) python(("`Python`")) -.-> python/ModulesandPackagesGroup(["`Modules and Packages`"]) python(("`Python`")) -.-> python/DataScienceandMachineLearningGroup(["`Data Science and Machine Learning`"]) python(("`Python`")) -.-> python/FunctionsGroup(["`Functions`"]) python/BasicConceptsGroup -.-> python/comments("`Comments`") pandas/DataSelectionGroup -.-> pandas/select_columns("`Select Columns`") pandas/DataAnalysisGroup -.-> pandas/basic_statistics("`Basic Statistics`") python/BasicConceptsGroup -.-> python/booleans("`Booleans`") python/DataStructuresGroup -.-> python/lists("`Lists`") python/DataStructuresGroup -.-> python/tuples("`Tuples`") python/DataStructuresGroup -.-> python/dictionaries("`Dictionaries`") python/ModulesandPackagesGroup -.-> python/importing_modules("`Importing Modules`") python/DataScienceandMachineLearningGroup -.-> python/numerical_computing("`Numerical Computing`") python/DataScienceandMachineLearningGroup -.-> python/data_analysis("`Data Analysis`") python/FunctionsGroup -.-> python/build_in_functions("`Build-in Functions`") subgraph Lab Skills python/comments -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} pandas/select_columns -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} pandas/basic_statistics -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} python/booleans -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} python/lists -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} python/tuples -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} python/dictionaries -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} python/importing_modules -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} python/numerical_computing -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} python/data_analysis -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} python/build_in_functions -.-> lab-68664{{"`Pandas DataFrame Memory Usage Method`"}} end

Import the necessary libraries and create a DataFrame

Before we start, let's import the pandas library and create a DataFrame.
Create a DataFrame with some sample data.

## Import pandas library
import pandas as pd

## Create a DataFrame
df = pd.DataFrame({'Name': ['Abhishek', 'Anurag', 'Divya'],
                   'Roll No': [100, 101, 104]})

View the DataFrame and calculate memory usage

Now, let's view the created DataFrame and calculate its memory usage using the DataFrame.memory_usage() method.

## View the DataFrame
print("----------The DataFrame is---------")
print(df)
print("-----------------------------------")

## Calculate memory usage
print(df.memory_usage())

Exclude index in memory usage calculation

By default, the DataFrame.memory_usage() method includes the memory usage by the index of the DataFrame. If we want to exclude the index from the memory usage calculation, we can set the index parameter to False.

## Calculate memory usage excluding index
print(df.memory_usage(index=False))

Get overall memory consumption

We can also get the overall memory consumption of the DataFrame columns by using the DataFrame.memory_usage() method along with the sum() function.

## Get overall memory consumption
print(df.memory_usage(index=False).sum())

Summary

In this lab, we learned how to use the DataFrame.memory_usage() method in Python Pandas. This method allows us to calculate the memory usage of each column in a DataFrame. We can include or exclude the index in the memory usage calculation based on our requirement, and also get the overall memory consumption of the DataFrame columns. Understanding the memory usage of a DataFrame can help optimize our code and improve performance.

Pandas DataFrame Memory Usage Method