How to use docker buildx prune command to manage build cache

Introduction

In this lab, we will explore how to effectively manage Docker build cache using the docker buildx prune command. Build cache is a crucial mechanism for accelerating Docker image builds by reusing previously built layers. However, over time, this cache can accumulate and consume significant disk space.

We will begin by building a simple Docker image to generate some build cache. Then, we will demonstrate various ways to prune this cache using docker buildx prune. This includes pruning all cache without filters, selectively pruning while keeping recent data using the --filter option, limiting the cache size with --keep-storage, and finally, pruning all build cache, including internal images, using the --all flag. By the end of this lab, you will have a solid understanding of how to manage your Docker build cache efficiently.

Skills Graph

Build an image to create build cache

In this step, we will build a simple Docker image to create some build cache. Docker build cache is a mechanism that speeds up the build process by reusing layers from previous builds. When you build a Docker image, each instruction in the Dockerfile creates a layer. If an instruction hasn't changed since the last build, Docker can reuse the existing layer instead of rebuilding it.

First, let's create a directory for our project and navigate into it.

mkdir ~/project/docker-cache-demo
cd ~/project/docker-cache-demo

Now, we will create a simple Dockerfile. This Dockerfile will copy a file into the image and then run a command.

nano Dockerfile

Add the following content to the Dockerfile:

FROM ubuntu:latest
COPY . /app
RUN echo "Hello, Docker Cache!" > /app/message.txt
CMD ["cat", "/app/message.txt"]

Save and close the file.

Next, let's create a simple file that we will copy into the image.

nano message.txt

Add some content to message.txt:

This is a test message.

Save and close the file.

Now, we will build the Docker image. We will tag the image as cache-demo.

docker build -t cache-demo .

You will see output indicating that Docker is building the image step by step. Each step corresponds to an instruction in the Dockerfile. Docker will download the ubuntu:latest image if it's not already present, copy the message.txt file, and then run the echo command.

After the build is complete, you can verify that the image was created by listing the available images.

docker images

You should see cache-demo in the list of images.

Now, let's build the image again without making any changes to the Dockerfile or message.txt.

docker build -t cache-demo .

This time, you will notice that the build process is much faster. Docker reuses the existing layers from the previous build because the instructions in the Dockerfile and the content of message.txt have not changed. The output will show "Using cache" for most of the steps. This demonstrates how Docker build cache works to speed up subsequent builds.

Prune build cache without filters

In this step, we will learn how to prune Docker build cache without using any filters. Pruning the build cache helps free up disk space by removing unused build cache entries.

The command to prune the build cache is docker builder prune. By default, this command will remove all build cache entries that are not used by a currently existing image.

Let's run the command to prune the build cache.

docker builder prune

You will be prompted to confirm the action. Type y and press Enter to proceed.

WARNING! This will remove all dangling build cache.
Are you sure you want to continue? [y/N] y

The output will show you the build cache entries that were deleted and the total amount of reclaimed space.

To see the effect of pruning, let's try building the image again.

cd ~/project/docker-cache-demo
docker build -t cache-demo .

This time, the build process might take longer than the second build in the previous step, as some or all of the build cache might have been removed. Docker will rebuild the layers that were pruned.

Pruning without filters is useful when you want to clean up all unused build cache to maximize free disk space. However, it might impact the build speed of subsequent builds if you frequently rebuild images. In the next steps, we will explore how to use filters to have more control over which build cache entries are pruned.

Prune build cache keeping recent data using --filter

In this step, we will learn how to prune Docker build cache while keeping recent data using the --filter flag. This is useful when you want to free up disk space but still want to benefit from the cache for recent builds.

The docker builder prune command supports various filters. One useful filter is until, which allows you to remove build cache entries older than a specific timestamp. The timestamp can be a date and time, or a duration relative to the current time.

Let's first build our image again to ensure we have some recent build cache.

cd ~/project/docker-cache-demo
docker build -t cache-demo .

Now, let's prune the build cache, keeping entries from the last 5 minutes. We can use the until filter with a duration like 5m.

docker builder prune --filter "until=5m"

You will be prompted to confirm the action. Type y and press Enter to proceed.

WARNING! This will remove all dangling build cache.
Are you sure you want to continue? [y/N] y

The output will show you the build cache entries that were deleted. Only entries older than 5 minutes will be removed.

You can also specify a specific timestamp. For example, to remove cache entries older than a specific date and time, you would use a format like "YYYY-MM-DDTHH:MM:SS".

Using the until filter with a duration is a convenient way to keep the most recent build cache while cleaning up older entries, balancing disk space usage and build speed.

Prune build cache keeping a specific size using --keep-storage

In this step, we will learn how to prune Docker build cache while keeping a specific amount of storage using the --keep-storage flag. This is useful when you want to limit the amount of disk space used by the build cache.

The docker builder prune command with the --keep-storage flag allows you to specify the maximum amount of storage to keep for the build cache. Docker will remove the oldest build cache entries until the total size of the remaining cache is below the specified limit.

Let's first build our image a couple of times to create more build cache entries.

cd ~/project/docker-cache-demo
docker build -t cache-demo:v1 .
docker build -t cache-demo:v2 .

Now, let's prune the build cache, keeping only 1GB of storage. You can specify the size in bytes, kilobytes (k), megabytes (m), or gigabytes (g).

docker builder prune --keep-storage 1g

You will be prompted to confirm the action. Type y and press Enter to proceed.

WARNING! This will remove all dangling build cache.
Are you sure you want to continue? [y/N] y

The output will show you the build cache entries that were deleted and the total amount of reclaimed space. Docker will remove the oldest cache entries until the total size of the remaining cache is approximately 1GB or less.

Using the --keep-storage flag is a good way to manage the disk space consumed by the build cache, especially in environments with limited storage.

Prune all build cache including internal images using --all

In this step, we will learn how to prune all Docker build cache, including internal images, using the --all flag. This is the most aggressive way to clean up build cache and will remove all cache entries regardless of whether they are used by existing images.

The docker builder prune command with the --all flag (-a is a shorthand) will remove all build cache entries. This includes cache entries that are associated with intermediate build images that are not tagged.

Let's run the command to prune all build cache.

docker builder prune --all

You will be prompted to confirm the action. Type y and press Enter to proceed.

WARNING! This will remove all build cache.
Are you sure you want to continue? [y/N] y

The output will show you all the build cache entries that were deleted and the total amount of reclaimed space. This command will typically free up the most disk space compared to pruning without the --all flag or using filters.

After running this command, if you build the cache-demo image again, it will likely take the longest time because all the build cache has been removed, and Docker will need to rebuild all the layers.

cd ~/project/docker-cache-demo
docker build -t cache-demo .

Using docker builder prune --all is useful when you need to free up as much disk space as possible or when you want to ensure a clean build without relying on any existing cache. However, be aware that it will significantly slow down subsequent builds until new cache is generated.

Summary

In this lab, we learned how to manage Docker build cache using the docker buildx prune command. We started by building a simple Docker image to generate build cache, observing how Docker reuses layers for faster subsequent builds.

We then explored different ways to prune the build cache. We learned how to prune all build cache without filters, how to keep recent cache data using the --filter option, how to limit the cache size using --keep-storage, and finally, how to remove all build cache, including internal images, with the --all flag. These techniques are essential for managing disk space and optimizing build environments.