How to use docker builder prune command to remove build cache

Introduction

In this lab, you will learn how to effectively manage Docker build cache using the docker builder prune command. We will begin by building a simple Docker image to understand how build cache is created and utilized to speed up subsequent builds.

Following the image build, you will explore different ways to clean up the build cache. This includes pruning dangling build cache, removing all unused build cache, and using filters to selectively prune cache. Finally, you will learn how to prune build cache while preserving storage space, providing a comprehensive understanding of build cache management techniques.

Skills Graph

Build an image to create build cache

In this step, we will build a simple Docker image to understand how build cache is created. Docker uses a build cache to speed up subsequent builds. Each instruction in a Dockerfile creates a layer, and if the instruction hasn't changed since the last build, Docker can reuse the existing layer from the cache.

First, navigate to the ~/project directory if you are not already there.

cd ~/project

Now, let's create a simple Dockerfile. We will name it Dockerfile.cache.

nano Dockerfile.cache

Add the following content to the Dockerfile.cache file:

FROM ubuntu:latest
RUN apt-get update && apt-get install -y --no-install-recommends curl
COPY . /app
WORKDIR /app
CMD ["curl", "https://www.example.com"]

Save the file and exit the nano editor (Press Ctrl + X, then Y, then Enter).

This Dockerfile does the following:

• FROM ubuntu:latest: Starts with the latest Ubuntu base image.
• RUN apt-get update && apt-get install -y --no-install-recommends curl: Updates the package list and installs the curl package. This is a common operation and will likely be cached on subsequent builds if the base image and this instruction don't change.
• COPY . /app: Copies the current directory's contents into the /app directory in the image.
• WORKDIR /app: Sets the working directory to /app.
• CMD ["curl", "https://www.example.com"]: Specifies the command to run when a container is started from this image.

Before building, let's make sure we have the ubuntu:latest image locally. If not, Docker will pull it automatically during the build, but explicitly pulling it first can sometimes make the build output clearer.

docker pull ubuntu:latest

Now, let's build the image using the Dockerfile.cache file. We will tag the image as myimage:cachetest.

docker build -t myimage:cachetest -f Dockerfile.cache .

You will see output indicating the steps of the build process. Docker will execute each instruction in the Dockerfile. Since this is the first time building with this Dockerfile, Docker will not use any build cache for the layers defined in this Dockerfile.

After the build completes, you can list your images to see the newly created myimage:cachetest.

docker images

You should see myimage with the tag cachetest in the list. This build process has created build cache for each layer defined in the Dockerfile.cache. In the next steps, we will explore how to manage this build cache.

Prune dangling build cache

In this step, we will learn how to prune dangling build cache. Dangling build cache refers to build cache layers that are no longer associated with any named image. This can happen when you rebuild an image and a previous layer is replaced by a new one. Pruning dangling cache can help free up disk space.

First, let's modify our Dockerfile.cache slightly to create some dangling cache. We will change the RUN instruction.

Navigate to the ~/project directory if you are not already there.

cd ~/project

Open the Dockerfile.cache file for editing.

nano Dockerfile.cache

Change the RUN instruction from:

RUN apt-get update && apt-get install -y --no-install-recommends curl

to:

RUN apt-get update && apt-get install -y --no-install-recommends wget

Save the file and exit the nano editor (Press Ctrl + X, then Y, then Enter).

Now, let's rebuild the image with the same tag.

docker build -t myimage:cachetest -f Dockerfile.cache .

Observe the output. Docker will likely reuse the FROM layer from the cache, but the RUN instruction has changed, so it will execute that step again, creating a new layer. The previous layer created by the old RUN instruction is now dangling build cache because it's no longer part of the myimage:cachetest image.

To see the build cache, you can use the docker builder prune --dry-run command, although it's not always easy to distinguish dangling cache specifically without more advanced tools or understanding of the buildkit output. A simpler way to see the effect is to prune the dangling cache and see the space reclaimed.

To prune only the dangling build cache, we use the docker builder prune command with the --filter dangling=true option.

docker builder prune --filter dangling=true

You will be prompted to confirm the action. Type y and press Enter.

WARNING! This will remove all dangling build cache.
Are you sure you want to continue? [y/N] y

The output will show the amount of space reclaimed. This space was occupied by the dangling build cache layers that were no longer needed.

By pruning dangling build cache, you can recover disk space that is no longer being used by any active image layers.

Prune all unused build cache

In this step, we will learn how to prune all unused build cache. While pruning dangling cache removes layers not associated with any image, pruning all unused cache removes any build cache that is not currently being used by a running container or referenced by a local image. This is a more aggressive way to free up disk space.

First, let's create another image using a different Dockerfile to add more build cache.

Navigate to the ~/project directory if you are not already there.

cd ~/project

Create a new Dockerfile named Dockerfile.another.

nano Dockerfile.another

Add the following content to Dockerfile.another:

FROM alpine:latest
RUN apk update && apk add --no-cache git
COPY . /app2
WORKDIR /app2
CMD ["git", "--version"]

Save the file and exit the nano editor (Press Ctrl + X, then Y, then Enter).

This Dockerfile uses the alpine base image and installs git.

Now, build an image using this new Dockerfile. We will tag it as anotherimage:latest.

docker build -t anotherimage:latest -f Dockerfile.another .

This build will create new build cache layers specific to this Dockerfile.

To prune all unused build cache, we use the docker builder prune command without any filters.

docker builder prune -a

The -a flag stands for "all" and indicates that you want to remove all unused build cache, not just dangling cache.

You will be prompted to confirm the action. Type y and press Enter.

WARNING! This will remove all unused build cache.
Are you sure you want to continue? [y/N] y

The output will show the amount of space reclaimed. This command removes any build cache layers that are not currently referenced by a local image or a running container. This can free up a significant amount of disk space, especially after many builds and image updates.

Keep in mind that pruning all unused cache might mean that subsequent builds of images you still have locally might take longer if they relied on the removed cache layers.

Prune build cache with filter

In this step, we will explore how to prune build cache using filters. Filters allow you to selectively remove build cache based on certain criteria, giving you more granular control over what is removed.

We have already seen the dangling=true filter in a previous step. Another useful filter is until, which allows you to remove build cache created before a specific timestamp.

First, let's build our myimage:cachetest image again to ensure we have recent build cache.

Navigate to the ~/project directory if you are not already there.

cd ~/project

Build the image again:

docker build -t myimage:cachetest -f Dockerfile.cache .

Now, let's simulate a scenario where you want to remove build cache that is older than a certain time. For demonstration purposes, we will use a filter to remove cache older than a few minutes ago.

To prune build cache older than a specific time, you can use the --filter until=<timestamp> option. The timestamp can be in various formats, including a duration like 5m (5 minutes) or 2h (2 hours).

Let's prune build cache that is older than 5 minutes.

docker builder prune --filter until=5m

You will be prompted to confirm the action. Type y and press Enter.

WARNING! This will remove all build cache created before 5 minutes ago.
Are you sure you want to continue? [y/N] y

The output will show the amount of space reclaimed from build cache that was older than 5 minutes. If you just built the image, there might not be much cache older than 5 minutes, but in a real-world scenario with many builds over time, this filter can be very useful for cleaning up old cache.

You can also combine filters. For example, to prune dangling cache older than a certain time, you could use:

docker builder prune --filter dangling=true --filter until=24h

This would remove only the dangling build cache that is older than 24 hours.

Using filters provides flexibility in managing your build cache and helps you free up disk space more strategically.

Prune build cache while keeping storage

In this step, we will learn how to prune build cache while keeping a certain amount of storage. This is useful when you want to free up disk space but still retain some recent build cache to speed up future builds.

The docker builder prune command has a --keep-storage option that allows you to specify the maximum amount of storage to keep for build cache. You can specify the size in bytes, or use units like k, m, or g.

First, let's ensure we have some build cache to prune. We have built images in previous steps, which created cache.

Navigate to the ~/project directory if you are not already there.

cd ~/project

To prune build cache while keeping a specific amount of storage, use the --keep-storage option followed by the desired size. For example, to keep a maximum of 1GB of build cache:

docker builder prune --keep-storage 1g

You will be prompted to confirm the action. Type y and press Enter.

WARNING! This will remove all build cache exceeding 1GB.
Are you sure you want to continue? [y/N] y

The output will show the amount of space reclaimed. Docker will remove the oldest build cache layers until the total build cache size is below the specified --keep-storage limit.

You can adjust the value of --keep-storage based on your available disk space and how much cache you want to retain for faster builds. For example, to keep a maximum of 500MB:

docker builder prune --keep-storage 500m

Again, confirm the action by typing y and pressing Enter.

This method provides a balance between freeing up disk space and maintaining some build cache for performance. It's a good practice to regularly prune your build cache to prevent it from consuming excessive disk space.

Summary

In this lab, we learned how to use the docker builder prune command to manage Docker build cache. We started by building a simple Docker image using a Dockerfile to understand how build cache is created and utilized by Docker to speed up subsequent builds. Each instruction in the Dockerfile creates a layer, and Docker reuses existing layers from the cache if the instruction hasn't changed.

We then explored different ways to prune the build cache. We learned how to prune dangling build cache, which refers to cache entries that are no longer associated with any build. We also covered how to prune all unused build cache, effectively cleaning up all cache entries that are not currently being used by any image. Furthermore, we demonstrated how to use filters with the prune command to selectively remove build cache based on specific criteria. Finally, we explored the option to prune build cache while keeping the storage associated with certain cache entries, which can be useful in specific scenarios.