👁6views

Multistage Build Optimisation: The Difference Between Amateur Containers and Production Systems

Most teams assume containers are lightweight by default, but that assumption does not survive contact with a real production system. Containers become bloated, slow, insecure, and operationally expensive when left unmanaged, and the penalties compound at scale as CI pipelines slow down, deployments lag, autoscaling becomes inefficient, and infrastructure costs quietly rise. This post goes deep into how containers are built, why they bloat, the real performance impact of large images, and how multistage builds combined with advanced techniques produce minimal, production grade containers.

1. How Containers Are Actually Built

At a mechanical level, a container image is a stack of immutable filesystem layers generated from a Dockerfile, where each instruction creates a new layer that is cached independently and reused across builds.

FROM ubuntu:22.04
COPY . /app
RUN npm install
CMD ["node", "app.js"]

Docker builds images top to bottom, and any change in an earlier layer invalidates all layers that follow. This makes Dockerfile structure one of the most important optimisation levers available, and the consequences of getting it wrong compound over every build in a pipeline.

Try it yourself. Create a minimal project to feel this directly:

mkdir cache-demo && cd cache-demo
echo '{"name":"demo","version":"1.0.0"}' > package.json
echo 'console.log("hello")' > app.js

# bad.Dockerfile
FROM node:20
COPY . /app
WORKDIR /app
RUN npm install
CMD ["node", "app.js"]

docker build -f bad.Dockerfile -t cache-demo:bad .
# Change app.js and rebuild
echo 'console.log("hello world")' > app.js
docker build -f bad.Dockerfile -t cache-demo:bad .
# Watch npm install run again despite no dependency changes

Every app.js change forces npm install to re-run from scratch because the COPY . /app instruction precedes it and is now invalidated. Reordering the Dockerfile fixes this entirely:

# good.Dockerfile
FROM node:20
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["node", "app.js"]

docker build -f good.Dockerfile -t cache-demo:good .
echo 'console.log("hello world v2")' > app.js
docker build -f good.Dockerfile -t cache-demo:good .
# npm install is now cached. COPY . . is the only step that re-runs.

This structural change alone can reduce rebuild time dramatically in larger systems where dependency installation is expensive.

2. Why Containers Become Bloated

Container bloat is structural rather than accidental, and it typically comes from a combination of predictable patterns that individually seem minor but compound into serious operational problems.

Oversized base images are one of the biggest contributors. Ubuntu commonly exceeds 77MB on disk, while Alpine sits under 7MB and distroless images are smaller still, meaning that choosing the wrong base image can inflate a final image by an order of magnitude before a single line of application code has been written.

Build tools leaking into runtime environments means compilers, package managers, and debugging utilities ship into production even though they are never used, increasing both image size and attack surface in ways that are invisible until a security scan surfaces them.

Excessive or poorly ordered layers accumulate unnecessary duplication, since each Dockerfile instruction creates a layer and an unstructured file means wasted bytes that cannot be recovered without a full rebuild.

Missing or incomplete .dockerignore causes source control directories, test data, logs, and local environment files to land inside the build context and then the image, often including .env files containing credentials that were never intended to leave a developer’s laptop.

Package manager residue such as apt list caches and pip download directories persists unless explicitly cleaned, adding hidden weight that is easy to avoid but rarely is.

Try it yourself. Pull a few base images and measure the damage directly:

docker pull ubuntu:22.04 && docker image inspect ubuntu:22.04 --format '{{.Size}}'
docker pull alpine:3.19 && docker image inspect alpine:3.19 --format '{{.Size}}'
docker pull node:20 && docker image inspect node:20 --format '{{.Size}}'
docker pull node:20-alpine && docker image inspect node:20-alpine --format '{{.Size}}'

Typical results:

The node:20 full image carries the entire Debian base, build toolchain, and npm ecosystem, and for a production runtime serving a single compiled app, almost none of that content is needed.

3. Performance Penalties of Bloated Containers

Bloated containers do not just consume storage; they create systemic performance issues across the entire delivery pipeline that are easy to underestimate until you are operating at scale.

Network transfer becomes a major bottleneck because every deployment requires pulling images across nodes. A 500MB image deployed to 50 nodes results in 25GB of network transfer per rollout, which directly impacts deployment speed and recovery time in failure scenarios where minutes matter.

Cold start latency is influenced less by image size than most engineers assume, because runtime execution dominates startup time. However, image size heavily impacts distribution and scaling speed, which means large images slow down cluster expansion and autoscale recovery during load spikes precisely when fast response is most important.

CI and build performance degrades because larger images cause more cache invalidation and longer rebuild chains, and simply reordering Dockerfile instructions can cut rebuild time by more than 50% in typical projects without any other change.

Security exposure increases as more packages are included, expanding the attack surface and the number of vulnerabilities requiring patches. Research from the Open Container Initiative indicates that images built with multistage workflows show roughly a 40% reduction in known CVEs compared to traditional single stage builds.

Disk and memory pressure on nodes increases as larger images consume more space, leading to degraded IO performance, more frequent evictions, and increased operational complexity under resource pressure in environments where node capacity is shared across many workloads.

4. Multistage Builds: The Core Optimisation Primitive

Multistage builds solve the fundamental problem that build dependencies are not the same as runtime dependencies, allowing you to separate the build environment from the final runtime image entirely. A multistage Dockerfile contains multiple FROM statements, each defining an isolated stage:

FROM builder-image AS build
# build steps

FROM runtime-image AS runtime
# runtime steps only

Artifacts are copied selectively between stages, ensuring only the necessary runtime components are included in the final image:

COPY --from=build /app/bin /app/bin

This removes build tools, reduces image size, improves security posture, and speeds up deployments because only the minimal runtime environment is ever shipped. One detail worth knowing is that Docker only builds stages the final image actually depends on, so unused stages are skipped entirely, which means you can include test and debug stages in the same file without any penalty in production builds.

5. Real World Example: Node.js Before and After

Single stage (the common mistake):

FROM node:20
WORKDIR /app
COPY . .
RUN npm install
RUN npm run build
CMD ["node", "dist/app.js"]

This ships development dependencies, build tools, and all source files into production, producing a typical image size of over 1.1GB for even a modest application.

Optimised multistage build:

# Stage 1: Build
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Stage 2: Runtime
FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
CMD ["node", "dist/app.js"]

Try it yourself. Build both and compare:

docker build -f single.Dockerfile -t node-demo:single .
docker build -f multi.Dockerfile -t node-demo:multi .
docker images | grep node-demo

Expected output (approximate):

node-demo    multi     abc123def   20 seconds ago   145MB
node-demo    single    def456abc   45 seconds ago   1.1GB

That is an 87% reduction from restructuring the Dockerfile alone, before any further optimisation has been applied.

6. Advanced Multistage Patterns by Language

Different languages benefit from specialised patterns that push optimisation considerably further than the basic two-stage split.

6.1 Go: Compile to Scratch

Go compiles to a single static binary, making it ideal for running in a scratch image with no operating system at all. The -s -w ldflags strip the symbol table and debug information, reducing binary size further, and the typical result is that a 1.1GB builder image produces a final image of 4 to 8MB, a 99% reduction.

FROM golang:1.22 AS build
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -ldflags="-s -w" -o app .

FROM scratch
COPY --from=build /app/app /app
CMD ["/app"]

docker build -t go-demo:scratch .
docker image inspect go-demo:scratch --format '{{.Size}}'
# Typical output: ~6MB

6.2 Java: JDK to JRE

FROM eclipse-temurin:21-jdk AS build
WORKDIR /app
COPY . .
RUN ./mvnw package -DskipTests

FROM eclipse-temurin:21-jre
COPY --from=build /app/target/app.jar /app.jar
CMD ["java", "-jar", "/app.jar"]

For Java 11 and later, jlink can go further by assembling a custom minimal JRE containing only the modules your application actually uses, which removes the portions of the JRE that your code never touches:

FROM eclipse-temurin:21-jdk AS jlink-stage
RUN jlink \
  --add-modules java.base,java.logging,java.sql \
  --strip-debug \
  --no-man-pages \
  --no-header-files \
  --compress=2 \
  --output /jre-minimal

FROM debian:bookworm-slim
COPY --from=jlink-stage /jre-minimal /jre
COPY --from=build /app/target/app.jar /app.jar
CMD ["/jre/bin/java", "-jar", "/app.jar"]

6.3 Python: Slim Runtime

FROM python:3.11 AS build
WORKDIR /app
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

FROM python:3.11-slim
COPY --from=build /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["python", "app.py"]

6.4 Rust: Scratch Final Stage

Rust compiles to static binaries similarly to Go, and images can go from 1.4GB at build time to under 5MB in the final stage when using scratch as the runtime base.

FROM rust:1.77 AS build
WORKDIR /app
COPY . .
RUN cargo build --release && strip target/release/myapp

FROM scratch
COPY --from=build /app/target/release/myapp /myapp
CMD ["/myapp"]

7. Beyond Multistage: The Full Optimisation Stack

Multistage builds are essential, but several additional techniques are required for fully production grade optimisation, and each one compounds the gains from the others.

7.1 Minimal Base Images

Choosing the right base image for the runtime stage is as important as the multistage split itself.

# Google distroless for Node.js
FROM gcr.io/distroless/nodejs20

7.2 Layer Consolidation

Every RUN instruction creates a layer, so package installation and cleanup should be combined into a single instruction to ensure the package manager cache is never committed to any layer:

RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*

7.3 A Comprehensive .dockerignore

Without this file, your entire working directory enters the build context, including .git, node_modules, test fixtures, and secret files. A minimal baseline looks like this:

.git
node_modules
*.log
.env
.env.*
coverage/
dist/
*.md
.DS_Store

7.4 BuildKit Cache Mounts

BuildKit has been the default builder since Docker Engine 23.0 and supports persistent cache mounts that survive between builds, which is particularly powerful for package managers. The cache mount persists between builds on the same builder so only new or changed packages are downloaded, which can reduce npm install or pip install times by 80 to 90% on subsequent builds.

# syntax=docker/dockerfile:1
FROM node:20-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY . .
RUN npm run build

# Python equivalent
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

# apt-get equivalent
RUN --mount=type=cache,target=/var/cache/apt \
    --mount=type=cache,target=/var/lib/apt \
    apt-get update && apt-get install -y curl

7.5 BuildKit Secret Mounts

Build args and COPY instructions bake secrets into image layers where they remain visible in docker history, so credentials should never be copied into an image even temporarily. Secret mounts solve this cleanly, making the secret available only during that specific RUN instruction without writing anything to any layer.

# syntax=docker/dockerfile:1
FROM node:20-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

docker build --secret id=npmrc,src=$HOME/.npmrc -t myapp .

You can verify the secret is absent from the final image by inspecting the layer history and running the container:

docker history myapp
# No .npmrc reference in any layer
docker run --rm myapp cat /root/.npmrc
# cat: can't open '/root/.npmrc': No such file or directory

7.6 BuildKit Parallel Stage Execution

BuildKit automatically identifies independent build stages and runs them in parallel, so structuring your Dockerfile to maximise independent work translates directly into faster total build time:

# These two stages run in parallel because they share no dependencies
FROM node:20-alpine AS frontend-deps
WORKDIR /frontend
COPY frontend/package*.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM python:3.12-slim AS backend-deps
WORKDIR /backend
COPY backend/requirements.txt ./
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt

# Final stage assembles both
FROM nginx:alpine
COPY --from=frontend-deps /frontend/node_modules ./node_modules
COPY --from=backend-deps /backend /app/backend

7.7 Partial Stage Builds for Development

Named stages allow you to build only the portion of the Dockerfile you need locally, which keeps iteration loops fast without maintaining a separate development Dockerfile:

FROM node:20-alpine AS deps
RUN npm ci

FROM deps AS test
RUN npm test

FROM deps AS build
RUN npm run build

FROM gcr.io/distroless/nodejs20 AS production
COPY --from=build /app/dist ./dist

# Run just the test stage locally without building production
docker build --target test -t myapp:test .

# Build only the production image in CI
docker build --target production -t myapp:prod .

7.8 Image Analysis

Before declaring an image production ready, it is worth inspecting what is actually inside it using the tools Docker provides alongside purpose-built analysis utilities:

# Show layer sizes and commands
docker history --no-trunc myapp:prod

# Inspect image metadata
docker image inspect myapp:prod

# Use dive for interactive layer exploration (install separately)
dive myapp:prod

# Generate SBOM for supply chain compliance
docker sbom myapp:prod

8. Production Grade Example: Node.js

A fully optimised Node.js container combining all of the techniques above demonstrates how the individual gains stack together into a substantially different outcome:

# syntax=docker/dockerfile:1
# Stage 1: Production dependencies only
FROM node:20-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --only=production

# Stage 2: Build (includes dev dependencies)
FROM node:20-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY . .
RUN npm run build

# Stage 3: Minimal production runtime
FROM gcr.io/distroless/nodejs20
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER nonroot
CMD ["dist/app.js"]

Build and measure:

docker build -t myapp:prod .
docker image inspect myapp:prod --format '{{.Size}}'

9. Real Impact

Optimised containers deliver measurable improvements across every dimension of the delivery pipeline, and the numbers below reflect what a straightforward migration from a single stage naive build to a multistage optimised build looks like in practice.

These improvements translate directly into faster deployments, lower egress and storage costs, improved autoscaling responsiveness, reduced operational risk from supply chain exposure, and cleaner architectural separation between build and runtime concerns.

10. What Most Teams Miss

Multistage builds are not an advanced optimisation technique reserved for platform teams with dedicated container expertise. They are a baseline requirement for any production system, and teams that do not use them are shipping their build toolchain into production environments and introducing unnecessary risk, inefficiency, and complexity at every level of the stack. The BuildKit cache mount feature is frequently overlooked despite being available since Docker 18.09 and the default since Docker Engine 23.0, and for teams running CI pipelines where every build installs hundreds of packages from scratch, enabling cache mounts is one of the highest leverage changes available and requires minimal effort to implement. The difference between amateur and production container systems is not tooling. It is discipline in how images are constructed, what enters each layer, and whether the runtime environment contains only what execution actually requires.

References

• Docker multistage builds documentation
• Docker build best practices
• BuildKit documentation
• Optimise cache usage in builds
• moby/buildkit on GitHub
• DevToolbox: Docker multistage builds guide 2026
• OneUptime: BuildKit cache mounts and secrets

Academic research

• https://arxiv.org/abs/2602.15214
• https://arxiv.org/abs/2312.13888
• https://arxiv.org/abs/2504.01742