0%
Deploying to Production

Production

What separates a container that works from a service you can operate.

10-15 min

Deploying to Production

An image that starts correctly on your workstation is not a production-ready image, and the gap between the two can’t be closed with one more command. It’s closed by a series of decisions that nothing forces you to make, which is exactly why they’re so often forgotten.

The incidents that result have a recognizable signature, because they repeat everywhere. A server whose disk hits 100% within three weeks: nobody configured log rotation, and Docker keeps logs without any limit by default. A container with a memory leak that ends up taking down the neighboring database, for lack of a cap. A deployment that cuts the service for forty seconds because the process never receives the stop signal. Traffic routed to a “started” container that isn’t accepting connections yet, because nothing distinguished “launched” from “ready.” And, more quietly, an image that nobody realizes has been shipping a vulnerable library for eight months.

None of these problems is hard. All of them are invisible in development, where the service runs for ten minutes, memory is plentiful, and an outage bothers no one. They only appear with time and load, which means at the worst possible moment.

This final lesson walks through these decisions one by one. The goal isn’t to turn a server into a cluster: Compose on a single host is a perfectly legitimate architecture for many services. The goal is for that server to run for six months without an emergency intervention.

A Production Image: Minimal and Pinned

A smaller image transfers faster, starts faster, and offers less exploitable code; a pinned image rebuilds identically six months from now. The multi-stage build from lesson 3 does most of the work; what remains is choosing the final base.

BaseApprox. sizeContentsDebugging
debian:12~120 MBFull distributionComfortable
python:3.12-slim~45 MBSlimmed-down Debian, glibcShell available
alpine:3.20~8 MBmusl instead of glibcsh shell
distroless~20 MBNo shell, no package managerNo exec
scratch0Nothing at allNone

The rule isn’t “the smallest wins.” Alpine imposes musl, which breaks some precompiled binaries and changes DNS resolution behavior; -slim variants often offer the best balance between size and peace of mind. Distroless and scratch give the smallest attack surface at the cost of debuggability: without a shell, docker exec is useless, and you have to rely entirely on logs and metrics.

For pinning, the tag isn’t enough in production: a tag is mutable, a digest is not.

# A tag can point to different content tomorrow
FROM node:22-slim

# A digest identifies exact content, forever
FROM node:22-slim@sha256:d3e2b17e0f5a9b1d9c8f6e5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d9e8f7a6b5c

You can read the digest to pin with docker inspect node:22-slim --format '{{index .RepoDigests 0}}'.

Finally, if your servers and workstations don’t share the same architecture — a common case since Apple Silicon Macs — build multi-architecture images with docker buildx build --platform linux/amd64,linux/arm64 --push. The --push is not optional: such an image is an index referencing several manifests, a structure the classic local cache can’t store.

Healthchecks: Distinguishing “Started” from “Ready”

Without a healthcheck, Docker knows only one useful state: the process is running, or it isn’t. But a running process may be stuck on a lock, may have lost its database connection, or may not have finished loading its cache.

# --interval     : how often to check under normal conditions
# --timeout      : beyond this, the check counts as a failure
# --start-period : grace period at startup, failures not counted
# --retries      : consecutive failures before the "unhealthy" status
HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
  CMD node -e "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"

The choice of test command matters. curl -f is the usual reflex, but it assumes curl is installed — not true on a minimal image, and installing it solely for the healthcheck increases the attack surface. Using the runtime that’s already present, as above with Node, avoids this trade-off.

--start-period is the most often overlooked parameter and the one that causes the most false positives. A Java application, or one that applies migrations at startup, can take thirty seconds to respond: without a grace period, it’s declared unhealthy, restarted, and never manages to start. The result is a restart loop with no apparent cause.

On the application side, the endpoint must check something real. A route that returns 200 without testing anything detects no failure.

app.get('/health', async (req, res) => {
  try {
    await db.query('SELECT 1');     // is the database really responding?
    await cache.ping();             // and the cache?
    res.status(200).json({ status: 'ok' });
  } catch (err) {
    res.status(503).json({ status: 'degraded', error: err.message });
  }
});
docker ps    # the health status appears in the STATUS column

# Check history, with the output of each attempt:
# essential to understand WHY a service is unhealthy
docker inspect --format '{{json .State.Health}}' mon-api

A word of honesty: Docker alone does nothing with an unhealthy container; it doesn’t restart it. The status is used by Compose (condition: service_healthy), by orchestrators, and by your monitoring — and monitoring is what should alert you.

Resource Limits: Preventing One Container from Killing the Others

Without limits, a container can consume all of the host’s memory and CPU. The kernel eventually invokes its OOM killer, and nothing guarantees it will pick the culprit: it’s often the hungriest process that gets killed, which means your database.

docker run -d \
  --memory 512m \             # strict memory cap
  --memory-swap 512m \        # equal to --memory: forbids any swap
  --memory-reservation 256m \ # soft limit, enforced under pressure
  --cpus 1.5 \                # 1.5 cores at most
  --pids-limit 200 \          # max number of processes (anti fork bomb)
  --restart unless-stopped \
  mon-api:1.2.3

The --memory-swap setting deserves an explanation. If it isn’t specified, Docker allows a swap volume equal to the memory limit: a leaking process then slows the machine down enormously instead of being stopped. Setting both to the same value forbids swap and makes the container fail outright — almost always preferable, because a clean failure gets detected whereas diffuse slowness does not.

In Compose, these limits go through deploy.resources, honored by Compose v2 even outside a cluster:

services:
  api:
    image: registre.exemple.com/equipe/mon-api:1.2.3
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 512M
        reservations:
          memory: 256M
    pids_limit: 200
    restart: unless-stopped

How do you choose the values? Observe consumption under normal load with docker stats, then add 50 to 100% headroom. And keep in mind the signature of a cap that’s too low: exit code 137, which is 128 + 9, a SIGKILL sent by the kernel.

docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
docker inspect mon-api --format '{{.State.ExitCode}} {{.State.OOMKilled}}'

Logging: The Leading Cause of Full Disks

The default driver, json-file, applies no rotation. A chatty application writes indefinitely to /var/lib/docker/containers/<id>/<id>-json.log. On a server that has been running for six months, this single file can reach several tens of gigabytes and fill the partition — which brings every other service on the machine to its knees.

The fix can be set per container (--log-opt max-size=10m --log-opt max-file=3, meaning at most three files of 10 MB), but best practice is to set it once and for all at the daemon level, so the default is safe for everything that will be launched on the machine.

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "live-restore": true
}

A sudo systemctl reload docker applies the file, and docker info --format '{{.LoggingDriver}}' confirms it was picked up.

live-restore is worth knowing: it lets containers keep running while the daemon restarts, which makes Docker updates far less nerve-racking. A caution, though: the daemon configuration only applies to containers created after the reload; existing ones keep theirs.

The other drivers address centralization needs — local (more compact), journald, syslog, fluentd, gelf, awslogs. One thing to know before switching: with most of them, docker logs stops working, because the engine no longer keeps anything locally. Only json-file, local, and journald allow it.

Two rules on the application side, which determine how useful everything else is. Write to standard output, never to a file inside the container. And write structured JSON rather than free text: across fifty containers, the difference between usable logs and a wall of text is decided here.

Graceful Shutdown: The Detail That Causes Service Outages

docker stop sends SIGTERM to the PID 1 process, waits ten seconds, then sends SIGKILL. If your application doesn’t handle SIGTERM, every deployment abruptly cuts in-flight requests. Three possible causes, three fixes.

First cause: the shell form of CMD. PID 1 is then /bin/sh, which doesn’t forward signals.

# PID 1 = sh. Your application NEVER receives SIGTERM.
CMD node server.js

# PID 1 = node. The signal arrives.
CMD ["node", "server.js"]

Second cause: the application doesn’t handle the signal.

process.on('SIGTERM', () => {
  server.close(async () => {      // no new connections
    await db.end();               // close the pool cleanly
    process.exit(0);
  });
  setTimeout(() => process.exit(1), 25_000);  // safety net
});

Third cause: zombie processes. A real PID 1 must reap orphaned processes, which most applications don’t do. The --init option inserts a minimal manager (tini) that handles this and forwards signals.

services:
  api:
    image: registre.exemple.com/equipe/mon-api:1.2.3
    init: true
    # Grace period: must exceed the duration of your graceful shutdown
    stop_grace_period: 30s
# Time the shutdown. Close to 10 s (or to stop_grace_period) =>
# the signal isn't being handled. Under one second => you're good.
time docker stop mon-api

Private Registry: Pushing and Pulling Your Images

In production, you don’t build on the server: you pull an image that has already been built and tested. That requires a registry.

# --password-stdin prevents the secret from appearing in
# the shell history or in the output of "ps".
echo "$REGISTRY_TOKEN" | docker login registre.exemple.com \
  -u ci-bot --password-stdin

docker tag mon-api:1.2.3 registre.exemple.com/equipe/mon-api:1.2.3
docker push registre.exemple.com/equipe/mon-api:1.2.3
docker logout registre.exemple.com

A tagging convention to adopt from the start: two tags per build, one readable, one traceable.

DEPOT="registre.exemple.com/equipe/mon-api"
docker build -t "$DEPOT:1.2.3" -t "$DEPOT:$(git rev-parse --short HEAD)" .
docker push --all-tags "$DEPOT"

The commit tag answers the only question that matters at 3 a.m.: what exact code is running in production? On the server side, deployment then boils down to three commands.

docker compose pull                 # fetch the new image
docker compose up -d                # recreate services whose image changed
docker compose ps

# Touch only one service, without restarting its dependencies
docker compose up -d --no-deps api

Let’s be clear about the limits: this operation recreates the container, and therefore briefly interrupts the service. A zero-downtime deployment requires two instances behind a load balancer, which is beyond the scope of Compose alone. For most services, a one-to-two-second interruption is acceptable; when it isn’t, that’s the signal that an orchestrator has become necessary.

Scanning for Vulnerabilities and Hardening Runtime

An image ships hundreds of packages, of which you chose only a handful. Vulnerabilities are published against them continuously, even on an image you haven’t modified in six months — hence the need to rebuild regularly, and not only when the code changes.

# Summary: number of vulnerabilities by severity level
docker scout quickview mon-api:1.2.3

# Details, filtered on the levels that require action
docker scout cves --only-severity critical,high mon-api:1.2.3

# Suggestions for updating the base image
docker scout recommendations mon-api:1.2.3

Trivy, a widely used open-source tool, runs as a container and fits well into continuous integration. A warning about this command: mounting /var/run/docker.sock gives the container full control of the daemon, and therefore of the machine. Acceptable for a tool you’ve vetted, on a dedicated build agent; never harmless.

# --exit-code 1 makes the command fail if vulnerabilities of the
# requested level remain: this is how you block a delivery pipeline.
docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  aquasec/trivy:latest image \
  --severity HIGH,CRITICAL --exit-code 1 \
  mon-api:1.2.3

Runtime hardening usefully complements scanning: these options reduce what an attacker can do even in the event of an application flaw.

docker run -d \
  --read-only \                            # read-only filesystem
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \  # except /tmp, in memory
  --cap-drop ALL \                         # drop all capabilities
  --cap-add NET_BIND_SERVICE \             # then add back the only useful one
  --security-opt no-new-privileges \       # no escalation via setuid
  --user 1001:1001 \                       # never root
  mon-api:1.2.3

The --read-only flag is particularly effective: it prevents an attacker from writing a binary into the container. You just need to identify the few paths the application must be able to write to, and mount them separately.

A complete production stack

Let’s put it all together in a file you can use alongside the development compose.yaml.

# compose.prod.yaml
# Usage: docker compose -f compose.yaml -f compose.prod.yaml up -d --wait

services:
  api:
    # Image from the registry, pinned: no build on the server
    image: registre.exemple.com/equipe/mon-api:${VERSION:?VERSION est obligatoire}
    build: !reset null        # disables the build inherited from the base file
    restart: unless-stopped
    init: true
    stop_grace_period: 30s
    read_only: true
    tmpfs:
      - /tmp:size=64m
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    user: "1001:1001"
    environment:
      NODE_ENV: production
      DATABASE_URL: postgres://app:${DB_PASSWORD:?}@db:5432/appdb
    ports:
      - "127.0.0.1:3000:3000"   # a front-end proxy will handle public exposure
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 20s
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 512M
        reservations:
          memory: 256M
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    networks:
      - frontend
      - backend

  db:
    image: postgres:16-alpine
    restart: unless-stopped
    stop_grace_period: 1m     # let Postgres finish its checkpoint
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: ${DB_PASSWORD:?}
      POSTGRES_DB: appdb
    volumes:
      - db-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    deploy:
      resources:
        limits:
          memory: 1G
    networks:
      - backend               # never reachable from outside

networks:
  frontend:
  backend:
    internal: true

volumes:
  db-data:
# Always check the merged configuration before deploying
docker compose -f compose.yaml -f compose.prod.yaml config

VERSION=1.2.3 docker compose -f compose.yaml -f compose.prod.yaml up -d --wait

Note ${VERSION:?...}: a deployment without an explicit version fails immediately instead of silently falling back to latest. Also note restart: unless-stopped rather than always: the difference is that a container you stopped on purpose won’t come back after a server reboot, which avoids waking up a service you deliberately took offline.

What to monitor next

Deployment is not the end. Disk space comes first, because it is the most common cause of outages. Next come health status and the restart counter, because a rising value signals a problem that no alert may ever surface. Finally, watch resource consumption against the limits you set, to detect drift before an incident occurs.

docker system df ; df -h /var/lib/docker
docker ps --format 'table {{.Names}}\t{{.Status}}'
docker inspect mon-api --format '{{.RestartCount}}'

Then there’s image freshness: an image that hasn’t been rebuilt in months accumulates the vulnerabilities of its dependencies even if your code hasn’t changed. Periodically rebuilding and redeploying, with no functional change, is a maintenance practice, not a waste of time.

Finally, automated cleanup calls for caution. Running docker system prune -af on a server deletes unused images, including the previous version you would want to roll back to if something goes wrong. Prefer a targeted command such as docker image prune -a --filter "until=720h" --force, which only removes images that have been unused for more than thirty days.

Key takeaways

  • A production image is minimal and pinned: multi-stage build, a -slim or distroless base depending on your diagnostic needs, and referenced by digest rather than by a mutable tag.
  • A healthcheck with a realistic --start-period distinguishes “started” from “ready.” The endpoint must test the database and the cache, otherwise it detects nothing. Docker does not restart an unhealthy container: it is a signal for Compose and for your monitoring.
  • Always set --memory and --cpus. Without a cap, a memory leak in one container brings down its neighbors. Exit code 137 means “killed for exceeding memory.”
  • Configure log rotation in /etc/docker/daemon.json, once and for all. The default json-file driver without rotation is the leading cause of full disks on a Docker server.
  • Check that docker stop takes less than a second. Otherwise your application is not handling SIGTERM — shell form of CMD, or missing signal handler — and every deployment cuts off in-flight requests.
  • Don’t build on the production server: push an image tagged with both its version and its commit, then run docker compose pull && docker compose up -d.
  • Scan your images (docker scout cves, Trivy) and rebuild them periodically even without code changes. Harden runtime with --read-only, --cap-drop ALL, no-new-privileges, and a non-root user.

Going further: when a single host is no longer enough — high availability, rolling deployments with no downtime, automatic scaling — you’re in orchestrator territory. Everything you’ve learned here still applies: Kubernetes consumes the same OCI images and expects exactly the same healthchecks, resource limits, and logs on standard output. The complete reference for all options remains docs.docker.com.

Comments

Comments are powered by GitHub Discussions

Sign in with GitHub to join the discussion

Link copied!