Running Docker in development is easy. Running it reliably in production is a different challenge. When containers fail silently or consume all available memory, you need monitoring, logging, and troubleshooting skills. This lesson covers how to keep your production containers healthy, visible, and debuggable — a critical bridge between running Docker locally and migrating to Kubernetes orchestration.

1. Learning Objectives

By the end of this lesson, you will be able to:

  • Monitor container resource usage with docker stats and docker events
  • Configure structured logging with Docker log drivers
  • Implement health checks for automatic container recovery
  • Set CPU and memory limits to prevent resource starvation
  • Use docker exec and docker inspect for live debugging
  • Set up cAdvisor and Prometheus for production monitoring
  • Troubleshoot common production container failures

2. Why This Matters

Real-world scenario: Your team deploys a Node.js API in a container to production. Two days later, users report timeouts. The container is still running, but memory usage has climbed to 95% and the health check endpoint returns 503. Without monitoring, you wouldn't even know the container is degrading — it's still running, just not serving traffic. Production Docker requires observability: CPU/memory stats, structured logs, health probes, and alerting. These skills are also your foundation for Kubernetes, where Pod health probes and resource limits are mandatory.

3. Core Concepts

Container Resource Monitoring with docker stats

# Live resource usage for all running containers
docker stats

# Stats for specific containers (with no-stream for one-shot)
docker stats --no-stream myapp nginx redis

# Format as JSON for scripting
docker stats --no-stream --format '{{json .}}' myapp

# CPU usage as percentage, memory in MB
docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'

# Watch container events in real time
docker events --filter 'event=oom' --filter 'event=die'

# Historical events (since a timestamp)
docker events --since '2025-02-10T00:00:00' --until '2025-02-11T00:00:00'
Real-time container monitoring

Resource Constraints: CPU and Memory Limits

# Hard memory limit (container hits the wall at 512MB)
docker run -d --name myapp --memory=512m myapp:latest

# Soft memory limit (warns before hard limit)
docker run -d --name myapp --memory=512m --memory-reservation=256m myapp:latest

# CPU limits
# --cpus: fractional CPU cores
docker run -d --name myapp --cpus=1.5 myapp:latest

# CPU shares (relative weight, default 1024)
docker run -d --name myapp --cpu-shares=2048 myapp:latest

# Pin to specific CPU cores
docker run -d --name myapp --cpuset-cpus=0,1 myapp:latest

# Combined production run
docker run -d \
  --name myapp \
  --memory=512m \
  --memory-reservation=384m \
  --cpus=1.0 \
  --restart=unless-stopped \
  -p 8080:3000 \
  myapp:latest

# Check current limits
docker inspect myapp --format '{{json .HostConfig.Memory}}' | xargs echo "Memory limit bytes:"
docker inspect myapp --format '{{json .HostConfig.NanoCpus}}' | xargs echo "CPU limit (nano-CPUs):"
Resource constraints in Docker

Health Checks for Automatic Recovery

# HEALTHCHECK instruction in Dockerfile
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

# Or at runtime (overrides Dockerfile HEALTHCHECK)
docker run -d \
  --name myapp \
  --health-cmd='curl -f http://localhost:3000/health || exit 1' \
  --health-interval=30s \
  --health-timeout=3s \
  --health-start-period=15s \
  --health-retries=3 \
  myapp:latest

# Check health status
docker inspect --format '{{.State.Health.Status}}' myapp
docker inspect --format '{{json .State.Health}}' myapp | jq

# View health check logs
docker inspect myapp | jq '.[].State.Health.Log[-1]'

# Auto-restart only on non-healthy state (with restart policy)
docker update --restart=unless-stopped myapp
Health checks for self-healing containers

Structured Logging with Log Drivers

# Default: json-file driver (may fill disk)
docker run -d --name myapp myapp:latest

# Limit log file size and rotation
docker run -d \
  --name myapp \
  --log-driver=json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  myapp:latest

# Syslog driver (ship to central logging)
docker run -d \
  --name myapp \
  --log-driver=syslog \
  --log-opt syslog-address=tcp://logs.example.com:514 \
  --log-opt tag='{{.Name}}/{{.ID}}' \
  myapp:latest

# Fluentd driver (stream to EFK/ELK)
docker run -d \
  --name myapp \
  --log-driver=fluentd \
  --log-opt fluentd-address=localhost:24224 \
  --log-opt tag='docker.{{.Name}}' \
  myapp:latest

# GELF driver (Graylog Extended Log Format)
docker run -d \
  --name myapp \
  --log-driver=gelf \
  --log-opt gelf-address=udp://graylog.example.com:12201 \
  myapp:latest

# Configure log driver globally in daemon.json
# /etc/docker/daemon.json
# {
#   'log-driver': 'json-file',
#   'log-opts': {
#     'max-size': '10m',
#     'max-file': '3'
#   }
# }

# View logs with filtering and timestamps
docker logs --tail=50 --timestamps myapp
docker logs --since=5m myapp
docker logs --until=2025-02-10T12:00:00 myapp

# Follow logs from multiple containers
docker logs -f myapp redis nginx
Docker logging drivers and best practices

cAdvisor: Production Container Monitoring

# Deploy cAdvisor (container advisor) - single host
# From Google: collects CPU, memory, network, and disk metrics
docker run -d \
  --name=cadvisor \
  --restart=unless-stopped \
  -p 8080:8080 \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --privileged \
  --device=/dev/kmsg \
  gcr.io/cadvisor/cadvisor:latest

# Access cAdvisor UI
# http://localhost:8080

# Docker Compose setup with Prometheus
version: '3.8'
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    ports:
      - '8080:8080'
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    privileged: true
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    ports:
      - '9090:9090'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    restart: unless-stopped

# prometheus.yml
scrape_configs:
  - job_name: 'cadvisor'
    scrape_interval: 15s
    static_configs:
      - targets: ['cadvisor:8080']

# Access Prometheus: http://localhost:9090
# Query examples in PromQL:
#   rate(container_cpu_usage_seconds_total[1m])
#   container_memory_usage_bytes
#   container_network_receive_bytes_total
cAdvisor and Prometheus for production monitoring

Live Debugging and Troubleshooting

# Inspect container configuration
docker inspect myapp

# Check exit code and reason
docker inspect --format '{{.State.Status}}: {{.State.ExitCode}} ({{.State.Error}})' myapp

# View container logs
docker logs --tail=100 myapp

# Execute commands in running container
docker exec -it myapp sh -c 'top -b -n 1'
docker exec myapp sh -c 'ps aux'
docker exec myapp sh -c 'netstat -tulpn'
docker exec myapp sh -c 'df -h'
docker exec myapp sh -c 'free -m'

# Debug with a sidecar container (same network namespace)
docker run -it --rm --network container:myapp nicolaka/netshoot /bin/bash
# Inside netshoot: tcpdump, curl, dig, nmap available

# Copy files from container
docker cp myapp:/var/log/app.log ./app.log

# Save and inspect container filesystem
docker export myapp -o myapp_fs.tar
tar tf myapp_fs.tar | grep error

# Check container resource usage history
docker stats --no-stream --all
Live container debugging techniques

4. Hands-On Practice: Production Docker Setup

Set up a production-ready Docker deployment with monitoring, health checks, and proper resource limits. This exercise uses a sample Node.js application with a health endpoint.

# 1. Create a sample app Dockerfile with health check
cat > Dockerfile << 'DOCKERFILE'
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000

# Health check endpoint at /health
HEALTHCHECK --interval=15s --timeout=5s --start-period=30s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1

USER node
CMD ['node', 'server.js']
DOCKERFILE

# 2. Simple health endpoint (server.js snippet)
const express = require('express');
const app = express();
app.get('/health', (req, res) => {
  const mem = process.memoryUsage();
  const uptime = process.uptime();
  res.json({ status: 'healthy', uptime, memory: mem });
});

# 3. Build and run with production flags
docker build -t myapp-prod .
docker run -d \
  --name myapp-prod \
  --memory=256m \
  --memory-reservation=192m \
  --cpus=0.5 \
  --restart=unless-stopped \
  -p 3000:3000 \
  myapp-prod

# 4. Verify health status
docker inspect --format '{{.State.Health.Status}}' myapp-prod

# 5. Check resource usage
docker stats --no-stream myapp-prod

# 6. Simulate a memory spike (watch for OOM)
docker exec myapp-prod sh -c 'stress --vm 1 --vm-bytes 300M --timeout 10s'
# The container will crash with OOM since limit is 256m
# docker restart will bring it back (restart policy)

# 7. View the crash event
docker events --since '1m' --filter 'container=myapp-prod'

# 8. Deploy cAdvisor for full visibility
docker run -d \
  --name=cadvisor \
  --restart=unless-stopped \
  -p 8080:8080 \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --privileged \
  gcr.io/cadvisor/cadvisor:latest

# 9. Access cAdvisor
# Open http://localhost:8080 in your browser

# 10. Cleanup
docker stop myapp-prod cadvisor
docker rm myapp-prod cadvisor
Complete production Docker setup walkthrough

5. Common Errors and Solutions

Error 1: OOMKilled (Exit Code 137)

  • Cause: Container exceeded its memory limit; kernel OOM killer terminated it
  • Fix: Increase --memory or optimize the app. Use --memory-reservation as a soft limit. Monitor with docker events --filter 'event=oom'

Error 2: Container exits immediately (Exit Code 1 or 139)

  • Cause: App crashed on startup (code 1) or segfault (code 139)
  • Fix: Check docker logs --tail=50 myapp for the error. Use docker run -it --rm myapp sh to debug interactively. Ensure ENTRYPOINT and CMD are correct

Error 3: Disk full from container logs

  • Cause: Default json-file log driver has no rotation; logs fill /var/lib/docker
  • Fix: Add --log-opt max-size=10m --log-opt max-file=3. Truncate existing logs: truncate -s 0 /var/lib/docker/containers/*/*-json.log

Error 4: Health check endpoint unreachable

  • Cause: Health check fires before the app is ready (too short --start-period)
  • Fix: Increase --health-start-period to 30–60s for apps with slow startup. Verify the port is correct in the health check command

Error 5: Container is running but not serving traffic

  • Cause: App is alive (not crashing) but stuck or deadlocked (e.g., database connection pool exhausted)
  • Fix: Implement a proper health check that tests dependencies. Use docker exec to inspect threads: docker exec myapp sh -c 'kill -3 1' (triggers thread dump in Java, stack trace in Node)

Error 6: Port already allocated

  • Cause: Another container or process is using the host port
  • Fix: Find the conflict: netstat -tulpn | grep :8080. Use docker ps to check running containers. Change the host port mapping

6. Summary Checklist

  • I can monitor containers with docker stats and docker events
  • I can set CPU and memory limits on containers
  • I can add HEALTHCHECK to a Dockerfile or at runtime
  • I can configure log rotation and structured log drivers
  • I can deploy cAdvisor to monitor container health
  • I can debug a crashed container using logs and inspect
  • I understand OOMKilled and how to prevent it
  • I can troubleshoot port conflicts and disk space issues

7. Practice Exercise

Challenge: Debug and fix a failing production container

  1. Run docker run -d --name debug-challenge --memory=64m nginx:alpine
  2. This container has too little memory. Check its health: docker ps -a — notice it may crash or be unhealthy
  3. Use docker logs debug-challenge and docker inspect debug-challenge to diagnose
  4. Fix by running with --memory=128m and adding a health check that tests http://localhost:80
  5. Set up log rotation with max-size=5m and max-file=2
  6. Add a restart policy of unless-stopped
  7. Deploy cAdvisor and verify the container appears in its dashboard (http://localhost:8080)
  8. Bonus: Write a shell script that alerts when a container's health status becomes 'unhealthy' for more than 3 checks

8. Next Steps

You've covered the full lifecycle of Docker in production — from resource limits and health checks to monitoring and troubleshooting. The next category in the DevOps roadmap is Kubernetes: Container Orchestration. Kubernetes builds on everything you've learned here: Pod health probes (liveness, readiness, startup), resource requests and limits, container logging with kubectl logs, and monitoring with Prometheus operator. Your Docker production skills are the direct foundation for K8s operations.

Next lesson in the Docker series: This concludes the Docker series. Continue to the Kubernetes category for container orchestration at scale.

Tags: docker, monitoring, logging, troubleshooting, production, containers, health-checks, resource-management, devops

Gataya Med

DevOps Engineer & Backend Developer. Sharing insights on cloud, automation, and scalable systems.

gatayaMed View Profile @https://github.com/gatayaMed

Comments (0)

Sarah Chen July 25, 2026

This is exactly what I needed! The initContainer approach solved our migration issues completely. Thanks for the detailed guide!

Reply

Leave a Comment