Skip to main content

Docker Compose Fundamentals — Defining Multi-Container Applications

Define and run multi-container applications with Docker Compose — covering the compose file schema, service configuration, and essential CLI commands.

44 Terms

Overview and What You Will Learn

Running a web application locally used to mean installing PostgreSQL, Redis, and RabbitMQ on your laptop, managing three different service configs, and hoping nothing conflicted with another project. Docker Compose replaces all of that with a single YAML file and a single command.

In this guide you will learn the complete Docker Compose file schema, how services, networks, and volumes work together, and every essential CLI command for managing a Compose application.

Core Principles

Bash
+------------------------------------------+
| docker-compose.yml |
| |
| services: <- what to run |
| api: <- one container |
| postgres: <- another container |
| redis: <- another container |
| |
| networks: <- how they connect |
| app-network: |
| |
| volumes: <- where data lives |
| postgres-data: |
+------------------------------------------+
|
| docker compose up -d
|
v
+------------------------------------------+
| Three containers running |
| Connected on app-network |
| Data persisted in postgres-data volume |
| api reaches postgres by name |
+------------------------------------------+

Detailed Step-by-Step Practical Lab

Milestone 1: The Complete docker-compose.yml Schema
YAML
version: "3.8" # Compose file format version — use 3.8 for most projects
services:
# ── Service Definition ─────────────────────────────
payment-api:
# What image to use (or build)
image: registry.razorpay.in/payment-api:v3.1.0
# Or build from a Dockerfile
build:
context: . # Build context directory
dockerfile: Dockerfile # Which Dockerfile (default: Dockerfile)
args:
BUILD_ENV: production
# Container name (otherwise: projectname_servicename_1)
container_name: payment-api
# Port mapping: host:container
ports:
- "8080:8080"
- "127.0.0.1:9090:9090" # Only localhost can access this port
# Environment variables
environment:
NODE_ENV: production
DB_HOST: postgres # Use service name as hostname
DB_PORT: "5432"
DB_NAME: payments
# Load env vars from a file (keeps secrets out of compose file)
env_file:
- .env
- .env.production
# Volume mounts
volumes:
- ./config:/app/config:ro # Bind mount, read-only
- payment-data:/app/data # Named volume
- type: tmpfs
target: /tmp # In-memory storage
# Which networks this service joins
networks:
- app-network
# Service dependency ordering
depends_on:
postgres:
condition: service_healthy # Wait until postgres is healthy
redis:
condition: service_started # Just wait until redis starts
# Restart policy
restart: unless-stopped
# Resource limits
deploy:
resources:
limits:
cpus: "1.5"
memory: 512M
reservations:
cpus: "0.25"
memory: 128M
# Health check
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 15s
# Override the default command
command: ["node", "dist/server.js", "--port", "8080"]
# ── Database Service ───────────────────────────────
postgres:
image: postgres:15-alpine
environment:
POSTGRES_DB: payments
POSTGRES_USER: api_user
POSTGRES_PASSWORD_FILE: /run/secrets/db-password # Secret from file
volumes:
- postgres-data:/var/lib/postgresql/data
- ./db/init.sql:/docker-entrypoint-initdb.d/init.sql # Init script
networks:
- app-network
healthcheck:
test: ["CMD-SHELL", "pg_isready -U api_user -d payments"]
interval: 10s
timeout: 5s
retries: 5
# ── Cache Service ──────────────────────────────────
redis:
image: redis:7-alpine
command: redis-server --appendonly yes # Enable persistence
volumes:
- redis-data:/data
networks:
- app-network
# ── Networks ─────────────────────────────────────────
networks:
app-network:
driver: bridge
# ── Volumes ──────────────────────────────────────────
volumes:
postgres-data: # Docker creates and manages this
redis-data:
payment-data:
Milestone 2: Essential Compose CLI Commands
Bash
# Start all services in background
docker compose up -d
# Start and rebuild images first
docker compose up -d --build
# Stop and remove containers (keeps volumes and networks)
docker compose down
# Stop, remove containers, AND remove named volumes (DELETES DATA)
docker compose down -v
# View status of all services
docker compose ps
# NAME IMAGE STATUS PORTS
# payment-api payment-api Up 5 minutes 0.0.0.0:8080->8080/tcp
# postgres postgres:15 Up 5 minutes (healthy)
# redis redis:7 Up 5 minutes
# View logs from all services
docker compose logs
# Follow logs from a specific service
docker compose logs -f payment-api
# Run a command in a running service container
docker compose exec postgres psql -U api_user -d payments
# Run a one-off command (starts a new container)
docker compose run --rm payment-api npm run migrate
# Scale a service (run multiple instances)
docker compose up -d --scale payment-api=3
# Restart a specific service
docker compose restart payment-api
# Pull latest images for all services
docker compose pull
# Build all services that have a build context
docker compose build
# See resource usage of running services
docker compose top
Milestone 3: Compose v1 vs Compose v2
Bash
# Compose v1: separate binary (docker-compose)
# Installed separately, uses Python, slower
docker-compose up -d # Old way
# Compose v2: built into Docker CLI as a plugin
# Faster, Go-based, installed with Docker Desktop and Docker Engine 20.10+
docker compose up -d # New way (space, not hyphen)
# Check which version you have
docker compose version
# Docker Compose version v2.23.0
# The compose file format is the same — only the binary changed
# Always use 'docker compose' (v2) on modern systems

Common Mistakes

Mistake Consequence Fix
Using depends_on without health checks Service starts before dependency is ready Add condition: service_healthy with a healthcheck
Hardcoding secrets in compose file Secrets in version control Use env_file with .env in .gitignore
No resource limits One service starves others Always set deploy.resources.limits
docker compose down -v by accident All database data deleted permanently Know exactly what -v does before using it
Using container_name with scaling Conflict when scaling to multiple instances Remove container_name for services you want to scale
Tip

Use docker compose config to see the fully resolved compose file with all variable substitutions applied. This is the fastest way to verify that your .env file variables are being read correctly and that all service configurations look as expected.

Remember

docker compose down keeps your volumes (your data is safe). docker compose down -v deletes your volumes (your data is gone permanently). This is the most common cause of accidental data loss with Docker Compose.


JSON
{
"title": "Docker Compose for Local Development — Full Stack in One Command",
"slug": "docker-compose-local-development",
"cluster": "docker",
"description": "Build a complete local development stack with Docker Compose — API server, PostgreSQL, Redis, and a message queue — with hot reloading and proper config.",
"primaryKeyword": "docker compose local development"
}

Docker Compose for Local Development — Full Stack in One Command

Overview and What You Will Learn

Every developer at CRED or Zerodha runs a complex application stack locally. Instead of installing PostgreSQL, Redis, and RabbitMQ on every developer's laptop — and dealing with version conflicts, OS differences, and "works on my machine" bugs — the team uses a single docker compose up command that gives every developer the exact same environment.

Core Principles

The ideal local development Compose file has three properties: hot reloading (code changes appear instantly without rebuild), isolated data (each developer has their own database), and production parity (the same images and config as production, with minor development overrides).

Complete Local Development Stack

YAML
# docker-compose.yml — base configuration
version: "3.8"
services:
api:
build:
context: .
dockerfile: Dockerfile.dev # Development dockerfile with dev tools
ports:
- "3000:3000"
- "9229:9229" # Node.js debugger port
volumes:
- ./src:/app/src # Bind mount for hot reloading
- /app/node_modules # Keep container's node_modules
environment:
NODE_ENV: development
DB_HOST: postgres
REDIS_HOST: redis
AMQP_URL: amqp://rabbitmq:5672
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_started
command: ["npm", "run", "dev"] # Dev command with nodemon/ts-node
postgres:
image: postgres:15-alpine
ports:
- "5432:5432" # Expose for local DB GUI (TablePlus, etc.)
environment:
POSTGRES_DB: myapp_dev
POSTGRES_USER: developer
POSTGRES_PASSWORD: devpassword123
volumes:
- postgres-dev-data:/var/lib/postgresql/data
- ./db/seed.sql:/docker-entrypoint-initdb.d/seed.sql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U developer -d myapp_dev"]
interval: 5s
timeout: 3s
retries: 5
redis:
image: redis:7-alpine
ports:
- "6379:6379" # Expose for redis-cli on host
rabbitmq:
image: rabbitmq:3-management-alpine
ports:
- "5672:5672" # AMQP port
- "15672:15672" # Management UI — http://localhost:15672
environment:
RABBITMQ_DEFAULT_USER: developer
RABBITMQ_DEFAULT_PASS: devpassword123
volumes:
- rabbitmq-dev-data:/var/lib/rabbitmq
mailhog:
image: mailhog/mailhog # Email capture for development
ports:
- "1025:1025" # SMTP port
- "8025:8025" # Web UI — http://localhost:8025
# Send emails to localhost:1025, view them in browser at :8025
volumes:
postgres-dev-data:
rabbitmq-dev-data:
Dockerfile
# Dockerfile.dev — development-specific
FROM node:20-alpine
WORKDIR /app
# Install development tools
RUN apk add --no-cache git curl
COPY package.json package-lock.json ./
RUN npm install # Install ALL deps including devDependencies
# Source is mounted via bind mount — not copied here
EXPOSE 3000 9229
CMD ["npm", "run", "dev"] # nodemon watches for changes
Hot Reloading Setup
YAML
# The critical volume mounts for hot reloading:
volumes:
- ./src:/app/src # Your source code mounted into container
- /app/node_modules # Anonymous volume — keeps container's node_modules
# Without this, host's node_modules overrides container's
# (Different OS = different native modules)
TEXT
// package.json
{
"scripts": {
"dev": "nodemon --watch src --ext ts,js,json --exec ts-node src/index.ts"
}
}
Bash
# Start the full dev stack
docker compose up -d
# Watch API logs while developing
docker compose logs -f api
# Run database migrations
docker compose exec api npm run migrate
# Reset the database
docker compose exec postgres psql -U developer -c "DROP DATABASE myapp_dev;"
docker compose exec postgres psql -U developer -c "CREATE DATABASE myapp_dev;"
docker compose restart postgres
# Open a postgres shell
docker compose exec postgres psql -U developer -d myapp_dev
Override File for Development
YAML
# docker-compose.override.yml
# Auto-merged with docker-compose.yml when both are present
version: "3.8"
services:
api:
environment:
DEBUG: "*" # Enable all debug logging
LOG_LEVEL: debug
volumes:
- ./tests:/app/tests # Mount test files for running tests inside container
Tip

Expose all service ports in your local development Compose file (postgres 5432, redis 6379, rabbitmq 15672) so you can connect with GUI tools like TablePlus, RedisInsight, and the RabbitMQ management UI. Remove these port exposures in production compose files — external access is not needed.

Remember

The anonymous volume trick (- /app/node_modules) is critical on macOS and Windows. Without it, your macOS node_modules (compiled for macOS) override the container's node_modules (compiled for Linux), and native modules fail at runtime.


JSON
{
"title": "Docker Compose Health Checks and Dependency Ordering",
"slug": "docker-compose-healthchecks-dependencies",
"cluster": "docker",
"description": "Configure service health checks and correct startup ordering in Docker Compose to prevent applications from starting before their dependencies are ready.",
"primaryKeyword": "docker compose health check"
}

Docker Compose Health Checks and Dependency Ordering

Overview and What You Will Learn

The most common Docker Compose failure pattern: your API starts, tries to connect to PostgreSQL, gets connection refused because PostgreSQL is still initialising, crashes, and gets stuck in a restart loop. The fix is health checks combined with depends_on condition: service_healthy.

Core Principles

◈ DIAGRAM
WRONG: depends_on without health check
postgres starts -> api starts immediately
postgres still initialising (takes 3-5 seconds)
api tries to connect -> connection refused
api crashes -> restart loop
CORRECT: depends_on with service_healthy
postgres starts
postgres health check runs every 5s
postgres becomes healthy (pg_isready returns 0)
THEN api starts
api connects successfully -> no crash

Health Check Configuration

YAML
version: "3.8"
services:
api:
image: registry.razorpay.in/payment-api:v3.1.0
depends_on:
postgres:
condition: service_healthy # Wait until postgres passes health check
redis:
condition: service_healthy
rabbitmq:
condition: service_healthy
postgres:
image: postgres:15-alpine
environment:
POSTGRES_DB: payments
POSTGRES_USER: api_user
POSTGRES_PASSWORD: secret
healthcheck:
test: ["CMD-SHELL", "pg_isready -U api_user -d payments"]
interval: 5s # Run health check every 5 seconds
timeout: 3s # Health check must complete in 3 seconds
retries: 5 # Fail after 5 consecutive failures
start_period: 10s # Wait 10s after start before first check
# Prevents false failures during initialisation
redis:
image: redis:7-alpine
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 3
rabbitmq:
image: rabbitmq:3-alpine
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "ping"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s # RabbitMQ takes longer to start
Health Check Command Examples
YAML
# HTTP endpoint check (requires curl in image)
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
# Use wget if curl is not available (Alpine images)
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1",
"--spider", "http://localhost:8080/health"]
# PostgreSQL readiness
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
# MySQL readiness
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
# Redis ping
healthcheck:
test: ["CMD", "redis-cli", "ping"]
# MongoDB readiness
healthcheck:
test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
# Simple TCP port check
healthcheck:
test: ["CMD-SHELL", "nc -z localhost 8080 || exit 1"]
# Custom application check via exit code
healthcheck:
test: ["CMD-SHELL", "node /app/scripts/healthcheck.js || exit 1"]
Monitoring Health Status
Bash
# Check health status of all services
docker compose ps
# NAME STATUS
# api Up 2 minutes (healthy)
# postgres Up 3 minutes (healthy)
# redis Up 3 minutes (healthy)
# If a service is unhealthy:
# docker compose ps shows: Up 2 minutes (unhealthy)
# See health check output for a container
docker inspect postgres-container --format \
'{{json .State.Health}}' | jq
# {
# "Status": "healthy",
# "FailingStreak": 0,
# "Log": [
# {
# "Start": "2024-01-15T09:00:00Z",
# "End": "2024-01-15T09:00:00.123Z",
# "ExitCode": 0,
# "Output": "/var/run/postgresql:5432 - accepting connections"
# }
# ]
# }
# If health check is failing, the Output field shows why
docker inspect unhealthy-container \
--format '{{(index .State.Health.Log 0).Output}}'
# FATAL: database "payments" does not exist
# -> Database not created yet — check POSTGRES_DB env var
Tip

Always set start_period on health checks for services that take time to initialise (like PostgreSQL, RabbitMQ, Elasticsearch). Without start_period, health check failures during the normal startup period count toward the failure threshold and can mark the service as unhealthy before it even had a chance to start properly.

Common Mistake

Using depends_on without condition: service_healthy. The basic depends_on: postgres only waits for the container to start — not for PostgreSQL to be ready to accept connections. PostgreSQL takes 2-5 seconds after the container starts to become ready. Without service_healthy, your API will almost certainly try to connect before PostgreSQL is ready.


JSON
{
"title": "Docker Compose in CI/CD Pipelines — GitHub Actions Integration",
"slug": "docker-compose-ci-cd",
"cluster": "docker",
"description": "Use Docker Compose in GitHub Actions CI/CD pipelines to run integration tests against real service dependencies — databases, caches, and message queues.",
"primaryKeyword": "docker compose ci cd"
}

Docker Compose in CI/CD Pipelines — GitHub Actions Integration

Overview and What You Will Learn

Integration tests that mock the database are not real integration tests. Real integration tests hit a real PostgreSQL database, a real Redis cache, and a real message queue — the same services your application uses in production. Docker Compose makes running these real dependencies in GitHub Actions free and reliable.

Core Principles

Bash
GitHub Actions runner:
ubuntu-latest has Docker pre-installed
Your CI workflow:
1. docker compose up -d (start postgres, redis, etc.)
2. Wait for services to be healthy
3. Run your integration tests against real services
4. docker compose down (cleanup)
Result: integration tests that test real behavior
not mocked behavior

Complete GitHub Actions Workflow

YAML
# .github/workflows/ci.yml
name: CI Pipeline
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
jobs:
integration-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Start services with Docker Compose
run: |
docker compose -f docker-compose.ci.yml up -d
echo "Services started"
- name: Wait for services to be healthy
run: |
echo "Waiting for postgres..."
timeout 60 bash -c \
'until docker compose -f docker-compose.ci.yml exec -T postgres \
pg_isready -U api_user -d payments; do sleep 2; done'
echo "Waiting for redis..."
timeout 30 bash -c \
'until docker compose -f docker-compose.ci.yml exec -T redis \
redis-cli ping | grep -q PONG; do sleep 1; done'
echo "All services ready"
- name: Run database migrations
run: |
docker compose -f docker-compose.ci.yml exec -T api \
npm run migrate
- name: Run integration tests
run: |
docker compose -f docker-compose.ci.yml exec -T api \
npm run test:integration
env:
CI: true
- name: Collect logs on failure
if: failure()
run: |
docker compose -f docker-compose.ci.yml logs > ci-logs.txt 2>&1
- name: Upload logs as artifact on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: ci-logs
path: ci-logs.txt
- name: Teardown services
if: always() # Run even if tests failed
run: |
docker compose -f docker-compose.ci.yml down -v
# -v removes volumes — clean slate for next run
YAML
# docker-compose.ci.yml — CI-specific compose file
version: "3.8"
services:
api:
build: .
environment:
NODE_ENV: test
DB_HOST: postgres
DB_PORT: "5432"
DB_NAME: payments_test
DB_USER: api_user
DB_PASSWORD: testpassword
REDIS_HOST: redis
postgres:
image: postgres:15-alpine
environment:
POSTGRES_DB: payments_test
POSTGRES_USER: api_user
POSTGRES_PASSWORD: testpassword
healthcheck:
test: ["CMD-SHELL", "pg_isready -U api_user -d payments_test"]
interval: 5s
timeout: 3s
retries: 5
# No volume — data does not need to persist in CI
# Faster than using a volume, clean state every run
redis:
image: redis:7-alpine
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 3s
retries: 5
# No volume — clean state every run
Caching Docker Layers in CI
YAML
# Significantly speeds up CI builds by caching built layers
- name: Build with cache
uses: docker/build-push-action@v5
with:
context: .
push: false
load: true
tags: myapp:test
cache-from: type=gha # Read cache from GitHub Actions cache
cache-to: type=gha,mode=max # Write new layers to cache

Common Mistakes

Mistake Consequence Fix
Not waiting for health checks Tests fail intermittently — race condition Use timeout bash -c 'until condition; do sleep 1; done'
Using volumes in CI State bleeds between runs No volumes in CI compose — in-memory or ephemeral storage
Not cleaning up with always() Runner disk fills up with old containers and volumes Always use if: always() on teardown step
Running tests directly on runner Inconsistent dependencies between runners Run tests inside the compose service container
Tip

Use docker compose -f docker-compose.ci.yml exec -T service command (note the -T flag) for non-interactive commands in CI. Without -T, GitHub Actions may hang waiting for TTY input that never comes.

Remember

Add if: always() to your teardown step. If tests fail and you do not tear down, the next CI run may fail because ports are already in use from the previous run's containers still running on the same GitHub Actions runner.


JSON
{
"title": "Docker Compose Production Patterns — What to Use and What to Avoid",
"slug": "docker-compose-production-patterns",
"cluster": "docker",
"description": "Understand what Docker Compose is appropriate for in production, its limitations compared to Kubernetes, and patterns for small-team production deployments.",
"primaryKeyword": "docker compose production"
}

Docker Compose Production Patterns — What to Use and What to Avoid

Overview and What You Will Learn

Docker Compose is excellent for local development and CI. In production it is appropriate for specific situations — small teams, single-host deployments, internal tools — and completely wrong for others. Understanding where the line is prevents you from building on the wrong foundation.

When Compose Is Appropriate in Production

TEXT
Compose is GOOD for production when:
* Single host deployment (one server)
* Team of 1-5 engineers
* Non-critical internal tools
* Staging / preview environments
* Startups with simple infrastructure needs
Compose is NOT appropriate when:
* Multiple hosts needed (high availability)
* Self-healing required (container fails = outage)
* Rolling updates with zero downtime required
* Horizontal scaling across nodes required
* Production traffic is business-critical

Production-Ready Compose Configuration

YAML
# docker-compose.prod.yml
version: "3.8"
services:
api:
image: registry.razorpay.in/payment-api:v3.1.0
restart: unless-stopped # Restart on crash, not on manual stop
ports:
- "127.0.0.1:8080:8080" # Only localhost — nginx proxies externally
environment:
NODE_ENV: production
env_file:
- .env.production # Production secrets from file
logging:
driver: "json-file"
options:
max-size: "100m" # Log rotation — prevents disk fill
max-file: "3" # Keep 3 log files
deploy:
resources:
limits:
cpus: "1.5"
memory: 512M
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
interval: 30s
timeout: 5s
retries: 3
nginx:
image: nginx:1.25-alpine
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./nginx/ssl:/etc/nginx/ssl:ro
- nginx-logs:/var/log/nginx
postgres:
image: postgres:15-alpine
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql/data
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
volumes:
postgres-data:
driver: local
nginx-logs:

Deploying Updates Without Downtime

Compose does not have built-in rolling updates. This is the pattern for zero-downtime updates:

Bash
# Pull new images
docker compose pull
# Recreate only the service that changed (api, not postgres)
docker compose up -d --no-deps api
# --no-deps: do not recreate dependencies (postgres, redis)
# This pulls new api image and recreates only the api container
# Note: there IS a brief downtime during the container recreation
# For true zero downtime, use Kubernetes or Traefik with multiple replicas
# For simple single-host zero-downtime:
# Run nginx in front, bring up new container on different port,
# update nginx config, reload nginx (zero downtime), remove old container

Compose vs Kubernetes Decision

Need Compose Kubernetes
Local development Excellent Overkill
Single host production Good Overkill
Multi-host / HA Cannot do it Designed for this
Self-healing No (manual restart) Yes (automatic)
Rolling updates (zero downtime) Manual workaround Built-in
Horizontal autoscaling No Built-in (HPA)
Service mesh / mTLS No Yes (Istio, Linkerd)
Complexity Low High
Ops cost Very low Significant
Tip

Use unless-stopped restart policy (not always) for production services. always restarts even when you intentionally stop the container for maintenance. unless-stopped restarts on crash but respects manual docker stop.

Common Mistake

Using Compose in production for business-critical services without understanding its limitations. If the server goes down, all your containers go down. If a container crashes and the restart fails (e.g., port conflict), the service is down until you SSH in and fix it manually. Plan for this reality before choosing Compose over Kubernetes for production.

Resources

Explore More in Docker Compose and Multi-Container Applications

All 6 Topics

Frequently Asked Questions

Is Docker Compose Fundamentals — Defining Multi-Container Applications free to learn on DevOps Network?

Yes - this topic, like everything on DevOps Network, is 100% free with no paywall or sign-up gate.

What does the Docker Compose Fundamentals — Defining Multi-Container Applications topic cover?

Define and run multi-container applications with Docker Compose — covering the compose file schema, service configuration, and essential CLI commands.