Overview and What You Will Learn
Unit tests with mocked database calls catch some bugs. They miss the ones that matter most — a SQL query with a subtle JOIN error, a Redis key expiry race condition, a RabbitMQ message acknowledgement bug. Catching those requires running tests against the real thing.
Docker Compose makes this practical inside CI. The same compose file used for local development can spin up real Postgres, Redis, and RabbitMQ instances inside a GitHub Actions runner, run the full integration test suite against them, and tear everything down — all in a few minutes, on every pull request.
By the end of this lab you will:
- Run a Compose stack inside a GitHub Actions workflow
- Use Compose profiles to define services that only start in CI
- Override compose config for CI-specific ports and settings
- Wait correctly for services to be ready before running tests
- Collect logs from a failed CI run for debugging
- Cache Docker layers in GitHub Actions to speed up repeated runs
Why This Matters in Production
At CRED, a backend pull request cannot merge unless the integration test suite passes against a real Postgres and Redis instance spun up inside the CI runner itself — not mocks, not a shared staging database that ten engineers are also hitting at the same time. This catches data-layer bugs before they reach staging, where they are far more expensive to trace back to a single commit.
The key engineering insight is that CI dependencies should be ephemeral and isolated per run. Using Compose for this means every PR gets a brand new, throwaway Postgres instance — no test pollution between runs, no shared state, no "works on CI sometimes" flakiness caused by leftover data from a previous run.
Core Principles
CI pipeline with Compose service lifecycle:
+------------------------------------------+| GitHub Actions job starts |+------------------------------------------+ | v+------------------------------------------+| docker compose up -d --wait | <- starts postgres, redis,+------------------------------------------+ rabbitmq, waits for healthy | v+------------------------------------------+| Run integration test suite | <- tests hit real services+------------------------------------------+ over localhost ports | v+------------------------------------------+| Collect logs if tests failed | <- docker compose logs+------------------------------------------+ uploaded as CI artifact | v+------------------------------------------+| docker compose down --volumes | <- full teardown, no+------------------------------------------+ leftover state for next runProfiles separate CI-only services from default services:
+------------------------+ +------------------------------+| Default services | | CI-profile services || | | || api, postgres, redis | <------> | test-seeder, mock-payment-gw || start with plain | | start only with --profile ci || docker compose up | | docker compose --profile ci |+------------------------+ +------------------------------+Detailed Step-by-Step Practical Lab
Milestone 1 — Write a CI-specific compose file
## docker-compose.ci.yml## Overrides ports and adds CI-only services on top of the base fileservices: postgres: ports: # Different host port avoids collisions with other CI jobs # running on the same self-hosted runner - "55432:5432" healthcheck: test: ["CMD-SHELL", "pg_isready -U cred_test"] interval: 3s timeout: 2s retries: 10 redis: ports: - "56379:6379" test-seeder: build: ./scripts/seed profiles: ["ci"] depends_on: postgres: condition: service_healthy environment: DATABASE_URL: postgres://cred_test:test_pw@postgres:5432/cred_testMilestone 2 — Write the GitHub Actions workflow
## .github/workflows/integration-tests.ymlname: Integration Tests on: pull_request: branches: [main] jobs: integration-tests: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Cache Docker layers uses: actions/cache@v4 with: path: /tmp/.buildx-cache key: ${{ runner.os }}-buildx-${{ hashFiles('**/Dockerfile') }} restore-keys: | ${{ runner.os }}-buildx- - name: Start dependency stack run: | docker compose \ -f docker-compose.yml \ -f docker-compose.ci.yml \ --profile ci \ up -d --wait - name: Run integration test suite run: | export DATABASE_URL=postgres://cred_test:test_pw@localhost:55432/cred_test export REDIS_URL=redis://localhost:56379 npm run test:integration - name: Collect logs on failure if: failure() run: docker compose logs --no-color > compose-logs.txt - name: Upload failure logs if: failure() uses: actions/upload-artifact@v4 with: name: compose-logs path: compose-logs.txt - name: Tear down stack if: always() run: docker compose -f docker-compose.yml -f docker-compose.ci.yml down --volumesMilestone 3 — Use --wait instead of manual sleep or polling scripts
## --wait blocks until all services with a healthcheck report healthy## Replaces fragile custom polling loops entirelydocker compose up -d --wait ## Set a custom timeout if the default is too short for slow servicesdocker compose up -d --wait --wait-timeout 60Milestone 4 — Activate CI-only services with profiles
## Locally: test-seeder does NOT start (no profile flag passed)docker compose up -d ## In CI: test-seeder DOES start, populating fixtures before tests rundocker compose --profile ci up -d --waitMilestone 5 — Collect logs from a failed CI run
## Dump all service logs, no color codes (cleaner in CI log viewers)docker compose logs --no-color > compose-logs.txt ## Dump logs for just one suspicious servicedocker compose logs --no-color postgres > postgres-logs.txt ## Include timestamps to correlate with test failure timingdocker compose logs --no-color -t > compose-logs-timestamped.txtMilestone 6 — Guarantee teardown even when tests fail
## The 'if: always()' condition is critical -- without it,## a failed test step would skip teardown and leak containers## on the runner, eventually exhausting disk space- name: Tear down stack if: always() run: docker compose down --volumes --remove-orphansProduction Best Practices and Common Pitfalls
| Scenario | Wrong | Correct |
|---|---|---|
| Waiting for services | sleep 10 before running tests | docker compose up -d --wait |
| CI-only fixtures | Always running seed/mock services locally too | Use profiles: ["ci"] to isolate them |
| Port collisions in CI | Same ports as local dev compose file | Override file maps different host ports |
| Failed test cleanup | Teardown step runs only on success | if: always() on the teardown step |
| Debugging a flaky CI failure | Re-run and hope | Upload docker compose logs as a CI artifact every failure |
Quick Reference and Troubleshooting Commands
| Task | Command |
|---|---|
| Start and wait for healthy | docker compose up -d --wait |
| Start with CI profile | docker compose --profile ci up -d --wait |
| Use multiple compose files | docker compose -f base.yml -f ci.yml up -d |
| Dump logs for debugging | docker compose logs --no-color > out.txt |
| Full teardown including orphans | docker compose down --volumes --remove-orphans |
| View merged CI config | docker compose -f base.yml -f ci.yml config |
TipUse
docker compose up -d --waitinstead of hand-written polling loops orsleepstatements. It directly uses each service'shealthcheckdefinition, so the CI pipeline and local development share exactly the same readiness logic.
RememberMultiple
-fflags merge compose files in the order given, with later files overriding earlier ones.docker compose -f docker-compose.yml -f docker-compose.ci.ymlapplies the base file first, then layers CI-specific overrides on top.
Common MistakeForgetting
if: always()on the teardown step in a GitHub Actions workflow. If a test step fails, the workflow skips remaining steps by default — leaving containers and volumes running on the runner and silently consuming resources on every failed PR.
SecurityNever reuse production database credentials in a CI compose override file, even temporarily. Use clearly fake, CI-only credentials (e.g.
cred_test/test_pw) that have zero validity outside the ephemeral CI environment, so an accidental leak in CI logs causes no real damage.
