It is your second week on the job. You clone the team's repository, and the README says "just run the pipeline." Thirty minutes later you have three version conflicts - your Python is 3.11, the pipeline needs 3.9. Postgres will not start because port 5432 is already used by something else on your laptop. A teammate's pull request works perfectly on his machine and breaks the moment it reaches the staging server. None of these are code problems. They are environment problems. The code is correct - the environment underneath it is different every time it runs. **Docker** solves this by packaging an application together with everything it depends on - the exact Python version, the exact libraries, the exact configuration - into one portable unit called a **container**. That unit runs the same way on your laptop, your teammate's laptop, and a CI runner, because it is not relying on whatever happens to already be installed on each machine. A production deployment may still differ in configuration or scale, but the core environment - the OS layer, the language version, the dependencies - stays consistent. > 💡 **Tip:** Think of a container like a shipping container on a cargo ship. The container itself does not care whether it is sitting on a truck, a train, or a ship - the contents inside stay exactly the same. Docker containers work the same way across your laptop, a CI runner, or a cloud server. ### Why This Matters Specifically for Data Engineering Data engineering work rarely involves just one tool. A single pipeline might need Postgres as a source, Kafka for streaming events, Spark for processing, and Airflow for orchestration - four separate systems that all need to run together, talk to each other, and not conflict with anything else already installed on your machine. Installing all four directly on your laptop is painful and fragile. Uninstalling them cleanly afterward is worse. Docker lets you run all four in isolated containers, wire them together with one configuration file, and tear the whole thing down with a single command when you are done - leaving your actual machine untouched. > 📌 **Engineering Decision:** For local, multi-service data infrastructure - Postgres, Kafka, Spark, Airflow running together - Docker is usually the easiest choice, because it isolates each tool's dependencies and makes the whole setup reproducible for every teammate. Native installs of multiple database and orchestration tools tend to fight over ports and versions, and cleaning them up afterward is painful. For a single simple tool with no dependency conflicts, a native install can still be reasonable - the decision matters most once you are running several services together. ### Concept Check * Why does "it works on my machine" happen in the first place, and what specifically does Docker do to prevent it? * Name two data tools you would likely run together in one local pipeline, and explain why running them natively instead of in containers would be harder to manage. ---
Three terms get used constantly in Docker and confused just as often. Getting this right early makes everything else in this module click faster. ### The Blueprint and the Building - Images vs Containers An **image** is a read-only template - the blueprint. It defines everything the application needs: the base operating system, the installed packages, the configuration, the code itself. An image never runs by itself; it just sits there as a definition. A **container** is a running instance created from that image. You can start ten containers from the same image, and each one runs independently, with its own memory and its own process - like ten identical buildings built from the same blueprint, each with people living inside doing different things. +------------------+ docker run +------------------+ | Docker Image | ------------------------> | Docker Container | | (the blueprint) | | (the running app) | +------------------+ +------------------+ Read-only, static Running process, isolated Built once Can start many from one image > **Note:** This is the same relationship as a class and an object in Python. A class defines what an object looks like; an object is one specific instance of that class, running in memory. An image defines a container; a container is one running instance of that image. ### The Recipe - Dockerfile A **Dockerfile** is a plain text file containing step-by-step instructions for building an image. Docker reads it top to bottom and executes each line in order, producing the final image. ```dockerfile FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY . . CMD ["python", "pipeline.py"] ``` > **Note:** `FROM` picks the starting point (a minimal Python environment here). `WORKDIR` sets the folder inside the container where later commands run. `COPY` brings files from your project into the image. `RUN` executes a command while building the image - here, installing dependencies. `CMD` is the command that runs when a container starts from this image. ### Concept Check * If you start three containers from the same image and one of them crashes, what happens to the other two? Why? * What is the difference between something that happens when an image is *built* versus something that happens when a container *starts*? ---
Docker Desktop bundles the Docker engine, the command-line client, Docker Compose, and a management GUI into one install. It is available for macOS, Windows, and Linux. ### macOS and Windows ```bash ## Download Docker Desktop from docker.com/products/docker-desktop ## Windows users: enable WSL2 (Windows Subsystem for Linux) first - Docker Desktop ## will prompt you to do this automatically if it is not already enabled ## After installing, verify from a terminal docker --version ## Docker version 27.x.x, build xxxxxxx ``` ### Linux (Ubuntu/Debian) ```bash ## Update package index sudo apt-get update ## Install Docker Engine directly (no GUI needed on a server) sudo apt-get install -y docker.io docker-compose-v2 ## Start Docker and enable it on boot sudo systemctl start docker sudo systemctl enable docker ## Allow your user to run docker without sudo every time sudo usermod -aG docker $USER ## Log out and back in for this to take effect ``` ### Verify the Installation ```bash docker run hello-world ``` ```text Hello from Docker! This message shows that your installation appears to be working correctly. ``` > 🔴 **Common Mistake:** Running `docker` commands with `sudo` on every single command because the user was never added to the `docker` group. This works but is tedious and can create files owned by root inside mounted folders, causing permission errors later. Run `sudo usermod -aG docker $USER` once, log out and back in, and `sudo` is no longer needed for normal Docker use. ---
### Writing a Dockerfile for a Data Pipeline Script Say you have a small Python script that reads a CSV and writes a summary. Here is a complete, minimal Dockerfile for it. ```dockerfile ## Base image - a small Python environment, not the full OS FROM python:3.11-slim ## All following commands run from this folder inside the container WORKDIR /app ## Copy just the requirements file first (see caching note below) COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt ## Now copy the rest of the pipeline code COPY . . ## Command that runs when the container starts CMD ["python", "pipeline.py"] ``` > 📌 **Remember:** Copy `requirements.txt` and run `pip install` before copying the rest of your code. Docker caches each instruction as a layer, and a layer is only rebuilt if something in it changed. If you copy all your code first, Docker reinstalls every dependency any time you edit a single line of Python. Copying `requirements.txt` separately means dependencies are only reinstalled when they actually change. ### Building the Image ```bash ## -t names and tags the image; . means "use the current folder as build context" docker build -t swiggy-order-pipeline:v1 . ``` ```text [+] Building 12.4s (10/10) FINISHED => [1/5] FROM docker.io/library/python:3.11-slim => [2/5] WORKDIR /app => [3/5] COPY requirements.txt . => [4/5] RUN pip install --no-cache-dir -r requirements.txt => [5/5] COPY . . => exporting to image => naming to docker.io/library/swiggy-order-pipeline:v1 ``` ### Running the Container ```bash docker run swiggy-order-pipeline:v1 ``` ```text Processed 4,213 orders from orders_2026_08_14.csv Summary written to output/daily_summary.csv ``` > **Note:** By default the container's filesystem disappears when the container stops - `output/daily_summary.csv` was written *inside* the container, not on your laptop. The Volumes section further down covers how to persist that file to your real filesystem. ### Concept Check * Why does moving `COPY requirements.txt .` before `COPY . .` speed up rebuilds during development? * What happens to a file your pipeline writes inside the container if you have not set up a volume? ---
### Managing Images ```bash ## List every image stored locally docker images ## Remove one image by name or ID docker rmi swiggy-order-pipeline:v1 ## Remove every image not currently used by a container - frees disk space docker image prune -a ``` ### Managing Containers ```bash ## Run in the foreground - blocks your terminal, shows live output docker run swiggy-order-pipeline:v1 ## Run detached (in the background) - gives your terminal back immediately docker run -d --name pipeline-run swiggy-order-pipeline:v1 ## List running containers docker ps ## List every container including stopped ones docker ps -a ## Stop a running container docker stop pipeline-run ## Remove a stopped container docker rm pipeline-run ## Stop and remove in one line - useful while iterating quickly docker rm -f pipeline-run ``` > 💡 **Tip:** Add `--rm` to `docker run` while testing (`docker run --rm swiggy-order-pipeline:v1`). This automatically deletes the container the moment it stops, so you do not accumulate dozens of dead test containers while iterating on a Dockerfile. ### Looking Inside a Running Container When a pipeline behaves unexpectedly, you often need to look around inside the container itself - check if a file actually landed where you expected, confirm an environment variable is set, or just poke around. ```bash ## Open an interactive shell inside a running container docker exec -it pipeline-run bash ## Now you are inside the container's filesystem ls /app cat requirements.txt exit ## leave the container's shell, container keeps running ``` ```bash ## View logs from a container - essential for debugging a failed pipeline run docker logs pipeline-run ## Follow logs live, like tail -f docker logs -f pipeline-run ``` ```bash ## Full configuration and runtime details for a container, as JSON docker inspect pipeline-run ## Pull out just one thing you need instead of reading the whole JSON blob docker inspect pipeline-run --format '{{.NetworkSettings.IPAddress}}' docker inspect pipeline-run --format '{{.Mounts}}' ``` > **Note:** `docker inspect` shows everything Docker knows about a container - its environment variables, mounted volumes, networks it is attached to, its IP address, and its exact configuration. Reach for it when `docker logs` shows an application-level error but you need to confirm the container's setup itself is what you expect - for example, checking whether an environment variable actually made it into the container, or which network it is really attached to. > 🔴 **Common Mistake:** Assuming a container crashed for no reason and rebuilding the image repeatedly without ever checking `docker logs`. The logs almost always contain the exact Python traceback or error message explaining what went wrong. Check logs before rebuilding. > 💡 **Practice:** Build the pipeline image above (or any small Python script of your own with a Dockerfile), run it with `--rm`, then intentionally break something - misspell a filename it reads - rebuild, run again, and use `docker logs` to find the exact error. Confirm you can read a failure from logs alone without opening the code. ---
Data written to a container's own writable layer survives a `docker stop` and a `docker restart` - stopping a container does not touch its filesystem. That data is only lost when the container itself is removed and recreated with `docker rm`. For a database or any pipeline output you actually care about, this is a serious problem the moment someone runs `docker rm` - or Compose recreates a container during a routine `docker compose up --build`. **Volumes** solve this by storing data outside the container's own writable layer, either in a location Docker manages or in a folder on your real machine. docker stop -> container paused, filesystem untouched, data still there docker start -> same container resumes, same data docker rm -> container and its writable layer are deleted any data NOT in a volume is gone permanently Without a volume: container runs -> writes data inside itself -> container removed -> data GONE With a volume: container runs -> writes data to volume -> container removed -> data SURVIVES (the volume lives independently of any one container) ### Named Volumes vs Bind Mounts A **named volume** is fully managed by Docker - you do not need to know or care where it physically lives on disk. This is the right choice for database data you want to persist reliably. A **bind mount** links a specific folder on your own machine directly into the container. This is the right choice when you want to edit pipeline code on your laptop and see the change reflected inside the container immediately, without rebuilding the image. ```bash ## Named volume - Docker manages where this actually lives docker run -d --name pg-data -v pgdata:/var/lib/postgresql/data postgres:16-alpine ## Bind mount - maps your local ./dags folder directly into the container docker run -d -v ./dags:/opt/airflow/dags apache/airflow:2.9.0 ``` > 📌 **Engineering Decision:** Use named volumes for persistent database storage in local development - Postgres data is the clearest example. Use bind mounts for source code you are actively editing, like Airflow DAGs or a pipeline script you are iterating on. Kafka's persistent storage is a more advanced case - the right approach depends on the broker deployment architecture and Kafka version, and it is designed as part of that deployment rather than treated as a simple Docker volume decision. Mixing volumes and bind mounts up the other way means either your database data lives in a folder you might accidentally delete, or your code edits require a full rebuild to take effect. ### Concept Check * You ran `docker stop pg-data` on a Postgres container without a volume attached, then `docker start pg-data` again. Is the data still there? Now suppose you had run `docker rm -f pg-data` instead - what happened to everything in that database? * Your Airflow DAG file is bind-mounted from your laptop. You edit the DAG file locally - do you need to rebuild the image to see the change take effect? ---
It is your second week on the job. You clone the team's repository, and the README says "just run the pipeline." Thirty ...
Three terms get used constantly in Docker and confused just as often. Getting this right early makes everything else in ...
Docker Desktop bundles the Docker engine, the command-line client, Docker Compose, and a management GUI into one install...
Writing a Dockerfile for a Data Pipeline Script Say you have a small Python script that reads a CSV and writes a summary...
Managing Images Managing Containers > 💡 Tip: Add --rm to docker run while testing (docker run --rm swiggy-order-pipelin...
Data written to a container's own writable layer survives a docker stop and a docker restart - stopping a container does...
The pipeline script earlier connected to Postgres with host="postgres", dbname="pipeline", user="pipeline", password="pi...
A single container is rarely enough. A real pipeline needs Postgres to talk to Airflow, or Spark workers to talk to a Sp...
Running several separate docker run commands with matching network flags, volume flags, and port flags every time you wa...
Here is a Compose file with three real, common mistakes baked in. Find and fix all three before continuing. > 📌 Remembe...
Pin Your Base Image Version A pipeline that works today on python:latest can silently break next month when that tag poi...
Create a project folder and a Python pipeline script that reads its database connection details from environment variabl...
Command What it Does docker build -t name:tag . Build an image from a Dockerfile docker run -d --name x image Start a co...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.