Build a Production Application Platform

Deploy a three-tier app on Kubernetes with Ingress, NetworkPolicies, HPA, Prometheus alerts and ArgoCD GitOps, then measure and document the results.

Architecture Overview & Problem Statement

This project builds a three-tier order app (React, Node.js, PostgreSQL, Redis) and deploys it to Kubernetes with an Ingress, default-deny NetworkPolicies, an HPA, PodDisruptionBudgets, Prometheus alerts and ArgoCD GitOps delivery. You finish with a repository and measured results that show production practices, not just a running app.

flowchart TD A[Browser]:::slate --> B[Traefik Ingress]:::orange B -->|/| C[Frontend React]:::blue B -->|/api| D[Backend Node.js]:::blue D --> E[(PostgreSQL)]:::green D --> F[(Redis cache)]:::red classDef blue fill:#326CE5,color:#fff classDef green fill:#10B981,color:#fff classDef orange fill:#F97316,color:#fff classDef red fill:#EF4444,color:#fff classDef slate fill:#475569,color:#fff

What You Build, Learn and Show

Most people learn Docker, then Kubernetes, then monitoring, then GitOps, each in isolation. This capstone connects them. You build one small application and then everything that makes it run like a production service.

You build an order management app with a React frontend, a Node.js API, PostgreSQL for storage and Redis as a cache. Around it you build the platform: container images, Kubernetes manifests, an Ingress, network isolation, autoscaling, disruption protection, correct health probes, Prometheus metrics and alerts, and delivery through ArgoCD.

You learn:

Why the work around an application (limits, probes, policies, budgets) is most of production readiness
How liveness and readiness probes differ, and what goes wrong when they are mixed up
How to isolate services with default-deny NetworkPolicies and add only the connections you need
How HPA, PodDisruptionBudgets and resource requests work together
How GitOps changes the way you ship: Git is the source of truth and the cluster follows it

You can show a Git repository with the application, Dockerfiles, manifests and ArgoCD application; screenshots of the running app, the Grafana dashboard, the HPA scaling under load and ArgoCD showing Synced and Healthy; and a RESULTS.md with the numbers you measured on your own cluster.

Plan on about 6 hours the first time, including debugging. That is normal for a capstone.

Tip
this is a learning platform designed to teach the patterns used by product teams, not a hardened production system. The "Before Real Production" section tells you what you would add next.

Prerequisites

You should be comfortable with the earlier roadmap steps: Docker basics, core Kubernetes objects (Pod, Deployment, Service), and the command line.

Install these tools:

Docker: builds container images
kubectl: controls the cluster
helm: installs Traefik, Prometheus and Grafana
Node.js 22 (current LTS): runs the backend and builds the frontend locally
Git and a GitHub account
A Kubernetes cluster with at least 2 CPUs and 4 GB RAM free: minikube, kind, or a managed cluster (EKS, GKE, AKS). Local clusters are free. Managed clusters cost money while they run.

Bash

## Verify everything is installed
docker --version
kubectl version --client
helm version --short
node --version     ## should print v22.x
git --version
 
## Local cluster option A: minikube with a CNI that enforces NetworkPolicies
minikube start --cpus=4 --memory=6144 --cni=calico
minikube addons enable metrics-server
 
## Local cluster option B: kind with the default CNI does NOT enforce NetworkPolicies.
## Use minikube with calico, or a managed cluster, to complete Milestone 3 properly.
 
## Verify kubectl reaches your cluster
kubectl get nodes
## Expected: a node with STATUS Ready

What You Are Building

Think of a restaurant. The React app is the menu, the Node.js API is the waiter, PostgreSQL is the kitchen's permanent order book, Redis is a whiteboard of recent orders for quick lookup, and the Ingress is the front door that sends people to the right place.

This is a three-tier architecture: presentation (frontend), application (API), and data (database and cache). Most real systems follow this shape, so understanding how to run it well transfers almost anywhere.

The final repository looks like this:

Bash

orders-platform/
  backend/            Node.js API
  frontend/           React app (Vite)
  k8s/                all Kubernetes manifests
  argocd-app.yaml     ArgoCD Application
  docker-compose.yml  local testing only
  RESULTS.md          what you measured

Keeping application code (backend/, frontend/) separate from infrastructure code (k8s/) means that when something breaks you know where to look.

Milestone 1: Build the Orders Application

This milestone builds the Node.js API, the React frontend and the database schema that the rest of the platform runs.

We write the application first because the platform only makes sense once you see how the code and the infrastructure connect: environment variables, health endpoints, metrics, and graceful shutdown are all code decisions that Kubernetes depends on.

1.1 Create the Project Structure

Bash

mkdir orders-platform && cd orders-platform
git init
 
mkdir -p backend/src/routes
mkdir -p k8s/database k8s/cache k8s/backend k8s/frontend k8s/ingress k8s/monitoring
 
echo "Project structure created"

1.2 Backend: The Node.js API

Bash

cd backend
npm init -y
 
## express  : web framework
## pg       : PostgreSQL client
## redis    : Redis client (version 4 uses promises)
## cors     : lets the browser frontend call the API during local development
## helmet   : adds security headers
## morgan   : request logging
## prom-client : exposes Prometheus metrics at /metrics
npm install express pg redis@4 cors helmet morgan prom-client
npm install --save-dev nodemon
 
## Add run scripts
npm pkg set scripts.start="node src/index.js"
npm pkg set scripts.dev="nodemon src/index.js"
 
## Keep build context small and secrets out of the image
printf "node_modules\n.env\nnpm-debug.log\n" > .dockerignore

Database connection

A connection pool keeps a few database connections open and reuses them, which is much faster than opening a new connection for every request. Note that this file does not crash the process if the database is unavailable at startup. The readiness probe (below) reports the problem and Kubernetes stops sending traffic, which is the correct behaviour in a cluster where start order is not guaranteed.

JAVASCRIPT

// backend/src/db.js
const { Pool } = require('pg');
 
// Connection details come from environment variables.
// In Kubernetes they come from a Secret, never from the code.
const pool = new Pool({
  host:     process.env.DB_HOST || 'localhost',
  port:     parseInt(process.env.DB_PORT || '5432', 10),
  database: process.env.DB_NAME || 'appdb',
  user:     process.env.DB_USER || 'appuser',
  password: process.env.DB_PASSWORD,
  max: 10,                        // at most 10 open connections
  idleTimeoutMillis: 30000,       // close idle connections after 30s
  connectionTimeoutMillis: 2000,  // fail fast if the DB is unreachable
});
 
// An idle client error must not crash the process
pool.on('error', (err) => console.error('Unexpected PG pool error:', err.message));
 
module.exports = pool;

Redis connection

JAVASCRIPT

// backend/src/cache.js
const { createClient } = require('redis');
 
const host = process.env.REDIS_HOST || 'localhost';
const port = process.env.REDIS_PORT || '6379';
 
const client = createClient({
  url: `redis://${host}:${port}`,
  // If Redis is down, commands fail immediately instead of queueing and hanging requests
  disableOfflineQueue: true,
  socket: {
    // Reconnect with growing delay: 100ms, 200ms ... up to 3 seconds
    reconnectStrategy: (retries) => Math.min(retries * 100, 3000),
  },
});
 
client.on('error', (err) => console.error('Redis error:', err.message));
client.on('ready', () => console.log('Redis connected'));
 
client.connect().catch((err) => console.error('Initial Redis connect failed:', err.message));
 
module.exports = client;

Metrics

Prometheus needs something to scrape. This file records how long each request takes and exposes default process metrics.

JAVASCRIPT

// backend/src/metrics.js
const client = require('prom-client');
 
client.collectDefaultMetrics(); // CPU, memory, event loop lag, and so on
 
const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5],
});
 
// Middleware: measure every request
function metricsMiddleware(req, res, next) {
  const end = httpDuration.startTimer();
  res.on('finish', () => {
    const route = req.route ? req.baseUrl + req.route.path : req.baseUrl || 'unmatched';
    end({ method: req.method, route, status_code: res.statusCode });
  });
  next();
}
 
async function metricsHandler(req, res) {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
}
 
module.exports = { metricsMiddleware, metricsHandler };

Orders routes

A route is an HTTP method plus a URL path (for example GET /api/orders) with a handler that runs when a matching request arrives. GET uses the cache-aside pattern: check Redis first, fall back to PostgreSQL, then store the result in Redis for 60 seconds. If Redis is unavailable the API still works, just slower. A cache should never be a single point of failure.

JAVASCRIPT

// backend/src/routes/orders.js
const express = require('express');
const router  = express.Router();
const pool    = require('../db');
const cache   = require('../cache');
 
const CACHE_KEY = 'orders:all';
 
// GET /api/orders: list the latest 50 orders
router.get('/', async (req, res) => {
  try {
    // 1. Try the cache. If Redis is down this throws, and we fall through to the database.
    try {
      const cached = await cache.get(CACHE_KEY);
      if (cached) return res.json({ source: 'cache', data: JSON.parse(cached) });
    } catch (err) {
      console.warn('Cache read skipped:', err.message);
    }
 
    // 2. Cache miss: query the database
    const result = await pool.query(
      'SELECT * FROM orders ORDER BY created_at DESC LIMIT 50'
    );
 
    // 3. Store for 60 seconds (best effort)
    try {
      await cache.setEx(CACHE_KEY, 60, JSON.stringify(result.rows));
    } catch (err) {
      console.warn('Cache write skipped:', err.message);
    }
 
    res.json({ source: 'database', data: result.rows });
  } catch (err) {
    console.error('GET /orders error:', err);
    res.status(500).json({ error: 'Failed to fetch orders' });
  }
});
 
// POST /api/orders: create an order
router.post('/', async (req, res) => {
  const { item, quantity, customer_name } = req.body || {};
  const qty = Number(quantity);
 
  // Always validate on the server. Never trust what the client sends.
  if (!item || !customer_name || !Number.isInteger(qty) || qty < 1) {
    return res.status(400).json({
      error: 'item, customer_name and a whole-number quantity of at least 1 are required',
    });
  }
 
  try {
    // $1, $2, $3 are parameter placeholders. They prevent SQL injection.
    const result = await pool.query(
      `INSERT INTO orders (item, quantity, customer_name, status)
       VALUES ($1, $2, $3, 'pending')
       RETURNING *`,
      [item, qty, customer_name]
    );
 
    // The list just changed, so drop the cached copy
    try { await cache.del(CACHE_KEY); } catch (err) { console.warn('Cache invalidate skipped:', err.message); }
 
    res.status(201).json(result.rows[0]);
  } catch (err) {
    console.error('POST /orders error:', err);
    res.status(500).json({ error: 'Failed to create order' });
  }
});
 
// GET /api/orders/:id: one order
router.get('/:id', async (req, res) => {
  const id = Number(req.params.id);
  if (!Number.isInteger(id)) return res.status(400).json({ error: 'id must be a number' });
 
  try {
    const result = await pool.query('SELECT * FROM orders WHERE id = $1', [id]);
    if (result.rows.length === 0) return res.status(404).json({ error: 'Order not found' });
    res.json(result.rows[0]);
  } catch (err) {
    console.error('GET /orders/:id error:', err);
    res.status(500).json({ error: 'Failed to fetch order' });
  }
});
 
module.exports = router;

Health checks: liveness and readiness are different jobs

Kubernetes asks every pod two questions:

Liveness: "Is the process alive enough to keep running?" If this fails repeatedly, Kubernetes restarts the container. Keep it trivial. Never check the database here. If the database is slow and every pod fails liveness, Kubernetes restarts all of them at once and turns a database problem into a full outage.
Readiness: "Should this pod receive traffic right now?" If this fails, Kubernetes removes the pod from the Service but does not restart it. Check the dependencies the pod truly cannot work without. Here that is PostgreSQL. Redis is only a cache, so it is reported but does not make the pod unready.

JAVASCRIPT

// backend/src/routes/health.js
const express = require('express');
const router  = express.Router();
const pool    = require('../db');
const cache   = require('../cache');
 
// Liveness: always 200 if the process can answer
router.get('/live', (req, res) => {
  res.status(200).json({ status: 'alive' });
});
 
// Readiness: 200 only if the database answers
router.get('/ready', async (req, res) => {
  const checks = {};
 
  try {
    await pool.query('SELECT 1');
    checks.database = 'ok';
  } catch (err) {
    checks.database = `failing: ${err.message}`;
  }
 
  // Reported for visibility, but does not affect readiness
  try {
    await cache.ping();
    checks.cache = 'ok';
  } catch (err) {
    checks.cache = 'degraded (API still serves from the database)';
  }
 
  const ready = checks.database === 'ok';
  res.status(ready ? 200 : 503).json({ status: ready ? 'ready' : 'not_ready', checks });
});
 
module.exports = router;

The server

The server handles SIGTERM. When Kubernetes stops a pod (a rolling update, a scale-down, a node drain) it sends SIGTERM and waits. A server that ignores it is killed abruptly and drops in-flight requests.

JAVASCRIPT

// backend/src/index.js
const express = require('express');
const cors    = require('cors');
const helmet  = require('helmet');
const morgan  = require('morgan');
 
const pool = require('./db');
const cache = require('./cache');
const { metricsMiddleware, metricsHandler } = require('./metrics');
const ordersRouter = require('./routes/orders');
const healthRouter = require('./routes/health');
 
const app  = express();
const PORT = process.env.PORT || 4000;
 
app.use(helmet());                // security headers
app.use(cors());                  // needed only for local development on another port
app.use(morgan('combined'));      // request log
app.use(express.json());          // parse JSON bodies
app.use(metricsMiddleware);       // record request durations
 
app.use('/api/orders', ordersRouter);
app.use('/health', healthRouter);
app.get('/metrics', metricsHandler);
 
app.use((req, res) => res.status(404).json({ error: `Route ${req.path} not found` }));
app.use((err, req, res, next) => {
  console.error('Unhandled error:', err);
  res.status(500).json({ error: 'Internal server error' });
});
 
const server = app.listen(PORT, () => console.log(`API listening on port ${PORT}`));
 
// Graceful shutdown: stop accepting connections, finish in-flight requests, close clients
function shutdown(signal) {
  console.log(`${signal} received, shutting down`);
  server.close(async () => {
    await pool.end().catch(() => {});
    await cache.quit().catch(() => {});
    process.exit(0);
  });
  setTimeout(() => process.exit(1), 25000).unref(); // safety net below the 30s grace period
}
process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT',  () => shutdown('SIGINT'));

Database schema

SQL

-- backend/src/init.sql
-- PostgreSQL runs files in /docker-entrypoint-initdb.d once, when the data directory is empty.
 
CREATE TABLE IF NOT EXISTS orders (
  id            SERIAL PRIMARY KEY,
  item          VARCHAR(255) NOT NULL,
  quantity      INTEGER NOT NULL CHECK (quantity > 0),
  customer_name VARCHAR(255) NOT NULL,
  status        VARCHAR(50) NOT NULL DEFAULT 'pending',
  created_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
 
-- Indexes speed up lookups that would otherwise scan every row
CREATE INDEX IF NOT EXISTS idx_orders_status     ON orders(status);
CREATE INDEX IF NOT EXISTS idx_orders_created_at ON orders(created_at DESC);
 
INSERT INTO orders (item, quantity, customer_name, status) VALUES
  ('Butter Chicken', 2, 'Arjun Sharma', 'delivered'),
  ('Paneer Tikka',   1, 'Priya Nair',   'preparing'),
  ('Biryani',        3, 'Vikram Das',   'pending'),
  ('Masala Dosa',    2, 'Ananya Reddy', 'delivered'),
  ('Chole Bhature',  1, 'Rohit Gupta',  'preparing');

1.3 Frontend: The React App

The frontend never talks to the database. It only knows the API address. This separation is fundamental to the three-tier design.

Bash

cd ../frontend
 
## Create a React project with Vite (Create React App is no longer maintained)
npm create vite@latest . -- --template react
npm install
npm install axios

Replace frontend/src/App.jsx with:

React JSX

// frontend/src/App.jsx
import { useState, useEffect } from 'react';
import axios from 'axios';
 
// In Kubernetes the Ingress sends /api to the backend, so a relative path works.
// For local Docker Compose we pass a full URL at build time.
const API_BASE = import.meta.env.VITE_API_URL || '/api';
 
const statusColor = {
  pending:   '#FFA500',
  preparing: '#3B82F6',
  delivered: '#22C55E',
};
 
export default function App() {
  const [orders, setOrders]   = useState([]);
  const [loading, setLoading] = useState(true);
  const [error, setError]     = useState(null);
  const [form, setForm]       = useState({ item: '', quantity: 1, customer_name: '' });
 
  const fetchOrders = async () => {
    try {
      setLoading(true);
      const response = await axios.get(`${API_BASE}/orders`);
      setOrders(response.data.data);
      setError(null);
    } catch (err) {
      setError('Could not load orders. Is the API running?');
    } finally {
      setLoading(false);
    }
  };
 
  // Run once when the page first loads
  useEffect(() => { fetchOrders(); }, []);
 
  const createOrder = async (e) => {
    e.preventDefault();
    try {
      await axios.post(`${API_BASE}/orders`, { ...form, quantity: Number(form.quantity) });
      setForm({ item: '', quantity: 1, customer_name: '' });
      fetchOrders();
    } catch (err) {
      setError(err.response?.data?.error || 'Failed to create order');
    }
  };
 
  const input = { padding: '10px 12px', borderRadius: 6, border: 'none', fontSize: 14 };
 
  return (
    <div style={{ maxWidth: 800, margin: '0 auto', padding: 24, background: '#111827',
                  minHeight: '100vh', color: '#F3F4F6', fontFamily: 'system-ui, sans-serif' }}>
      <h1 style={{ color: '#F97316', borderBottom: '2px solid #F97316', paddingBottom: 16 }}>
        Order Management
      </h1>
 
      <div style={{ background: '#1F2937', padding: 24, borderRadius: 8, marginBottom: 32 }}>
        <h2 style={{ marginTop: 0 }}>Place New Order</h2>
        <form onSubmit={createOrder} style={{ display: 'flex', gap: 12, flexWrap: 'wrap' }}>
          <input placeholder="Item (e.g. Biryani)" value={form.item} required
                 onChange={(e) => setForm({ ...form, item: e.target.value })}
                 style={{ ...input, flex: 1, minWidth: 140 }} />
          <input type="number" min="1" placeholder="Qty" value={form.quantity} required
                 onChange={(e) => setForm({ ...form, quantity: e.target.value })}
                 style={{ ...input, width: 70 }} />
          <input placeholder="Your name" value={form.customer_name} required
                 onChange={(e) => setForm({ ...form, customer_name: e.target.value })}
                 style={{ ...input, flex: 1, minWidth: 140 }} />
          <button type="submit"
                  style={{ background: '#F97316', color: 'white', border: 'none',
                           padding: '10px 20px', borderRadius: 6, cursor: 'pointer', fontWeight: 600 }}>
            Place Order
          </button>
        </form>
        {error && <p style={{ color: '#EF4444', marginBottom: 0 }}>{error}</p>}
      </div>
 
      <div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
        <h2>All Orders {!loading && `(${orders.length})`}</h2>
        <button onClick={fetchOrders}
                style={{ background: 'transparent', border: '1px solid #F97316',
                         color: '#F97316', padding: '6px 14px', borderRadius: 6, cursor: 'pointer' }}>
          Refresh
        </button>
      </div>
 
      {loading && <p style={{ color: '#9CA3AF' }}>Loading orders...</p>}
 
      {orders.map((order) => (
        <div key={order.id}
             style={{ background: '#1F2937', padding: '16px 20px', borderRadius: 8, marginBottom: 8,
                      display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
          <div>
            <strong>{order.item} x {order.quantity}</strong>
            <p style={{ color: '#9CA3AF', margin: '4px 0 0', fontSize: 14 }}>{order.customer_name}</p>
          </div>
          <span style={{ background: statusColor[order.status] || '#6B7280', color: 'white',
                         padding: '4px 14px', borderRadius: 20, fontSize: 13 }}>
            {order.status}
          </span>
        </div>
      ))}
    </div>
  );
}

Milestone 2: Containerise the Application

This milestone packages both services into small non-root container images and proves they work together with Docker Compose before Kubernetes is involved.

A container image packages your app with its runtime and libraries so it runs the same on your laptop, in Kubernetes and in any cloud. A Dockerfile is the recipe, an image is the built result, and a container is a running instance of it.

Both Dockerfiles use multi-stage builds: one stage installs and builds, and the final stage copies in only what is needed to run. The result is a smaller image with fewer tools for an attacker to use.

2.1 Backend Dockerfile

Dockerfile

# backend/Dockerfile
 
# Stage 1: install production dependencies
FROM node:22-alpine AS deps
WORKDIR /app
 
# Copy package files first. If they have not changed, Docker reuses the cached layer
# and rebuilds are much faster when only your source code changed.
COPY package*.json ./
RUN npm ci --omit=dev
 
# Stage 2: final image
FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
 
# Never run application code as root
RUN addgroup -g 1001 -S nodejs && adduser -S nodeuser -u 1001 -G nodejs
 
COPY --from=deps --chown=nodeuser:nodejs /app/node_modules ./node_modules
COPY --chown=nodeuser:nodejs package.json ./
COPY --chown=nodeuser:nodejs src/ ./src/
 
USER 1001
 
EXPOSE 4000
 
CMD ["node", "src/index.js"]

2.2 Frontend Dockerfile and Nginx Config

Nginx serves static files far more efficiently than Node.js. The unprivileged Nginx image runs as a non-root user and listens on port 8080 (unprivileged users cannot bind port 80).

TEXT

# frontend/nginx.conf
server {
    listen 8080;
 
    location / {
        root   /usr/share/nginx/html;
        index  index.html;
        # React is a single-page app. Unknown paths must fall back to index.html.
        try_files $uri $uri/ /index.html;
    }
 
    # Kubernetes probes use this
    location = /health {
        default_type text/plain;
        return 200 'ok';
    }
}

In Kubernetes the Ingress routes /api to the backend, so Nginx does not need to proxy API calls.

Dockerfile

# frontend/Dockerfile
 
# Stage 1: build the static files
FROM node:22-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
 
# The API address is baked into the JavaScript at build time
ARG VITE_API_URL=/api
ENV VITE_API_URL=$VITE_API_URL
RUN npm run build
 
# Stage 2: serve with Nginx running as a non-root user
FROM nginxinc/nginx-unprivileged:stable-alpine AS runner
COPY --from=builder /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 8080

Bash

printf "node_modules\ndist\n" > frontend/.dockerignore

2.3 Test Everything Locally First

Docker Compose runs several containers together on one machine. It is for local testing only, because it cannot scale, heal or schedule across machines. Debugging here is much faster than debugging inside Kubernetes.

YAML

# docker-compose.yml (LOCAL TESTING ONLY)
services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: appdb
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD: local-dev-only
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./backend/src/init.sql:/docker-entrypoint-initdb.d/init.sql
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U appuser -d appdb"]
      interval: 5s
      retries: 10
 
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
 
  backend:
    build: ./backend
    ports:
      - "4000:4000"
    environment:
      DB_HOST: postgres
      DB_NAME: appdb
      DB_USER: appuser
      DB_PASSWORD: local-dev-only
      REDIS_HOST: redis
      PORT: "4000"
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
 
  frontend:
    build:
      context: ./frontend
      args:
        VITE_API_URL: http://localhost:4000/api   # browser calls the API directly in Compose
    ports:
      - "3000:8080"
    depends_on:
      - backend
 
volumes:
  postgres_data:

Bash

cd ..   ## back to the orders-platform root
docker compose up --build
 
## In another terminal, after about 30 seconds:
curl http://localhost:4000/health/ready
## Expected: {"status":"ready","checks":{"database":"ok","cache":"ok"}}
 
curl http://localhost:4000/api/orders
## Expected: "source":"database" and 5 sample orders
 
curl -X POST http://localhost:4000/api/orders \
  -H "Content-Type: application/json" \
  -d '{"item":"Dosa","quantity":2,"customer_name":"Test User"}'
## Expected: a JSON object with "id":6 and "status":"pending"
 
curl http://localhost:4000/api/orders
## Expected: "source":"database" (the POST cleared the cache)
curl http://localhost:4000/api/orders
## Expected: "source":"cache" (Redis is working)
 
curl -s http://localhost:4000/metrics | head -5
## Expected: Prometheus text such as "# HELP http_request_duration_seconds ..."
 
## Open http://localhost:3000 and place an order in the UI
 
## Prove the cache is optional: stop Redis and call the API again
docker compose stop redis
curl http://localhost:4000/api/orders      ## still works, "source":"database"
curl http://localhost:4000/health/ready    ## still ready, cache shows "degraded"
 
docker compose down

2.4 Build and Push Images to a Registry

Kubernetes pulls images from a container registry. Docker Hub is free for public images.

Bash

export REGISTRY=your-dockerhub-username     ## replace with your username
export VERSION=v1.0.0
 
docker login
 
## Use a specific version tag. The "latest" tag is mutable, which makes rollbacks and audits unreliable.
docker build -t ${REGISTRY}/orders-backend:${VERSION}  ./backend
docker build -t ${REGISTRY}/orders-frontend:${VERSION} ./frontend
 
docker push ${REGISTRY}/orders-backend:${VERSION}
docker push ${REGISTRY}/orders-frontend:${VERSION}
 
## Alternative with no registry (local clusters only):
## minikube image load ${REGISTRY}/orders-backend:${VERSION}
## kind load docker-image ${REGISTRY}/orders-backend:${VERSION}
## If you do this, set imagePullPolicy: IfNotPresent on the Deployments.

Milestone 3: Write the Kubernetes Manifests

This milestone writes every manifest the application needs: namespace, database, cache, API, frontend, Ingress, autoscaling, disruption budgets and network policies.

Docker Compose runs on one machine. Kubernetes manages containers across many: it restarts crashed containers, reschedules pods when a node fails, scales on load, performs rolling updates and manages configuration and secrets. You describe the desired state in YAML (declarative configuration) and Kubernetes works out how to reach it.

Building blocks used in this project:

Namespace: a virtual folder that groups our resources
Secret / ConfigMap: sensitive and non-sensitive configuration kept out of images
StatefulSet: runs pods that need a stable identity and their own storage (databases)
Deployment: runs interchangeable, stateless pods with rolling updates
Service: a stable network address in front of changing pods
Ingress: routes outside traffic to Services by path or host
HPA: adjusts replica count based on load
PodDisruptionBudget: limits how many pods maintenance may remove at once
NetworkPolicy: pod-level firewall rules

3.1 Namespace and Default Resource Limits

A LimitRange gives every container a default request and limit when its manifest does not set one, so a forgotten limit cannot let one pod consume a whole node.

YAML

# k8s/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: orders
  labels:
    app: orders
    pod-security.kubernetes.io/enforce: baseline
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: orders
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: 50m
      memory: 64Mi
    default:
      cpu: 250m
      memory: 256Mi

Bash

kubectl apply -f k8s/namespace.yaml
kubectl get namespace orders

3.2 The Database Secret (not stored in Git)

Never commit passwords. Create the Secret directly in the cluster with a generated password. Because it is not in Git, ArgoCD will neither overwrite nor delete it.

Bash

kubectl create secret generic postgres-secret \
  --namespace orders \
  --from-literal=POSTGRES_USER=appuser \
  --from-literal=POSTGRES_PASSWORD="$(openssl rand -base64 24 | tr -d '/+=')"
 
kubectl get secret postgres-secret -n orders

Security
a Kubernetes Secret is only base64-encoded unless you enable encryption at rest, and anyone who can read Secrets in the namespace can read the password. Before real production, use External Secrets Operator or Sealed Secrets backed by a secrets manager such as AWS Secrets Manager or Vault.

3.3 PostgreSQL

Why a StatefulSet? A Deployment treats pods as interchangeable. A database pod must keep reattaching to its own data. A StatefulSet gives each pod a stable name and its own PersistentVolumeClaim (via volumeClaimTemplates), so the data follows the pod identity across restarts.

Generate the init ConfigMap from the same SQL file you tested locally, so there is one source of truth:

Bash

kubectl create configmap postgres-init \
  --namespace orders \
  --from-file=init.sql=backend/src/init.sql \
  --dry-run=client -o yaml > k8s/database/postgres-init-configmap.yaml

YAML

# k8s/database/postgres.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: orders
spec:
  serviceName: postgres        # must match the headless Service below
  replicas: 1                  # a single instance. See "Before real production" at the end.
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16-alpine
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_DB
          value: appdb
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: POSTGRES_USER
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: POSTGRES_PASSWORD
        # Use a subdirectory: a freshly formatted volume contains a lost+found folder,
        # and PostgreSQL refuses to initialise in a non-empty directory.
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
        - name: init-script
          mountPath: /docker-entrypoint-initdb.d
        livenessProbe:
          exec:
            command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\""]
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\""]
          initialDelaySeconds: 5
          periodSeconds: 5
      volumes:
      - name: init-script
        configMap:
          name: postgres-init
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes: [ReadWriteOnce]
      resources:
        requests:
          storage: 5Gi
---
# Headless Service (clusterIP: None): gives the pod a stable DNS name.
# The backend connects to the host "postgres".
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: orders
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
  - port: 5432
    targetPort: 5432

3.4 Redis

Redis here is only a cache. If it restarts, the cache is empty and the API reads from PostgreSQL. So it needs no persistent storage, and a plain Deployment is correct.

YAML

# k8s/cache/redis.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: orders
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      containers:
      - name: redis
        image: redis:7-alpine
        # Pure cache: no snapshots, no append-only file, evict old keys when memory is full
        command: ["redis-server", "--save", "", "--appendonly", "no",
                  "--maxmemory", "128mb", "--maxmemory-policy", "allkeys-lru"]
        ports:
        - containerPort: 6379
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi
        livenessProbe:
          exec:
            command: ["redis-cli", "ping"]
          initialDelaySeconds: 15
          periodSeconds: 10
        readinessProbe:
          exec:
            command: ["redis-cli", "ping"]
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: orders
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379

3.5 Backend API

A Deployment keeps the requested number of identical pods running and performs rolling updates. Notice what is missing: there is no replicas field. The HorizontalPodAutoscaler (the HPA section of Milestone 3) owns the replica count. If both the Deployment and the HPA set it, every re-apply (and every ArgoCD sync) would fight the autoscaler.

YAML

# k8s/backend/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: orders
  labels:
    app: backend
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0        # never drop below the desired count during an update
      maxSurge: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001        # the user created in the Dockerfile
      containers:
      - name: backend
        image: your-dockerhub-username/orders-backend:v1.0.0   # replace with your image
        ports:
        - name: http
          containerPort: 4000
        env:
        - name: PORT
          value: "4000"
        - name: DB_HOST
          value: postgres      # the headless Service name (Kubernetes DNS)
        - name: DB_PORT
          value: "5432"
        - name: DB_NAME
          value: appdb
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: POSTGRES_USER
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: POSTGRES_PASSWORD
        - name: REDIS_HOST
          value: redis
        - name: REDIS_PORT
          value: "6379"
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]
        # requests = what the scheduler reserves (and what HPA percentages are based on)
        # limits   = the hard ceiling (CPU is throttled, memory overuse is OOMKilled)
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
        # Liveness: "restart me if I am stuck". Keep it dependency-free.
        livenessProbe:
          httpGet:
            path: /health/live
            port: http
          initialDelaySeconds: 15
          periodSeconds: 10
          failureThreshold: 3
        # Readiness: "send me traffic only if the database works"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: http
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 2
---
# The Service gives a stable name and load-balances across ready pods.
# The label app=backend is used by the Prometheus ServiceMonitor later.
apiVersion: v1
kind: Service
metadata:
  name: backend-service
  namespace: orders
  labels:
    app: backend
spec:
  selector:
    app: backend
  ports:
  - name: http
    port: 4000
    targetPort: http

3.6 Frontend

YAML

# k8s/frontend/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: orders
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 101          # the nginx user in the unprivileged image
      containers:
      - name: frontend
        image: your-dockerhub-username/orders-frontend:v1.0.0   # replace with your image
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 256Mi
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  namespace: orders
spec:
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 8080

3.7 Ingress: The Front Door

Without an Ingress, every public application would need its own cloud load balancer. An Ingress controller is one load balancer that routes traffic to many Services, and an Ingress resource holds the routing rules.

Common Mistake
using the community ingress-nginx controller, which was retired by the Kubernetes project in March 2026 and no longer receives security fixes, so do not use it for new work. This project uses Traefik, which implements the same Ingress API. In real environments you will increasingly meet the newer Gateway API as well; the Ingress resource below is the simplest way to learn the routing idea.

Bash

helm repo add traefik https://traefik.github.io/charts
helm repo update
helm install traefik traefik/traefik --namespace traefik --create-namespace
 
kubectl get pods -n traefik
## Expected: a traefik pod Running

YAML

# k8s/ingress/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: orders-ingress
  namespace: orders
spec:
  ingressClassName: traefik
  rules:
  - host: orders.local
    http:
      paths:
      # Longest, most specific path first
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: backend-service
            port:
              number: 4000
      # Everything else goes to the React app
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 80

To reach it from your machine without a cloud load balancer, forward the Traefik port and use a Host header:

Bash

## Terminal 1 (keep running)
kubectl port-forward -n traefik svc/traefik 8000:80
 
## Terminal 2: send requests with the host name the Ingress expects
curl -H "Host: orders.local" http://localhost:8000/api/orders
 
## For the browser, add this line to /etc/hosts, then open http://orders.local:8000
## 127.0.0.1 orders.local

3.8 HPA and PodDisruptionBudgets

HPA (HorizontalPodAutoscaler) adds pods when load rises and removes them when it falls. It computes utilisation as a percentage of the pod's CPU request, so containers without requests cannot be autoscaled. It needs the Metrics Server (already enabled on minikube above; on other clusters install it from the Kubernetes SIG metrics-server release).

A PodDisruptionBudget limits how many pods voluntary disruptions (node drains, cluster upgrades) may remove at once.

YAML

# k8s/backend/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backend-hpa
  namespace: orders
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backend
  minReplicas: 2     # redundancy: one pod can fail and the service stays up
  maxReplicas: 6     # cost control
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Pods
        value: 2
        periodSeconds: 30
    # Scale down slowly to avoid flapping (adding and removing pods repeatedly)
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60

YAML

# k8s/backend/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: backend-pdb
  namespace: orders
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: backend
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: frontend-pdb
  namespace: orders
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: frontend

3.9 NetworkPolicies: Security Boundaries Between Services

By default every pod can talk to every other pod. If the frontend pod were compromised, an attacker could connect straight to PostgreSQL. A NetworkPolicy is a pod-level firewall. The production pattern is default deny, then allow only the connections that are needed (the principle of least privilege).

Two details cause most NetworkPolicy bugs:

Both ends need permission. With default-deny egress and ingress, a connection needs an egress rule on the sender and an ingress rule on the receiver.
from and ports in the same rule are combined with AND, but separate list items are combined with OR. Write them as keys of the same rule.

YAML

# k8s/network-policies.yaml
# 1. Deny everything by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: orders
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# 2. Allow all pods to look up DNS names in kube-system
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: orders
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - { port: 53, protocol: UDP }
    - { port: 53, protocol: TCP }
---
# 3. Traefik may reach the frontend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-frontend
  namespace: orders
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: traefik
    ports:
    - port: 8080
---
# 4. Traefik may reach the backend API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-backend
  namespace: orders
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: traefik
    ports:
    - port: 4000
---
# 5. Prometheus (in the monitoring namespace) may scrape the backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-to-backend
  namespace: orders
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
    ports:
    - port: 4000
---
# 6. Only the backend may connect to PostgreSQL
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-to-postgres
  namespace: orders
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - port: 5432
---
# 7. Only the backend may connect to Redis
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-to-redis
  namespace: orders
spec:
  podSelector:
    matchLabels:
      app: redis
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - port: 6379
---
# 8. The backend may send traffic to PostgreSQL and Redis (the sender side of rules 6 and 7)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-egress
  namespace: orders
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes: [Egress]
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - port: 5432
  - to:
    - podSelector:
        matchLabels:
          app: redis
    ports:
    - port: 6379

Security
NetworkPolicies only work if the cluster's network plugin (CNI) enforces them, for example Calico or Cilium. Managed clusters need the feature enabled (for example the network policy option on EKS's VPC CNI). On a cluster without enforcement, the policies are accepted silently and do nothing, so always run the isolation test in Milestone 4.

Milestone 4: Deploy and Test on Kubernetes

This milestone applies the manifests in dependency order and checks that traffic, isolation and autoscaling behave as designed.

Order matters: the namespace and Secret first, then the data layer, then the application, then routing and policies.

Bash

## 0. Point the manifests at your images (run once, from the repo root)
sed -i.bak "s#your-dockerhub-username#${REGISTRY}#g" \
  k8s/backend/deployment.yaml k8s/frontend/deployment.yaml && rm k8s/backend/*.bak k8s/frontend/*.bak
 
## 1. Namespace and Secret (created in Milestone 3)
 
## 2. Data layer
kubectl apply -f k8s/database/
kubectl apply -f k8s/cache/
 
kubectl wait --for=condition=Ready pod -l app=postgres -n orders --timeout=180s
kubectl wait --for=condition=Ready pod -l app=redis    -n orders --timeout=60s
 
## 3. Application layer
kubectl apply -f k8s/backend/
kubectl apply -f k8s/frontend/
 
## 4. Routing
kubectl apply -f k8s/ingress/
 
## 5. Isolation (last, so you can compare behaviour before and after)
kubectl get pods -n orders
kubectl apply -f k8s/network-policies.yaml

Verify Everything Is Working

Bash

## All pods Running and READY 1/1 (0/1 means the readiness probe is failing)
kubectl get pods -n orders
## Expected: backend x2, frontend x2, postgres-0, redis-...
 
kubectl get svc,ingress,hpa,pdb,pvc -n orders
## postgres-data-postgres-0 should be Bound; the HPA should show a CPU target
 
## Through the front door (port-forward from Milestone 3 (Ingress section) must be running)
curl -H "Host: orders.local" http://localhost:8000/api/orders
curl -s -o /dev/null -w "%{http_code}\n" -H "Host: orders.local" http://localhost:8000/
## Expected: orders JSON, then 200
 
## The backend's /metrics and /health/ready are not routed by the Ingress.
## Only /api reaches the backend, so these requests are answered by the React app (HTML), not the API:
curl -s -H "Host: orders.local" http://localhost:8000/metrics | head -c 80
## Expected: HTML from the frontend, not Prometheus text

Prove the NetworkPolicies Work

Bash

## The frontend must NOT be able to reach PostgreSQL (expect a timeout and the message)
kubectl exec -n orders deploy/frontend -- nc -w 3 -z postgres 5432 \
  && echo "FAIL: frontend reached postgres" || echo "PASS: frontend blocked from postgres"
 
## The backend MUST be able to reach PostgreSQL
kubectl exec -n orders deploy/backend -- nc -w 3 -z postgres 5432 \
  && echo "PASS: backend reaches postgres" || echo "FAIL: backend blocked"
 
## The app still works end to end
curl -H "Host: orders.local" http://localhost:8000/api/orders

If the frontend test prints FAIL, your cluster is not enforcing NetworkPolicies. Fix the CNI before continuing. This is the most common silent failure in this project.

Test Auto-Scaling

Bash

## Install hey, a simple load generator
## Mac: brew install hey      Linux: go install github.com/rakyll/hey@latest (or download the release binary)
 
## Terminal A: watch the HPA
kubectl get hpa -n orders --watch
 
## Terminal B: generate load through the Ingress
hey -z 3m -c 50 -host orders.local http://localhost:8000/api/orders
 
## Watch REPLICAS rise from 2 as CPU passes 70% of requests.
## After the load stops, scale-down takes about 5 minutes because of the stabilisation window.

Note down the highest replica count reached, the CPU percentage at that point, and how long scale-up and scale-down took. These are your own measurements for RESULTS.md. They depend on your hardware, so do not compare them with anyone else's.

Tip
if the API is so cheap that one pod never reaches 70% CPU, lower averageUtilization to 30 for the test, or raise -c. The goal is to see the HPA react, and the right threshold in production comes from load testing your real workload.

Milestone 5: Add Prometheus Monitoring and Alerts

This milestone installs Prometheus and Grafana, scrapes the backend and adds alerts that fire when the platform is unhealthy.

Prometheus scrapes metrics from /metrics endpoints and stores them as time series. Grafana draws them as dashboards. Alertmanager routes alerts. They answer "is it healthy right now?" and "what happened two hours ago?".

5.1 Install the Monitoring Stack

Bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
 
## kube-prometheus-stack bundles Prometheus, Grafana, Alertmanager, node exporter and kube-state-metrics.
## The two "NilUsesHelmValues=false" flags let Prometheus discover ServiceMonitors and rules from any namespace.
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false
 
kubectl get pods -n monitoring --watch
## Wait for all pods to be Running (2 to 5 minutes)

5.2 ServiceMonitor: Tell Prometheus About the Backend

YAML

# k8s/monitoring/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: backend-monitor
  namespace: monitoring
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: backend             # matches the label on backend-service
  namespaceSelector:
    matchNames:
    - orders
  endpoints:
  - port: http                 # the named port on backend-service
    path: /metrics
    interval: 15s

5.3 Alerts

YAML

# k8s/monitoring/alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: orders-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
  - name: orders
    interval: 30s
    rules:
    - alert: BackendNoReadyPods
      expr: kube_deployment_status_replicas_ready{deployment="backend", namespace="orders"} < 1
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Backend has no ready pods, so users cannot reach the API"
 
    - alert: DatabaseNotReady
      expr: kube_pod_status_ready{condition="true", pod="postgres-0", namespace="orders"} == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "PostgreSQL is not ready, so database operations are failing"
 
    - alert: BackendHighErrorRate
      expr: |
        sum(rate(http_request_duration_seconds_count{status_code=~"5..", namespace="orders"}[5m]))
        /
        sum(rate(http_request_duration_seconds_count{namespace="orders"}[5m])) > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "More than 5% of API requests are failing with 5xx errors"
 
    - alert: BackendHPAMaxedOut
      expr: |
        kube_horizontalpodautoscaler_status_current_replicas{horizontalpodautoscaler="backend-hpa", namespace="orders"}
        ==
        kube_horizontalpodautoscaler_spec_max_replicas{horizontalpodautoscaler="backend-hpa", namespace="orders"}
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Backend HPA is at maximum replicas. Consider raising maxReplicas or investigating load."

Bash

kubectl apply -f k8s/monitoring/servicemonitor.yaml
kubectl apply -f k8s/monitoring/alerts.yaml

5.4 Verify Scraping and Open Grafana

Bash

## Prometheus targets: the backend should be "up"
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090 &
sleep 3
curl -s http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets[] | select(.labels.namespace=="orders") | {pod: .labels.pod, health: .health}'
## Expected: "health": "up" for each backend pod
 
## Grafana: read the generated admin password instead of using a hardcoded one
kubectl get secret monitoring-grafana -n monitoring \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo
kubectl port-forward -n monitoring svc/monitoring-grafana 3001:80
## Open http://localhost:3001   user: admin

In Grafana, the stack already includes Kubernetes dashboards (Dashboards -> Kubernetes / Compute Resources / Namespace (Pods), choose namespace orders). Then build one panel of your own using your app's metrics:

Request rate: sum(rate(http_request_duration_seconds_count{namespace="orders"}[1m]))
p95 latency: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{namespace="orders"}[5m])) by (le))

Run the load test again and screenshot request rate, latency and replica count together for your portfolio.

Trigger one alert on purpose to prove the chain works: kubectl scale statefulset postgres -n orders --replicas=0, wait about 90 seconds, open Prometheus -> Alerts and see DatabaseNotReady fire. Then restore it with --replicas=1.

Milestone 6: Deliver with ArgoCD GitOps

This milestone installs ArgoCD and makes Git the source of truth, so the cluster follows your repository.

Right now you deliver with kubectl apply. Problems with that: someone can kubectl edit a live object and leave no record, a rebuilt cluster needs you to remember every file and its order, and there is no history of what changed and why.

GitOps makes Git the single source of truth. An operator inside the cluster (ArgoCD) watches the repository and makes the cluster match it. You ship by committing and pushing, and you roll back with git revert.

6.1 Install ArgoCD

Bash

kubectl create namespace argocd
 
## --server-side avoids the "annotation too long" error on ArgoCD's large CRDs
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
 
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s
 
## Initial admin password
kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath="{.data.password}" | base64 -d; echo
 
kubectl port-forward svc/argocd-server -n argocd 8080:443
## Open https://localhost:8080   user: admin

Tip
stable moves as new releases come out. For anything beyond a lab, pin the manifest URL to a specific ArgoCD version so upgrades are deliberate.

6.2 Push the Repository to GitHub

Create an empty GitHub repository named orders-platform, then:

Bash

printf "node_modules\ndist\n.env\n*.bak\n" > .gitignore
git add .
git commit -m "feat: orders platform application and Kubernetes manifests"
git branch -M main
git remote add origin https://github.com/YOUR-USERNAME/orders-platform.git
git push -u origin main
 
## Check that no secret was committed
git grep -i "password" -- k8s/ || echo "no passwords in k8s/ (good)"
## Only secretKeyRef references and the key names should appear, never a password value

6.3 Create the ArgoCD Application

An ArgoCD Application connects a Git path to a cluster namespace. Two settings matter:

directory.recurse: true: our manifests live in subfolders (k8s/backend, k8s/database, and so on). Without it ArgoCD reads only the top-level files of k8s/ and silently deploys almost nothing.
selfHeal and prune: ArgoCD reverts manual changes and deletes resources that were removed from Git.

YAML

# argocd-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-platform
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR-USERNAME/orders-platform.git   # replace
    targetRevision: main
    path: k8s
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: orders
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Bash

kubectl apply -f argocd-app.yaml
 
kubectl get application orders-platform -n argocd --watch
## Expected: SYNC STATUS Synced, HEALTH STATUS Healthy

If the repository is private, add it under ArgoCD Settings -> Repositories with a read-only token first.

6.4 Prove GitOps Works

Test 1: change through Git. Raise the HPA minimum from 2 to 3:

Bash

## edit k8s/backend/hpa.yaml: minReplicas: 2  ->  minReplicas: 3
git add k8s/backend/hpa.yaml
git commit -m "scale: raise backend minReplicas to 3"
git push
 
kubectl get pods -n orders -l app=backend --watch
## A third backend pod appears within a few minutes (ArgoCD polls Git about every 3 minutes by default)

Test 2: self-heal. Make a change outside Git and watch ArgoCD undo it:

Bash

kubectl patch hpa backend-hpa -n orders -p '{"spec":{"minReplicas":5}}'
kubectl get hpa backend-hpa -n orders -w
## ArgoCD detects the drift and restores minReplicas to the value in Git

Test 3: rollback. Run git revert HEAD && git push and watch the cluster return to the earlier state. That is your audit trail and your undo button.

Validation & Testing

This final topic checks the whole platform end to end and collects the evidence for your portfolio.

Production Checklist

Run each check and tick it off in your RESULTS.md.

Bash

## 1. Every pod is Running and Ready
kubectl get pods -n orders
 
## 2. Probes are configured and passing
kubectl describe pod -l app=backend -n orders | grep -E "Liveness|Readiness"
 
## 3. Every container has requests and limits
kubectl get pods -n orders -o json | \
  jq '.items[].spec.containers[] | {name: .name, requests: .resources.requests, limits: .resources.limits}'
 
## 4. Containers do not run as root
kubectl get pods -n orders -o json | \
  jq '.items[] | {pod: .metadata.name, runAsNonRoot: .spec.securityContext.runAsNonRoot}'
## postgres-0 is the one documented exception: its image starts as root to initialise, then drops privileges
 
## 5. Database storage is persistent and bound
kubectl get pvc -n orders
 
## 6. No password values in the Deployment
kubectl get deployment backend -n orders -o yaml | grep -iA3 "PASSWORD"
## Expected: only a secretKeyRef, never a literal value
 
## 7. HPA is active and reading metrics
kubectl get hpa -n orders       ## TARGETS shows something like 12%/70%, not <unknown>
 
## 8. PodDisruptionBudgets exist
kubectl get pdb -n orders
 
## 9. Network isolation holds (re-run the Milestone 4 tests)
 
## 10. ArgoCD is Synced and Healthy
kubectl get application orders-platform -n argocd
 
## 11. Prometheus scrapes the backend (Milestone 5) and the alerts are loaded
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].name' | grep orders
 
## 12. A node drain does not cause downtime (needs 2+ nodes; skip on single-node minikube)
## In one terminal: while true; do curl -s -o /dev/null -w "%{http_code}\n" -H "Host: orders.local" http://localhost:8000/api/orders; sleep 0.5; done
## In another: kubectl drain <node-with-backend-pod> --ignore-daemonsets --delete-emptydir-data
## You should see 200s throughout. Then: kubectl uncordon <node>

Common Production Mistakes

Common Mistake
Using a Deployment instead of a StatefulSet for PostgreSQL. The pod is rescheduled and the shared claim cannot attach where the new pod lands, or two pods end up writing to the same data. The database refuses to start or, worse, data is corrupted. Fix: Use a StatefulSet with volumeClaimTemplates so each pod keeps its own volume and identity. For real production, use a database operator or a managed database (RDS, Cloud SQL).

Common Mistake
A liveness probe that checks the database. PostgreSQL slows down under load. The liveness probe times out on every backend pod. Kubernetes restarts them all at once, and now you have a full outage caused by a slow dependency. Fix: Liveness only answers "is this process responsive?". Readiness checks the dependencies and removes the pod from the Service without restarting it.

Common Mistake
Forgetting resource requests. The HPA cannot compute a percentage without a CPU request, shows <unknown>, and never scales. During a spike the pods sit at their limits and no new ones appear. Fix: Set requests on every container. The LimitRange gives defaults, but explicit values are better.

Common Mistake
A password in a Deployment file committed to Git. Git history is permanent. If the repository ever becomes public or is shared, the password is exposed, and rotating it means updating the database, the Secret and restarting pods under pressure. Fix: Reference Secrets with secretKeyRef. Keep the Secret itself out of Git, or encrypt it with Sealed Secrets or manage it with External Secrets Operator.

Common Mistake
No PodDisruptionBudget during a node upgrade. A drain evicts both backend pods at once. For 30 to 60 seconds there are no ready pods and users see 503 errors. Fix: minAvailable: 1 makes the drain wait until another pod is ready before evicting the next.

Common Mistake
NetworkPolicy rules written with from and ports as separate list items. Separate items mean OR, so "allow from the ingress namespace OR allow port 4000 from anywhere". The policy looks right but is far more open than intended. Fix: Put from and ports inside the same rule so they combine as AND, and test that blocked paths are really blocked.

Common Mistake
Default-deny without allowing DNS. Every pod instantly loses the ability to resolve postgres or redis. The app fails with errors that look like database failures. Fix: Apply an explicit DNS egress rule together with default-deny.

Common Mistake
Mutable latest image tags. Two pods in the same Deployment run different code because the tag changed between pulls, and you cannot tell which version is live. Fix: Use immutable version tags, and let Git history record which version was deployed when.

Debugging Playbook

Pod in CrashLoopBackOff

Bash

## Logs from the previous, crashed run
kubectl logs -n orders deploy/backend --previous
 
## ECONNREFUSED or timeout to postgres:
kubectl get pods -n orders -l app=postgres
kubectl get endpoints postgres -n orders
## Password authentication failed:
kubectl describe pod -l app=backend -n orders | grep -A12 "Environment"
## Did the Secret exist before the pods started? Recreate pods after fixing it:
kubectl rollout restart deployment/backend -n orders

Pod stuck in Pending

Bash

kubectl describe pod <pod-name> -n orders | tail -20
## "Insufficient cpu/memory": reduce requests or add capacity (kubectl describe nodes | grep -A5 "Allocated")
## "unbound immediate PersistentVolumeClaims": kubectl get storageclass (the cluster needs a default class)

502 or 404 from the Ingress

Bash

## Does the Service have endpoints? <none> means the selector matches no ready pods
kubectl get endpoints backend-service frontend-service -n orders
 
## Is the controller seeing the Ingress?
kubectl describe ingress orders-ingress -n orders
kubectl logs -n traefik deploy/traefik --tail=30
 
## Does the NetworkPolicy allow the controller's namespace? The namespace label must match:
kubectl get ns traefik --show-labels | grep kubernetes.io/metadata.name

HPA shows `` or does not scale

Bash

kubectl top pods -n orders              ## fails? Metrics Server is missing or broken
kubectl get deployment metrics-server -n kube-system
kubectl describe hpa backend-hpa -n orders
## "missing request for cpu": add resources.requests.cpu

ArgoCD says OutOfSync or deploys nothing

Bash

kubectl describe application orders-platform -n argocd | tail -30
## "app path does not exist" or empty: check path, targetRevision, and directory.recurse: true
## Permission or auth errors: add repository credentials in ArgoCD Settings

Backend returns 503 on /health/ready

Bash

kubectl exec -n orders deploy/backend -- wget -qO- http://localhost:4000/health/ready
## The "checks" object names the failing dependency

Cost and Cleanup

Local clusters cost nothing. A managed cluster with load balancers, volumes and several nodes is billed hourly, so delete it when you are done.

Bash

kubectl delete application orders-platform -n argocd     ## also removes the managed resources
helm uninstall monitoring -n monitoring
helm uninstall traefik -n traefik
kubectl delete namespace orders monitoring traefik argocd
## Managed clusters: delete the cluster and check for leftover volumes and load balancers in the cloud console

Before Real Production: What Is Still Missing

This project teaches the patterns, but a real production system would add:

A highly available database: a managed service (RDS, Cloud SQL) or a Postgres operator with replicas, backups and tested restores. A single-replica StatefulSet is a single point of failure.
Managed secrets: External Secrets Operator or Sealed Secrets, plus encryption at rest and rotation
TLS: cert-manager issuing certificates, with HTTPS at the Ingress
CI: a pipeline that tests, scans, builds and pushes images, then opens a pull request that updates the image tag in k8s/
Image supply-chain checks: vulnerability scanning and signed images
Alert routing: Alertmanager sending to Slack or PagerDuty, with runbooks
Logs and tracing: a log pipeline and distributed tracing alongside metrics
Gateway API: the successor to the Ingress API, increasingly standard for new clusters

Your Portfolio Deliverable

Your public repository orders-platform should contain:

All application code, Dockerfiles, docker-compose.yml, k8s/ manifests and argocd-app.yaml
A README.md with the architecture diagram, how to run locally, how to deploy to Kubernetes, and how to clean up
Screenshots: the running app, ArgoCD Synced and Healthy, the Grafana panels under load, and the HPA scaling
RESULTS.md with your own measurements: replica count and CPU at peak, scale-up and scale-down times, the output of the NetworkPolicy isolation test, the alert you triggered on purpose, and the result of the GitOps self-heal test
A "Decisions and trade-offs" section: why StatefulSet for PostgreSQL, why readiness ignores Redis, why Ingress and not LoadBalancers, why default-deny, and what you would change for real production (use the section above)

When an interviewer asks whether you have deployed an application to Kubernetes with production practices, you can open the repository, walk through the decisions, and show the measurements.

What You Have Built

A three-tier application with validation, caching and graceful shutdown
Multi-stage, non-root container images
A StatefulSet database with persistent storage, and a stateless cache and API
Separate liveness and readiness probes with different jobs
Path-based routing through an Ingress controller that is still maintained
Autoscaling with an HPA, and availability protection with PodDisruptionBudgets
Default-deny NetworkPolicies with tested allow rules
Prometheus metrics, alerts and a Grafana dashboard
GitOps delivery with ArgoCD, including a proven self-heal and rollback

Next up: Capstone 2, Build Production AWS Infrastructure with Terraform. That capstone provisions the cloud infrastructure this platform would run on in a real company.

Found a mistake, an outdated step or a broken link?

Build a Production Application Platform

Blueprint Walkthrough

Architecture Overview & Problem Statement

What You Build, Learn and Show

Prerequisites

What You Are Building

Milestone 1: Build the Orders Application

1.1 Create the Project Structure

1.2 Backend: The Node.js API

Database connection

Redis connection

Metrics

Orders routes

Health checks: liveness and readiness are different jobs

The server

Database schema

1.3 Frontend: The React App

Milestone 2: Containerise the Application

2.1 Backend Dockerfile

2.2 Frontend Dockerfile and Nginx Config

2.3 Test Everything Locally First

2.4 Build and Push Images to a Registry

Milestone 3: Write the Kubernetes Manifests

3.1 Namespace and Default Resource Limits

3.2 The Database Secret (not stored in Git)

3.3 PostgreSQL

3.4 Redis

3.5 Backend API

3.6 Frontend

3.7 Ingress: The Front Door

3.8 HPA and PodDisruptionBudgets

3.9 NetworkPolicies: Security Boundaries Between Services

Milestone 4: Deploy and Test on Kubernetes

Verify Everything Is Working

Prove the NetworkPolicies Work

Test Auto-Scaling

Milestone 5: Add Prometheus Monitoring and Alerts

5.1 Install the Monitoring Stack

5.2 ServiceMonitor: Tell Prometheus About the Backend

5.3 Alerts

5.4 Verify Scraping and Open Grafana

Milestone 6: Deliver with ArgoCD GitOps

6.1 Install ArgoCD

6.2 Push the Repository to GitHub

6.3 Create the ArgoCD Application

6.4 Prove GitOps Works

Validation & Testing

Production Checklist

Common Production Mistakes

Debugging Playbook

Pod in CrashLoopBackOff

Pod stuck in Pending

502 or 404 from the Ingress

HPA shows `` or does not scale

ArgoCD says OutOfSync or deploys nothing

Backend returns 503 on /health/ready

Cost and Cleanup

Before Real Production: What Is Still Missing

Your Portfolio Deliverable

What You Have Built