In this project you will build the complete observability stack for a Kubernetes cluster — the same stack used by SRE teams at Zerodha, Razorpay, and Freshworks. You will deploy Prometheus for metrics collection, Grafana for visualisation and dashboards, Loki for log aggregation, and Alertmanager for routing alerts to Slack or PagerDuty. By the end, you will have a single Grafana interface where you can correlate metrics, logs, and traces — answering "what happened?" in any production incident. Kubernetes Cluster +------------------------------------------------------------------+ | | | Applications (pods) --- metrics/logs --► | | | | | Node Exporters (DaemonSet) -------------+ | | v | | kube-state-metrics ----------► Prometheus (scrapes all) | | | | | Promtail (DaemonSet) --------------► Loki | | (collects pod logs) | | | v | | Grafana UI | | (queries both) | | | | | Alertmanager | | (routes to Slack/PagerDuty) | +------------------------------------------------------------------+
Without observability, when something breaks in production you are flying blind. You SSH into pods, scroll through `kubectl logs`, run `kubectl top`, and try to piece together what happened from fragments. It takes hours. With this stack, every production incident investigation starts with the same 3-step workflow: open Grafana, find the time of the anomaly on the metrics dashboard, click the log correlation button to see the exact log lines from that pod at that time. Most incidents are diagnosed in under 5 minutes. The Loki integration specifically solves the log retention problem — pod logs disappear when pods are deleted or restarted. Loki ships logs to persistent storage as they are generated, so you always have the logs from a crashed pod even after it is gone.
### Step 1: Prepare Kubernetes Cluster and Install Helm ```bash ## Verify cluster is running kubectl cluster-info kubectl get nodes ## Install Helm 3 curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash helm version ## Add the monitoring Helm chart repositories helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add grafana https://grafana.github.io/helm-charts helm repo update ## Create a dedicated namespace for monitoring kubectl create namespace monitoring ``` ### Step 2: Deploy the kube-prometheus-stack The `kube-prometheus-stack` Helm chart installs Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics in a single deployment — pre-configured with Kubernetes-specific dashboards and alert rules. Create `prometheus-values.yaml`: ```yaml ## prometheus-values.yaml grafana: enabled: true adminPassword: "change-me-in-production" # Change this! ingress: enabled: false # Enable and configure for production persistence: enabled: true size: 10Gi # Pre-load the Loki datasource additionalDataSources: * name: Loki type: loki url: http://loki:3100 access: proxy isDefault: false prometheus: prometheusSpec: retention: 15d # Keep 15 days of metrics storageSpec: volumeClaimTemplate: spec: resources: requests: storage: 50Gi # Adjust based on your metric volume # Scrape all ServiceMonitors and PodMonitors across namespaces serviceMonitorSelectorNilUsesHelmValues: false podMonitorSelectorNilUsesHelmValues: false alertmanager: alertmanagerSpec: storage: volumeClaimTemplate: spec: resources: requests: storage: 2Gi nodeExporter: enabled: true # Collects CPU, memory, disk metrics from every node kubeStateMetrics: enabled: true # Collects Kubernetes object state (pod status, deployment replicas, etc.) ## Disable components not needed in local dev kubeEtcd: enabled: false kubeControllerManager: enabled: false kubeScheduler: enabled: false ``` ```bash ## Install the kube-prometheus-stack helm install kube-prometheus-stack \ prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --values prometheus-values.yaml \ --version 58.0.0 # Pin the version for reproducibility ## Watch the pods come up (takes 3-5 minutes) kubectl get pods -n monitoring --watch ## All pods should be Running: ## alertmanager-kube-prometheus-stack-alertmanager-* ## kube-prometheus-stack-grafana-* ## kube-prometheus-stack-kube-state-metrics-* ## kube-prometheus-stack-operator-* ## kube-prometheus-stack-prometheus-* ## prometheus-kube-prometheus-stack-prometheus-* ## node-exporter pods (one per node) ``` > 💡 **Tip:** Run `helm list -n monitoring` to verify the chart version and release status. Always pin chart versions in production — `--version 58.0.0` — so your next `helm upgrade` does not pull unexpected changes. ### Step 3: Access Grafana and Explore Built-in Dashboards ```bash ## Port-forward Grafana to your local machine kubectl port-forward -n monitoring \ svc/kube-prometheus-stack-grafana 3000:80 ## Open http://localhost:3000 ## Username: admin ## Password: change-me-in-production (value from your prometheus-values.yaml) ``` The kube-prometheus-stack ships with pre-built dashboards. Navigate to **Dashboards** in Grafana and explore: * **Kubernetes / Compute Resources / Cluster** — cluster-wide CPU and memory usage * **Kubernetes / Compute Resources / Namespace** — per-namespace resource consumption * **Node Exporter / Full** — detailed per-node metrics (disk I/O, network, CPU wait) * **Kubernetes / Persistent Volumes** — PVC capacity and usage ### Step 4: Deploy Loki for Log Aggregation Loki stores logs efficiently in object storage (S3 or local filesystem). Promtail is the agent that collects logs from each node and ships them to Loki. Create `loki-values.yaml`: ```yaml ## loki-values.yaml — Single binary mode for simplicity loki: commonConfig: replication_factor: 1 # Single replica for demo; use 3 for production storage: type: filesystem # Use S3 for production auth_enabled: false limits_config: retention_period: 168h # 7 days ingestion_rate_mb: 16 max_streams_per_user: 10000 singleBinary: replicas: 1 persistence: enabled: true size: 20Gi ## Disable distributed components for single-binary mode read: replicas: 0 write: replicas: 0 backend: replicas: 0 ``` ```bash ## Install Loki helm install loki grafana/loki \ --namespace monitoring \ --values loki-values.yaml \ --version 6.0.0 ## Verify Loki is running kubectl get pods -n monitoring -l app.kubernetes.io/name=loki ``` **Install Promtail** (the log shipping agent — runs on every node as a DaemonSet): Create `promtail-values.yaml`: ```yaml ## promtail-values.yaml config: clients: * url: http://loki:3100/loki/api/v1/push scrape_configs: * job_name: kubernetes-pods kubernetes_sd_configs: * role: pod relabel_configs: # Add namespace as a label * source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace # Add pod name as a label * source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod # Add container name as a label * source_labels: [__meta_kubernetes_pod_container_name] action: replace target_label: container # Add app label from pod labels * source_labels: [__meta_kubernetes_pod_label_app] action: replace target_label: app ``` ```bash ## Install Promtail as a DaemonSet helm install promtail grafana/promtail \ --namespace monitoring \ --values promtail-values.yaml ## Verify Promtail is running on all nodes kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail ## Expected: One pod per node, all Running ``` ### Step 5: Configure Alertmanager for Slack Notifications Alertmanager routes Prometheus alerts to notification channels. Here you will configure it to send alerts to a Slack channel. **Create a Slack Incoming Webhook:** 1. Go to `https://api.slack.com/apps` -> Create New App 2. Add **Incoming Webhooks** feature 3. Activate and create a webhook for your `#alerts` channel 4. Copy the webhook URL Create the Alertmanager configuration as a Kubernetes secret: ```yaml ## alertmanager-config.yaml apiVersion: v1 kind: Secret metadata: name: alertmanager-kube-prometheus-stack-alertmanager namespace: monitoring stringData: alertmanager.yaml: | global: resolve_timeout: 5m route: group_by: ['alertname', 'namespace'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'slack-critical' routes: * match: severity: critical receiver: 'slack-critical' * match: severity: warning receiver: 'slack-warning' receivers: * name: 'slack-critical' slack_configs: * api_url: 'YOUR_SLACK_WEBHOOK_URL' channel: '#alerts-critical' color: 'danger' title: '{{ .CommonAnnotations.summary }}' text: |- {{ range .Alerts }} *Cluster:* {{ .Labels.cluster }} *Namespace:* {{ .Labels.namespace }} *Pod:* {{ .Labels.pod }} *Description:* {{ .Annotations.description }} {{ end }} * name: 'slack-warning' slack_configs: * api_url: 'YOUR_SLACK_WEBHOOK_URL' channel: '#alerts-warning' color: 'warning' title: '{{ .CommonAnnotations.summary }}' ``` ```bash kubectl apply -f alertmanager-config.yaml ## Restart Alertmanager to pick up new config kubectl rollout restart statefulset \ alertmanager-kube-prometheus-stack-alertmanager \ -n monitoring ``` > 📌 **Remember:** Alertmanager silences work on labels. If an alert fires too frequently and is causing noise, use `amtool silence add` to silence it for a specific duration rather than deleting the alert rule — you might need that rule for a real incident. ### Step 6: Deploy a Test Application with Custom Metrics Deploy a real application that exposes Prometheus metrics so you can see the full observability pipeline end to end. ```bash ## Deploy the Prometheus example app (exposes metrics at /metrics) kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: example-app namespace: default spec: replicas: 3 selector: matchLabels: app: example-app template: metadata: labels: app: example-app spec: containers: * name: example-app image: quay.io/prometheus/prometheus-example-app:latest ports: * name: http containerPort: 8080 * name: metrics containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: example-app namespace: default labels: app: example-app spec: ports: * name: http port: 8080 selector: app: example-app --- ## ServiceMonitor tells Prometheus to scrape this service apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: example-app namespace: default labels: release: kube-prometheus-stack # Must match Prometheus selector spec: selector: matchLabels: app: example-app endpoints: * port: http path: /metrics interval: 15s EOF ## Verify Prometheus is scraping the new target ## Port-forward Prometheus UI kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 ## Open http://localhost:9090/targets ## Expected: example-app target should appear with state UP ```
```bash ## 1. Verify all monitoring components are healthy kubectl get pods -n monitoring ## Expected: All pods Running ## 2. Check Prometheus targets (no DOWN targets) kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 ## Open http://localhost:9090/targets ## Check: All targets show state = UP ## 3. Test Loki log ingestion kubectl port-forward -n monitoring svc/loki 3100:3100 curl http://localhost:3100/ready ## Expected: ready ## Query Loki for recent logs from the monitoring namespace curl -G 'http://localhost:3100/loki/api/v1/query' \ --data-urlencode 'query={namespace="monitoring"}' \ --data-urlencode 'limit=5' ## Expected: JSON response with recent log lines ## 4. Test end-to-end in Grafana kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80 ## Open Grafana -> Explore -> Select Loki datasource ## Query: {namespace="default", app="example-app"} ## Expected: Log lines from the example app appear ## 5. Trigger a test alert ## Create a pod that consumes high CPU kubectl run cpu-stress --image=progrium/stress \ --restart=Never -- --cpu 2 --timeout 60s ## Wait 2-5 minutes, then check Alertmanager kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093 ## Open http://localhost:9093 ## Expected: An alert for high CPU usage should appear ## Clean up kubectl delete pod cpu-stress kubectl delete deployment example-app kubectl delete svc example-app kubectl delete servicemonitor example-app ``` > 🔴 **Common Mistake:** Creating a ServiceMonitor but Prometheus is not picking it up. Check that the `labels` on your ServiceMonitor match the `serviceMonitorSelector` configured on the Prometheus resource. Run `kubectl get prometheus -n monitoring -o yaml | grep serviceMonitorSelector` to see what labels Prometheus is looking for.
In this project you will build the complete observability stack for a Kubernetes cluster — the same stack used by SRE te...
Without observability, when something breaks in production you are flying blind. You SSH into pods, scroll through kubec...
Step 1: Prepare Kubernetes Cluster and Install Helm Step 2: Deploy the kube-prometheus-stack The kube-prometheus-stack H...
> 🔴 Common Mistake: Creating a ServiceMonitor but Prometheus is not picking it up. Check that the labels on your Servic...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.