Deploy Kubernetes Monitoring Stack with Prometheus, Grafana and Loki

Build a production observability platform on Kubernetes with Prometheus metrics, Grafana dashboards, Loki log aggregation, and alerting.

Related Concepts & TermsAlertmanager Filesystem Deployment grep Grafana DaemonSet

Domains & Technologies

Domains

MONITORING

Technologies

KUBERNETESPROMETHEUSGRAFANA

Blueprint Walkthrough

Architecture Overview

In this project you will build the complete observability stack for a Kubernetes cluster — the same stack used by SRE teams at Zerodha, Razorpay, and Freshworks. You will deploy Prometheus for metrics collection, Grafana for visualisation and dashboards, Loki for log aggregation, and Alertmanager for routing alerts to Slack or PagerDuty.

By the end, you will have a single Grafana interface where you can correlate metrics, logs, and traces — answering "what happened?" in any production incident.

◈ DIAGRAM

    Kubernetes Cluster
    +------------------------------------------------------------------+
    |                                                                  |
    |  Applications (pods)  --- metrics/logs --►                      |
    |                                          |                      |
    |  Node Exporters (DaemonSet) -------------+                      |
    |                                          v                      |
    |  kube-state-metrics ----------►  Prometheus (scrapes all)       |
    |                                          |                      |
    |  Promtail (DaemonSet) --------------►  Loki                     |
    |  (collects pod logs)                     |                      |
    |                                          v                      |
    |                                   Grafana UI                    |
    |                                   (queries both)               |
    |                                          |                      |
    |                               Alertmanager                      |
    |                               (routes to Slack/PagerDuty)       |
    +------------------------------------------------------------------+

Problem Solved

Without observability, when something breaks in production you are flying blind. You SSH into pods, scroll through kubectl logs, run kubectl top, and try to piece together what happened from fragments. It takes hours.

With this stack, every production incident investigation starts with the same 3-step workflow: open Grafana, find the time of the anomaly on the metrics dashboard, click the log correlation button to see the exact log lines from that pod at that time. Most incidents are diagnosed in under 5 minutes.

The Loki integration specifically solves the log retention problem — pod logs disappear when pods are deleted or restarted. Loki ships logs to persistent storage as they are generated, so you always have the logs from a crashed pod even after it is gone.

Step-by-Step Implementation Guide

Step 1: Prepare Kubernetes Cluster and Install Helm

Bash

## Verify cluster is running
kubectl cluster-info
kubectl get nodes
 
## Install Helm 3
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version
 
## Add the monitoring Helm chart repositories
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
 
## Create a dedicated namespace for monitoring
kubectl create namespace monitoring

Step 2: Deploy the kube-prometheus-stack

The kube-prometheus-stack Helm chart installs Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics in a single deployment — pre-configured with Kubernetes-specific dashboards and alert rules.

Create prometheus-values.yaml:

YAML

## prometheus-values.yaml
grafana:
  enabled: true
  adminPassword: "change-me-in-production"  # Change this!
  ingress:
    enabled: false  # Enable and configure for production
  persistence:
    enabled: true
    size: 10Gi
  # Pre-load the Loki datasource
  additionalDataSources:
  - name: Loki
    type: loki
    url: http://loki:3100
    access: proxy
    isDefault: false
 
prometheus:
  prometheusSpec:
    retention: 15d  # Keep 15 days of metrics
    storageSpec:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 50Gi  # Adjust based on your metric volume
    # Scrape all ServiceMonitors and PodMonitors across namespaces
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
 
alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 2Gi
 
nodeExporter:
  enabled: true  # Collects CPU, memory, disk metrics from every node
 
kubeStateMetrics:
  enabled: true  # Collects Kubernetes object state (pod status, deployment replicas, etc.)
 
## Disable components not needed in local dev
kubeEtcd:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false

Bash

## Install the kube-prometheus-stack
helm install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values prometheus-values.yaml \
  --version 58.0.0  # Pin the version for reproducibility
 
## Watch the pods come up (takes 3-5 minutes)
kubectl get pods -n monitoring --watch
 
## All pods should be Running:
## alertmanager-kube-prometheus-stack-alertmanager-*
## kube-prometheus-stack-grafana-*
## kube-prometheus-stack-kube-state-metrics-*
## kube-prometheus-stack-operator-*
## kube-prometheus-stack-prometheus-*
## prometheus-kube-prometheus-stack-prometheus-*
## node-exporter pods (one per node)

Tip
Run helm list -n monitoring to verify the chart version and release status. Always pin chart versions in production — --version 58.0.0 — so your next helm upgrade does not pull unexpected changes.

Step 3: Access Grafana and Explore Built-in Dashboards

Bash

## Port-forward Grafana to your local machine
kubectl port-forward -n monitoring \
  svc/kube-prometheus-stack-grafana 3000:80
 
## Open http://localhost:3000
## Username: admin
## Password: change-me-in-production (value from your prometheus-values.yaml)

The kube-prometheus-stack ships with pre-built dashboards. Navigate to Dashboards in Grafana and explore:

Kubernetes / Compute Resources / Cluster — cluster-wide CPU and memory usage
Kubernetes / Compute Resources / Namespace — per-namespace resource consumption
Node Exporter / Full — detailed per-node metrics (disk I/O, network, CPU wait)
Kubernetes / Persistent Volumes — PVC capacity and usage

Step 4: Deploy Loki for Log Aggregation

Loki stores logs efficiently in object storage (S3 or local filesystem). Promtail is the agent that collects logs from each node and ships them to Loki.

Create loki-values.yaml:

YAML

## loki-values.yaml — Single binary mode for simplicity
loki:
  commonConfig:
    replication_factor: 1  # Single replica for demo; use 3 for production
  storage:
    type: filesystem  # Use S3 for production
  auth_enabled: false
  limits_config:
    retention_period: 168h  # 7 days
    ingestion_rate_mb: 16
    max_streams_per_user: 10000
 
singleBinary:
  replicas: 1
  persistence:
    enabled: true
    size: 20Gi
 
## Disable distributed components for single-binary mode
read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0

Bash

## Install Loki
helm install loki grafana/loki \
  --namespace monitoring \
  --values loki-values.yaml \
  --version 6.0.0
 
## Verify Loki is running
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki

Install Promtail (the log shipping agent — runs on every node as a DaemonSet):

Create promtail-values.yaml:

YAML

## promtail-values.yaml
config:
  clients:
  - url: http://loki:3100/loki/api/v1/push
  
  scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    # Add namespace as a label
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: namespace
    # Add pod name as a label
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: pod
    # Add container name as a label
    - source_labels: [__meta_kubernetes_pod_container_name]
      action: replace
      target_label: container
    # Add app label from pod labels
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: replace
      target_label: app

Bash

## Install Promtail as a DaemonSet
helm install promtail grafana/promtail \
  --namespace monitoring \
  --values promtail-values.yaml
 
## Verify Promtail is running on all nodes
kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail
## Expected: One pod per node, all Running

Step 5: Configure Alertmanager for Slack Notifications

Alertmanager routes Prometheus alerts to notification channels. Here you will configure it to send alerts to a Slack channel.

Create a Slack Incoming Webhook:

Go to https://api.slack.com/apps -> Create New App
Add Incoming Webhooks feature
Activate and create a webhook for your #alerts channel
Copy the webhook URL

Create the Alertmanager configuration as a Kubernetes secret:

YAML

## alertmanager-config.yaml
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-kube-prometheus-stack-alertmanager
  namespace: monitoring
stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'slack-critical'
      routes:
      - match:
          severity: critical
        receiver: 'slack-critical'
      - match:
          severity: warning
        receiver: 'slack-warning'
 
    receivers:
    - name: 'slack-critical'
      slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#alerts-critical'
        color: 'danger'
        title: '{{ .CommonAnnotations.summary }}'
        text: |-
          {{ range .Alerts }}
          *Cluster:* {{ .Labels.cluster }}
          *Namespace:* {{ .Labels.namespace }}
          *Pod:* {{ .Labels.pod }}
          *Description:* {{ .Annotations.description }}
          {{ end }}
    - name: 'slack-warning'
      slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#alerts-warning'
        color: 'warning'
        title: '{{ .CommonAnnotations.summary }}'

Bash

kubectl apply -f alertmanager-config.yaml
 
## Restart Alertmanager to pick up new config
kubectl rollout restart statefulset \
  alertmanager-kube-prometheus-stack-alertmanager \
  -n monitoring

Remember
Alertmanager silences work on labels. If an alert fires too frequently and is causing noise, use amtool silence add to silence it for a specific duration rather than deleting the alert rule — you might need that rule for a real incident.

Step 6: Deploy a Test Application with Custom Metrics

Deploy a real application that exposes Prometheus metrics so you can see the full observability pipeline end to end.

Bash

## Deploy the Prometheus example app (exposes metrics at /metrics)
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      * name: example-app
        image: quay.io/prometheus/prometheus-example-app:latest
        ports:
        * name: http
          containerPort: 8080
        * name: metrics
          containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: example-app
  namespace: default
  labels:
    app: example-app
spec:
  ports:
  * name: http
    port: 8080
  selector:
    app: example-app
---
## ServiceMonitor tells Prometheus to scrape this service
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  namespace: default
  labels:
    release: kube-prometheus-stack  # Must match Prometheus selector
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  * port: http
    path: /metrics
    interval: 15s
EOF
 
## Verify Prometheus is scraping the new target
## Port-forward Prometheus UI
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
## Open http://localhost:9090/targets
## Expected: example-app target should appear with state UP

Validation & Testing

Bash

## 1. Verify all monitoring components are healthy
kubectl get pods -n monitoring
## Expected: All pods Running
 
## 2. Check Prometheus targets (no DOWN targets)
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
## Open http://localhost:9090/targets
## Check: All targets show state = UP
 
## 3. Test Loki log ingestion
kubectl port-forward -n monitoring svc/loki 3100:3100
curl http://localhost:3100/ready
## Expected: ready
 
## Query Loki for recent logs from the monitoring namespace
curl -G 'http://localhost:3100/loki/api/v1/query' \
  --data-urlencode 'query={namespace="monitoring"}' \
  --data-urlencode 'limit=5'
## Expected: JSON response with recent log lines
 
## 4. Test end-to-end in Grafana
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80
## Open Grafana -> Explore -> Select Loki datasource
## Query: {namespace="default", app="example-app"}
## Expected: Log lines from the example app appear
 
## 5. Trigger a test alert
## Create a pod that consumes high CPU
kubectl run cpu-stress --image=progrium/stress \
  --restart=Never -- --cpu 2 --timeout 60s
## Wait 2-5 minutes, then check Alertmanager
kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093
## Open http://localhost:9093
## Expected: An alert for high CPU usage should appear
 
## Clean up
kubectl delete pod cpu-stress
kubectl delete deployment example-app
kubectl delete svc example-app
kubectl delete servicemonitor example-app

Common Mistake
Creating a ServiceMonitor but Prometheus is not picking it up. Check that the labels on your ServiceMonitor match the serviceMonitorSelector configured on the Prometheus resource. Run kubectl get prometheus -n monitoring -o yaml | grep serviceMonitorSelector to see what labels Prometheus is looking for.

Videos & Guides

Video Guide

Grafana Loki — Complete Logging Stack Tutorial

Hands-on tutorial covering the full PLG stack (Prometheus, Loki, Grafana) deployment on Kubernetes with Promtail log shipping and dashboard creation.

View Reference

DOCUMENTATION

kube-prometheus-stack Helm Chart Documentation

Official documentation for the kube-prometheus-stack Helm chart covering all configuration values, CRDs, and upgrade procedures.

View Reference

DOCUMENTATION

Grafana Loki Documentation — LogQL Query Language

Official Loki LogQL documentation — learn how to query, filter, and aggregate log data in Grafana for production incident investigation.

View Reference