Kubernetes Monitoring with Prometheus and Loki

Deploy Prometheus, Grafana, Loki and Alertmanager on Kubernetes with Helm, then alert on a sample app and trace an issue from metric to log.

Architecture Overview & Problem Statement

This project deploys a complete observability stack on Kubernetes with Helm: Prometheus collects metrics, Loki stores logs, Grafana shows both in one place, and Alertmanager sends alerts to Slack. You deploy a sample app, write an alert rule for it, break the app on purpose, and follow the evidence from the alert to the matching log lines. You finish with a working stack and a short incident write-up you can show in a portfolio.

This project covers observability fundamentals: gathering metrics and logs, building a dashboard, and alerting on a symptom. It does not cover SLOs, error budgets, or burn-rate alerting. Those belong to the next project, Deploy Kubernetes Monitoring with Prometheus, Grafana, and SLO Alerting, which assumes you can already run this stack.

What you build:

kube-prometheus-stack (Prometheus, Grafana, Alertmanager, node-exporter, kube-state-metrics)
Loki in single-binary mode with a Grafana Alloy agent shipping pod logs
A sample app scraped through a ServiceMonitor
A custom alert rule that notifies Slack

What you learn: how Prometheus discovers targets, how Loki labels logs, how to correlate a metric spike with log lines by time and label, and how Alertmanager routes notifications.

flowchart LR subgraph K8s[Kubernetes cluster] APP[App pods] -->|metrics| PROM[Prometheus] NODE[node-exporter and
kube-state-metrics] --> PROM APP -->|logs| ALLOY[Alloy
DaemonSet] ALLOY --> LOKI[Loki] end PROM --> GRAF[Grafana] LOKI --> GRAF PROM --> AM[Alertmanager] AM --> SLACK[Slack] classDef metrics fill:#F97316,color:#fff classDef logs fill:#8B5CF6,color:#fff classDef ui fill:#326CE5,color:#fff classDef out fill:#475569,color:#fff class PROM metrics class LOKI,ALLOY logs class GRAF ui class AM,SLACK out

The problem this solves

Without observability, a production problem means running kubectl logs against pods one at a time and guessing. Worse, when a pod crashes or is replaced, its logs are gone. This stack keeps both signals: metrics tell you something is wrong and when, and Loki keeps the logs of pods that no longer exist. In Grafana you find the time of the anomaly on a dashboard, then query the logs for that same time and label set. Measure your own results: time yourself on the incident exercise in the validation step and note how long the diagnosis takes.

Tip
Run this on a local cluster such as kind or minikube, or a small managed cluster. You need roughly 4 CPUs and 8 GB of memory free.

Milestone 1: Prepare the Cluster and Helm

This milestone checks that the cluster works and adds the Helm repositories. Helm packages the whole stack as charts, so each component installs with one command and one values file.

Bash

kubectl cluster-info
kubectl get nodes
 
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
 
kubectl create namespace monitoring

Look up the current chart versions and pin them, so a later helm upgrade never pulls in surprise changes:

Bash

helm search repo prometheus-community/kube-prometheus-stack
helm search repo grafana/loki
helm search repo grafana/alloy
 
export PROM_VERSION=<version from the search above>
export LOKI_VERSION=<version from the search above>
export ALLOY_VERSION=<version from the search above>

Common Mistake
Copying a chart version from an old tutorial. Chart values change between major versions, and a stale pin can break the values files below. Always pin a version you have just looked up.

Milestone 2: Deploy kube-prometheus-stack

This milestone installs Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics in one release, with Kubernetes dashboards and alert rules already included. Node-exporter reports machine metrics such as CPU and disk, and kube-state-metrics reports object state such as deployment replicas.

Create prometheus-values.yaml:

YAML

grafana:
  adminPassword: "change-me"
  persistence:
    enabled: true
    size: 10Gi
  additionalDataSources:
    - name: Loki
      type: loki
      url: http://loki.monitoring.svc:3100
      access: proxy
 
prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 20Gi
    # Pick up ServiceMonitors, PodMonitors and rules from every namespace
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
 
# Managed clusters hide these control plane components
kubeEtcd:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false

Bash

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values prometheus-values.yaml \
  --version $PROM_VERSION
 
kubectl get pods -n monitoring --watch

Wait three to five minutes until every pod is Running: Alertmanager, Grafana, kube-state-metrics, the operator, Prometheus, and one node-exporter pod per node.

Security
The admin password above is a placeholder for a local lab. In a shared cluster, use a Kubernetes Secret through grafana.admin.existingSecret and never commit real credentials.

Milestone 3: Explore the Built-in Dashboards

This milestone opens Grafana so you know what the stack already gives you before you add anything. Learning the default dashboards first helps you spot what is missing later.

Bash

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Open http://localhost:3000 and sign in as admin with the password from your values file. Under Dashboards, open:

Kubernetes / Compute Resources / Cluster for cluster-wide CPU and memory
Kubernetes / Compute Resources / Namespace (Pods) for per-namespace usage
Node Exporter / Nodes for per-node disk, network, and CPU detail
Kubernetes / Persistent Volumes for volume capacity and usage

Milestone 4: Deploy Loki for Log Storage

This milestone installs Loki, which stores logs and indexes only their labels, not their full text. That makes it cheap to run, and it is why consistent labels such as namespace, pod, and app matter so much.

Create loki-values.yaml for single-binary mode, which suits a lab. The schema block is required by recent chart versions.

YAML

deploymentMode: SingleBinary
 
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  limits_config:
    retention_period: 168h
  compactor:
    retention_enabled: true
    delete_request_store: filesystem
 
singleBinary:
  replicas: 1
  persistence:
    enabled: true
    size: 20Gi
 
# Turn off the distributed components and extras
read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0
chunksCache:
  enabled: false
resultsCache:
  enabled: false
gateway:
  enabled: false
lokiCanary:
  enabled: false
test:
  enabled: false

Bash

helm install loki grafana/loki \
  --namespace monitoring \
  --values loki-values.yaml \
  --version $LOKI_VERSION
 
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki

Common Mistake
Installing the Loki chart with default values on a small cluster. The defaults assume a distributed deployment with object storage and caches, and the pods stay Pending. Single-binary mode with filesystem storage is for labs only; use object storage such as S3 in production.

Milestone 5: Ship Pod Logs with Grafana Alloy

This milestone installs the agent that reads pod logs and sends them to Loki. Promtail was the usual agent, but it reached end of life in March 2026, so new setups use Grafana Alloy, which Grafana now recommends for log collection.

Alloy runs as a DaemonSet and uses a small config in its own language. This config discovers pods, copies useful metadata into labels, tails the logs through the Kubernetes API, and writes them to Loki.

Create alloy-values.yaml:

YAML

controller:
  type: daemonset
 
alloy:
  configMap:
    content: |
      discovery.kubernetes "pods" {
        role = "pod"
      }
      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pods.targets
 
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app"]
          target_label  = "app"
        }
      }
 
      loki.source.kubernetes "pods" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [loki.write.default.receiver]
      }
 
      loki.write "default" {
        endpoint {
          url = "http://loki.monitoring.svc:3100/loki/api/v1/push"
        }
      }

Bash

helm install alloy grafana/alloy \
  --namespace monitoring \
  --values alloy-values.yaml \
  --version $ALLOY_VERSION
 
kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy

Expect one Alloy pod per node, all Running. If your chart version expects a different key for the config, check helm show values grafana/alloy for the current location.

Milestone 6: Deploy a Sample App with Metrics

This milestone adds an application that exposes Prometheus metrics. A ServiceMonitor is a Kubernetes object that tells Prometheus which Service to scrape, so you do not edit Prometheus configuration by hand.

Bash

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
        - name: example-app
          image: quay.io/brancz/prometheus-example-app:v0.5.0
          ports:
            - name: http
              containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: example-app
  namespace: default
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
    - name: http
      port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  namespace: default
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
EOF

Check that Prometheus found it:

Bash

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
# Open http://localhost:9090/targets
# Expected: the example-app target shows state UP

Common Mistake
A ServiceMonitor that Prometheus never picks up. Check that its labels match the Prometheus selector, and that the port name in the ServiceMonitor equals the port name in the Service. Run kubectl get prometheus -n monitoring -o yaml | grep -A3 serviceMonitorSelector to see what Prometheus expects.

Milestone 7: Route Alerts to Slack

This milestone configures Alertmanager, the component that groups alerts, removes duplicates, and sends them to a channel. Prometheus decides when an alert fires; Alertmanager decides who hears about it.

Create a Slack incoming webhook (Slack API, Create New App, Incoming Webhooks, add one for your #alerts channel) and store the URL as a Secret so it never lands in Git:

Bash

kubectl create secret generic slack-webhook -n monitoring \
  --from-literal=url='https://hooks.slack.com/services/REPLACE_ME'

Create alertmanager-values.yaml:

YAML

alertmanager:
  alertmanagerSpec:
    secrets:
      - slack-webhook
  config:
    route:
      receiver: slack-alerts
      group_by: ["alertname", "namespace"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      routes:
        - receiver: "null"
          matchers:
            - alertname = "Watchdog"
    receivers:
      - name: "null"
      - name: slack-alerts
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/slack-webhook/url
            channel: "#alerts"
            send_resolved: true

Bash

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values prometheus-values.yaml \
  --values alertmanager-values.yaml \
  --version $PROM_VERSION

The Watchdog alert always fires by design, to prove the pipeline works, so the config sends it to a receiver that does nothing.

Security
Treat the webhook URL like a password. Anyone who has it can post to your channel.

Milestone 8: Write a Custom Alert Rule

This milestone adds a rule for your own app. A PrometheusRule object holds the alert expression, how long it must stay true before firing, and the labels and text that Alertmanager uses.

Bash

kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-app-rules
  namespace: default
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: example-app
      rules:
        - alert: ExampleAppUnavailable
          expr: kube_deployment_status_replicas_available{namespace="default",deployment="example-app"} < 1
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "example-app has no available replicas"
            description: "No pods of example-app have been ready for over a minute."
EOF

This rule alerts on a symptom users would feel (no replicas serving), not on a low-level cause. Writing alerts around symptoms, and measuring them against reliability targets, is the subject of the SLO project.

Validation & Testing

Run these checks to prove the whole pipeline works.

Bash

# 1. Every monitoring pod is Running
kubectl get pods -n monitoring
 
# 2. Prometheus has no DOWN targets
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
# Open http://localhost:9090/targets and check for UP everywhere
 
# 3. Loki is ready and holds logs
kubectl port-forward -n monitoring svc/loki 3100:3100
curl http://localhost:3100/ready
# Expected: ready
 
curl -G 'http://localhost:3100/loki/api/v1/query' \
  --data-urlencode 'query={namespace="monitoring"}' \
  --data-urlencode 'limit=5'
# Expected: JSON containing recent log lines

Next, run the incident exercise. Break the app on purpose and follow the evidence:

Bash

# Start a timer, then cause the outage
kubectl scale deployment example-app --replicas=0

Wait about two minutes. Open Alertmanager (kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093, then http://localhost:9093) and confirm ExampleAppUnavailable is firing. Confirm the Slack message arrived.
In Grafana, open the Kubernetes / Compute Resources / Namespace (Pods) dashboard, set the namespace to default, and find the moment the pods disappeared.
Open Explore, choose the Loki data source, and query {namespace="default", app="example-app"} with the time range set around that moment. The pods are gone, but their logs are still here, which is the main reason to run Loki.

Bash

# Restore the app and confirm the alert resolves
kubectl scale deployment example-app --replicas=3

Write down in your README how long it took from the alert to the log lines, and what you would add to speed it up. That measured result, with the dashboard screenshot and the alert message, is the portfolio deliverable.

Clean up the lab when you finish:

Bash

kubectl delete deployment,service,servicemonitor,prometheusrule example-app example-app-rules -n default
helm uninstall alloy loki kube-prometheus-stack -n monitoring
kubectl delete namespace monitoring

Common Mistake
Querying Loki with a label that your Alloy config never created. If {app="example-app"} returns nothing, run {namespace="default"} first to see which labels exist, then adjust the query.

Found a mistake, an outdated step or a broken link?

Kubernetes Monitoring with Prometheus and Loki

Blueprint Walkthrough

Architecture Overview & Problem Statement

The problem this solves

Milestone 1: Prepare the Cluster and Helm

Milestone 2: Deploy kube-prometheus-stack

Milestone 3: Explore the Built-in Dashboards

Milestone 4: Deploy Loki for Log Storage

Milestone 5: Ship Pod Logs with Grafana Alloy

Milestone 6: Deploy a Sample App with Metrics

Milestone 7: Route Alerts to Slack

Milestone 8: Write a Custom Alert Rule

Validation & Testing