PromQL

Kubernetes hub

Prometheus Query Language — the functional query language used to select, filter, aggregate, and calculate metrics stored in Prometheus. PromQL is used to build Grafana dashboard panels, define alert thresholds in PrometheusRule objects, and explore live metric data in the Prometheus web UI.

PromQL — Querying Your Kubernetes Metrics

What is PromQL in Simple Terms?

PromQL is the language you use to ask questions of Prometheus. Questions like: what is the average CPU usage of all payment-api pods over the last 5 minutes? Which pods have restarted more than 3 times in the last hour? What percentage of requests are returning 5xx errors right now?

Every panel in a Grafana dashboard, every alert rule in Alertmanager, and every metric you see in the Prometheus web UI is powered by a PromQL query. Understanding PromQL is the difference between having monitoring and understanding what your monitoring is telling you.

The Core Data Types in PromQL

◈ DIAGRAM

+------------------------------------------+
| Instant Vector                           |
| A set of time series with ONE value each |
| at a specific point in time              |
|                                          |
| Example: http_requests_total             |
| Returns: current counter value per pod  |
+------------------------------------------+
 
+------------------------------------------+
| Range Vector                             |
| A set of time series with a RANGE of    |
| values over a time window               |
|                                          |
| Example: http_requests_total[5m]         |
| Returns: all values in last 5 minutes   |
| Used with: rate(), increase(), delta()  |
+------------------------------------------+
 
+------------------------------------------+
| Scalar                                   |
| A single number with no labels           |
|                                          |
| Example: 3.14                            |
| Used in: mathematical expressions        |
+------------------------------------------+

Building PromQL Queries — Step by Step

Bash

# Step 1 — Select a metric by name
http_requests_total
# Returns: all time series with this metric name
# Includes ALL labels: namespace, pod, status, method, etc.
 
# Step 2 — Filter with label matchers
http_requests_total{namespace="production", service="payment-api"}
# Returns: only payment-api time series in production namespace
 
# Label matcher operators:
# =  exact match
# != not equal
# =~ regex match     (e.g. status=~"5.." matches 500, 501, 502...)
# !~ regex not match (e.g. status!~"2.." excludes 200, 201, 202...)
 
# Step 3 — Apply a function
rate(http_requests_total{namespace="production"}[5m])
# rate() calculates: (value now - value 5m ago) / 300 seconds
# Result: requests per SECOND averaged over the last 5 minutes
# This is the correct way to query counters
 
# Step 4 — Aggregate across labels
sum(rate(http_requests_total{namespace="production"}[5m]))
# sum() adds up the per-second rate across all pods
# Result: total requests per second for all production pods combined
 
# Step 5 — Group by label
sum by (pod) (rate(http_requests_total{namespace="production"}[5m]))
# Result: requests per second broken down per individual pod
# Shows which pods are handling more or less traffic

The Functions You Will Use Most

Bash

# rate() — per-second rate of a counter over a time window
# Use for: request rates, error rates, bytes transferred
rate(http_requests_total[5m])
 
# increase() — total increase of a counter over a time window
# Use for: total events (restarts, errors) in a period
increase(kube_pod_container_status_restarts_total[1h])
# How many times did this pod restart in the last hour?
 
# irate() — per-second rate using only the last two data points
# Use for: real-time spikes (more sensitive, more noisy)
irate(http_requests_total[5m])
 
# avg_over_time() — average of a gauge over a time window
# Use for: average memory usage over the last 30 minutes
avg_over_time(container_memory_usage_bytes{pod="payment-api-xyz"}[30m])
 
# topk() — top N time series by value
# Use for: which pods are using the most memory?
topk(5, container_memory_usage_bytes{namespace="production"})
 
# histogram_quantile() — calculate percentile from histogram
# Use for: P99 latency, P95 request duration
histogram_quantile(0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{service="payment-api"}[5m])
  )
)
# Returns: 99th percentile latency for payment-api over last 5 minutes

Production-Ready PromQL Queries for Kubernetes

Bash

# -- Pod Health Queries -----------------------------
 
# Pods that are NOT running (stuck Pending, Failed, Unknown)
kube_pod_status_phase{phase!="Running", namespace="production"} == 1
 
# Pods with high restart count (flapping or CrashLoopBackOff)
rate(kube_pod_container_status_restarts_total{
  namespace="production"
}[30m]) * 1800 > 3
# Restarts more than 3 times in the last 30 minutes
 
# Pods that are not ready (passing liveness but failing readiness)
kube_pod_status_ready{condition="false", namespace="production"} == 1
 
# -- Resource Usage Queries -------------------------
 
# CPU usage as percentage of request (shows over-provisioning)
sum by (pod) (
  rate(container_cpu_usage_seconds_total{namespace="production"}[5m])
) /
sum by (pod) (
  kube_pod_container_resource_requests{
    resource="cpu", namespace="production"
  }
) * 100
 
# Memory usage as percentage of limit (shows pods near OOMKill)
container_memory_usage_bytes{namespace="production"} /
container_spec_memory_limit_bytes{namespace="production"} * 100
# Alert when this exceeds 85%
 
# Nodes approaching memory capacity
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) /
node_memory_MemTotal_bytes * 100 > 85
 
# -- Traffic and Error Queries ---------------------
 
# HTTP error rate percentage per service
sum by (service) (
  rate(http_requests_total{status=~"5..", namespace="production"}[5m])
) /
sum by (service) (
  rate(http_requests_total{namespace="production"}[5m])
) * 100
 
# P99 API latency per service
histogram_quantile(0.99,
  sum by (service, le) (
    rate(http_request_duration_seconds_bucket{namespace="production"}[5m])
  )
) * 1000
# Result in milliseconds
 
# -- Scaling Queries -------------------------------
 
# HPA current vs desired replicas (shows scaling activity)
kube_horizontalpodautoscaler_status_current_replicas
  / kube_horizontalpodautoscaler_spec_max_replicas
# Values close to 1.0 mean the HPA is near its ceiling
 
# Pending pods (not enough capacity — CA should kick in)
kube_pod_status_phase{phase="Pending"} == 1

Common PromQL Mistakes to Avoid

Bash

# WRONG — querying a counter directly (shows total since start, not rate)
http_requests_total{service="payment-api"}
# Returns: 47,832,091 (meaningless cumulative count)
 
# CORRECT — use rate() to get per-second throughput
rate(http_requests_total{service="payment-api"}[5m])
# Returns: 245.3 (requests per second over last 5 minutes)
 
# WRONG — using irate() for alert rules (too noisy)
irate(http_requests_total[5m]) > 500
# irate() spikes on single data point anomalies — alert fires and resolves constantly
 
# CORRECT — use rate() for alert rules (smoothed average)
rate(http_requests_total[5m]) > 500
# rate() averages over the full window — stable, reliable alerting
 
# WRONG — forgetting by() clause when summing across pods
sum(container_memory_usage_bytes{namespace="production"})
# Returns: one number (total memory for all pods combined)
# You lose which pod is causing the problem
 
# CORRECT — add by (pod) to keep pod-level granularity
sum by (pod) (container_memory_usage_bytes{namespace="production"})
# Returns: memory per pod — now you can see which pod is the problem

Testing PromQL Queries

Bash

# Access the Prometheus web UI for interactive query testing
kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring
# Open http://localhost:9090
# Use the Graph tab to test queries and visualise results
 
# Query Prometheus from the command line
kubectl exec -it prometheus-kube-prometheus-stack-prometheus-0 \
  -n monitoring -- \
  promtool query instant http://localhost:9090 \
  'rate(http_requests_total{namespace="production"}[5m])'

Tip
Always use rate() over a 5-minute window as your default time window for dashboard panels — it provides a smooth curve without too much lag. For alerting rules, use rate() over 5 minutes with a for: 2m clause so the alert only fires if the condition holds for 2 minutes continuously, preventing false alerts from single-point metric spikes.

Remember
PromQL metrics from Kubernetes have many labels — namespace, pod, container, node, service. Always filter by namespace first in every query (namespace="production") otherwise you get metrics from all namespaces mixed together, which makes graphs unreadable and alert thresholds meaningless.

Common Mistake
Querying container_memory_usage_bytes to alert on memory pressure. This metric includes cache memory which Linux can reclaim at any time. It shows higher than actual memory pressure. Use container_memory_working_set_bytes instead — this is the metric the kubelet itself uses for OOMKill decisions and eviction calculations. It excludes reclaimable cache and gives you the true memory pressure number.

Security
Never embed Prometheus or Grafana access tokens directly in PromQL queries or dashboard JSON that gets committed to Git. Use Grafana data source variables for credentials and store sensitive values in Kubernetes Secrets referenced by the Grafana deployment — not hardcoded in dashboard configuration files.

Frequently Asked Questions

What makes PromQL different from a SQL query against a metrics table?

PromQL isn't relational — it operates on time series identified by a metric name plus a set of key-value labels, and every query returns either an instant vector (one value per series at a point in time) or a range vector (a window of samples per series). Functions like `rate()`, `irate()`, and `histogram_quantile()` are purpose-built for counters and histograms, letting you compute a per-second request rate or a p99 latency directly, which would require much more manual work in SQL.

Why does rate() give wrong or missing results on a metric that isn't a counter?

`rate()` assumes the input is a monotonically increasing counter and automatically handles resets (e.g., on process restart), but applying it to a gauge — a value that goes up and down, like current memory usage — produces meaningless numbers. A common mistake is also using `rate()` over too short a range relative to the scrape interval; as a rule of thumb the range should span at least 4x your scrape interval, or you risk gaps and noisy, unstable results.