Question 1

What is Prometheus, and why is a pull-based model its default?

Accepted Answer

Prometheus is an open-source metrics-based monitoring system: it scrapes numeric time-series data from targets, stores it in its own time-series database, and lets you query it with PromQL.

It defaults to pulling metrics (Prometheus fetches `/metrics` from each target) rather than targets pushing to it, for a few practical reasons: Prometheus can tell a target is unreachable (a failed scrape *is* a signal, whereas a missing push looks the same as "nothing happened"), you don't need to give every application credentials to a central collector, and you can run Prometheus locally against a target to debug it without touching production config. Short-lived jobs that finish before a scrape could ever reach them are the one case this doesn't fit, which is why Prometheus ships a separate Pushgateway for batch jobs rather than changing the core model.

[Prometheus overview](https://prometheus.io/docs/introduction/overview/)

Question 2

What are the four Prometheus metric types, and when do you use each?

Accepted Answer

The four types are Counter, Gauge, Histogram, and Summary.

- **Counter** - a value that only goes up (or resets to zero on restart), like total requests handled. Use it for anything you'd describe as "how many so far."
- **Gauge** - a value that goes up and down, like current memory usage or number of active connections.
- **Histogram** - buckets observations (e.g. request durations) into configurable ranges and also exposes a count and sum, so you can compute quantiles server-side in PromQL.
- **Summary** - similar to a histogram but calculates quantiles client-side over a sliding window; cheaper to query but the quantiles can't be aggregated across instances.

The practical rule of thumb: reach for a histogram unless you have a specific reason not to, because it aggregates correctly across many instances, which is the normal case in a distributed system.

Question 3

Why should you never use rate() or irate() directly on a Gauge?

Accepted Answer

`rate()` and `irate()` are built to detect and correct counter resets - if a value drops between two samples, they assume the process restarted and the counter went back to zero, so they add the pre-reset value back in rather than reporting a negative rate.

A gauge is allowed to go down on its own (queue length dropping, memory being freed), and PromQL cannot tell the difference between "this gauge legitimately decreased" and "this counter reset." If you run `rate()` on a gauge, every normal decrease gets misread as a reset and silently added into the result, producing numbers that are too high and don't correspond to anything real.

The common wrong answer here is "rate() just calculates speed of change, so it works on any number that changes over time." It doesn't - `rate()` is defined specifically in terms of counter semantics. For a gauge, use `deriv()` if you want a per-second rate of change, or just graph the raw gauge.

[PromQL functions](https://prometheus.io/docs/prometheus/latest/querying/functions/)

Question 4

You run `increase(http_requests_total[5m])` and get a fractional, non-integer number back. Why, and is that a bug?

Accepted Answer

It's not a bug - it's extrapolation, and it's part of how `increase()` and `rate()` are defined.

Prometheus only has samples at your scrape interval (say every 15s), and those samples rarely land exactly on the start and end of your requested time range. So instead of just subtracting the two closest raw values, Prometheus takes the increase observed *between* the first and last sample inside the window and extrapolates it slightly outward to cover the full requested range, based on the average interval between samples. That extrapolation is what produces a non-integer result even though the underlying counter only ever moves in whole numbers.

There are safety limits on this: if the gap between the window edge and the nearest sample is bigger than about 1.1 times the average sample interval, Prometheus extrapolates only half an interval instead of all the way to the edge, and it never extrapolates a counter below zero (since counters start at zero and can't go negative). If you need an exact integer total rather than an estimate, you generally don't get one from `increase()` on a counter with any realistic scrape gaps - that's a trade-off of the whole approach, not a misconfiguration to fix.

The trap version of this question is being asked to explain *why* a dashboard shows "142.7 requests" and treating it as an application bug. It's the query function working as designed.

[How exactly does PromQL calculate rates? - PromLabs](https://promlabs.com/blog/2021/01/29/how-exactly-does-promql-calculate-rates/)

Question 5

What does Prometheus service discovery do, and how does it work in Kubernetes?

Accepted Answer

Service discovery lets Prometheus find scrape targets automatically instead of you hand-listing every IP and port in `prometheus.yml`. That matters anywhere targets come and go - pods restart with new IPs, autoscaling adds and removes instances - because a static target list would go stale immediately.

In Kubernetes, Prometheus's `kubernetes_sd_config` queries the Kubernetes API for objects like Pods, Services, and Endpoints, and refreshes that list on an interval. It doesn't know on its own which of those it should actually scrape, though - that's handled by relabeling rules that read pod/service annotations (commonly `prometheus.io/scrape: "true"`, `prometheus.io/port`, `prometheus.io/path`) and use them to decide the final target list. In most real clusters this whole flow is delegated to the Prometheus Operator's `ServiceMonitor` and `PodMonitor` custom resources, which express "scrape anything with this label" declaratively instead of hand-writing relabel configs.

[Kubernetes SD config docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)

Question 6

What is relabeling in Prometheus, and what's the difference between relabel_configs and metric_relabel_configs?

Accepted Answer

Relabeling rewrites, filters, or drops labels using regex rules, and it runs at two different points in the scrape pipeline with very different consequences.

`relabel_configs` runs **before the scrape happens**, against the metadata service discovery produced (pod labels, annotations, addresses). You use it to decide *which targets get scraped at all* and to set things like the scrape port or path from an annotation. If a rule here drops a target, Prometheus never even connects to it - no load is generated.

`metric_relabel_configs` runs **after the scrape**, against the actual metric samples returned. You use it to drop unwanted metrics or labels, or rename messy label names, after the exporter has already been hit. This is the tool for controlling cardinality on data you don't control the source of - but the target was still scraped, so it doesn't save any scrape load, only storage and query cost.

The trap: people reach for `metric_relabel_configs` to "stop scraping" a noisy exporter and are confused when scrape duration and load don't change - that requires `relabel_configs` on the target set itself, not the post-scrape metrics.

[Relabeling - Prometheus configuration docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)

Question 7

What is high cardinality in Prometheus, and why does it become a problem at scale?

Accepted Answer

Cardinality is the number of unique time series a metric produces - every distinct combination of label values creates a separate series. `http_requests_total{method="GET", path="/users"}` and `http_requests_total{method="POST", path="/users"}` are two series from one metric name.

The problem is that Prometheus keeps an in-memory index of every active series so it can answer queries fast, and that index's memory footprint scales with the *number of series*, not the amount of underlying data. A label that takes unbounded values - a user ID, a raw URL path with IDs embedded, a request ID, a full email address - can turn one metric into millions of series. That shows up as ballooning memory usage, slow queries, and in bad cases an out-of-memory crash of the Prometheus server itself, all from a single badly-labeled metric.

Common fixes: never put unbounded values in labels (aggregate the path template, not the resolved URL, for instance), use `metric_relabel_configs` to drop or hash offending labels before they're stored, and if you're running Thanos/Mimir/Cortex, apply cardinality limits or drop rules at the ingest layer where the majority of cost overruns actually originate.

[Instrumentation best practices: do not overuse labels](https://prometheus.io/docs/practices/instrumentation/#do-not-overuse-labels)

Question 8

What's the difference between a recording rule and an alerting rule?

Accepted Answer

Both live in Prometheus rule files and both run on a schedule, but they do different jobs.

A **recording rule** pre-computes an expensive or frequently-used PromQL expression and saves the result as a new time series. You use these to speed up dashboards that would otherwise re-run a heavy aggregation on every page load, and as a naming convention (`level:metric:operation`, e.g. `job:http_errors:rate5m`) that other rules and dashboards can build on cheaply.

An **alerting rule** evaluates a PromQL expression and, when it's true for at least the configured `for` duration, fires an alert that gets sent to Alertmanager. The `for` duration matters: without it, a single noisy spike that clears itself in the next scrape would still page someone.

A common real pattern is to define the underlying calculation once as a recording rule, then have the alerting rule threshold against that recorded series - so the dashboard and the alert are guaranteed to agree on what the number actually is.

Question 9

How do you achieve high availability for Prometheus, given it's normally a single stateful process?

Accepted Answer

Vanilla Prometheus has no built-in clustering or replication - each server scrapes and stores independently. The standard HA pattern is to run two (or more) identical Prometheus instances scraping the same targets with the same config, so either one alone can answer queries and fire alerts if the other is down. This gets you availability but not deduplication: Alertmanager has to dedupe identical alerts coming from both replicas, which it does natively via its own clustering (gossip protocol) as long as both Prometheus instances are configured to point at the same Alertmanager cluster.

What plain replication doesn't solve is long-term storage, global querying across many Prometheus servers, and downsampling for older data - for that you add **Thanos**, **Cortex**, or **Mimir** in front of or alongside Prometheus. These typically work by having a sidecar or remote-write path ship blocks to object storage (S3/GCS), and a separate Querier component fan out queries across all Prometheus replicas and the long-term store to present one global view.

The trap: describing "HA Prometheus" as if it were one clustered system with shared storage. It's stateless duplication at the collection layer, with a separate system bolted on for anything that needs a unified, long-retention view.

[Thanos overview](https://thanos.io/tip/thanos/design.md/)

Question 10

What does the Prometheus Alertmanager actually do, separate from Prometheus itself?

Accepted Answer

Alertmanager handles what happens *after* an alert fires - Prometheus itself only evaluates rules and decides something is wrong; it hands the resulting alert off to Alertmanager over HTTP.

Alertmanager then deduplicates identical alerts coming from multiple Prometheus replicas, groups related alerts together so one incident produces one notification instead of fifty, silences alerts that match a pattern during planned maintenance, can inhibit lower-priority alerts when a related higher-priority one is already firing, and finally routes the result to the right receiver - Slack, PagerDuty, email, a generic webhook - based on label matching rules you define in `alertmanager.yml`.

A useful way to separate the two mentally: Prometheus decides *if* something is wrong; Alertmanager decides *who hears about it, how often, and in what shape*.

Question 11

Your team is getting paged 50 times for what's clearly one outage. What Alertmanager features do you reach for, and how are they different?

Accepted Answer

There are three distinct tools here and they solve different parts of the problem, so the interviewer is usually checking you don't conflate them.

**Grouping** batches multiple *firing* alerts that share labels (say `alertname`, `cluster`, `service`) into a single notification instead of one message per alert. If ten pods all trip `PodNotReady` at once, grouping turns that into one Slack message listing ten pods rather than ten separate pages.

**Inhibition** suppresses a *lower-priority* alert when a *related, higher-priority* alert is already active, based on a source/target label match you define. The classic example: if `DatabaseDown` is firing, you don't also need fifty `APIErrors` alerts from every service that depends on that database - inhibition rules stop the noisy downstream symptoms from paging separately.

**Silencing** is manual and time-boxed: you tell Alertmanager to mute alerts matching a label set for a defined window, typically during planned maintenance, so expected noise doesn't page anyone.

In practice, fixing a 50-page incident usually means: `group_by` includes enough shared labels to collapse duplicates, an inhibition rule ties the leaf symptoms to their root cause, and going forward, each alert added to the system should be reviewed for whether it's actually a distinct actionable signal or just noise from something upstream. A quarterly audit of alerts that never fire, or always fire in bulk, is standard operational hygiene for keeping this from recurring.

```yaml
route:
  group_by: ['alertname', 'cluster', 'service']
inhibit_rules:
  - source_matchers:
      - alertname = DatabaseDown
    target_matchers:
      - alertname = APIErrors
    equal: ['cluster']
```

[Alertmanager configuration docs](https://prometheus.io/docs/alerting/latest/configuration/)

Question 12

What is a dead man's switch (Watchdog) alert, and why would you deliberately configure an alert to always be firing?

Accepted Answer

It's an alert rule written to be permanently true - for example, `vector(1) == 1` - routed to a receiver that expects a heartbeat and pages if the heartbeat *stops*.

The reason to want this at all: your alerting pipeline has multiple points of silent failure - Prometheus could crash, Alertmanager could lose its config, a network path to your paging tool could break - and in every one of those cases, the failure mode is that you simply stop getting paged, which looks identical to "everything is fine." A Watchdog alert converts "absence of a signal" into "presence of a different signal": if the heartbeat notification stops arriving, something in the alerting chain itself is broken, independent of whether your actual services are healthy.

This pairs naturally with using `absent()` on critical exporters, which alerts when a metric that should always exist has stopped being scraped at all - catching the case where an exporter or scrape target silently disappeared rather than reporting bad values.

Question 13

What is an exporter, and how is it different from instrumenting your own application?

Accepted Answer

An exporter is a small standalone process that translates metrics from something that doesn't natively speak Prometheus's format into a `/metrics` HTTP endpoint Prometheus can scrape. `node_exporter` exposes host-level metrics (CPU, memory, disk); `mysqld_exporter` queries a MySQL instance and republishes its internal stats the same way.

This is different from instrumenting your own application, where you add a Prometheus client library directly into your code and expose `/metrics` from the app process itself, with custom business metrics you define (orders processed, queue depth, cache hit ratio).

The rule of thumb: use an exporter for third-party systems you can't or don't want to modify; instrument directly for code you own, since it lets you emit metrics that are actually meaningful to your application's logic rather than generic infrastructure stats.

Question 14

What is Thanos (or Cortex/Mimir), and what problem does it solve that Prometheus alone doesn't?

Accepted Answer

Prometheus by itself is a single server with local disk storage, a fixed retention window, and no way to query across multiple Prometheus instances as one dataset. Thanos, Cortex, and Mimir all solve variations of the same three problems: **long-term retention** (shipping old data to cheap object storage like S3 instead of local disk), **global query view** (a Querier component that fans a query out across many Prometheus servers and the long-term store, then merges the result), and **horizontal scalability** for ingesting metrics beyond what one Prometheus process can handle.

Thanos specifically is designed to sit alongside existing Prometheus servers with minimal changes - a sidecar container uploads blocks to object storage and serves recent data, while separate Store Gateway and Querier components handle historical data and cross-cluster queries. Cortex and Mimir take a more centralized, remote-write-based ingestion approach instead of the sidecar model. Which one to pick usually comes down to whether you want to keep existing Prometheus servers as-is (favors Thanos) or centralize ingestion from the start (favors Mimir/Cortex).

[Thanos architecture](https://thanos.io/tip/thanos/design.md/)

Question 15

What is Grafana, and how does it relate to Prometheus?

Accepted Answer

Grafana is an open-source visualization and dashboarding tool - it doesn't store or collect data itself, it connects to data sources (Prometheus, Loki, Elasticsearch/OpenSearch, InfluxDB, and dozens of others) and lets you build graphs, tables, and alerts on top of whatever they return.

Prometheus and Grafana are commonly paired because Prometheus is good at collecting and storing metrics but has a fairly minimal built-in UI, while Grafana specializes in rich, shareable dashboards, templated variables, and multi-source panels. In that pairing, Prometheus does the querying via PromQL under the hood; Grafana is just the presentation and alerting-on-top-of-visualization layer.

[Grafana docs - what is Grafana](https://grafana.com/docs/grafana/latest/introduction/)

Question 16

What are template variables in Grafana, and why would you use them instead of hardcoding a query?

Accepted Answer

A template variable is a placeholder - shown as a dropdown at the top of a dashboard - that gets substituted into every panel's query at render time, instead of the query hardcoding one specific value.

Without them, monitoring ten services means either building ten near-identical dashboards or manually editing a query every time you want to look at a different one. With a `$service` variable driving the query, one dashboard serves all ten: the viewer picks a value from the dropdown and every panel re-queries for that selection. Variables can be static lists, populated dynamically from a query against the data source itself (e.g. "list all distinct `job` label values"), or chained so one variable's options depend on another's selection (pick a cluster, then the service dropdown only shows services in that cluster).

The practical payoff is that a single well-built templated dashboard replaces an entire folder of copy-pasted, drifting ones - which is usually the actual point behind this question, not the mechanics of the dropdown.

Question 17

How does Grafana manage user access across teams and organizations?

Accepted Answer

Access in Grafana is layered. **Organizations** are the top-level tenant boundary - each one has its own dashboards, data sources, and users, useful when genuinely separate teams or customers shouldn't see each other's content at all. Within an organization, users get a **role** - Viewer, Editor, or Admin - that governs what they can do (view only, build and edit dashboards, or manage users and data sources). Newer Grafana versions add **folder-level permissions** and fine-grained RBAC on top of this, so you can grant edit access to one team's folder without giving them admin over the whole instance.

For authentication itself, Grafana doesn't require you to manage passwords directly - it integrates with LDAP, OAuth (Google, GitHub, generic OIDC), and SAML, which is the standard approach in any organization that already has centralized identity, since it avoids a second set of credentials to manage and revoke.

[Grafana role-based access control docs](https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/)

Question 18

What's the difference between a Grafana panel, a dashboard, and a data source?

Accepted Answer

A **data source** is the connection to wherever your data actually lives - a Prometheus server, an OpenSearch cluster, a database. It's configured once and then reused by many dashboards.

A **panel** is a single visualization - one graph, table, heatmap, or stat - built from a query against one or more data sources.

A **dashboard** is a collection of panels arranged on a grid, usually all related to one system or purpose (a service overview, an infrastructure health page). Dashboards are what people actually look at day to day; panels are the individual building blocks inside them.

Question 19

How would you design Grafana alerting so an SRE isn't paged for every threshold breach across fifty near-identical services?

Accepted Answer

Grafana's unified alerting lets an alert rule query across a label dimension and evaluate the same condition per series, so you write one rule - not fifty - and it fans out to one alert instance per matching label set (per service, per pod, whatever you group by). That alone reduces maintenance overhead, but it doesn't by itself reduce paging volume if fifty of those instances trip at once.

The actual noise-reduction techniques mirror what you'd do in Alertmanager: notification policies group related firing alerts into a single message using shared labels, mute timings suppress notifications during known maintenance windows, and you route by severity so only genuinely critical instances page immediately while warnings land in a dashboard or low-priority channel. For a fleet of similar services specifically, the higher-leverage move is often to alert on an aggregate - "more than 20% of the fleet is unhealthy" - rather than firing individually per instance, since one service blipping is noise but a fifth of the fleet failing is a real incident.

The deeper principle interviewers are checking for here is symptom-based, SLO-aligned alerting over static per-instance thresholds - alerting on the burn rate of an error budget catches real user impact, while alerting on every CPU spike on every box does not.

[Grafana alerting docs](https://grafana.com/docs/grafana/latest/alerting/)

Question 20

What is a Grafana transformation, and how is it different from modifying the underlying query?

Accepted Answer

A transformation reshapes the *result* of a query after it comes back from the data source - renaming fields, joining two queries together, filtering rows, doing math between series - without touching the query itself or the data source.

This matters because not every data source's query language can do everything you need. If you're combining a Prometheus series with a CSV-sourced table, or you want to show the ratio of two independently-queried metrics as one panel, that join happens in Grafana's transformation pipeline, not in PromQL. The trade-off is that transformations run in the browser at render time on whatever data was returned, so they're the wrong tool for anything that needs to filter or aggregate at the source to keep query size manageable - for large datasets, push filtering into the query itself and reserve transformations for reshaping the smaller result set for display.

Question 21

What is the LGTM stack, and what does each letter cover?

Accepted Answer

LGTM is Grafana Labs' name for a self-contained observability stack made of four open-source components: **L**oki for logs, **G**rafana for visualization and alerting across all of it, **T**empo for distributed traces, and **M**imir for long-term, horizontally-scalable metrics storage (a Prometheus-compatible remote-write target).

The point of the stack is that all three signal types - metrics, logs, traces - share the same query and label model as much as possible, so you can pivot from a metric spike in Grafana straight into the logs or traces for that same service and time window without switching tools or re-learning a query language. It's Grafana Labs' answer to running Prometheus, an ELK/OpenSearch stack, and a separate tracing backend as three disconnected systems.

Question 22

What is Fluent Bit, and how is it different from Fluentd?

Accepted Answer

Fluent Bit is a lightweight log and metrics collector and forwarder - it reads logs from a source (files, systemd journal, container runtimes), optionally parses and transforms them, and ships them to a destination like OpenSearch, Elasticsearch, Kafka, or S3.

It's a sibling project to Fluentd, from the same Fluent open-source project, but built for a different niche: Fluent Bit is written in C, has a much smaller memory and CPU footprint, and is designed to run as a per-node DaemonSet in resource-constrained environments like Kubernetes edge nodes. Fluentd is written in Ruby (with C extensions), has a larger plugin ecosystem, and is typically used as a more feature-rich aggregation layer sitting behind lightweight Fluent Bit collectors - a common Kubernetes pattern runs Fluent Bit as a DaemonSet on every node forwarding to a smaller number of centralized Fluentd (or Logstash) instances that do heavier processing before the final destination.

[Fluent Bit vs Fluentd](https://docs.fluentbit.io/manual/about/fluentd-and-fluent-bit)

Question 23

Walk through the stages of a Fluent Bit pipeline.

Accepted Answer

A Fluent Bit pipeline has four stages that a log record moves through in order.

1. **Input** - where records enter: tailing a log file, reading the systemd journal, an HTTP endpoint, a Kubernetes container log path.
2. **Parser** - converts unstructured text into structured key-value fields, typically applied at the input using a regex, JSON, or a built-in format like the Docker/CRI log format.
3. **Filter** - modifies, enriches, or drops records after parsing: adding Kubernetes metadata (pod name, namespace, labels) via the `kubernetes` filter, redacting fields, or dropping records matching a pattern with `grep`.
4. **Output** - where the record finally goes: OpenSearch, Elasticsearch, S3, Kafka, stdout for debugging, or dozens of other destinations.

Records flow through these in the order they're declared in the config, tagged by a `Tag` that routing rules (`Match`) use to decide which filters and outputs apply to which inputs - this is what lets one Fluent Bit process handle several unrelated log sources with different processing rules in a single config file.

Question 24

How does Fluent Bit's multiline parser handle a Java stack trace spread across many log lines?

Accepted Answer

By default, a log collector treats every line as one record, which shreds a stack trace into dozens of unrelated log entries - the exception message becomes one record and every subsequent frame becomes its own separate, contextless record. Fluent Bit's multiline parsing solves this by defining a start pattern (typically a regex matching a timestamp or log-level prefix, since that's what marks the beginning of a *new* log entry) and treating every following line that doesn't match that start pattern as a continuation of the previous one, concatenating them into a single record.

There are two places this can be configured, and the choice matters: doing it in the **Tail input plugin itself** is the recommended approach when reading from a log file, because concatenation happens as lines are read, which is more efficient. The separate **multiline filter** is for cases where you can't do it at the input (e.g., records already arrived via a different input plugin) - but it comes with real constraints: it isn't affected by `buffer_max_size`, so a runaway match can grow a record indefinitely, it uses an internal re-emitter, and you can't define two multiline filters that match the same tag or you'll cause an infinite loop in the pipeline. For that reason, if you need to try multiple multiline patterns against one input, you configure a single filter with a comma-separated list of parsers, not multiple filter blocks.

```ini
[INPUT]
    Name              tail
    Path              /var/log/app/*.log
    multiline.parser  java
```

[Fluent Bit multiline parsing docs](https://docs.fluentbit.io/manual/data-pipeline/parsers/multiline-parsing)

Question 25

What happens when Fluent Bit's output can't keep up with incoming log volume - where does backpressure show up, and how do you control it?

Accepted Answer

Fluent Bit buffers incoming records (in memory, on disk, or both, depending on the `storage.type` setting) while waiting for an output to accept them. If the output - say OpenSearch under load, or a network blip to a remote endpoint - falls behind, that buffer keeps growing. Left unchecked, that eventually exhausts memory or disk and can crash the process or, worse, silently drop records.

Fluent Bit's answer is to pause the affected input once its buffer crosses a configured limit (`Mem_Buf_Limit` for memory-buffered inputs), rather than let it grow unbounded. Once paused, the engine stops accepting new data from that specific input plugin - it's the plugin's own responsibility to decide what to do while paused, and for HTTP-based inputs like `http`, `elasticsearch`, or `opentelemetry`, that means the listener itself stops accepting new connections until the buffer drains back under the limit and a resume callback fires.

The practical levers: use filesystem-backed buffering (`storage.type filesystem`) for inputs where you can't afford to lose data during an output outage, since it survives a Fluent Bit restart that memory buffering doesn't; set `Mem_Buf_Limit` deliberately rather than leaving it at a default that doesn't match your actual traffic; and size retries and flush intervals on the output side so a transient outage doesn't immediately trigger backpressure on the input side.

[Fluent Bit backpressure docs](https://docs.fluentbit.io/manual/administration/backpressure)

Question 26

What does a Fluent Bit parser do, and what's the difference between a regex parser and the built-in JSON parser?

Accepted Answer

A parser turns a raw log line - usually a single unstructured string - into a structured record with named fields, which is what makes the log queryable and filterable downstream instead of being one opaque blob of text.

The **JSON parser** is the simple case: if the application already logs structured JSON, Fluent Bit just decodes it directly into fields with no pattern-matching required, and it's fast and unambiguous. The **regex parser** is for anything not already structured - a traditional `combined` or custom application log format - where you write a named-capture-group regex that maps parts of the line to field names, plus (usually) a time format string so Fluent Bit can pull the timestamp out as a proper time field rather than leaving it as a text substring.

The practical guidance: if you control the application, log JSON from the start and skip regex parsing entirely - it's more reliable, cheaper to process, and doesn't break silently the next time someone changes a log message's wording.

Question 27

What is a sidecar vs a DaemonSet pattern for shipping logs in Kubernetes, and when would you choose one over the other?

Accepted Answer

A **DaemonSet** runs one instance of the log collector (e.g. Fluent Bit) per node, reading every container's log files from the node's filesystem directly. This is the default and by far the most common pattern for general-purpose cluster logging: one collector process per node handles every pod on that node, which is far cheaper in aggregate resource usage than one collector per pod, and it keeps working even if the application container itself crashes, since it's reading from the node's log files rather than talking to the app process.

A **sidecar** runs a collector container inside the same pod as the application, typically because the application can't or doesn't write to stdout/stderr in a way the node-level collector can pick up - it writes to a file inside its own container filesystem, or emits logs in a format that needs per-application-specific parsing that would be awkward to centralize. The sidecar tails that file and forwards it, at the cost of one extra container per pod rather than per node.

The common wrong answer is defaulting to sidecars "for isolation" - for standard stdout/stderr logging, that isolation buys nothing and multiplies resource cost by however many pods you're running, so DaemonSet is the right default and sidecar is the exception for a specific per-application constraint.

Question 28

What is OpenSearch, and how does it relate to Elasticsearch?

Accepted Answer

OpenSearch is a distributed search and analytics engine - you index documents (often log lines or structured events) into it and query them with full-text search, aggregations, and dashboards. It's a fork of Elasticsearch and Kibana, created by AWS in 2021 after Elasticsearch changed its license away from a fully open-source model, and it's maintained under the Apache 2.0 license with its own governance.

Because of the fork's timing, OpenSearch and Elasticsearch share the same core architecture and much of the same API surface from that point, but they've diverged in features since - newer Elasticsearch releases and newer OpenSearch releases are not drop-in compatible with each other, so "it's basically Elasticsearch" is true for the fundamentals (shards, the query DSL, cluster health) but not a safe assumption for anything added after the split.

[OpenSearch project overview](https://opensearch.org/docs/latest/about/)

Question 29

What are shards and replicas in OpenSearch, and why does a single-node cluster with the default settings sit at yellow, not green?

Accepted Answer

An index in OpenSearch is split into **shards** - each shard is a self-contained Lucene index holding a subset of the index's documents - so that a dataset too large or too fast for one machine can be spread across many. A **replica** is a full copy of a primary shard, kept on a different node, for both fault tolerance (the replica takes over if the node holding the primary fails) and read scalability (queries can be served from either copy).

A replica is never placed on the same node as its primary, because that would provide zero protection - losing the node loses both copies at once. On a **single-node cluster**, the default index settings still ask for one replica per shard, but there's nowhere else to put it, so those replica shards stay permanently unassigned. Cluster health reflects exactly that: green means every primary and every replica is assigned; yellow means every primary is assigned but some replicas aren't; red means at least one primary itself is unassigned (real data loss risk, not just redundancy risk). A single-node dev cluster sitting at yellow forever is expected behavior, not a misconfiguration - the fix, if you actually want green on a single node, is to explicitly set `number_of_replicas: 0` for that index.

[OpenSearch cluster health API](https://docs.opensearch.org/latest/api-reference/cluster-api/cluster-health/)

Question 30

How do you diagnose and fix an OpenSearch cluster that's constantly yellow, even with multiple nodes available?

Accepted Answer

Start with the cluster health API against a specific index, not just the cluster as a whole, since "yellow" only tells you *something* has an unassigned replica, not which index or why.

```text
GET _cluster/health?level=indices
```

From there, common root causes and how to tell them apart: **not enough nodes for the replica count** - if an index wants one replica per shard but you only have as many data nodes as primary shards with no room for a distinct copy, the replica can't be placed; the fix is either add a node or lower `number_of_replicas`. **Disk watermark thresholds** - OpenSearch stops allocating new shards to a node once it crosses a configured disk-usage percentage, which shows up as replicas stuck unassigned even though there's technically a free node; check `_cat/allocation` and either free disk space or adjust the watermark. **Shard allocation awareness / zone constraints** - if you've configured awareness attributes (e.g. spreading replicas across availability zones) but don't have nodes in enough zones, replicas can't satisfy the constraint and stay unassigned. **Over-sharding** - too many small shards per node drives up JVM heap pressure from shard metadata overhead alone, which can cause allocation to fail or the cluster to become unstable well before you'd expect from data volume; the standard mitigation here is Index State Management (ISM), rolling over to a new index by size or age and deleting old ones on a retention schedule rather than letting shard count grow unbounded.

The trap is assuming yellow always means "something is broken" - on a single node it's expected, and even on a healthy multi-node cluster it can be a transient, self-resolving state during a rolling restart or node replacement, not an incident on its own.

[OpenSearch shards and replicas](https://opster.com/blogs/opensearch-shards-and-replicas/)

Question 31

What is OpenSearch Index Lifecycle Management (ISM), and why is it standard practice for log data specifically?

Accepted Answer

ISM (Index State Management in OpenSearch, ILM in the Elasticsearch lineage) automates moving an index through phases - typically hot, warm, cold, and delete - based on age or size, without a human running those actions manually.

Log data is the textbook use case because it's naturally time-partitioned and has a natural expiry: you almost never need to search six-month-old debug logs with the same performance as today's, and keeping every index around forever is how clusters silently accumulate thousands of small, mostly-idle shards, which drives up per-node JVM heap pressure and eventually destabilizes the cluster (the over-sharding problem). A typical policy rolls an index over once it hits a size or age threshold (so "today's logs" is always a fresh, appropriately-sized index rather than one index growing forever), then deletes indices past a retention window automatically.

```json
PUT _plugins/_ism/policies/logs_rollover_policy
{
  "policy": {
    "description": "Roll over daily or at 50 GB, delete after 7 days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          { "rollover": { "min_size": "50gb", "min_index_age": "1d" } }
        ],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "7d" } }
        ]
      },
      {
        "name": "delete",
        "actions": [ { "delete": {} } ],
        "transitions": []
      }
    ],
    "ism_template": [
      { "index_patterns": ["logs-*"], "priority": 100 }
    ]
  }
}
```

[OpenSearch Index State Management](https://docs.opensearch.org/latest/im-plugin/ism/index/)

Question 32

How many primary shards should you configure for a new OpenSearch index?

Accepted Answer

There's no single correct number - it's a trade-off, and the honest answer names the trade-off rather than a formula.

Too few shards for the data volume means each shard grows very large, which slows queries against it and makes rebalancing (moving that shard to another node) slow and expensive when it eventually needs to happen. Too many shards, especially many small ones, means the overhead - each shard carries its own Lucene index structures and metadata held in JVM heap - starts to dominate: a cluster with thousands of tiny shards can run out of usable heap and become unstable well before actual data volume would justify it, since shard *count* costs memory independent of shard *size*.

The practical guidance most teams converge on: aim for shard sizes in roughly the tens of gigabytes range (commonly cited as 10-50GB) rather than picking a shard *count* up front, keep JVM heap pressure from shard overhead under about 75% as a safety margin, and for continuously-growing data like logs, don't try to pick the final number at index-creation time at all - use rollover (via ILM/ISM) to create a new index once the current one hits a size threshold, so shard sizing becomes self-correcting instead of a one-time guess that ages badly as volume grows.

The trap is answering with a fixed number like "5 shards" as if it's a rule - the correct framing is "it depends on expected index size and node count, and for time-series data you avoid the question by rolling over instead of guessing."

[Sizing OpenSearch shards](https://opster.com/blogs/opensearch-shards-and-replicas/)

Question 33

What is the difference between OpenSearch's Query DSL term-level queries and full-text queries?

Accepted Answer

**Term-level queries** (`term`, `terms`, `range`) match exact values against a field without any text analysis - they're for structured data like an exact status code, a keyword field, or a numeric range, and the value you search for has to match precisely what's stored.

**Full-text queries** (`match`, `match_phrase`, `query_string`) run the search text through the same analyzer used when the field was indexed - lowercasing, tokenizing on word boundaries, sometimes stemming - before comparing, which is what lets a search for "running" match a document containing "ran" or "run" depending on the analyzer, and lets "Error Connecting" match a log line however it was cased or punctuated.

The common mistake this question is checking for: running a `term` query against a `text` field expecting it to behave like a full-text search, and getting zero results because the field was analyzed at index time (broken into lowercase tokens) but `term` compares against the raw, unanalyzed input. The fix is either querying the field's `.keyword` sub-field for exact matching, or switching to a `match` query if full-text behavior is actually what's wanted.

[OpenSearch query DSL](https://docs.opensearch.org/latest/query-dsl/)

Question 34

What is OpenTelemetry, and how does it relate to tools like Prometheus and Fluent Bit?

Accepted Answer

OpenTelemetry (OTel) is a vendor-neutral standard and set of SDKs/APIs for generating and exporting the three observability signal types - metrics, logs, and traces - from your application code, without tying that instrumentation to a specific backend.

It sits at a different layer than Prometheus or Fluent Bit rather than competing with them directly: OTel defines how signals are *generated and shaped* in your application and how they're transported (often via an OTel Collector, which can receive, process, and export to many backends), while Prometheus and OpenSearch are backends that *store and query* metrics and logs respectively, and Fluent Bit is a collector focused specifically on log shipping. In practice, the OTel Collector can scrape Prometheus-format metrics as an input and export them to Mimir or Prometheus remote-write as an output, or receive OTLP-format traces and logs and forward them to whatever backend you use - the appeal is instrumenting an application once, in a standard format, and being able to swap backends later without re-instrumenting.

The distinction worth naming clearly: OTel doesn't replace Prometheus or OpenSearch as storage, and it doesn't replace Fluent Bit as a node-level log collector - it's a common instrumentation and transport layer that can feed all of them.

[OpenTelemetry overview](https://opentelemetry.io/docs/what-is-opentelemetry/)

Question 35

What is the RED method, and how is it different from the USE method?

Accepted Answer

Both are frameworks for deciding what to actually monitor, aimed at two different layers of a system.

**RED** (Rate, Errors, Duration) is for request-driven services: how many requests per second is this handling, what fraction are failing, and how long are they taking. It's the natural fit for an API, a microservice, or anything you'd describe in terms of requests coming in and responses going out - these three signals catch the vast majority of user-facing problems in a service.

**USE** (Utilization, Saturation, Errors) is for resources: CPU, memory, disk, network. Utilization is how busy the resource is, saturation is how much work is queued waiting for it, and errors are, well, errors. This is the right lens for infrastructure and hardware-level components that don't have a "request" concept in the same way a service does.

A well-instrumented system typically uses both at their appropriate layer: RED dashboards for services, USE dashboards for the nodes and infrastructure underneath them, rather than trying to force one framework to cover everything.

Question 36

What's the difference between monitoring and observability?

Accepted Answer

Monitoring is watching a predefined set of signals against known thresholds - you decide in advance what to measure (CPU usage, error rate, response time) and get alerted when it crosses a line. It answers questions you thought to ask beforehand.

Observability is the broader property of a system: how much you can understand about its *internal* state just from the external outputs it produces (metrics, logs, traces), including questions you didn't anticipate needing to ask. A highly observable system lets you debug a novel failure mode you've never seen before by exploring the data you already have, rather than needing to ship new instrumentation first and wait for the bug to recur.

In practice the two aren't opposites - monitoring is a subset of what observability enables. The distinction matters in interviews mainly because "observability" gets used as a buzzword; being able to define it concretely (rich, high-cardinality, explorable telemetry vs. a fixed dashboard of known metrics) signals you've actually thought about it rather than just absorbed the term.

Question 37

What is an SLO, and how does error-budget burn-rate alerting improve on a static threshold alert?

Accepted Answer

An SLO (Service Level Objective) is a target for how reliable a service should be over a window - "99.9% of requests succeed over a rolling 30 days" - derived from an SLI (the actual measured indicator, like success rate) and backed by an error budget: the amount of unreliability you're allowed before breaching the objective (0.1% of requests, in that example).

A static threshold alert ("page if error rate > 5% for 5 minutes") has two failure modes: it can miss a slow, sustained low-grade problem that never crosses 5% but steadily eats the entire monthly error budget over three weeks, and it can also page unnecessarily for a brief spike that, in the context of the full 30-day window, barely dents the budget at all.

Burn-rate alerting instead asks "at the current rate of failure, how fast is the error budget being consumed?" - a burn rate of 1x means the budget is being used exactly on pace to be fully spent right at the end of the window (fine); a burn rate of 14x means the entire 30-day budget will be gone in about two days if this continues, and 2% of it is already gone after one hour (page immediately). The standard implementation, multi-window multi-burn-rate (MWMBR) alerting from Google's SRE practice, checks burn rate over two windows at once - a short one (catches fast, severe burns) and a long one (avoids paging on a brief blip that self-corrects) - before firing, which is what makes it both fast to catch real incidents and resistant to noise from short-lived spikes.

The trap in this question is describing burn-rate alerting as "just a different threshold" - the conceptual shift is alerting on rate of budget consumption relative to a time window, not on an absolute value in isolation.

[Google SRE workbook - alerting on SLOs](https://sre.google/workbook/alerting-on-slos/)

Question 38

What's the difference between structured and unstructured logging, and why does it matter for a centralized logging pipeline?

Accepted Answer

Unstructured logging is free-text: `2026-01-15 ERROR Failed to connect to database after 3 retries`. It's easy for a human to read in isolation but hard for a machine to reliably parse, because extracting the retry count or the error type requires regex that breaks the moment someone rewords the message.

Structured logging emits the same information as key-value fields, typically JSON: `{"level": "error", "msg": "db connection failed", "retries": 3, "service": "orders"}`. Every field is queryable and filterable directly by a log backend like OpenSearch without any parsing step, aggregations work correctly (you can sum `retries` across all events, which you can't reliably do against a free-text field), and it survives message wording changes without breaking downstream dashboards or alerts built on those fields.

The practical cost is on the collection side: a structured-JSON log can be ingested directly by Fluent Bit's JSON parser with no pattern matching, while an unstructured log needs a regex parser that has to be kept in sync with the application's log format - the moment a developer tweaks a log message's wording, that regex silently stops matching and the pipeline starts losing fields (or entire records) without any error. This is the concrete reason "log in JSON from day one" is standard advice rather than a nice-to-have.

Question 39

What is log sampling, and why would you deliberately drop logs before storing them?

Accepted Answer

Log sampling means only keeping a subset of log records that match some criteria, rather than every record generated, typically to control storage and ingestion cost on very high-volume, low-information log types.

The case for it: a busy service can generate log volume that's mostly repetitive and low-value - the same successful health-check line thousands of times a minute - and indexing all of it in OpenSearch or shipping all of it to a paid logging backend costs real money and storage for information you'll essentially never query. Sampling strategies range from simple (keep 1 in every N successful requests, keep 100% of errors) to more deliberate tail-based sampling in tracing contexts (decide whether to keep a trace based on its outcome - an error or unusually slow request - after it completes, rather than sampling uniformly up front).

The trade-off, and the reason this is a hard question rather than an easy one: sampling is a bet that what you're dropping was actually low-value, and you find out you're wrong exactly when you need the dropped data most - debugging an intermittent issue that only shows up in the 99% of successful requests you didn't keep. The safer version of this pattern keeps all error and warning-level logs unconditionally and only samples verbose/debug-level or routine-success logs, so the sampling risk is concentrated on the data least likely to matter during an incident.

Question 40

In Grafana, what does it mean for an alert to be in the "No Data" state, and why is that different from "OK"?

Accepted Answer

"OK" means the alert rule's query returned data and that data didn't meet the alert condition - the thing being monitored is actively confirmed healthy. "No Data" means the query returned nothing at all to evaluate, which is a meaningfully different and often more concerning situation: it could mean the metric genuinely doesn't exist (a typo in the query, a data source that isn't scraping), or it could mean the target the metric comes from has disappeared entirely - a crashed exporter, a dead service, a broken network path to the data source.

The trap this question is checking for is treating "No Data" and "OK" as roughly equivalent ("no alert fired, so we're fine"). A silently-broken metrics pipeline produces "No Data," not a firing alert, so if "No Data" isn't itself configured to notify someone, an entire class of failure - the monitoring itself going dark - passes by unnoticed. Grafana lets you explicitly configure how a rule should behave on "No Data" (treat as OK, treat as Alerting, or keep the last state), and the safe default for anything critical is to treat it as an alerting condition, not a clean pass.

Question 41

What is cardinality-aware cost control in a logging pipeline, and how does log volume differ from log cardinality as a cost driver?

Accepted Answer

Log **volume** is simply how many bytes or records you're ingesting - the obvious cost driver, and the one most teams think of first, controlled with sampling, filtering, and retention.

Log **cardinality** is a subtler cost driver that behaves differently: it's the number of distinct label or field-value combinations attached to your logs, and it matters most in systems (like Grafana Loki, or index-mapping-heavy OpenSearch setups) where the storage or indexing structure scales with the number of *distinct combinations*, not just total bytes. Putting a user ID, a request ID, or a raw unbounded URL into an indexed label rather than into the log line's body can multiply the number of distinct index entries even if total log volume didn't change at all - this is structurally the same failure mode as Prometheus metric-label cardinality blowing up a metrics index, just applied to logs.

The practical fix mirrors the metrics-side fix: keep genuinely high-cardinality fields (user IDs, request IDs, raw paths) in the log body where they can be searched via full-text query, and reserve indexed labels/fields for genuinely low-cardinality dimensions (service name, environment, log level, region) that you actually want to filter and aggregate by. Conflating "we have a lot of logs" (volume) with "our labels have too many distinct values" (cardinality) leads to the wrong fix - sampling won't help a cardinality problem, and stricter label discipline won't help a pure volume problem.

Question 42

What is a golden signal, and where does the term come from?

Accepted Answer

"Golden signals" refers to a small set of metrics - commonly latency, traffic, errors, and saturation - that Google's SRE book identifies as the highest-value things to monitor for a user-facing service, on the reasoning that if you can only instrument a handful of things well, these four catch the overwhelming majority of real problems users actually experience.

It's closely related to (and sometimes merged with, as "RED plus Saturation") the RED method - Rate, Errors, Duration map fairly directly onto Traffic, Errors, Latency, with Saturation added as a fourth axis pulled in from the resource-monitoring side (USE). Interviewers ask this mostly to see whether a candidate defaults to instrumenting everything they *can* measure versus starting from a deliberately small, well-chosen set that's actually actionable.

[Google SRE book - monitoring distributed systems](https://sre.google/sre-book/monitoring-distributed-systems/)

Question 43

What is the difference between Grafana's Explore view and a dashboard?

Accepted Answer

A dashboard is a fixed, saved arrangement of panels meant to be looked at repeatedly - a service overview you check every morning, an on-call landing page. It's built in advance and designed to answer known, recurring questions at a glance.

Explore is a free-form, ad-hoc query interface for digging into raw data one query at a time - no saved layout, no fixed panels, just a query box against a chosen data source with the ability to quickly switch between metrics and logs while preserving label filters. It's the tool you reach for while actively debugging something you don't have a pre-built dashboard for, since building and saving a dashboard for a one-off investigation would be wasted effort.

Question 44

Your Grafana dashboard shows a metric spiking, but the corresponding logs from the same time window show nothing unusual. What are the likely explanations?

Accepted Answer

This is a correlation-debugging question, and a good answer walks through several concrete possibilities rather than picking one.

**Clock or timezone mismatch** - the metrics data source and the log data source may be using different time bases (UTC vs local, or a scrape-time vs event-time discrepancy), so "the same time window" in the dashboard isn't actually the same wall-clock window in both systems. Always check the underlying timestamps, not just what the dashboard UI shows.

**The spike is infrastructure-level, not application-level** - a CPU or network saturation spike can happen with zero corresponding application log lines, because nothing in the *application's* code path logged anything; the problem is at a layer below what the app instruments. This is exactly why RED (application-level) and USE (resource-level) signals are tracked separately - one can spike without the other.

**Log sampling or filtering dropped the relevant records** - if the pipeline samples routine/successful logs, and the spike corresponds to a burst of otherwise-successful requests (a traffic spike, not an error spike), the records that would explain it may have been sampled out before storage.

**The metric itself is noisy or mislabeled** - a cardinality issue, a bad relabeling rule, or a metric that doesn't mean what its name implies can produce a spike that isn't a real event at all; verify what's actually generating the metric before assuming the underlying system did something.

The answer that's usually wrong: assuming the logs are complete and concluding "nothing happened" - the correct instinct is to question whether the two data sources are actually looking at the same thing, in the same window, with nothing filtered out, before concluding there's no correlation.

Question 45

What is the difference between Grafana Loki's approach to indexing logs and OpenSearch's approach, and why does it matter for cost?

Accepted Answer

OpenSearch (like Elasticsearch) indexes the full content of every log line by default - it builds an inverted index over the text itself, which is what makes arbitrary full-text search across the entire log body fast, but that indexing work and the resulting index size scale with total log volume and is a major driver of both storage cost and cluster resource usage.

Loki takes a different approach: it only indexes a small set of labels (service, environment, level - deliberately low-cardinality metadata), not the log content itself, and stores the actual log lines as compressed chunks in cheap object storage. Querying Loki means first narrowing down to the right chunks via label matching (fast, since the label index is small), then grep-style scanning the log content within those chunks at query time rather than at index time. This makes ingestion cheap and storage cheap, at the cost of query-time full-text search being comparatively slower than OpenSearch's pre-built index, especially over very broad label selections.

The trade-off in one line: OpenSearch pays indexing cost up front for fast arbitrary search later; Loki defers that cost to query time in exchange for far cheaper ingestion and storage. Which one is the right fit depends heavily on your actual query pattern - if you frequently need broad free-text search across huge volumes, OpenSearch's model earns its cost; if most queries are already scoped by service/environment/time and content search is secondary, Loki's model is dramatically cheaper for the same log volume.

[Grafana Loki architecture](https://grafana.com/docs/loki/latest/get-started/architecture/)

Question 46

What is Prometheus federation, and why is it usually the wrong tool if what you actually want is long-term global storage?

Accepted Answer

Federation is a Prometheus server scraping a filtered subset of time series from another Prometheus server's `/federate` endpoint, treating that remote server as just another scrape target. It's still pull-based scraping under the hood - it doesn't add clustering, shared storage, or automatic sharding, it just lets you compose a hierarchy of Prometheus servers.

The pattern it's genuinely good for is **hierarchical aggregation**: many per-cluster or per-datacenter Prometheus servers each scrape detailed, instance-level metrics locally, and a higher-level "global" Prometheus federates only the already-aggregated, job-level series from each of them - enough to build a single cross-cluster dashboard or a global alert, without ever pulling the full-cardinality local data into one place. A second valid pattern, cross-service federation, is one team's Prometheus pulling a specific handful of series from another team's Prometheus for a joint alert or sanity check.

Where it breaks down is being used as a substitute for actual long-term, multi-cluster storage: federation only exposes what you explicitly filter for at scrape time, it inherits the local server's retention window, and pulling large or high-cardinality metric sets through federation just recreates the cardinality and scale problems you were trying to solve, one hop later. For genuine long-term retention and a real global query view across many Prometheus servers, `remote_write` into Thanos, Cortex, or Mimir is the current standard approach - federation has narrowed to a single legitimate niche (a small, deliberately filtered set of aggregated series for one global view) rather than a general scaling mechanism.

The trap: describing federation as "how Prometheus scales," full stop. It scales *specific, pre-aggregated queries* across a hierarchy; it does not scale storage or cardinality.

[Prometheus federation docs](https://prometheus.io/docs/prometheus/latest/federation/)

Question 47

What is the Prometheus Blackbox Exporter, and how is blackbox monitoring different from the exporters you'd normally scrape?

Accepted Answer

Most exporters are whitebox: `node_exporter` or an application's own `/metrics` endpoint reports on internal state - CPU load, queue depth, internal counters - from inside the system being monitored. The Blackbox Exporter does the opposite: it sits outside the target and actively probes it over HTTP(S), TCP, DNS, or ICMP, then reports the result (did it respond, how long did it take, was the certificate valid, did the response match an expected pattern) as Prometheus metrics like `probe_success` and `probe_duration_seconds`.

The distinction matters because whitebox and blackbox monitoring catch different failure classes. A service can be internally "healthy" by every whitebox metric - CPU fine, no error logs - while still being unreachable from outside due to a firewall rule, a DNS misconfiguration, or a load balancer misroute; only an external probe catches that. Conversely, blackbox monitoring alone can't tell you *why* something is failing, just that it is - you still need whitebox metrics and logs to diagnose the root cause once a blackbox probe flags an outage.

A specific trap worth naming: `probe_success` (did this specific probe succeed) is not the same signal as the standard `up` metric (is Prometheus able to scrape this target's `/metrics` endpoint) - a target can be `up` for scraping purposes while the actual service being probed through it is down, since Blackbox Exporter itself is what Prometheus scrapes, not the target directly. Configuring alerts against the wrong one of these two metrics is a common early mistake.

[Blackbox Exporter](https://github.com/prometheus/blackbox_exporter)

Question 48

What is configuration-as-code (provisioning) in Grafana, and why do teams avoid building dashboards by hand in the UI for anything long-lived?

Accepted Answer

Provisioning means defining dashboards, data sources, alert rules, and notification policies as version-controlled files (JSON or YAML) that Grafana loads automatically on startup, rather than an engineer clicking through the UI and hitting Save.

The case for it mirrors infrastructure-as-code generally: a hand-built dashboard exists only in one Grafana instance's database, with no history of who changed what threshold or why, no code review before a change ships, and no reliable way to recreate it if that instance is lost or you need the same dashboard in a second environment. Provisioned dashboards live in a git repository like any other config, get reviewed in a pull request, and can be deployed identically to dev, staging, and production Grafana instances - which also means the dashboard a developer tests against locally is guaranteed to match what's running in production, rather than drifting the moment someone tweaks a panel directly on the live instance.

The practical trade-off: provisioned dashboards are typically read-only in the UI (edits made there don't persist, or get flagged as diverging from the source file), which is a deliberate choice to keep the file the single source of truth - teams that want to prototype visually first, then export the finished JSON into the provisioning repo once it's right, use the UI as a draft tool rather than the system of record.

[Grafana provisioning docs](https://grafana.com/docs/grafana/latest/as-code/observability-as-code/provision-resources/)

Question 49

What is a Kubernetes liveness probe versus a readiness probe, and why does confusing them cause monitoring gaps or outages?

Accepted Answer

A **liveness probe** answers "is this container in a state where it should be killed and restarted?" - if it fails repeatedly, Kubernetes kills the pod and starts a fresh one. A **readiness probe** answers a different question: "should this pod currently receive traffic?" - if it fails, Kubernetes removes the pod from the Service's load-balanced endpoints, but does *not* restart the container; the pod keeps running and can rejoin once readiness passes again.

The common, expensive mistake is wiring the same check to both, especially a check that depends on an external dependency (a database connection, a downstream API). If that dependency has a brief outage, a shared check makes both probes fail: Kubernetes kills and restarts every pod that depends on it (via liveness) at the exact moment they're already struggling, which can turn a transient downstream blip into a full restart storm across your fleet - the opposite of what you want, since restarting doesn't fix a problem that's external to the pod, it just adds a slow, disruptive recovery on top of it.

The correct split: liveness should only fail for problems the container itself can't recover from without a restart (a deadlocked process, a broken internal state) and generally shouldn't call out to external dependencies at all. Readiness should reflect actual serving capability, including dependency health, so traffic gets pulled away from a struggling pod without killing it - letting it recover on its own once the dependency comes back, without a disruptive restart in the middle.

This connects directly to monitoring because liveness/readiness flapping shows up as a very specific, recognizable signature in metrics and logs - restart counts climbing in lockstep with a downstream outage, rather than with anything actually wrong in the pod itself - and being able to name that pattern is often exactly what this question is testing.

[Kubernetes liveness/readiness/startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Question 50

What is trace-log-metric correlation, and what has to be true of your telemetry for it to actually work?

Accepted Answer

Correlation means being able to jump directly from one signal to another for the *same event* - from a metric spike straight to the specific logs and the specific distributed trace that explain it - instead of manually guessing a time window and hoping you land on the right entries in a completely separate system.

For this to actually work, a few things have to be true that don't happen automatically. First, every signal needs a shared identifier that survives the trip across systems - typically a `trace_id` generated at the edge of a request and propagated through every service call, attached to the spans in your tracing backend, and also injected into the structured logs each service emits while handling that request. Without deliberately propagating that ID through log statements, your logs and your traces are just two systems that happen to cover the same time period, not two views of the same event - and "same time period" breaks down under load, when hundreds of unrelated requests are in flight at once and a time-window guess returns noise.

Second, clocks and time bases need to be consistent - NTP-synced across hosts, and ideally all signals timestamped in UTC - since a correlation UI that lines up a metric spike with logs from a clock that's even a few seconds off will show you the wrong window's logs and lead you to the wrong conclusion.

This is precisely the problem OpenTelemetry's semantic conventions and context propagation exist to standardize: instrumenting metrics, logs, and traces so that a `trace_id` (and related context) flows consistently through all three, which is what makes tools like Grafana's Explore (pivoting from a Tempo trace straight into the matching Loki logs) actually work, rather than being three separately-useful tools you happen to have open in adjacent tabs.

[OpenTelemetry - correlating signals](https://opentelemetry.io/docs/concepts/signals/)

Question 51

What is a time series in Prometheus, and what are labels?

Accepted Answer

A time series is a stream of numeric values for one metric, each stamped with the time it was recorded. Labels are key-value pairs attached to a metric that split it into separate series.

For example, `http_requests_total{method="GET", status="200"}` and `http_requests_total{method="POST", status="500"}` share one metric name but are two different time series, because their label values differ.

Labels are what make Prometheus data easy to slice: you can total requests per `method`, or filter to one `status`. The cost is that every new combination of label values creates another series, so labels must stay bounded (see high cardinality).

Reference: [Data model](https://prometheus.io/docs/concepts/data_model/)

Question 52

What is PromQL?

Accepted Answer

PromQL (Prometheus Query Language) is the query language you use to read and calculate over Prometheus data. It is used in dashboards, alert rules, and recording rules.

A query returns either an instant value ("what is it now?") or a range of values over time. You select series by metric name and labels, then apply functions to them:

```promql
# per-second request rate over 5 minutes, summed per service
sum by (service) (rate(http_requests_total[5m]))
```

PromQL is built for time-series maths: rates, sums, ratios, and percentiles. It is not SQL, and it cannot join arbitrary tables, but it is very good at questions like "what share of requests failed in the last 10 minutes?".

Reference: [Querying basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)

Question 53

What does the `up` metric tell you in Prometheus?

Accepted Answer

`up` is a metric Prometheus creates automatically for every scrape target. It is `1` when the last scrape succeeded and `0` when it failed.

It answers a basic question: "is Prometheus able to collect metrics from this target?". An alert such as `up == 0` for a few minutes is the standard way to notice that a target or exporter has disappeared.

Be careful about what it does **not** tell you. `up == 1` means the `/metrics` endpoint responded. It does not mean the service is working for users. For that, you need request error rates or an external probe.

Reference: [Jobs and instances](https://prometheus.io/docs/concepts/jobs_instances/)

Question 54

What are the scrape interval and scrape timeout in Prometheus?

Accepted Answer

The **scrape interval** is how often Prometheus fetches metrics from a target (commonly 15s to 60s). The **scrape timeout** is how long it waits for one scrape to answer before giving up and marking it failed.

The timeout must be less than or equal to the interval. If a target regularly takes longer than the timeout to respond, you will see gaps in the data and `up == 0`.

Choosing the interval is a trade-off. A shorter interval gives finer detail but costs more storage and load. A longer one is cheaper but can miss short spikes. Also remember that `rate()` needs at least two samples in its window, so a range such as `[5m]` should be several times larger than the scrape interval.

Reference: [Scrape configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)

Question 55

What is the Prometheus Pushgateway, and when should you use it?

Accepted Answer

The Pushgateway is a small service that accepts metrics pushed to it and then exposes them for Prometheus to scrape. It exists for short-lived jobs (like a nightly batch script) that finish before Prometheus could ever scrape them.

The job pushes its final result (for example, "last success time") to the Pushgateway, and Prometheus scrapes it from there.

It is easy to misuse. It is **not** a way to turn Prometheus into a push system for normal services. The Pushgateway keeps each pushed value forever until you delete it, so a job that stopped running still looks like it is reporting, and you lose the automatic `up` health check per instance. Use it for batch jobs only.

Reference: [When to use the Pushgateway](https://prometheus.io/docs/practices/pushing/)

Question 56

What is Grafana Loki?

Accepted Answer

Loki is Grafana's log storage system. It is designed to be cheap to run: instead of indexing the full text of every log line, it indexes only a small set of labels (such as `app`, `namespace`, `level`) and stores the log lines themselves as compressed chunks.

You query it with LogQL and view results in Grafana. A common query first picks streams by label, then searches inside them:

```logql
{app="checkout", env="prod"} |= "timeout"
```

Because it indexes so little, Loki is inexpensive for large log volumes. The trade-off is that searching inside log text is done at query time, so it works best when you narrow by labels and time range first.

Reference: [Loki overview](https://grafana.com/docs/loki/latest/get-started/overview/)

Question 57

What are a trace and a span in distributed tracing?

Accepted Answer

A **trace** is the full journey of one request as it moves through all the services that handle it. A **span** is one timed step inside that journey, such as "call the payment service" or "run this SQL query".

Spans are linked in a parent-child tree, so you can see that a slow checkout spent 800 ms in the database call and only 20 ms everywhere else. Each span records a start time, a duration, a name, and attributes (status, HTTP route, and so on).

Metrics tell you that something is slow. Logs tell you what happened in one place. Traces show you **where** across many services the time went.

Reference: [Traces](https://opentelemetry.io/docs/concepts/signals/traces/)

Question 58

What is OpenSearch Dashboards?

Accepted Answer

OpenSearch Dashboards is the web interface for OpenSearch. You use it to search and filter logs (the Discover page), build charts and dashboards, run queries, and manage the cluster. It is the OpenSearch counterpart of Kibana in the Elasticsearch world, and was forked from Kibana.

For log work, the usual flow is: create an **index pattern** that matches your indices (for example `logs-*`), open **Discover** to search by time range and fields, then save useful searches and build visualizations from them.

It reads data from OpenSearch and does not store log data itself.

Reference: [OpenSearch Dashboards](https://opensearch.org/docs/latest/dashboards/)

Question 59

What are log levels, and how should you use them?

Accepted Answer

Log levels label how important or severe a message is. Common ones, from least to most severe, are `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`.

A good habit:
- **DEBUG**: detail for developers, usually off in production.
- **INFO**: normal events worth recording (a service started, a job finished).
- **WARN**: something unexpected that the system recovered from.
- **ERROR**: an operation failed and needs attention.
- **FATAL**: the process cannot continue.

Levels matter for operations because you can filter and alert on them (for example, "error logs per minute"), and you can lower the level in production to save cost. A common mistake is logging every normal event as `ERROR`, which makes the level meaningless and your alerts noisy.

Question 60

What are an SLI, an SLO, and an SLA?

Accepted Answer

They are three related ideas about reliability, and people often mix them up.

- **SLI (Service Level Indicator)**: a measurement of how the service is doing, such as "the percentage of requests that succeed" or "the share of requests faster than 300 ms".
- **SLO (Service Level Objective)**: the target you set for that measurement, such as "99.9% of requests succeed over 30 days".
- **SLA (Service Level Agreement)**: a promise to customers, usually with consequences (such as refunds) if it is missed. It is a business contract, and it is normally looser than the internal SLO.

The SLO is the one engineers work with day to day. The gap between 100% and the SLO is the **error budget**: the amount of failure you can afford.

Reference: [Service Level Objectives](https://sre.google/sre-book/service-level-objectives/)

Question 61

What is log rotation, and why is it needed?

Accepted Answer

Log rotation means closing the current log file at a set time or size, starting a new one, and compressing or deleting old ones. It stops log files from growing until they fill the disk.

On Linux this is usually done by `logrotate`. A typical rule rotates daily, keeps 7 old files, and compresses them. In containers, the runtime (or Kubernetes) rotates the container log files for you.

It matters for log shipping because the collector must cope with files being renamed or replaced. A good agent tracks files by inode and remembers its read position, so it does not lose lines or re-send old ones when rotation happens.

Reference: [logrotate(8)](https://man7.org/linux/man-pages/man8/logrotate.8.html)

Question 62

What are MTTD and MTTR, and why do teams track them?

Accepted Answer

**MTTD** (Mean Time To Detect) is the average time between a problem starting and someone knowing about it. **MTTR** (Mean Time To Recover, sometimes Repair or Respond) is the average time from detection until the service is working again.

Monitoring mainly shrinks MTTD: good alerts mean you learn about an outage in minutes, not from an angry customer. Good dashboards, logs, traces, and runbooks shrink MTTR, because the on-call engineer finds the cause faster.

Be careful with averages. A few very long incidents can hide behind a healthy mean, so many teams also look at the median and the worst cases.

Question 63

What does node_exporter expose, and what is it used for?

Accepted Answer

`node_exporter` is the standard Prometheus exporter for Linux hosts. It runs on each machine and exposes hardware and operating system metrics on `/metrics`: CPU time, memory, disk space and I/O, network traffic, load average, and file system usage.

You use it to answer infrastructure questions such as "is this disk about to fill?" or "is this host CPU-bound?". Metric names start with `node_`, for example `node_filesystem_avail_bytes`.

It reports on the host, not on your application. In Kubernetes it usually runs as a DaemonSet (one per node), and container-level metrics come from a different source (cAdvisor).

Reference: [node_exporter](https://github.com/prometheus/node_exporter)

Question 64

What is the difference between rate(), irate(), and increase() in PromQL?

Accepted Answer

All three work on counters, but they answer slightly different questions.

- **`rate(x[5m])`**: the average per-second increase over the whole window. Smooth and stable. This is the right default, especially for alerts.
- **`irate(x[5m])`**: the per-second rate using only the **last two samples** in the window. It reacts quickly and looks spiky, so it suits zoomed-in graphs of fast-moving counters, but it can miss short events and is a poor choice for alerting.
- **`increase(x[5m])`**: the total increase over the window. It is roughly `rate()` multiplied by the window length in seconds.

Rule of thumb: use `rate()` for alerts and normal dashboards, `increase()` when you want "how many in the last hour?", and `irate()` only for detailed views. Make the range at least four times the scrape interval so there are enough samples.

Reference: [Query functions](https://prometheus.io/docs/prometheus/latest/querying/functions/)

Source	What it reports	Example question it answers
node_exporter	Host (node) OS and hardware metrics	Is this node's disk nearly full?
cAdvisor (built into the kubelet)	Resource use per container	How much CPU and memory is this container using?
kube-state-metrics	The desired and current state of Kubernetes objects, read from the API	Are all replicas of this Deployment available?

Severity	Burn rate	Long window	Short window	Budget used
Page	14.4x	1 hour	5 minutes	2%
Page	6x	6 hours	30 minutes	5%
Ticket	1x	3 days	6 hours	10%

Monitoring & Logging Interview Questions

What is Prometheus, and why is a pull-based model its default?

What are the four Prometheus metric types, and when do you use each?

What does the Prometheus Alertmanager actually do, separate from Prometheus itself?

What is an exporter, and how is it different from instrumenting your own application?

What is Grafana, and how does it relate to Prometheus?

What's the difference between a Grafana panel, a dashboard, and a data source?

What is the LGTM stack, and what does each letter cover?

What is Fluent Bit, and how is it different from Fluentd?

What is OpenSearch, and how does it relate to Elasticsearch?

What's the difference between monitoring and observability?

What is a golden signal, and where does the term come from?

What is the difference between Grafana's Explore view and a dashboard?

What is a time series in Prometheus, and what are labels?

What is PromQL?

What does the `up` metric tell you in Prometheus?

What are the scrape interval and scrape timeout in Prometheus?

What is the Prometheus Pushgateway, and when should you use it?

What is Grafana Loki?

What are a trace and a span in distributed tracing?

What is OpenSearch Dashboards?

What are log levels, and how should you use them?

What are an SLI, an SLO, and an SLA?

What is log rotation, and why is it needed?

What are MTTD and MTTR, and why do teams track them?

What does node_exporter expose, and what is it used for?

Why should you never use rate() or irate() directly on a Gauge?

What does Prometheus service discovery do, and how does it work in Kubernetes?

What's the difference between a recording rule and an alerting rule?

What is a dead man's switch (Watchdog) alert, and why would you deliberately configure an alert to always be firing?

What is Thanos (or Cortex/Mimir), and what problem does it solve that Prometheus alone doesn't?

What are template variables in Grafana, and why would you use them instead of hardcoding a query?

How does Grafana manage user access across teams and organizations?

What is a Grafana transformation, and how is it different from modifying the underlying query?

Walk through the stages of a Fluent Bit pipeline.

What does a Fluent Bit parser do, and what's the difference between a regex parser and the built-in JSON parser?

What are shards and replicas in OpenSearch, and why does a single-node cluster with the default settings sit at yellow, not green?

What is OpenSearch Index Lifecycle Management (ISM), and why is it standard practice for log data specifically?

What is the difference between OpenSearch's Query DSL term-level queries and full-text queries?

What is OpenTelemetry, and how does it relate to tools like Prometheus and Fluent Bit?

What is the RED method, and how is it different from the USE method?

What's the difference between structured and unstructured logging, and why does it matter for a centralized logging pipeline?

In Grafana, what does it mean for an alert to be in the "No Data" state, and why is that different from "OK"?

What is the Prometheus Blackbox Exporter, and how is blackbox monitoring different from the exporters you'd normally scrape?

What is configuration-as-code (provisioning) in Grafana, and why do teams avoid building dashboards by hand in the UI for anything long-lived?

What is the difference between rate(), irate(), and increase() in PromQL?

How do you calculate a 95th percentile latency from a Prometheus histogram?

What is the difference between `sum by` and `sum without` in PromQL?

What does the `for` field do in a Prometheus alert rule, and what is `keep_firing_for`?

How would you design Alertmanager routing so the right people get the right alerts?

How does Prometheus store data locally, and how does retention work?

What is `remote_write` in Prometheus, and what is it used for?

What is the difference between node_exporter, kube-state-metrics, and cAdvisor in a Kubernetes cluster?

What is the Prometheus Operator, and what do ServiceMonitor and PrometheusRule do?

How do labels and streams work in Loki, and what makes a good label?

What is LogQL, and how can you turn logs into metrics with it?

How would you choose between Fluent Bit, Grafana Alloy, and the OpenTelemetry Collector for collecting telemetry?

What does the Fluent Bit Kubernetes filter do, and how can pods control their own parsing?

How does Fluent Bit's tail input avoid losing or repeating log lines after a restart or log rotation?

What happens when a Fluent Bit output fails to deliver, and how do retries work?

What is the difference between dynamic and explicit mapping in OpenSearch, and what is a mapping explosion?

What is the difference between `text` and `keyword` field types in OpenSearch?

What are hot, warm, and cold tiers for log data in OpenSearch, and why use them?

How do snapshots work in OpenSearch, and are replicas a substitute for them?

What is trace context propagation, and what breaks a trace into pieces?

What are index templates and data streams in OpenSearch, and why are they used for logs?

How is an OpenTelemetry Collector pipeline structured, and why does processor order matter?

What is the difference between head sampling and tail sampling for traces?

How do you keep sensitive data (passwords, tokens, personal data) out of logs?

Why are percentiles (p95, p99) better than averages for latency, and what are the pitfalls?

What makes an alert good, and how do you reduce alert fatigue?

What makes a good Grafana dashboard?

What are the key signals to monitor for applications running in Kubernetes?

What is CPU throttling in Kubernetes, and how do you detect it?

What is synthetic monitoring, and how does it differ from monitoring internal metrics?

How do you keep monitoring and logging costs under control as systems grow?

You run `increase(http_requests_total[5m])` and get a fractional, non-integer number back. Why, and is that a bug?

What is relabeling in Prometheus, and what's the difference between relabel_configs and metric_relabel_configs?

What is high cardinality in Prometheus, and why does it become a problem at scale?

How do you achieve high availability for Prometheus, given it's normally a single stateful process?