50 real Prometheus, Grafana, Fluent Bit and OpenSearch interview questions with answers on PromQL, alerting, shards and log pipelines.
Prometheus is an open-source metrics-based monitoring system: it scrapes numeric time-series data from targets, stores it in its own time-series database, and lets you query it with PromQL.
It defaults to pulling metrics (Prometheus fetches /metrics from each target) rather than targets pushing to it, for a few practical reasons: Prometheus can tell a target is unreachable (a failed scrape is a signal, whereas a missing push looks the same as "nothing happened"), you don't need to give every application credentials to a central collector, and you can run Prometheus locally against a target to debug it without touching production config. Short-lived jobs that finish before a scrape could ever reach them are the one case this doesn't fit, which is why Prometheus ships a separate Pushgateway for batch jobs rather than changing the core model.
The four types are Counter, Gauge, Histogram, and Summary.
The practical rule of thumb: reach for a histogram unless you have a specific reason not to, because it aggregates correctly across many instances, which is the normal case in a distributed system.
Alertmanager handles what happens after an alert fires — Prometheus itself only evaluates rules and decides something is wrong; it hands the resulting alert off to Alertmanager over HTTP.
Alertmanager then deduplicates identical alerts coming from multiple Prometheus replicas, groups related alerts together so one incident produces one notification instead of fifty, silences alerts that match a pattern during planned maintenance, can inhibit lower-priority alerts when a related higher-priority one is already firing, and finally routes the result to the right receiver — Slack, PagerDuty, email, a generic webhook — based on label matching rules you define in alertmanager.yml.
A useful way to separate the two mentally: Prometheus decides if something is wrong; Alertmanager decides who hears about it, how often, and in what shape.
An exporter is a small standalone process that translates metrics from something that doesn't natively speak Prometheus's format into a /metrics HTTP endpoint Prometheus can scrape. node_exporter exposes host-level metrics (CPU, memory, disk); mysqld_exporter queries a MySQL instance and republishes its internal stats the same way.
This is different from instrumenting your own application, where you add a Prometheus client library directly into your code and expose /metrics from the app process itself, with custom business metrics you define (orders processed, queue depth, cache hit ratio).
The rule of thumb: use an exporter for third-party systems you can't or don't want to modify; instrument directly for code you own, since it lets you emit metrics that are actually meaningful to your application's logic rather than generic infrastructure stats.
Grafana is an open-source visualization and dashboarding tool — it doesn't store or collect data itself, it connects to data sources (Prometheus, Loki, Elasticsearch/OpenSearch, InfluxDB, and dozens of others) and lets you build graphs, tables, and alerts on top of whatever they return.
Prometheus and Grafana are commonly paired because Prometheus is good at collecting and storing metrics but has a fairly minimal built-in UI, while Grafana specializes in rich, shareable dashboards, templated variables, and multi-source panels. In that pairing, Prometheus does the querying via PromQL under the hood; Grafana is just the presentation and alerting-on-top-of-visualization layer.
A data source is the connection to wherever your data actually lives — a Prometheus server, an OpenSearch cluster, a database. It's configured once and then reused by many dashboards.
A panel is a single visualization — one graph, table, heatmap, or stat — built from a query against one or more data sources.
A dashboard is a collection of panels arranged on a grid, usually all related to one system or purpose (a service overview, an infrastructure health page). Dashboards are what people actually look at day to day; panels are the individual building blocks inside them.
LGTM is Grafana Labs' name for a self-contained observability stack made of four open-source components: Loki for logs, Grafana for visualization and alerting across all of it, Tempo for distributed traces, and Mimir for long-term, horizontally-scalable metrics storage (a Prometheus-compatible remote-write target).
The point of the stack is that all three signal types — metrics, logs, traces — share the same query and label model as much as possible, so you can pivot from a metric spike in Grafana straight into the logs or traces for that same service and time window without switching tools or re-learning a query language. It's Grafana Labs' answer to running Prometheus, an ELK/OpenSearch stack, and a separate tracing backend as three disconnected systems.
Fluent Bit is a lightweight log and metrics collector and forwarder — it reads logs from a source (files, systemd journal, container runtimes), optionally parses and transforms them, and ships them to a destination like OpenSearch, Elasticsearch, Kafka, or S3.
It's a sibling project to Fluentd, from the same Fluent open-source project, but built for a different niche: Fluent Bit is written in C, has a much smaller memory and CPU footprint, and is designed to run as a per-node DaemonSet in resource-constrained environments like Kubernetes edge nodes. Fluentd is written in Ruby (with C extensions), has a larger plugin ecosystem, and is typically used as a more feature-rich aggregation layer sitting behind lightweight Fluent Bit collectors — a common Kubernetes pattern runs Fluent Bit as a DaemonSet on every node forwarding to a smaller number of centralized Fluentd (or Logstash) instances that do heavier processing before the final destination.
OpenSearch is a distributed search and analytics engine — you index documents (often log lines or structured events) into it and query them with full-text search, aggregations, and dashboards. It's a fork of Elasticsearch and Kibana, created by AWS in 2021 after Elasticsearch changed its license away from a fully open-source model, and it's maintained under the Apache 2.0 license with its own governance.
Because of the fork's timing, OpenSearch and Elasticsearch share the same core architecture and much of the same API surface from that point, but they've diverged in features since — newer Elasticsearch releases and newer OpenSearch releases are not drop-in compatible with each other, so "it's basically Elasticsearch" is true for the fundamentals (shards, the query DSL, cluster health) but not a safe assumption for anything added after the split.
Monitoring is watching a predefined set of signals against known thresholds — you decide in advance what to measure (CPU usage, error rate, response time) and get alerted when it crosses a line. It answers questions you thought to ask beforehand.
Observability is the broader property of a system: how much you can understand about its internal state just from the external outputs it produces (metrics, logs, traces), including questions you didn't anticipate needing to ask. A highly observable system lets you debug a novel failure mode you've never seen before by exploring the data you already have, rather than needing to ship new instrumentation first and wait for the bug to recur.
In practice the two aren't opposites — monitoring is a subset of what observability enables. The distinction matters in interviews mainly because "observability" gets used as a buzzword; being able to define it concretely (rich, high-cardinality, explorable telemetry vs. a fixed dashboard of known metrics) signals you've actually thought about it rather than just absorbed the term.
"Golden signals" refers to a small set of metrics — commonly latency, traffic, errors, and saturation — that Google's SRE book identifies as the highest-value things to monitor for a user-facing service, on the reasoning that if you can only instrument a handful of things well, these four catch the overwhelming majority of real problems users actually experience.
It's closely related to (and sometimes merged with, as "RED plus Saturation") the RED method — Rate, Errors, Duration map fairly directly onto Traffic, Errors, Latency, with Saturation added as a fourth axis pulled in from the resource-monitoring side (USE). Interviewers ask this mostly to see whether a candidate defaults to instrumenting everything they can measure versus starting from a deliberately small, well-chosen set that's actually actionable.
A dashboard is a fixed, saved arrangement of panels meant to be looked at repeatedly — a service overview you check every morning, an on-call landing page. It's built in advance and designed to answer known, recurring questions at a glance.
Explore is a free-form, ad-hoc query interface for digging into raw data one query at a time — no saved layout, no fixed panels, just a query box against a chosen data source with the ability to quickly switch between metrics and logs while preserving label filters. It's the tool you reach for while actively debugging something you don't have a pre-built dashboard for, since building and saving a dashboard for a one-off investigation would be wasted effort.
rate() and irate() are built to detect and correct counter resets — if a value drops between two samples, they assume the process restarted and the counter went back to zero, so they add the pre-reset value back in rather than reporting a negative rate.
A gauge is allowed to go down on its own (queue length dropping, memory being freed), and PromQL cannot tell the difference between "this gauge legitimately decreased" and "this counter reset." If you run rate() on a gauge, every normal decrease gets misread as a reset and silently added into the result, producing numbers that are too high and don't correspond to anything real.
The common wrong answer here is "rate() just calculates speed of change, so it works on any number that changes over time." It doesn't — rate() is defined specifically in terms of counter semantics. For a gauge, use deriv() if you want a per-second rate of change, or just graph the raw gauge.
Service discovery lets Prometheus find scrape targets automatically instead of you hand-listing every IP and port in prometheus.yml. That matters anywhere targets come and go — pods restart with new IPs, autoscaling adds and removes instances — because a static target list would go stale immediately.
In Kubernetes, Prometheus's kubernetes_sd_config queries the Kubernetes API for objects like Pods, Services, and Endpoints, and refreshes that list on an interval. It doesn't know on its own which of those it should actually scrape, though — that's handled by relabeling rules that read pod/service annotations (commonly prometheus.io/scrape: "true", prometheus.io/port, prometheus.io/path) and use them to decide the final target list. In most real clusters this whole flow is delegated to the Prometheus Operator's ServiceMonitor and PodMonitor custom resources, which express "scrape anything with this label" declaratively instead of hand-writing relabel configs.
Both live in Prometheus rule files and both run on a schedule, but they do different jobs.
A recording rule pre-computes an expensive or frequently-used PromQL expression and saves the result as a new time series. You use these to speed up dashboards that would otherwise re-run a heavy aggregation on every page load, and as a naming convention (level:metric:operation, e.g. job:http_errors:rate5m) that other rules and dashboards can build on cheaply.
An alerting rule evaluates a PromQL expression and, when it's true for at least the configured for duration, fires an alert that gets sent to Alertmanager. The for duration matters: without it, a single noisy spike that clears itself in the next scrape would still page someone.
A common real pattern is to define the underlying calculation once as a recording rule, then have the alerting rule threshold against that recorded series — so the dashboard and the alert are guaranteed to agree on what the number actually is.
It's an alert rule written to be permanently true — for example, vector(1) == 1 — routed to a receiver that expects a heartbeat and pages if the heartbeat stops.
The reason to want this at all: your alerting pipeline has multiple points of silent failure — Prometheus could crash, Alertmanager could lose its config, a network path to your paging tool could break — and in every one of those cases, the failure mode is that you simply stop getting paged, which looks identical to "everything is fine." A Watchdog alert converts "absence of a signal" into "presence of a different signal": if the heartbeat notification stops arriving, something in the alerting chain itself is broken, independent of whether your actual services are healthy.
This pairs naturally with using absent() on critical exporters, which alerts when a metric that should always exist has stopped being scraped at all — catching the case where an exporter or scrape target silently disappeared rather than reporting bad values.
Prometheus by itself is a single server with local disk storage, a fixed retention window, and no way to query across multiple Prometheus instances as one dataset. Thanos, Cortex, and Mimir all solve variations of the same three problems: long-term retention (shipping old data to cheap object storage like S3 instead of local disk), global query view (a Querier component that fans a query out across many Prometheus servers and the long-term store, then merges the result), and horizontal scalability for ingesting metrics beyond what one Prometheus process can handle.
Thanos specifically is designed to sit alongside existing Prometheus servers with minimal changes — a sidecar container uploads blocks to object storage and serves recent data, while separate Store Gateway and Querier components handle historical data and cross-cluster queries. Cortex and Mimir take a more centralized, remote-write-based ingestion approach instead of the sidecar model. Which one to pick usually comes down to whether you want to keep existing Prometheus servers as-is (favors Thanos) or centralize ingestion from the start (favors Mimir/Cortex).
A template variable is a placeholder — shown as a dropdown at the top of a dashboard — that gets substituted into every panel's query at render time, instead of the query hardcoding one specific value.
Without them, monitoring ten services means either building ten near-identical dashboards or manually editing a query every time you want to look at a different one. With a $service variable driving the query, one dashboard serves all ten: the viewer picks a value from the dropdown and every panel re-queries for that selection. Variables can be static lists, populated dynamically from a query against the data source itself (e.g. "list all distinct job label values"), or chained so one variable's options depend on another's selection (pick a cluster, then the service dropdown only shows services in that cluster).
The practical payoff is that a single well-built templated dashboard replaces an entire folder of copy-pasted, drifting ones — which is usually the actual point behind this question, not the mechanics of the dropdown.
Access in Grafana is layered. Organizations are the top-level tenant boundary — each one has its own dashboards, data sources, and users, useful when genuinely separate teams or customers shouldn't see each other's content at all. Within an organization, users get a role — Viewer, Editor, or Admin — that governs what they can do (view only, build and edit dashboards, or manage users and data sources). Newer Grafana versions add folder-level permissions and fine-grained RBAC on top of this, so you can grant edit access to one team's folder without giving them admin over the whole instance.
For authentication itself, Grafana doesn't require you to manage passwords directly — it integrates with LDAP, OAuth (Google, GitHub, generic OIDC), and SAML, which is the standard approach in any organization that already has centralized identity, since it avoids a second set of credentials to manage and revoke.
A transformation reshapes the result of a query after it comes back from the data source — renaming fields, joining two queries together, filtering rows, doing math between series — without touching the query itself or the data source.
This matters because not every data source's query language can do everything you need. If you're combining a Prometheus series with a CSV-sourced table, or you want to show the ratio of two independently-queried metrics as one panel, that join happens in Grafana's transformation pipeline, not in PromQL. The trade-off is that transformations run in the browser at render time on whatever data was returned, so they're the wrong tool for anything that needs to filter or aggregate at the source to keep query size manageable — for large datasets, push filtering into the query itself and reserve transformations for reshaping the smaller result set for display.
A Fluent Bit pipeline has four stages that a log record moves through in order.
kubernetes filter, redacting fields, or dropping records matching a pattern with grep.Records flow through these in the order they're declared in the config, tagged by a Tag that routing rules (Match) use to decide which filters and outputs apply to which inputs — this is what lets one Fluent Bit process handle several unrelated log sources with different processing rules in a single config file.
A parser turns a raw log line — usually a single unstructured string — into a structured record with named fields, which is what makes the log queryable and filterable downstream instead of being one opaque blob of text.
The JSON parser is the simple case: if the application already logs structured JSON, Fluent Bit just decodes it directly into fields with no pattern-matching required, and it's fast and unambiguous. The regex parser is for anything not already structured — a traditional combined or custom application log format — where you write a named-capture-group regex that maps parts of the line to field names, plus (usually) a time format string so Fluent Bit can pull the timestamp out as a proper time field rather than leaving it as a text substring.
The practical guidance: if you control the application, log JSON from the start and skip regex parsing entirely — it's more reliable, cheaper to process, and doesn't break silently the next time someone changes a log message's wording.
An index in OpenSearch is split into shards — each shard is a self-contained Lucene index holding a subset of the index's documents — so that a dataset too large or too fast for one machine can be spread across many. A replica is a full copy of a primary shard, kept on a different node, for both fault tolerance (the replica takes over if the node holding the primary fails) and read scalability (queries can be served from either copy).
A replica is never placed on the same node as its primary, because that would provide zero protection — losing the node loses both copies at once. On a single-node cluster, the default index settings still ask for one replica per shard, but there's nowhere else to put it, so those replica shards stay permanently unassigned. Cluster health reflects exactly that: green means every primary and every replica is assigned; yellow means every primary is assigned but some replicas aren't; red means at least one primary itself is unassigned (real data loss risk, not just redundancy risk). A single-node dev cluster sitting at yellow forever is expected behavior, not a misconfiguration — the fix, if you actually want green on a single node, is to explicitly set number_of_replicas: 0 for that index.
ISM (Index State Management in OpenSearch, ILM in the Elasticsearch lineage) automates moving an index through phases — typically hot, warm, cold, and delete — based on age or size, without a human running those actions manually.
Log data is the textbook use case because it's naturally time-partitioned and has a natural expiry: you almost never need to search six-month-old debug logs with the same performance as today's, and keeping every index around forever is how clusters silently accumulate thousands of small, mostly-idle shards, which drives up per-node JVM heap pressure and eventually destabilizes the cluster (see the over-sharding failure mode above). A typical policy rolls an index over once it hits a size or age threshold (so "today's logs" is always a fresh, appropriately-sized index rather than one index growing forever), then deletes indices past a retention window automatically.
PUT _ilm/policy/logs_rollover_policy{ "policy": { "phases": { "hot": { "actions": { "rollover": { "max_size": "50GB", "max_age": "1d" } } }, "delete": { "min_age": "7d", "actions": { "delete": {} } } } }}Term-level queries (term, terms, range) match exact values against a field without any text analysis — they're for structured data like an exact status code, a keyword field, or a numeric range, and the value you search for has to match precisely what's stored.
Full-text queries (match, match_phrase, query_string) run the search text through the same analyzer used when the field was indexed — lowercasing, tokenizing on word boundaries, sometimes stemming — before comparing, which is what lets a search for "running" match a document containing "ran" or "run" depending on the analyzer, and lets "Error Connecting" match a log line however it was cased or punctuated.
The common mistake this question is checking for: running a term query against a text field expecting it to behave like a full-text search, and getting zero results because the field was analyzed at index time (broken into lowercase tokens) but term compares against the raw, unanalyzed input. The fix is either querying the field's .keyword sub-field for exact matching, or switching to a match query if full-text behavior is actually what's wanted.
OpenTelemetry (OTel) is a vendor-neutral standard and set of SDKs/APIs for generating and exporting the three observability signal types — metrics, logs, and traces — from your application code, without tying that instrumentation to a specific backend.
It sits at a different layer than Prometheus or Fluent Bit rather than competing with them directly: OTel defines how signals are generated and shaped in your application and how they're transported (often via an OTel Collector, which can receive, process, and export to many backends), while Prometheus and OpenSearch are backends that store and query metrics and logs respectively, and Fluent Bit is a collector focused specifically on log shipping. In practice, the OTel Collector can scrape Prometheus-format metrics as an input and export them to Mimir or Prometheus remote-write as an output, or receive OTLP-format traces and logs and forward them to whatever backend you use — the appeal is instrumenting an application once, in a standard format, and being able to swap backends later without re-instrumenting.
The distinction worth naming clearly: OTel doesn't replace Prometheus or OpenSearch as storage, and it doesn't replace Fluent Bit as a node-level log collector — it's a common instrumentation and transport layer that can feed all of them.
Both are frameworks for deciding what to actually monitor, aimed at two different layers of a system.
RED (Rate, Errors, Duration) is for request-driven services: how many requests per second is this handling, what fraction are failing, and how long are they taking. It's the natural fit for an API, a microservice, or anything you'd describe in terms of requests coming in and responses going out — these three signals catch the vast majority of user-facing problems in a service.
USE (Utilization, Saturation, Errors) is for resources: CPU, memory, disk, network. Utilization is how busy the resource is, saturation is how much work is queued waiting for it, and errors are, well, errors. This is the right lens for infrastructure and hardware-level components that don't have a "request" concept in the same way a service does.
A well-instrumented system typically uses both at their appropriate layer: RED dashboards for services, USE dashboards for the nodes and infrastructure underneath them, rather than trying to force one framework to cover everything.
Unstructured logging is free-text: 2026-01-15 ERROR Failed to connect to database after 3 retries. It's easy for a human to read in isolation but hard for a machine to reliably parse, because extracting the retry count or the error type requires regex that breaks the moment someone rewords the message.
Structured logging emits the same information as key-value fields, typically JSON: {"level": "error", "msg": "db connection failed", "retries": 3, "service": "orders"}. Every field is queryable and filterable directly by a log backend like OpenSearch without any parsing step, aggregations work correctly (you can sum retries across all events, which you can't reliably do against a free-text field), and it survives message wording changes without breaking downstream dashboards or alerts built on those fields.
The practical cost is on the collection side: a structured-JSON log can be ingested directly by Fluent Bit's JSON parser with no pattern matching, while an unstructured log needs a regex parser that has to be kept in sync with the application's log format — the moment a developer tweaks a log message's wording, that regex silently stops matching and the pipeline starts losing fields (or entire records) without any error. This is the concrete reason "log in JSON from day one" is standard advice rather than a nice-to-have.
"OK" means the alert rule's query returned data and that data didn't meet the alert condition — the thing being monitored is actively confirmed healthy. "No Data" means the query returned nothing at all to evaluate, which is a meaningfully different and often more concerning situation: it could mean the metric genuinely doesn't exist (a typo in the query, a data source that isn't scraping), or it could mean the target the metric comes from has disappeared entirely — a crashed exporter, a dead service, a broken network path to the data source.
The trap this question is checking for is treating "No Data" and "OK" as roughly equivalent ("no alert fired, so we're fine"). A silently-broken metrics pipeline produces "No Data," not a firing alert, so if "No Data" isn't itself configured to notify someone, an entire class of failure — the monitoring itself going dark — passes by unnoticed. Grafana lets you explicitly configure how a rule should behave on "No Data" (treat as OK, treat as Alerting, or keep the last state), and the safe default for anything critical is to treat it as an alerting condition, not a clean pass.
Most exporters are whitebox: node_exporter or an application's own /metrics endpoint reports on internal state — CPU load, queue depth, internal counters — from inside the system being monitored. The Blackbox Exporter does the opposite: it sits outside the target and actively probes it over HTTP(S), TCP, DNS, or ICMP, then reports the result (did it respond, how long did it take, was the certificate valid, did the response match an expected pattern) as Prometheus metrics like probe_success and probe_duration_seconds.
The distinction matters because whitebox and blackbox monitoring catch different failure classes. A service can be internally "healthy" by every whitebox metric — CPU fine, no error logs — while still being unreachable from outside due to a firewall rule, a DNS misconfiguration, or a load balancer misroute; only an external probe catches that. Conversely, blackbox monitoring alone can't tell you why something is failing, just that it is — you still need whitebox metrics and logs to diagnose the root cause once a blackbox probe flags an outage.
A specific trap worth naming: probe_success (did this specific probe succeed) is not the same signal as the standard up metric (is Prometheus able to scrape this target's /metrics endpoint) — a target can be up for scraping purposes while the actual service being probed through it is down, since Blackbox Exporter itself is what Prometheus scrapes, not the target directly. Configuring alerts against the wrong one of these two metrics is a common early mistake.
Provisioning means defining dashboards, data sources, alert rules, and notification policies as version-controlled files (JSON or YAML) that Grafana loads automatically on startup, rather than an engineer clicking through the UI and hitting Save.
The case for it mirrors infrastructure-as-code generally: a hand-built dashboard exists only in one Grafana instance's database, with no history of who changed what threshold or why, no code review before a change ships, and no reliable way to recreate it if that instance is lost or you need the same dashboard in a second environment. Provisioned dashboards live in a git repository like any other config, get reviewed in a pull request, and can be deployed identically to dev, staging, and production Grafana instances — which also means the dashboard a developer tests against locally is guaranteed to match what's running in production, rather than drifting the moment someone tweaks a panel directly on the live instance.
The practical trade-off: provisioned dashboards are typically read-only in the UI (edits made there don't persist, or get flagged as diverging from the source file), which is a deliberate choice to keep the file the single source of truth — teams that want to prototype visually first, then export the finished JSON into the provisioning repo once it's right, use the UI as a draft tool rather than the system of record.
It's not a bug — it's extrapolation, and it's part of how increase() and rate() are defined.
Prometheus only has samples at your scrape interval (say every 15s), and those samples rarely land exactly on the start and end of your requested time range. So instead of just subtracting the two closest raw values, Prometheus takes the increase observed between the first and last sample inside the window and extrapolates it slightly outward to cover the full requested range, based on the average interval between samples. That extrapolation is what produces a non-integer result even though the underlying counter only ever moves in whole numbers.
There are two safety limits on this: extrapolation never projects further than half an average sample interval past the edge of the data, and it never extrapolates a counter below zero (since counters start at zero and can't go negative). If you need an exact integer total rather than an estimate, you generally don't get one from increase() on a counter with any realistic scrape gaps — that's a trade-off of the whole approach, not a misconfiguration to fix.
The trap version of this question is being asked to explain why a dashboard shows "142.7 requests" and treating it as an application bug. It's the query function working as designed.
Relabeling rewrites, filters, or drops labels using regex rules, and it runs at two different points in the scrape pipeline with very different consequences.
relabel_configs runs before the scrape happens, against the metadata service discovery produced (pod labels, annotations, addresses). You use it to decide which targets get scraped at all and to set things like the scrape port or path from an annotation. If a rule here drops a target, Prometheus never even connects to it — no load is generated.
metric_relabel_configs runs after the scrape, against the actual metric samples returned. You use it to drop unwanted metrics or labels, or rename messy label names, after the exporter has already been hit. This is the tool for controlling cardinality on data you don't control the source of — but the target was still scraped, so it doesn't save any scrape load, only storage and query cost.
The trap: people reach for metric_relabel_configs to "stop scraping" a noisy exporter and are confused when scrape duration and load don't change — that requires relabel_configs on the target set itself, not the post-scrape metrics.
Cardinality is the number of unique time series a metric produces — every distinct combination of label values creates a separate series. http_requests_total{method="GET", path="/users"} and http_requests_total{method="POST", path="/users"} are two series from one metric name.
The problem is that Prometheus keeps an in-memory index of every active series so it can answer queries fast, and that index's memory footprint scales with the number of series, not the amount of underlying data. A label that takes unbounded values — a user ID, a raw URL path with IDs embedded, a request ID, a full email address — can turn one metric into millions of series. That shows up as ballooning memory usage, slow queries, and in bad cases an out-of-memory crash of the Prometheus server itself, all from a single badly-labeled metric.
Common fixes: never put unbounded values in labels (aggregate the path template, not the resolved URL, for instance), use metric_relabel_configs to drop or hash offending labels before they're stored, and if you're running Thanos/Mimir/Cortex, apply cardinality limits or drop rules at the ingest layer where the majority of cost overruns actually originate.
Vanilla Prometheus has no built-in clustering or replication — each server scrapes and stores independently. The standard HA pattern is to run two (or more) identical Prometheus instances scraping the same targets with the same config, so either one alone can answer queries and fire alerts if the other is down. This gets you availability but not deduplication: Alertmanager has to dedupe identical alerts coming from both replicas, which it does natively via its own clustering (gossip protocol) as long as both Prometheus instances are configured to point at the same Alertmanager cluster.
What plain replication doesn't solve is long-term storage, global querying across many Prometheus servers, and downsampling for older data — for that you add Thanos, Cortex, or Mimir in front of or alongside Prometheus. These typically work by having a sidecar or remote-write path ship blocks to object storage (S3/GCS), and a separate Querier component fan out queries across all Prometheus replicas and the long-term store to present one global view.
The trap: describing "HA Prometheus" as if it were one clustered system with shared storage. It's stateless duplication at the collection layer, with a separate system bolted on for anything that needs a unified, long-retention view.
There are three distinct tools here and they solve different parts of the problem, so the interviewer is usually checking you don't conflate them.
Grouping batches multiple firing alerts that share labels (say alertname, cluster, service) into a single notification instead of one message per alert. If ten pods all trip PodNotReady at once, grouping turns that into one Slack message listing ten pods rather than ten separate pages.
Inhibition suppresses a lower-priority alert when a related, higher-priority alert is already active, based on a source/target label match you define. The classic example: if DatabaseDown is firing, you don't also need fifty APIErrors alerts from every service that depends on that database — inhibition rules stop the noisy downstream symptoms from paging separately.
Silencing is manual and time-boxed: you tell Alertmanager to mute alerts matching a label set for a defined window, typically during planned maintenance, so expected noise doesn't page anyone.
In practice, fixing a 50-page incident usually means: group_by includes enough shared labels to collapse duplicates, an inhibition rule ties the leaf symptoms to their root cause, and going forward, each alert added to the system should be reviewed for whether it's actually a distinct actionable signal or just noise from something upstream. A quarterly audit of alerts that never fire, or always fire in bulk, is standard operational hygiene for keeping this from recurring.
route: group_by: ['alertname', 'cluster', 'service']inhibit_rules: - source_match: alertname: 'DatabaseDown' target_match: alertname: 'APIErrors' equal: ['cluster']Grafana's unified alerting (v2) lets an alert rule query across a label dimension and evaluate the same condition per series, so you write one rule — not fifty — and it fans out to one alert instance per matching label set (per service, per pod, whatever you group by). That alone reduces maintenance overhead, but it doesn't by itself reduce paging volume if fifty of those instances trip at once.
The actual noise-reduction techniques mirror what you'd do in Alertmanager: notification policies group related firing alerts into a single message using shared labels, mute timings suppress notifications during known maintenance windows, and you route by severity so only genuinely critical instances page immediately while warnings land in a dashboard or low-priority channel. For a fleet of similar services specifically, the higher-leverage move is often to alert on an aggregate — "more than 20% of the fleet is unhealthy" — rather than firing individually per instance, since one service blipping is noise but a fifth of the fleet failing is a real incident.
The deeper principle interviewers are checking for here is symptom-based, SLO-aligned alerting over static per-instance thresholds — alerting on the burn rate of an error budget catches real user impact, while alerting on every CPU spike on every box does not.
By default, a log collector treats every line as one record, which shreds a stack trace into dozens of unrelated log entries — the exception message becomes one record and every subsequent frame becomes its own separate, contextless record. Fluent Bit's multiline parsing solves this by defining a start pattern (typically a regex matching a timestamp or log-level prefix, since that's what marks the beginning of a new log entry) and treating every following line that doesn't match that start pattern as a continuation of the previous one, concatenating them into a single record.
There are two places this can be configured, and the choice matters: doing it in the Tail input plugin itself is the recommended approach when reading from a log file, because concatenation happens as lines are read, which is more efficient. The separate multiline filter is for cases where you can't do it at the input (e.g., records already arrived via a different input plugin) — but it comes with real constraints: it isn't affected by buffer_max_size, so a runaway match can grow a record indefinity, it uses an internal re-emitter, and you can't define two multiline filters that match the same tag or you'll cause an infinite loop in the pipeline. For that reason, if you need to try multiple multiline patterns against one input, you configure a single filter with a comma-separated list of parsers, not multiple filter blocks.
[INPUT] Name tail Path /var/log/app/*.log multiline.parser javaFluent Bit buffers incoming records (in memory, on disk, or both, depending on the storage.type setting) while waiting for an output to accept them. If the output — say OpenSearch under load, or a network blip to a remote endpoint — falls behind, that buffer keeps growing. Left unchecked, that eventually exhausts memory or disk and can crash the process or, worse, silently drop records.
Fluent Bit's answer is to pause the affected input once its buffer crosses a configured limit (Mem_Buf_Limit for memory-buffered inputs), rather than let it grow unbounded. Once paused, the engine stops accepting new data from that specific input plugin — it's the plugin's own responsibility to decide what to do while paused, and for HTTP-based inputs like http, elasticsearch, or opentelemetry, that means the listener itself stops accepting new connections until the buffer drains back under the limit and a resume callback fires.
The practical levers: use filesystem-backed buffering (storage.type filesystem) for inputs where you can't afford to lose data during an output outage, since it survives a Fluent Bit restart that memory buffering doesn't; set Mem_Buf_Limit deliberately rather than leaving it at a default that doesn't match your actual traffic; and size retries and flush intervals on the output side so a transient outage doesn't immediately trigger backpressure on the input side.
A DaemonSet runs one instance of the log collector (e.g. Fluent Bit) per node, reading every container's log files from the node's filesystem directly. This is the default and by far the most common pattern for general-purpose cluster logging: one collector process per node handles every pod on that node, which is far cheaper in aggregate resource usage than one collector per pod, and it keeps working even if the application container itself crashes, since it's reading from the node's log files rather than talking to the app process.
A sidecar runs a collector container inside the same pod as the application, typically because the application can't or doesn't write to stdout/stderr in a way the node-level collector can pick up — it writes to a file inside its own container filesystem, or emits logs in a format that needs per-application-specific parsing that would be awkward to centralize. The sidecar tails that file and forwards it, at the cost of one extra container per pod rather than per node.
The common wrong answer is defaulting to sidecars "for isolation" — for standard stdout/stderr logging, that isolation buys nothing and multiplies resource cost by however many pods you're running, so DaemonSet is the right default and sidecar is the exception for a specific per-application constraint.
Start with the cluster health API against a specific index, not just the cluster as a whole, since "yellow" only tells you something has an unassigned replica, not which index or why.
GET _cluster/health?level=indicesFrom there, common root causes and how to tell them apart: not enough nodes for the replica count — if an index wants one replica per shard but you only have as many data nodes as primary shards with no room for a distinct copy, the replica can't be placed; the fix is either add a node or lower number_of_replicas. Disk watermark thresholds — OpenSearch stops allocating new shards to a node once it crosses a configured disk-usage percentage, which shows up as replicas stuck unassigned even though there's technically a free node; check _cat/allocation and either free disk space or adjust the watermark. Shard allocation awareness / zone constraints — if you've configured awareness attributes (e.g. spreading replicas across availability zones) but don't have nodes in enough zones, replicas can't satisfy the constraint and stay unassigned. Over-sharding — too many small shards per node drives up JVM heap pressure from shard metadata overhead alone, which can cause allocation to fail or the cluster to become unstable well before you'd expect from data volume; the standard mitigation here is Index Lifecycle Management, rolling over to a new index by size or age and deleting old ones on a retention schedule rather than letting shard count grow unbounded.
The trap is assuming yellow always means "something is broken" — on a single node it's expected, and even on a healthy multi-node cluster it can be a transient, self-resolving state during a rolling restart or node replacement, not an incident on its own.
There's no single correct number — it's a trade-off, and the honest answer names the trade-off rather than a formula.
Too few shards for the data volume means each shard grows very large, which slows queries against it and makes rebalancing (moving that shard to another node) slow and expensive when it eventually needs to happen. Too many shards, especially many small ones, means the overhead — each shard carries its own Lucene index structures and metadata held in JVM heap — starts to dominate: a cluster with thousands of tiny shards can run out of usable heap and become unstable well before actual data volume would justify it, since shard count costs memory independent of shard size.
The practical guidance most teams converge on: aim for shard sizes in roughly the tens of gigabytes range (commonly cited as 10–50GB) rather than picking a shard count up front, keep JVM heap pressure from shard overhead under about 75% as a safety margin, and for continuously-growing data like logs, don't try to pick the final number at index-creation time at all — use rollover (via ILM/ISM) to create a new index once the current one hits a size threshold, so shard sizing becomes self-correcting instead of a one-time guess that ages badly as volume grows.
The trap is answering with a fixed number like "5 shards" as if it's a rule — the correct framing is "it depends on expected index size and node count, and for time-series data you avoid the question by rolling over instead of guessing."
An SLO (Service Level Objective) is a target for how reliable a service should be over a window — "99.9% of requests succeed over a rolling 30 days" — derived from an SLI (the actual measured indicator, like success rate) and backed by an error budget: the amount of unreliability you're allowed before breaching the objective (0.1% of requests, in that example).
A static threshold alert ("page if error rate > 5% for 5 minutes") has two failure modes: it can miss a slow, sustained low-grade problem that never crosses 5% but steadily eats the entire monthly error budget over three weeks, and it can also page unnecessarily for a brief spike that, in the context of the full 30-day window, barely dents the budget at all.
Burn-rate alerting instead asks "at the current rate of failure, how fast is the error budget being consumed?" — a burn rate of 1x means the budget is being used exactly on pace to be fully spent right at the end of the window (fine); a burn rate of 14x means the entire month's budget will be gone in about two hours if this continues (page immediately). The standard implementation, multi-window multi-burn-rate (MWMBR) alerting from Google's SRE practice, checks burn rate over two windows at once — a short one (catches fast, severe burns) and a long one (avoids paging on a brief blip that self-corrects) — before firing, which is what makes it both fast to catch real incidents and resistant to noise from short-lived spikes.
The trap in this question is describing burn-rate alerting as "just a different threshold" — the conceptual shift is alerting on rate of budget consumption relative to a time window, not on an absolute value in isolation.
Log sampling means only keeping a subset of log records that match some criteria, rather than every record generated, typically to control storage and ingestion cost on very high-volume, low-information log types.
The case for it: a busy service can generate log volume that's mostly repetitive and low-value — the same successful health-check line thousands of times a minute — and indexing all of it in OpenSearch or shipping all of it to a paid logging backend costs real money and storage for information you'll essentially never query. Sampling strategies range from simple (keep 1 in every N successful requests, keep 100% of errors) to more deliberate tail-based sampling in tracing contexts (decide whether to keep a trace based on its outcome — an error or unusually slow request — after it completes, rather than sampling uniformly up front).
The trade-off, and the reason this is a hard question rather than an easy one: sampling is a bet that what you're dropping was actually low-value, and you find out you're wrong exactly when you need the dropped data most — debugging an intermittent issue that only shows up in the 99% of successful requests you didn't keep. The safer version of this pattern keeps all error and warning-level logs unconditionally and only samples verbose/debug-level or routine-success logs, so the sampling risk is concentrated on the data least likely to matter during an incident.
Log volume is simply how many bytes or records you're ingesting — the obvious cost driver, and the one most teams think of first, controlled with sampling, filtering, and retention.
Log cardinality is a subtler cost driver that behaves differently: it's the number of distinct label or field-value combinations attached to your logs, and it matters most in systems (like Grafana Loki, or index-mapping-heavy OpenSearch setups) where the storage or indexing structure scales with the number of distinct combinations, not just total bytes. Putting a user ID, a request ID, or a raw unbounded URL into an indexed label rather than into the log line's body can multiply the number of distinct index entries even if total log volume didn't change at all — this is structurally the same failure mode as Prometheus metric-label cardinality blowing up a metrics index, just applied to logs.
The practical fix mirrors the metrics-side fix: keep genuinely high-cardinality fields (user IDs, request IDs, raw paths) in the log body where they can be searched via full-text query, and reserve indexed labels/fields for genuinely low-cardinality dimensions (service name, environment, log level, region) that you actually want to filter and aggregate by. Conflating "we have a lot of logs" (volume) with "our labels have too many distinct values" (cardinality) leads to the wrong fix — sampling won't help a cardinality problem, and stricter label discipline won't help a pure volume problem.
This is a correlation-debugging question, and a good answer walks through several concrete possibilities rather than picking one.
Clock or timezone mismatch — the metrics data source and the log data source may be using different time bases (UTC vs local, or a scrape-time vs event-time discrepancy), so "the same time window" in the dashboard isn't actually the same wall-clock window in both systems. Always check the underlying timestamps, not just what the dashboard UI shows.
The spike is infrastructure-level, not application-level — a CPU or network saturation spike can happen with zero corresponding application log lines, because nothing in the application's code path logged anything; the problem is at a layer below what the app instruments. This is exactly why RED (application-level) and USE (resource-level) signals are tracked separately — one can spike without the other.
Log sampling or filtering dropped the relevant records — if the pipeline samples routine/successful logs, and the spike corresponds to a burst of otherwise-successful requests (a traffic spike, not an error spike), the records that would explain it may have been sampled out before storage.
The metric itself is noisy or mislabeled — a cardinality issue, a bad relabeling rule, or a metric that doesn't mean what its name implies can produce a spike that isn't a real event at all; verify what's actually generating the metric before assuming the underlying system did something.
The answer that's usually wrong: assuming the logs are complete and concluding "nothing happened" — the correct instinct is to question whether the two data sources are actually looking at the same thing, in the same window, with nothing filtered out, before concluding there's no correlation.
OpenSearch (like Elasticsearch) indexes the full content of every log line by default — it builds an inverted index over the text itself, which is what makes arbitrary full-text search across the entire log body fast, but that indexing work and the resulting index size scale with total log volume and is a major driver of both storage cost and cluster resource usage.
Loki takes a different approach: it only indexes a small set of labels (service, environment, level — deliberately low-cardinality metadata), not the log content itself, and stores the actual log lines as compressed chunks in cheap object storage. Querying Loki means first narrowing down to the right chunks via label matching (fast, since the label index is small), then grep-style scanning the log content within those chunks at query time rather than at index time. This makes ingestion cheap and storage cheap, at the cost of query-time full-text search being comparatively slower than OpenSearch's pre-built index, especially over very broad label selections.
The trade-off in one line: OpenSearch pays indexing cost up front for fast arbitrary search later; Loki defers that cost to query time in exchange for far cheaper ingestion and storage. Which one is the right fit depends heavily on your actual query pattern — if you frequently need broad free-text search across huge volumes, OpenSearch's model earns its cost; if most queries are already scoped by service/environment/time and content search is secondary, Loki's model is dramatically cheaper for the same log volume.
Federation is a Prometheus server scraping a filtered subset of time series from another Prometheus server's /federate endpoint, treating that remote server as just another scrape target. It's still pull-based scraping under the hood — it doesn't add clustering, shared storage, or automatic sharding, it just lets you compose a hierarchy of Prometheus servers.
The pattern it's genuinely good for is hierarchical aggregation: many per-cluster or per-datacenter Prometheus servers each scrape detailed, instance-level metrics locally, and a higher-level "global" Prometheus federates only the already-aggregated, job-level series from each of them — enough to build a single cross-cluster dashboard or a global alert, without ever pulling the full-cardinality local data into one place. A second valid pattern, cross-service federation, is one team's Prometheus pulling a specific handful of series from another team's Prometheus for a joint alert or sanity check.
Where it breaks down is being used as a substitute for actual long-term, multi-cluster storage: federation only exposes what you explicitly filter for at scrape time, it inherits the local server's retention window, and pulling large or high-cardinality metric sets through federation just recreates the cardinality and scale problems you were trying to solve, one hop later. For genuine long-term retention and a real global query view across many Prometheus servers, remote_write into Thanos, Cortex, or Mimir is the current standard approach — federation has narrowed to a single legitimate niche (a small, deliberately filtered set of aggregated series for one global view) rather than a general scaling mechanism.
The trap: describing federation as "how Prometheus scales," full stop. It scales specific, pre-aggregated queries across a hierarchy; it does not scale storage or cardinality.
A liveness probe answers "is this container in a state where it should be killed and restarted?" — if it fails repeatedly, Kubernetes kills the pod and starts a fresh one. A readiness probe answers a different question: "should this pod currently receive traffic?" — if it fails, Kubernetes removes the pod from the Service's load-balanced endpoints, but does not restart the container; the pod keeps running and can rejoin once readiness passes again.
The common, expensive mistake is wiring the same check to both, especially a check that depends on an external dependency (a database connection, a downstream API). If that dependency has a brief outage, a shared check makes both probes fail: Kubernetes kills and restarts every pod that depends on it (via liveness) at the exact moment they're already struggling, which can turn a transient downstream blip into a full restart storm across your fleet — the opposite of what you want, since restarting doesn't fix a problem that's external to the pod, it just adds a slow, disruptive recovery on top of it.
The correct split: liveness should only fail for problems the container itself can't recover from without a restart (a deadlocked process, a broken internal state) and generally shouldn't call out to external dependencies at all. Readiness should reflect actual serving capability, including dependency health, so traffic gets pulled away from a struggling pod without killing it — letting it recover on its own once the dependency comes back, without a disruptive restart in the middle.
This connects directly to monitoring because liveness/readiness flapping shows up as a very specific, recognizable signature in metrics and logs — restart counts climbing in lockstep with a downstream outage, rather than with anything actually wrong in the pod itself — and being able to name that pattern is often exactly what this question is testing.
Correlation means being able to jump directly from one signal to another for the same event — from a metric spike straight to the specific logs and the specific distributed trace that explain it — instead of manually guessing a time window and hoping you land on the right entries in a completely separate system.
For this to actually work, a few things have to be true that don't happen automatically. First, every signal needs a shared identifier that survives the trip across systems — typically a trace_id generated at the edge of a request and propagated through every service call, attached to the spans in your tracing backend, and also injected into the structured logs each service emits while handling that request. Without deliberately propagating that ID through log statements, your logs and your traces are just two systems that happen to cover the same time period, not two views of the same event — and "same time period" breaks down under load, when hundreds of unrelated requests are in flight at once and a time-window guess returns noise.
Second, clocks and time bases need to be consistent — NTP-synced across hosts, and ideally all signals timestamped in UTC — since a correlation UI that lines up a metric spike with logs from a clock that's even a few seconds off will show you the wrong window's logs and lead you to the wrong conclusion.
This is precisely the problem OpenTelemetry's semantic conventions and context propagation exist to standardize: instrumenting metrics, logs, and traces so that a trace_id (and related context) flows consistently through all three, which is what makes tools like Grafana's Explore (pivoting from a Tempo trace straight into the matching Loki logs) actually work, rather than being three separately-useful tools you happen to have open in adjacent tabs.