Databases, Caching and Message Queues Interview Questions

ACID is the set of four guarantees that make a database transaction safe: Atomicity, Consistency, Isolation and Durability.

• Atomicity: a transaction either fully happens or not at all. If a transfer debits one account and crashes before crediting the other, the debit is rolled back.
• Consistency: a transaction moves the database from one valid state to another, so rules such as foreign keys and CHECK constraints are never left broken.
• Isolation: concurrent transactions do not see each other's half-finished work. How strictly depends on the isolation level you choose.
• Durability: once COMMIT returns, the data survives a crash, because it was written to the write-ahead log on disk first.

It matters because without these guarantees every application would have to handle partial writes and race conditions on its own. The common trap is treating isolation as all or nothing. It is a dial, and most databases default to a level weaker than fully serializable, so some anomalies are still possible unless you ask for stronger isolation.

An index is a separate, sorted data structure that lets the database find rows without reading the whole table. Think of the index at the back of a book: you jump straight to the right pages instead of reading every page.

In PostgreSQL the default index type is a B-tree. It is good at equality (WHERE id = 42), ranges (WHERE created_at > ...) and sorting. Without a usable index the database does a sequential scan and reads every row.

Indexes are not free. Each one takes disk space and must be updated on every INSERT, UPDATE and DELETE, so too many indexes slow down writes. The common wrong answer is "index every column". Index the columns that appear in WHERE, JOIN and ORDER BY clauses of queries that actually run often, and check with EXPLAIN that the index is used.

Choose a relational database when your data has clear structure, relationships between entities, and you need transactions and flexible queries such as joins. Choose a NoSQL database when a specific access pattern or scale requirement fits its model better.

The common trap is the claim that "NoSQL is faster". It is only faster for the access pattern it was designed for. A well-indexed PostgreSQL with JSONB columns handles many document-style workloads, so default to relational unless you can name the specific problem a NoSQL store solves.

Replication continuously copies data from a primary database to one or more replicas, so the same data exists on several servers. A read replica is a replica that serves read-only queries.

It gives you three things. First, read scaling: reports and search pages can query replicas instead of the primary. Second, high availability: if the primary dies, a replica can be promoted. Third, a place to run backups or heavy analytics without hurting production writes.

Replicas do not scale writes, because every write still goes through the primary. They also lag slightly behind, so a user can save a profile and then read an old version from a replica. This is called a read-your-writes problem. The usual fix is to send a user's reads to the primary for a short time after they write.

A connection pool keeps a set of open database connections and hands them out to requests, instead of opening a new connection for every request. Opening a connection is slow, and each PostgreSQL connection is a separate server process that uses real memory.

Without pooling, a traffic spike creates thousands of connections and the database spends its time managing them instead of running queries. With pooling, many requests share a small number of connections.

Pools exist in two places: inside the application (for example HikariCP or SQLAlchemy's pool) and as a separate proxy (for example PgBouncer). The common trap is forgetting to multiply. A pool size of 20 across 50 application instances is 1000 connections to the database, which is usually far more than it can handle well.

A replica copies everything, including mistakes. If someone runs DROP TABLE orders or a bug corrupts rows, the replica applies the same change seconds later, so now both copies are wrong.

A backup is a copy from a point in time that does not change afterwards. It lets you go back to "yesterday at 3 PM, before the bad deploy". Replication protects you from a server failing. Backups protect you from data being wrong.

For PostgreSQL, the strong setup is a physical base backup plus archived write-ahead log files, which allows point-in-time recovery. And an untested backup is only a hope: restore it to a scratch server on a schedule and check that the data is really there.

EXPLAIN shows the plan the database intends to use for a query: which tables it scans, which indexes it uses and in what order it joins. EXPLAIN ANALYZE actually runs the query and adds the real timings and real row counts next to the estimates.

When reading a plan, look for three things: a Seq Scan on a large table where you expected an index, a big gap between estimated and actual rows (stale statistics), and which node takes most of the time.

The trap is that EXPLAIN ANALYZE really executes the statement. On an UPDATE or DELETE that means it changes data. Wrap it so you can undo it:

BUFFERS adds how many pages came from cache and how many from disk, which is often the real reason a query is slow.

Further reading: PostgreSQL: Using EXPLAIN

The N+1 problem is when code runs one query to fetch a list and then one more query for each item in that list. Fetching 100 orders and then looking up each customer separately is 1 + 100 = 101 queries instead of 1 or 2.

Each query is fast on its own, so nothing looks slow in isolation. But the round trips add up, and under load they crowd out the database. It often appears right after a deploy that changed an ORM relation, with database CPU climbing and no single slow query to blame.

Fix it by fetching related rows together, using a JOIN, a batched WHERE id IN (...) query, or the ORM's eager loading feature. To detect it, look at call counts in pg_stat_statements: a cheap query with an enormous calls number is the signature.

RPO (Recovery Point Objective) is how much data you can afford to lose, measured in time. RTO (Recovery Time Objective) is how long you can afford to be down before service is restored.

If the RPO is 5 minutes, you need backups or replication that never fall more than 5 minutes behind. If the RTO is 2 minutes, restoring from a backup (which may take hours for a large database) is not an option, so you need a ready standby and automated failover.

The design follows from the numbers:

Tighter targets cost more, so the right answer is to get the business to state the numbers first.

Caching means keeping a copy of data that is slow or expensive to produce in a faster place, so the next request can reuse it. Teams use it to cut latency, reduce load on the database, and lower cost.

A request can pass through several cache layers, and each one catches requests before they reach the next.

The measure that tells you whether a cache works is the hit ratio: hits divided by hits plus misses. A low ratio means the cache is not helping, often because keys are too specific, TTLs are too short, or memory is too small and keys get evicted.

Do not cache when the data must always be exactly fresh (account balances at checkout), when the query is already cheap, or when almost every request is unique. The price of a cache is stale data and one more system to run, so add it for a measured problem, not by habit.

In cache-aside (also called lazy loading) the application talks to both the cache and the database. It checks the cache first, and only on a miss does it read the database and then store the result in the cache.

The diagram shows the read path. On writes, the application updates the database and then deletes (invalidates) the cached key, so the next read reloads fresh data.

It is the most common pattern because it is simple and the cache only holds data that was actually requested. The weakness is that the first request after a miss is slow, and the cache and database can briefly disagree. The common wrong answer is to update the cache on writes instead of deleting the key, which causes stale-value races (covered in the invalidation question).

TTL (time to live) is how long a key stays in the cache before it expires automatically. It is your safety net: even if you forget to invalidate a key, it will not stay stale forever.

Choose the TTL from how stale the business can tolerate the data being. A product description can be cached for hours. A stock count on a flash sale page might be cached for a few seconds. Data that must never be stale should usually not be cached, or should be invalidated explicitly on every change.

Two practical rules help. Add random jitter (for example 300 seconds plus or minus 10 percent) so many keys do not all expire at the same moment. And remember that a TTL is a maximum staleness, not a refresh schedule, because the key is only reloaded when someone asks for it after expiry.

Both are in-memory key-value stores used for caching, but Redis does much more.

Choose Redis when you need more than plain caching: counters, rate limiting, leaderboards, queues, locks, sessions, or data that should survive a restart. Choose Memcached when you only need a simple, large, multi-threaded cache of small strings and want the simplest possible thing to run.

In practice most teams pick Redis (or a compatible fork) because one tool covers both caching and the extras. The trap is to say Memcached is "faster". For typical cache workloads both are fast enough that the network round trip dominates.

Redis stores values in several data types, and picking the right one often replaces application code.

Two details are worth knowing. Operations on a single key are atomic, so INCR is safe to call from many clients at once with no race. And large collections are a risk: a hash or list with millions of entries is a "big key", which makes some commands slow and blocks other clients.

A message queue lets one part of a system hand work to another without waiting for it. The sender puts a message on the queue and moves on, and a consumer processes it when it can.

It solves four problems:

• Decoupling: the order service does not need the email service to be up at the moment an order is placed.
• Absorbing spikes: a sudden burst of requests piles up in the queue instead of crashing the workers.
• Independent scaling: add more consumers when the backlog grows.
• Retries: a failed message can be tried again later without the user noticing.

The costs are real. The system becomes eventually consistent, because the work happens after the request returns. You must handle duplicates and failures, monitor queue depth, and debug across several services.

Do not use a queue when the caller needs the answer right now (checking a password), or when a simple function call would do. A queue is for work that can happen a little later.

They differ in who receives a message and what happens to it afterwards.

Use a work queue to share tasks among workers, such as resizing images. Use pub/sub to broadcast an event to several independent services. Use a log when you want several consumers to read the same history at their own pace, or to replay old events after fixing a bug.

A common wrong answer is to call Redis pub/sub a queue. It is fire and forget: if a subscriber is offline when a message is published, it never gets it.

A topic is a named stream of messages. A topic is split into partitions, and each partition is an ordered, append-only log. Each message in a partition has a number called an offset. Brokers are the servers that store the partitions.

Producers write to topics. A message with a key always goes to the same partition (the key is hashed), so all messages for one customer stay in order. Consumers read messages and track their position by committing offsets.

A consumer group is a set of consumers that share the work of one topic. Kafka gives each partition to exactly one consumer in the group, so a group reads each message once, and different groups each read everything independently.

The diagram shows the key rule: parallelism is limited by partitions. With 3 partitions, a fourth consumer in the same group has nothing to read. Order is guaranteed only inside a partition, not across the whole topic.

PostgreSQL accepts the four standard names but implements three behaviours, and the default is Read Committed.

Under Read Committed, two SELECTs in the same transaction can return different results if another transaction commits in between. Repeatable Read and Serializable fix that, but they can fail with a serialization error (could not serialize access), and the application must be ready to retry the whole transaction.

The common wrong answer is that PostgreSQL allows dirty reads at Read Uncommitted. It never does. Pick the lowest level that is correct for the operation, and always add retry logic when you move up.

Further reading: PostgreSQL: Transaction Isolation

MVCC (Multi-Version Concurrency Control) lets readers and writers work at the same time without blocking each other, by keeping several versions of a row. Readers see the version that was valid for their snapshot, and writers create a new version.

In PostgreSQL an UPDATE does not change a row in place. It writes a new row version and marks the old one as no longer current. A DELETE only marks the row as dead. Those dead versions must stay until no running transaction could still need them.

That is why VACUUM exists: it reclaims space from dead versions. The practical consequences are table bloat when vacuum falls behind, and the rule that a long-running transaction can hold back cleanup for the whole database. The trap is thinking DELETE frees disk space immediately. It does not, until vacuum runs and the space is reused.

VACUUM removes dead row versions so their space can be reused, updates statistics for the planner, and keeps the visibility map current. Autovacuum is the background process that runs it automatically when enough rows have changed.

Bloat is the extra space a table or index occupies because dead rows pile up faster than vacuum cleans them. Common causes are heavy update or delete workloads, autovacuum settings too gentle for a big hot table, and long-running or idle in transaction sessions that stop vacuum from removing rows.

To investigate, check n_dead_tup and last_autovacuum in pg_stat_user_tables, and look for old transactions in pg_stat_activity. Tuning usually means lowering the per-table scale factor so big tables are vacuumed sooner. Be careful with VACUUM FULL: it rewrites the table and holds an exclusive lock, so it blocks all access while it runs.

Further reading: PostgreSQL: Routine Vacuuming

Yes. A composite index on (a, b) is sorted by a first and by b within each a, so it is most useful when the query filters on a, or on a and b together.

The usual rule is to put equality columns first and range or sort columns last. Newer PostgreSQL versions can sometimes skip over a missing leading column, but do not design around that. The trap is creating one index per column and expecting the database to combine them as well as one well-ordered composite index.

The planner chooses a plan by estimated cost, and there are several common reasons it decides the index is not worth using.

• The query matches too many rows. If a filter returns a large share of the table, reading it sequentially is cheaper than jumping through an index.
• A function wraps the column. WHERE lower(email) = 'a@b.com' cannot use a plain index on email. Create an expression index on lower(email) instead.
• Type mismatch or implicit cast. Comparing a number column to a text value can disable the index.
• Leading wildcard. LIKE '%shirt' cannot use a normal B-tree index.
• Stale statistics. The planner thinks the table is small or the value is common. Run ANALYZE.
• The table is tiny. A sequential scan really is faster.

The way to find out is EXPLAIN ANALYZE, comparing estimated rows with actual rows. A big gap points to statistics. Do not force the plan with hints first, because fixing the cause is more reliable.

With asynchronous replication, the primary commits and tells the client "done" without waiting for the replica. With synchronous replication, the primary waits until at least one replica confirms it has the change.

So the trade-off is data safety against speed and availability. Asynchronous replication means a non-zero RPO. Synchronous replication gives an RPO near zero but ties write availability to the standby. A common compromise in PostgreSQL is to require confirmation from any one of several standbys (quorum), so a single standby failing does not stop writes.

Further reading: PostgreSQL: Log-Shipping Standby Servers

PgBouncer is a lightweight proxy that sits between applications and PostgreSQL. Thousands of client connections connect to PgBouncer, and it maps them onto a small number of real server connections.

It has three modes:

• Session pooling: a client keeps one server connection for its whole session. Everything works, but you get the least sharing.
• Transaction pooling: a server connection is used only while a transaction runs, then returned. This gives the best sharing and is the usual choice.
• Statement pooling: like transaction pooling but multi-statement transactions are not allowed.

The risk with transaction mode is anything that stores state on the connection, because your next transaction may run on a different one. Session-level SET values, LISTEN, session-level advisory locks, and session-level prepared statements do not work reliably. Protocol-level prepared statements need PgBouncer's max_prepared_statements setting to be non-zero. Mismatched expectations here are a classic cause of strange, intermittent production bugs.

Further reading: PgBouncer features

A logical backup (from pg_dump) stores the data as SQL statements or a portable archive. It is flexible, for example you can restore a single table or move to another major version, but restoring a large database is slow, and it captures only one moment in time.

A physical backup copies the database files themselves (for example with pg_basebackup, or tools like pgBackRest, Barman or WAL-G). It is much faster to restore on large databases.

Point-in-time recovery (PITR) combines a physical base backup with continuously archived write-ahead log (WAL) files. You restore the base backup and then replay the WAL up to a chosen moment, such as one minute before someone dropped a table.

PITR is what turns "we have last night's backup" into "we can lose at most seconds of data". Test it regularly, because a broken WAL archive often goes unnoticed until the day you need it.

Further reading: PostgreSQL: Continuous Archiving and PITR

All three spread data around, but they solve different problems.

• Replication keeps copies of the same data on several servers. It helps with availability and read scaling.
• Partitioning splits one large table into smaller pieces, usually by date or by key range, but inside a single database server. It speeds up queries that touch only one piece (partition pruning) and makes removing old data cheap, because you drop a whole partition instead of deleting millions of rows.
• Sharding splits the data across several independent servers, each holding a subset. It is how you scale writes and storage past what one machine can handle.

Sharding is the most expensive of the three, because cross-shard queries, transactions and re-balancing all become your problem. The common trap is jumping to sharding when a better index, a bigger instance, partitioning or read replicas would solve the actual bottleneck first.

A deadlock happens when two transactions each hold a lock the other one needs, so neither can continue. For example, transaction A updates row 1 then wants row 2, while transaction B updates row 2 then wants row 1.

PostgreSQL detects this after deadlock_timeout (one second by default), aborts one of the transactions with an error, and lets the other finish. Your application must treat that error as retryable.

Prevention is mostly about discipline:

• Always touch rows and tables in the same order in every code path.
• Keep transactions short and do no slow work (API calls, user input) inside them.
• Take the locks you need up front with SELECT ... FOR UPDATE in a consistent order.

The common wrong answer is to raise the timeout or add retries only. Retries hide the symptom, while consistent lock ordering removes the cause.

Further reading: PostgreSQL: Explicit Locking

Use small, backward-compatible steps, often called expand and contract, and avoid anything that holds a heavy lock for long.

1. Expand: add the new column as nullable, or with a constant default (in modern PostgreSQL this is a quick metadata change and does not rewrite the table).
2. Deploy code that writes to both old and new columns, and reads the new one when present.
3. Backfill old rows in small batches, with pauses, so you do not overload the primary or the replicas.
4. Contract: after everything uses the new column, drop the old one in a later release.

Useful safe tools:

Set a lock_timeout on migrations. The trap is that even a quick ALTER TABLE must wait for a lock, and while it waits it blocks every query queued behind it, so one long-running query can turn a "fast" migration into an outage.

It means a client started a transaction (BEGIN) and then stopped doing anything, without committing or rolling back. The database cannot treat it as finished.

It is dangerous for three reasons. It keeps any row locks it took, so other queries wait behind it. It holds an old snapshot, which stops vacuum from cleaning up dead rows and causes bloat. And it keeps a connection slot busy, which can contribute to running out of connections.

The usual causes are application bugs, such as an exception path that never rolls back, or code that opens a transaction and then calls a slow external API. Protect the database with server-side limits:

Find offenders with pg_stat_activity where state = 'idle in transaction'. Fix the application too, because timeouts are a safety net and not a design.

Use SELECT ... FOR UPDATE SKIP LOCKED so each worker claims different rows without waiting for the others.

SKIP LOCKED means a row locked by another worker is simply skipped instead of waited on, so workers do not block each other. It gives you transactional safety for free: the job and your business data can change in the same transaction.

The limits are throughput and housekeeping. Constant inserts, updates and deletes create bloat, so you need aggressive vacuuming. A crashed worker leaves rows stuck as running, so you need a lease time or a sweeper job. For very high rates or many consumers, a real broker such as Kafka, RabbitMQ or SQS is a better fit.

These patterns describe who talks to the database and when the cache gets updated.

Write-through gives a cache that is always current but costs write latency. Write-behind gives very fast writes and is good for absorbing bursts such as view counters, but you must accept losing recent writes on a crash.

The common trap is choosing write-behind for data that must not be lost, such as payments. For most web applications, cache-aside with explicit invalidation is simpler and good enough.

Eviction is controlled by two settings: maxmemory (the memory limit) and maxmemory-policy (what to do when the limit is reached).

• noeviction: do not remove anything. Writes fail with an out-of-memory error, and reads still work. This is Redis's own default.
• allkeys-lru: remove the least recently used key from all keys. The usual choice for a pure cache.
• allkeys-lfu: remove the least frequently used key. Better when a stable set of keys is much hotter than the rest.
• volatile-lru, volatile-lfu, volatile-ttl: only remove keys that have a TTL. Keys without a TTL are never evicted.
• allkeys-random and volatile-random: remove random keys.

Redis does not track exact LRU for every key. It samples a few keys and removes the best candidate, which is cheaper and close enough.

The trap is a volatile-* policy when most keys have no TTL: there is nothing eligible to evict, so you hit out-of-memory errors even though the policy "is set". Managed services can ship different defaults than Redis itself, so check the value with CONFIG GET maxmemory-policy.

Further reading: Redis key eviction

RDB writes a point-in-time snapshot of the whole dataset at intervals. AOF (append-only file) logs every write command, so Redis can replay it on restart.

The Redis documentation recommends using both if you want durability close to a traditional database. If you can accept losing a few minutes of data, RDB alone is enough. If Redis is only a cache and you can rebuild it from the database, you may turn persistence off.

One thing interviewers like: snapshots use fork(). On a large dataset the fork itself can pause Redis briefly, and copy-on-write can make memory use spike while writes continue. Leave memory headroom and watch latency during saves.

Further reading: Redis persistence

Sentinel is for high availability of a single dataset. Cluster is for splitting the dataset across several nodes (sharding), and it also includes failover.

With Sentinel, you run one primary and some replicas, and separate Sentinel processes watch them. If the primary fails, Sentinels vote and promote a replica, and clients ask Sentinel where the current primary is. Every node holds the whole dataset, so you are limited by one machine's memory.

With Cluster, the key space is divided into 16384 hash slots. Each primary owns a range of slots, and each primary can have replicas. A key's slot is CRC16(key) mod 16384, so data and traffic spread across nodes.

Choose Sentinel when your data fits on one node and you want automatic failover. Choose Cluster when you need more memory or throughput than one node offers. In Cluster, commands touching several keys only work if all keys are in the same slot. You can force that with hash tags: {user:981}:cart and {user:981}:profile use only the part in braces for the slot.

Further reading: Scale with Redis Cluster

Redis runs commands one at a time on a main thread. It is fast because the data is in memory, commands are mostly very quick, and there is no locking between threads. One command at a time also means every single command is atomic with no extra effort. (Newer versions can use extra threads for network I/O, but commands still execute serially.)

The danger is that one slow command blocks everyone. While KEYS * runs on a big database, every other client waits. The same is true for commands that touch a whole large collection, very big Lua scripts, and deleting a huge key with DEL.

Practical habits that follow from this:

• Use SCAN instead of KEYS.
• Use UNLINK instead of DEL for big keys, because it frees memory in the background.
• Use pipelining to send many commands in one network round trip.
• Keep keys and collections small, and keep Lua scripts short.

A common wrong answer is "single-threaded means it cannot use many CPU cores, so it does not scale". You scale it by running multiple instances, or by using Cluster.

The usual rule is to write to the database first and then delete the cached key, not update it. The next read misses and reloads the fresh value.

Updating the cache is risky because of races. Imagine two writers: writer A updates the database to 10, writer B updates it to 20, but the cache updates arrive in the opposite order, so the cache ends up showing 10 while the database holds 20. Deleting avoids this because a delete cannot leave a wrong value behind.

A small race still exists with deletes. A reader can miss, load the old value from the database, and then a writer updates the database and deletes the key, and finally the reader stores its old value into the cache. That stale value then lives until the TTL ends.

Ways to limit it:

• Always keep a TTL as a backstop.
• Delete the key a second time shortly after the write (delayed double delete).
• Drive invalidation from the database change log (CDC) so it is not tied to application code paths.
• Put a version number in the key so old versions are simply never read.

The honest interview answer is that perfect consistency between a cache and a database does not exist without paying heavily. You choose how much staleness is acceptable and design for it.

A cache stampede (also called thundering herd or dogpile) happens when a popular key expires and many requests miss at the same moment. All of them hit the database to rebuild the same value, which can overload it.

Prevention options, from simple to advanced:

• Single flight lock: the first request that misses takes a short lock (SET lock:key token NX PX 5000) and rebuilds the value, while others wait briefly or get the old value.
• Jittered TTLs: add a random amount to each TTL so popular keys do not all expire together.
• Serve stale while refreshing: keep a slightly longer physical expiry, return the stale value, and refresh in the background.
• Early or probabilistic refresh: refresh a hot key a little before it expires, with some randomness, so only one request does it.
• Pre-warming: load known hot keys before a launch or after a restart.

The trap is relying on the database to cope. A single expensive query repeated by thousands of clients at once can be enough to take it down, and the problem is worst exactly when traffic is highest.

Cache penetration is when requests ask for data that does not exist, so they never get cached and every request goes to the database. It happens by accident (bad IDs from a buggy client) or on purpose (an attacker requests random IDs to overload the database).

Normal caching does not help because there is nothing to store. The defences are:

• Cache the "not found" result with a short TTL (30 to 60 seconds), so repeated misses are answered by the cache.
• Validate input first. Reject IDs that are impossible, such as negative numbers or wrong formats, before touching the cache or database.
• Use a Bloom filter in front of the cache: a compact structure that can say "this key definitely does not exist", so those requests are stopped early.
• Rate limit clients that produce many misses.

Do not confuse it with a stampede. A stampede is many requests for one real key that just expired. Penetration is many requests for keys that were never there. Caching missing results has one catch: if the data is created later, the cached "not found" must expire or be deleted, hence the short TTL.

A cache avalanche is when a large number of keys expire at about the same time, or the whole cache goes down, so a flood of requests lands on the database at once. A stampede is about one hot key, while an avalanche is about many keys or the entire cache.

A typical cause is loading thousands of keys at startup with the same fixed TTL, so they all expire together. The other cause is a Redis outage or failover, where the database suddenly gets 100 percent of traffic that it was only ever sized for at 10 percent.

How to reduce the risk:

• Add random jitter to TTLs.
• Run the cache highly available (replicas with failover), so a single node loss is not a full loss.
• Protect the database with rate limits, request coalescing and a circuit breaker, so it degrades instead of collapsing.
• Warm the cache gradually after a restart rather than opening all traffic at once.

An interview point worth making: size and test the database for a cold cache, because that is the scenario that actually takes sites down.

KEYS pattern scans the entire key space in one go. Because Redis runs commands one at a time, a big database makes KEYS * block every other client until it finishes, which can look like an outage.

Use SCAN instead. It walks the key space in small steps with a cursor, so other commands can run in between.

Keep in mind that SCAN can return the same key more than once, and it gives no guarantee about keys changed during the scan, so your code must tolerate duplicates.

A big key is a single key holding a huge value, such as a hash with millions of fields or a list with millions of entries. Big keys cause latency spikes (slow commands, slow deletion), uneven memory across cluster nodes, and long network transfers. Find them with redis-cli --bigkeys or MEMORY USAGE key, split them into smaller keys, and remove them with UNLINK, which frees memory in the background.

The simplest version is a fixed window counter: one key per user per time window, incremented on each request, and expiring when the window ends.

There are two things to get right. First, INCR and EXPIRE as two separate calls are not atomic: if the client crashes in between, the key never expires. Fix it with a small Lua script, or with SET key 0 EX 60 NX before the INCR. Second, a fixed window allows bursts at the boundary: 100 requests at 09:30:59 and 100 more at 09:31:00 is 200 in two seconds.

Better options when that matters:

• Sliding window log: keep each request time in a sorted set and count the entries in the last 60 seconds. Accurate, but uses more memory.
• Token bucket: refill tokens at a steady rate and allow short bursts. Usually implemented in a Lua script so the read and update happen atomically.

Also decide what happens when Redis is down: fail open (allow traffic) or fail closed (block it). For most public APIs, failing open is the safer default.

These describe how often a consumer may see a message when failures happen.

• At-most-once: a message is delivered zero or one time. You commit the position before processing, so a crash mid-processing loses the message.
• At-least-once: a message is delivered one or more times. You process first and commit after, so a crash before the commit causes the message to be processed again.
• Exactly-once: the effect happens once. It is harder, because it needs cooperation from the producer, the broker and the place where the result is stored.

Most real systems use at-least-once and make the consumer idempotent, meaning that processing the same message twice has the same result as processing it once (for example, an INSERT with a unique order ID that ignores duplicates).

The common wrong answer is "turn on exactly-once and forget about duplicates". Kafka's exactly-once support covers reading from Kafka and writing back to Kafka. It cannot undo an email already sent or a payment API call already made, so side effects outside the broker still need idempotency.

Further reading: Apache Kafka documentation

They are built around different ideas, so the right choice depends on what you need to do with the messages.

Choose Kafka for event streaming, high volume, several consumers reading the same data, and replay. Choose RabbitMQ for task queues that need flexible routing (exchanges, priorities, per-message handling) and low latency. Choose SQS when you are on AWS and want a simple queue with no servers to manage.

The trap is picking Kafka because it is popular. If you need only a simple job queue for a few thousand messages an hour, Kafka's operational cost is hard to justify.

Consumer lag is how far a consumer group is behind the newest message, measured per partition as the latest offset minus the group's committed offset. Growing lag means consumers are slower than producers.

A steady small lag is normal. What matters is the trend, and lag in time ("this group is 4 minutes behind") is often more useful than lag in message count.

Typical causes and fixes:

• Consumers are too slow. Profile the processing, batch database writes, and avoid slow calls per message.
• A slow downstream system (database, API) is the real bottleneck. Scaling consumers will not help.
• Not enough consumers. Add more, but only up to the number of partitions. A consumer beyond that count sits idle.
• Frequent rebalances keep pausing consumption.
• A producer burst simply outpaced you. If it is temporary, lag will drain by itself.

Alert on sustained lag growth, not on a single number. The common trap is adding consumers beyond the partition count and seeing no improvement.

A rebalance is when Kafka redistributes a topic's partitions among the consumers in a group. During the older "eager" rebalance every consumer stops, gives up its partitions and waits for a new assignment, so consumption pauses for the whole group.

Triggers include:

• A consumer joins (a new pod starts) or leaves (a pod is stopped during a deploy).
• A consumer is considered dead because it missed heartbeats within session.timeout.ms.
• A consumer takes too long between poll() calls and exceeds max.poll.interval.ms, so it is removed from the group.
• The number of partitions in the topic changes.

Frequent rebalances hurt throughput, because the group spends its time reassigning instead of processing. Ways to reduce them:

• Use the cooperative sticky assignor, so only the affected partitions move.
• Use static membership (group.instance.id), so a restarting consumer gets its old partitions back without a rebalance.
• Keep per-message processing short, or lower max.poll.records.

Newer Kafka versions (4.0 onward) also include a new consumer group protocol (KIP-848) that moves coordination to the broker and avoids stopping the whole group.

Durability in Kafka is the result of three settings working together.

• Replication factor: how many copies of each partition exist (3 is common).
• acks (producer): how many confirmations the producer waits for. acks=0 waits for none, acks=1 waits for the leader only, and acks=all waits for every in-sync replica.
• min.insync.replicas: the minimum number of in-sync replicas that must acknowledge for a write with acks=all to succeed.

A safe and common setup is replication factor 3, acks=all and min.insync.replicas=2. A write is confirmed only when at least two copies have it, so you can lose one broker without losing confirmed data and without blocking writes.

The trap is acks=all with min.insync.replicas=1. If the other replicas fall out of sync, "all in-sync replicas" can mean just the leader, and a leader crash then loses data. Also keep unclean.leader.election.enable off, so a replica that missed messages is never elected leader. The price of this safety is higher write latency, and writes stop if too few replicas are available.

Both control how long Kafka keeps messages, but they answer different questions.

Retention deletes old data by age or size. For example, retention.ms of 7 days means messages older than a week are removed. Use it for event streams where only recent history matters, such as clickstream or logs.

Compaction keeps only the latest message for each key and removes older ones with the same key. Use it for "current state" topics, such as the latest profile for each user, or a changelog that rebuilds a table. A message with a key and a null value (a tombstone) marks a key for deletion.

You can combine them (cleanup.policy=compact,delete).

Two points often tested: compaction does not delete immediately, it runs in the background and the most recent segment is not compacted, so old values can still be read for a while. And compaction needs keys; messages without keys cannot be compacted. The wrong answer is to treat Kafka as unlimited storage with no cleanup policy, which fills the broker disks.

An idempotent consumer is one where processing the same message more than once has the same effect as processing it once. You need this because at-least-once delivery means duplicates will happen: after a crash, a rebalance, or a retry.

Common ways to build it:

• Natural idempotency. Operations like "set status to shipped" are safe to repeat. "Add 100 to the balance" is not.
• A unique key in the database. Store the message or business ID (such as payment_id) with a unique constraint. A duplicate insert fails or is ignored.
• A processed-messages table. Record the message ID in the same database transaction as the business change, and skip IDs already seen.
• Idempotency keys for outside calls. Many payment APIs accept an idempotency key, so a repeated request does not charge twice.

The key detail is doing the check and the change in the same transaction. If you check first and write later as two steps, a crash between them still leaves a gap.

A single bad message must not block everything behind it, and a temporary failure should not lose a message. Retries with delay plus a dead-letter queue (DLQ) handle both.

The diagram shows the flow: failed messages go to a delayed retry, and after a fixed number of attempts they are parked in the DLQ.

Good practice:

• Separate the error types. Retry temporary errors (timeouts, a busy database). Send permanent ones (invalid data, a bug) straight to the DLQ, because retrying cannot fix them.
• Use exponential backoff with jitter (for example 1s, 5s, 30s), so retries do not pile onto a recovering service.
• Cap the attempts, such as 5.
• Alert on the DLQ, and keep the original message with the error reason and attempt count so you can fix and replay it.

In SQS this is built in through a redrive policy with maxReceiveCount. In Kafka you create retry and DLQ topics yourself. The trap is an unmonitored DLQ, which just becomes a place where data quietly disappears.

When a consumer receives a message from SQS, the message is not deleted. It is hidden from other consumers for the visibility timeout. If the consumer deletes it in time, the work is done. If not, the message becomes visible again and another consumer receives it.

The default is 30 seconds and the maximum is 12 hours.

• Too short: processing takes longer than the timeout, so the message reappears while still being worked on. Two consumers process it at once, which creates duplicates.
• Too long: if a consumer crashes, the message stays hidden for a long time before it can be retried, so recovery is slow.

A good setting is a bit longer than your normal worst-case processing time. For variable jobs, extend the timeout while working (ChangeMessageVisibility) with a heartbeat.

Remember that standard SQS queues deliver at least once and do not guarantee order, so consumers must be idempotent anyway. Pair the queue with a dead-letter queue and maxReceiveCount, so a message that keeps failing stops cycling forever.

Further reading: Amazon SQS visibility timeout

RabbitMQ gives reliability at two separate points, and they are independent of each other.

• Consumer acknowledgements (acks): the consumer tells the broker "I finished this message", and only then is it removed. If the consumer's connection closes without an ack, the message is put back in the queue for another consumer. This protects against consumers failing.
• Publisher confirms: the broker tells the producer "I have accepted this message". For persistent messages in durable queues, that means it has been written to disk. This protects against the broker losing messages.

Having only one of the two leaves a gap, which is a common interview trap.

Prefetch (also called QoS) limits how many unacknowledged messages a consumer can hold at once. With no limit, one fast connection can grab thousands of messages and starve other consumers, and a crash then requeues all of them. A small prefetch (for example 10 to 50) spreads work evenly and bounds memory.

For safety also use durable queues and persistent messages, and consider quorum queues for replicated, highly available queues. Since redeliveries can happen, consumers should be idempotent.

Further reading: RabbitMQ consumer acknowledgements and publisher confirms

The partition count sets the maximum consumer parallelism for a group, and it also affects ordering and broker load. Pick it from the throughput you need and the number of consumers you expect to run.

A simple way to estimate: take your target throughput, divide by the throughput one partition (or one consumer) can handle, and round up with some headroom for growth. Too few partitions limits scaling. Too many costs more open files, more memory, longer leader elections after a failure, and more overhead for clients.

The important rules:

• You can increase partitions later, but you cannot decrease them.
• Increasing the count changes which partition a key maps to, because the hash is taken modulo the number of partitions. Messages for one key can end up in a new partition, so ordering for that key can break around the change.
• Existing messages are not moved.

Because of that, plan some headroom at the start, instead of expecting to fix it later. If ordering per key matters, increase partitions during a quiet period, or create a new topic with the right count and migrate to it.

Backpressure is the system telling producers to slow down because consumers cannot keep up. Without it, the queue just grows until memory or disk runs out, or messages become so old they are useless.

First work out why it grows. Is the arrival rate permanently higher than the processing rate, or is it a temporary spike? A queue that grows during a sale and drains afterwards is working as intended. One that never drains is a capacity or bug problem.

Ways to respond:

• Scale consumers, up to what the queue or partitions allow, and check that the downstream database or API can handle the extra load.
• Make processing faster through batching or removing slow calls.
• Limit producers with rate limits or bounded queues, so they get a clear "try again later" instead of silently piling up work.
• Shed or delay low-value work such as analytics events, so important work keeps flowing.
• Set message expiry (TTL) for work that is pointless after a deadline, such as a one-time login code.

Also monitor queue depth and the age of the oldest message. The age is the number users feel. The common trap is scaling consumers when the real bottleneck is a shared database, which just moves the overload there.

PostgreSQL numbers transactions with a 32-bit counter, which is only about 4 billion values. To decide which rows are visible, it treats roughly 2 billion IDs behind the current one as "the past" and 2 billion ahead as "the future". If old rows are never marked as frozen, the counter eventually wraps and old data would suddenly look like it is in the future, meaning invisible.

To prevent that, vacuum freezes old rows, marking them as visible to everyone. If vacuum cannot keep up, PostgreSQL first logs warnings as the database approaches the limit (the docs describe a warning about forty million transactions from the wrap point), and finally refuses new write transactions until a manual vacuum completes. That is a full write outage.

What blocks freezing is usually something holding the oldest snapshot back:

• A very old open transaction or idle in transaction session
• An abandoned replication slot
• A forgotten prepared transaction

Monitor that age, alert early, find and end the blocker, then run VACUUM FREEZE on the oldest tables. The common wrong answer is "just run VACUUM FULL". That does not help here and makes things worse by locking the table.

Further reading: PostgreSQL: Routine Vacuuming

A safe failover has four steps: detect, choose, promote and fence. After that you redirect traffic and rebuild.

The diagram shows the order. Fencing comes right after promotion and before traffic moves.

Split-brain is when two servers both believe they are the primary and both accept writes. The data then diverges, and merging it later is painful or impossible. It happens when the old primary was only unreachable, not dead, and comes back after a replica was promoted. Prevent it by fencing the old node (shut it down, cut its network or storage access) and by using a consensus store such as etcd, as Patroni does, so there is one agreed leader.

Also state what you lose. With asynchronous replication, the chosen replica may be missing the last few commits, so check its lag and tell the business what the RPO actually was. Afterwards, the old primary may need pg_rewind or a full rebuild before it can rejoin as a replica.

WAL (write-ahead log) files are normally recycled after a checkpoint. If they pile up, something is forcing PostgreSQL to keep them.

The usual causes are:

• An inactive or lagging replication slot. A slot tells the primary "keep every WAL file this consumer has not read yet". If the replica or a logical consumer (such as a CDC tool) is down, WAL grows without limit until the disk is full.
• A failing archive_command. If archiving to object storage keeps failing, PostgreSQL keeps the unarchived files.
• A replica falling far behind while using streaming replication with a slot.
• Write bursts such as a huge batch job or bulk load.

To fix it: get the consumer running again, or drop the slot if it is truly abandoned, and fix the failing archive command. To prevent a repeat, set max_slot_wal_keep_size so a stuck slot gets invalidated instead of taking the primary down, and alert on retained WAL size and archive failures. The trap is deleting WAL files by hand from the data directory, which can corrupt the database.

Work from "what changed" to "which query" to "why is it slow", and stabilise first if users are hurting.

1. Stabilise. If the deploy is the obvious trigger, roll it back or disable the feature flag. Kill runaway queries if needed (pg_terminate_backend).
2. See what is running now. Look at pg_stat_activity for long-running or numerous identical queries, and for sessions waiting on locks.
3. Find the top consumers. pg_stat_statements ordered by total time and by calls shows the heavy queries. A cheap query with a huge call count points to an N+1 pattern. A slow query with few calls points to a missing index or a bad plan.
4. Explain the culprit. Run EXPLAIN (ANALYZE, BUFFERS) and compare estimated and actual rows.
5. Check side causes. Connection count jumped (an app pool setting changed), autovacuum is struggling, or a plan flipped because statistics changed.

Fix with the smallest safe change: an index built with CONCURRENTLY, a rewritten query, a cache in front of a hot read, or a smaller pool. The common wrong answer is to scale the instance up first. That costs money and only hides a bad query for a few weeks.

This error means PostgreSQL has reached max_connections, so it refuses new connections. The proper fix is to reduce how many connections the application needs, not to raise the limit and move on.

First, work out who holds the connections:

Many rows in idle mean pools are too big. Many in idle in transaction mean an application bug. Many active mean the database is actually overloaded.

Then fix the cause:

• Put a pooler such as PgBouncer in front, so many clients share few server connections.
• Do the multiplication: pool size times number of instances. Autoscaling and serverless functions multiply connections without anyone noticing.
• Shrink application pools. More connections rarely means more throughput, because the database only has so many CPU cores.
• Add timeouts so leaked or idle sessions are closed.

Raising max_connections is a trap. Each connection costs memory and adds contention, so a higher limit often makes a struggling database slower, not faster.

A lock on a single Redis node is fine for avoiding duplicate work, but it is not enough when correctness depends on it.

The basic pattern is one atomic command, followed by a safe release:

The random token matters because without it a slow client could delete a lock that now belongs to someone else.

The problems are timing and failover:

• If the lock expires while the holder is paused (a long garbage collection pause, a stalled network call), a second client takes the lock and both think they own it.
• Replication is asynchronous, so if the primary fails after granting the lock, the promoted replica may never have heard of it.

Redlock tries to improve this by acquiring the lock on several independent Redis instances and requiring a majority. Critics, notably Martin Kleppmann, argue it still depends on timing assumptions, and that the real fix is a fencing token: a number that grows with each lock grant, which the protected resource checks and uses to reject writes from older holders.

A good answer is: use a Redis lock for efficiency (skip duplicate work), and use a database constraint, a transaction or a fencing token when a mistake would corrupt data.

Further reading: Redis distributed locks

A hot key is one key that receives a disproportionate share of requests. In Redis Cluster the key lives on exactly one primary, so adding more nodes does not help, and that node's CPU or network becomes the bottleneck.

First confirm it. Look at per-node CPU and network, run redis-cli --hotkeys (this needs an LFU eviction policy), or check client-side metrics, because a hot key is invisible when you only look at averages.

Then reduce the pressure on that one node:

• Local cache in the application with a very short TTL (even 1 to 5 seconds). Each app instance serves most reads from memory and Redis sees a tiny fraction.
• Replicate the key under several names (config:home#1 to config:home#8) and pick one at random per read. This spreads the load across slots and nodes.
• Read from replicas for data that tolerates slight staleness.
• Reshape the data so the hot item is not one huge value, if it is naturally partitionable.

Remember the cost: every copy has to be updated or invalidated together, and a short local TTL means users may see data a few seconds old. The wrong answer is to simply "add more nodes", which does nothing for a single key.

No. Redis Cluster does not give strong consistency. It can lose a write that it already acknowledged to the client.

There are two main reasons:

• Asynchronous replication. A primary replies "OK" before it has sent the write to its replicas. If it crashes right then and a replica that never received the write is promoted, the write is gone.
• Network partitions. If a client is cut off with a minority of primaries, it can keep writing to a primary until cluster-node-timeout passes. Meanwhile the majority side may already have promoted a replica, and the writes made on the minority side are lost when the partition heals.

The WAIT command can block until a number of replicas confirm a write, which lowers the chance of loss, but the Redis documentation is clear that it still does not make the system strongly consistent.

What to do with this knowledge:

• Treat Redis as a cache or a fast store whose loss you can tolerate or rebuild, not as the only copy of important data.
• If the data must not be lost, write it to a database of record first.
• For critical counters or locks, design for the possibility of lost or duplicate updates.

A related detail interviewers like is cluster-require-full-coverage. By default, if some hash slots have no available node, the cluster stops serving requests, which favours consistency over availability.

Further reading: Scale with Redis Cluster

Narrow it down in layers, from the network inward, because Redis itself is usually fast and the cause is often something around it.

1. Check the client and network first. Compare latency from the app with redis-cli --latency run on the Redis host. If the local number is low and the remote one is high, it is the network or the client.
2. Look for slow commands. SLOWLOG GET 10 lists the slowest recent commands. Common culprits are KEYS, big SMEMBERS or HGETALL, large SORT, and long Lua scripts.
3. Check for big keys and expiration storms. Many keys expiring at once makes Redis busy deleting them.
4. Check persistence. Forking for an RDB save or an AOF rewrite can pause Redis, and slow disk fsync can stall AOF.
5. Check memory and swap. If the host starts swapping, latency goes from microseconds to seconds.
6. Check CPU. The main thread uses one core, so one saturated core is the limit even if the machine looks idle overall.

Also check how many clients are connected and whether a client has a huge output buffer. The trap is restarting Redis first: it clears the evidence and often only hides a bad command or a big key that will return.

Plan for the cache being unavailable, because it will be, and the database was probably never sized to carry full traffic alone.

On the application side:

• Short timeouts on Redis calls (tens of milliseconds), so a dead cache does not tie up every thread waiting.
• Circuit breaker: after repeated failures stop calling Redis for a while and go straight to the fallback, then probe occasionally to recover.
• Serve a local copy (small in-process cache) for the hottest data, if staleness is acceptable.
• Avoid retry storms. Retrying every failed call multiplies the load on a struggling system.

On the database side:

• Coalesce identical requests so 1000 requests for one product become one query.
• Rate limit or shed load for non-critical features, such as recommendations, so checkout keeps working.
• Protect connections with a pool limit so the database does not fall over trying to serve every request.

Finally, test it. Run a game day where you deliberately stop Redis and watch what breaks. The wrong answer is "we have Redis replicas, so it never goes down". Failover takes time, and during it requests still need a plan. Also warm the cache gradually after recovery, otherwise the refill itself becomes the next outage.

Start by separating three numbers in INFO memory: used_memory (what Redis thinks it stores), used_memory_rss (what the operating system has given it), and their ratio, mem_fragmentation_ratio.

• Ratio well above 1 (for example 1.5 or more): fragmentation. Memory was freed inside Redis but not returned to the OS. Active defragmentation (activedefrag) or a controlled restart can help.
• Ratio below 1: the OS is swapping Redis out. This is serious because latency collapses. Add RAM or reduce data immediately.
• Used memory simply large: look for big keys and for keys without a TTL that grow forever (redis-cli --bigkeys, MEMORY USAGE key).

Other causes of sudden growth:

• Fork copy-on-write. During an RDB save or AOF rewrite, heavy writes make the child process copy many memory pages, so memory can nearly double in the worst case.
• Client output buffers, for example a slow subscriber or replica holding large buffers.
• A maxmemory set too close to physical RAM, leaving no room for those extras.

The fix is to set maxmemory well below total RAM (leave real headroom), choose an eviction policy that matches how keys are used, give every cache key a TTL, and alert on both memory and fragmentation. If the Linux out-of-memory killer ends Redis, the cause was almost always missing headroom, not a bug in Redis.

Kafka's exactly-once support guarantees that, inside Kafka, a record is processed and its results are written once, even with retries and failures. It does not extend to the outside world.

It is built from two features:

• Idempotent producer. Each producer has an ID and attaches sequence numbers, so the broker discards a duplicate caused by a retry. It is on by default in modern Kafka versions.
• Transactions. A consumer can read records, process them, write results to other topics, and commit its input offsets in one atomic transaction. Either all of it becomes visible or none of it does. Consumers read with isolation.level=read_committed to ignore aborted data.

This covers a read-process-write pipeline between Kafka topics, such as in Kafka Streams.

Where it stops: the moment your code does something outside Kafka, such as sending an email, calling a payment API or updating an external database, the transaction cannot roll that back. If the consumer crashes after the side effect but before the commit, the message is processed again and the side effect repeats.

So the practical answer is: use Kafka transactions for Kafka-to-Kafka flows, and use idempotent consumers or idempotency keys for everything that touches another system. "Exactly-once" is really "effectively once", achieved by combining at-least-once delivery with idempotent handling.

Each partition has one leader and some followers. Producers and consumers talk only to the leader. When a broker dies, the partitions it led get a new leader chosen from the followers that are in the in-sync replica set (ISR).

The sequence:

1. The cluster controller notices the broker is gone (a missed heartbeat or lost session).
2. For each affected partition it elects a new leader from the ISR and publishes the new metadata.
3. Producers and consumers get errors or refresh metadata, find the new leader and continue. Producer retries cover the short gap.
4. When the broker returns it rejoins as a follower and catches up before it rejoins the ISR.

Whether you lose data depends on the settings from the durability question. With replication factor 3, acks=all and min.insync.replicas=2, every confirmed write exists on at least two brokers, so one failure loses nothing. If too few replicas remain in sync, producers get NotEnoughReplicas errors and writes stop, which is a deliberate choice of consistency over availability.

Unclean leader election (allowing an out-of-sync replica to become leader) restores availability at the cost of losing messages, so leave it off for important data.

On the cluster side, modern Kafka no longer needs ZooKeeper: Kafka 4.0 completed the move to KRaft, where Kafka's own controller quorum stores the metadata. Expect questions about both older and newer setups.

The dual write problem is that writing to the database and publishing to a broker are two separate systems with no shared transaction. If you save the order and then crash before publishing, the event is lost. If you publish first and the database write fails, you announce an order that does not exist.

The outbox pattern fixes this by making only one write to one system. In a single database transaction, you save the business data and also insert an event row into an outbox table. A separate process then reads the outbox and publishes to the broker.

The diagram shows that the broker is only reached from the relay, after the data is safely committed.

The relay can poll the table, or use change data capture (CDC), for example Debezium reading the database log. Either way, it can crash after publishing but before marking the row as sent, so the same event may be published twice. That means consumers must be idempotent. The pattern gives you at-least-once publishing with no lost events, which is almost always what you want.

Order is only guaranteed within one partition (Kafka) or one message group (SQS FIFO), so the design must route all messages for one customer into the same ordered lane and process that lane one message at a time.

Things that break order:

• Different keys or no key. Messages for one customer land on different partitions. Fix: use the customer ID as the message key.
• Parallel processing inside a consumer. If a consumer takes a batch from one partition and handles it with a thread pool, messages finish out of order. Fix: process sequentially per key, or use one thread per key group.
• Retries. If message 1 fails and goes to a retry queue while message 2 succeeds, message 2 is applied before message 1. Fix: block the lane until message 1 succeeds or is dead-lettered, or make the handler tolerate out-of-order updates by comparing a version or timestamp.
• Changing the partition count, which remaps keys (see the partitions question).
• Producer retries without idempotence, which can reorder messages in flight. Enable the idempotent producer.

The trade-off is that strict ordering reduces parallelism, because one slow message holds up everything behind it for that key. The best design often avoids needing strict order: carry a sequence number or version in each event, and let the consumer ignore anything older than what it has already applied.

Start by confirming it is a rebalance problem, then find out what keeps removing consumers from the group.

1. Check the symptoms. Look at consumer logs for repeated "rebalance" and "revoked partitions" messages, and at lag that grows in sawtooth steps.
2. Find the trigger. Is a deploy rolling pods one by one? Are pods being restarted by out-of-memory kills or failing health checks? Is the autoscaler adding and removing consumers constantly?
3. Check for a slow loop. If processing a batch takes longer than max.poll.interval.ms, the consumer is kicked out and a rebalance starts. This is the most common cause. A slow downstream call, a big batch or a long garbage collection pause can do it.
4. Check heartbeats. Network blips or CPU starvation can miss session.timeout.ms.

Fixes follow the cause:

• Reduce max.poll.records or speed up processing, or raise max.poll.interval.ms if the work is legitimately long.
• Use the cooperative sticky assignor and static membership (group.instance.id), so restarts do not trigger a full rebalance.
• Stop the autoscaler from flapping with stabilisation windows.
• Shut consumers down gracefully on deploys.

The wrong answer is to only raise timeouts. That hides a slow consumer, and makes real failures take longer to detect.

The goal is to protect the database from the spike, never oversell, and keep the user experience honest. Each component has one job.

• Cache (Redis) for fast reads and a first gate. Serve product pages from the cache and a CDN. Keep the remaining stock as a counter in Redis and decrement it atomically (DECR). If the result goes below zero, reject immediately with "sold out". This turns most of the 50,000 requests into cheap in-memory operations.
• Queue for the slow part. Users who pass the gate get an accepted order placed on a queue, and the user sees "order received". Workers then create the order, take payment and update the database at a rate the database can handle.
• Database as the source of truth. Workers do a final guarded update, such as UPDATE stock SET qty = qty - 1 WHERE item_id = ? AND qty > 0, and check that one row changed. This guarantees no overselling even if the Redis counter was wrong.

The key trade-offs to mention:

• Redis is not durable truth. A failover can lose or repeat decrements, so the database check is the real guard, and the Redis counter is an optimisation.
• Idempotency. Duplicate clicks and retries must not create two orders, so use an order or idempotency key.
• Cleanup. If payment fails, release the reserved stock, and expire reservations after a timeout.
• Backpressure and fairness. Put a limit on the queue and return a clear response when it is full.

A strong answer names what happens when each part fails: Redis down (fall back to the database with rate limiting), queue backed up (show estimated wait), and database slow (workers slow down, the queue absorbs it).