Kinesis Data Streams
A real-time streaming service that collects and stores streaming data for 1 to 365 days. Multiple consumers can read the same stream simultaneously and replay data from any point in the retention window.
Kinesis Data Streams collects and stores real-time streaming data — clicks, sensor readings, application logs, financial transactions — and makes it available for multiple consumers to process simultaneously and replay from any point in the retention window.
The Core Difference from SQS and SNS
SQS: each message → one consumer, message deleted after consumption, no replaySNS: each message → all subscribers at moment of publish, not storedKinesis: each record → all consumers that want it, stored for 1-365 days, replayableThe replay capability is what makes Kinesis unique. A bug in your processing code at 2 PM? Fix the bug, redeploy, and replay all records from 2 PM. SQS cannot do this — once consumed, gone forever.
Shards — The Capacity Unit
Each shard handles: Ingest: up to 1 MB/s or 1,000 records/second Read: up to 2 MB/s per consumer Need 5 MB/s ingest → provision 5 shardsMultiple consumers reading same shard: each gets up to 2 MB/s independentlyProvisioned mode: you choose shard count, pay per shard per hour On-Demand mode: auto-scales based on peak traffic over last 30 days, pay per GB
Ordering and Partition Keys
Records with same Partition Key → always go to same shard → ordered within that shardRecords with different Partition Keys → may go to different shards Use case: partition by UserId so all events for one user arrive in orderGlobal ordering (all users in sequence) → use one shard (throughput limited)Producers and Consumers
Producers (send data in): AWS SDK direct API calls Kinesis Producer Library (KPL) — batching and compression AWS services: CloudWatch Logs, IoT Core, Database activity streams Consumers (read data): Lambda (automatic trigger on new records) Amazon Managed Flink (real-time stream processing) Kinesis Data Firehose (delivery to S3, Redshift, OpenSearch) Custom EC2/ECS applications using Kinesis Client Library (KCL)Retention and Replay
Default retention: 24 hoursExtended retention: up to 365 days (additional cost per shard per day)Replay from any timestamp within retention windowKinesis vs MSK (Kafka)
Kinesis: AWS-proprietary, simpler, AWS-native ecosystemMSK: Apache Kafka, open standard, richer ecosystem (Kafka Connect, more consumer types)RememberManaged Flink reads from Kinesis Data Streams — NOT from Kinesis Data Firehose. Firehose is a delivery pipeline you send data to. Kinesis Data Streams is a stream you subscribe to and read from. They are fundamentally different despite the similar naming.
Frequently Asked Questions
What makes Kinesis Data Streams different from a message queue like SQS?
SQS deletes a message once a consumer processes it, so only one consumer effectively gets each message without extra fan-out setup. Kinesis retains data for a configurable window between 1 and 365 days and lets multiple independent consumers read the same stream at their own pace, each tracking its own position (sequence number) — which is what makes it suited to scenarios needing both real-time processing and later replay, like feeding both a live dashboard and a batch analytics job from the same events.
What's a common scaling mistake with Kinesis Data Streams?
Under-provisioning shards for the actual write throughput — each shard supports up to 1MB/s or 1,000 records/s of ingest, and exceeding that causes ProvisionedThroughputExceededException errors that producers must handle with retries. A related mistake is picking a poor partition key (e.g., a low-cardinality field) that causes uneven data distribution across shards, creating 'hot shards' that throttle while others sit idle even though aggregate capacity looks sufficient.