Kinesis Firehose

AWS Cloud Engineering hub

A fully managed delivery service that loads streaming data into S3, Redshift, OpenSearch, or third-party destinations. Buffers data by size or time and delivers in near-real-time batches. No consumer code needed.

Kinesis Data Firehose is a fully managed delivery service that loads streaming data into destinations automatically. No consumer code. No servers. No scaling to manage. You point it at a source and a destination — it handles the rest.

How Firehose Works

◈ DIAGRAM

Producers send records to Firehose
    ↓
Firehose buffers records until size or time threshold is reached
    ↓
Optional: Lambda transform runs on each batch
    ↓
Firehose delivers the batch to the destination
 
Buffering triggers (whichever comes first):
Size:  1 MB to 128 MB (deliver when this much data has accumulated)
Time:  60 seconds to 900 seconds (deliver every N seconds regardless of size)
 
This buffering is why Firehose is near-real-time — there is always at least 60 seconds of delay. For millisecond latency you need Kinesis Data Streams with your own consumer.

Destinations

TEXT

AWS:          Amazon S3 (most common), Amazon Redshift (via S3 COPY), Amazon OpenSearch
Third-party:  Splunk, Datadog, New Relic, MongoDB, Coralogix, Dynatrace
Custom:       Any HTTP endpoint you own

Lambda Transform — Inline Processing

Before delivery, Firehose can invoke a Lambda function on each batch to transform records:

◈ DIAGRAM

Raw JSON records → Lambda transforms:
→ Convert to Parquet (Athena queries cost 90% less on Parquet)
→ Redact sensitive fields (mask credit card numbers before archiving)
→ Add metadata fields (timestamp, partition key, region)
→ Filter out records you do not want to store

Firehose vs Kinesis Data Streams

	Kinesis Data Streams	Kinesis Data Firehose
Purpose	Collect and store streaming data	Deliver streaming data to a destination
Consumer code	You write it	None needed — fully managed
Latency	Milliseconds	60+ seconds (buffered)
Data retention	1 to 365 days	No storage — routes and delivers only
Replay	Yes	No
Multiple consumers	Yes — each reads independently	No — one destination per stream

Common Architecture Using Both

◈ DIAGRAM

IoT devices → Kinesis Data Streams (collect, store 7 days, replay if needed)
            → Managed Flink consumer (real-time anomaly detection)
            → Kinesis Firehose consumer (delivery to S3 in Parquet format)
            → Athena queries S3 for historical analysis

Streams handles collection and replay. Firehose handles archiving. Each does what it is built for.

Remember
Firehose has no storage and no API for reading records. Data flows in and flows out to the destination. You cannot subscribe to a Firehose delivery stream the way you subscribe to a Kinesis Data Streams shard. Firehose is delivery, not storage.

Frequently Asked Questions

How does Kinesis Firehose differ from Kinesis Data Streams if both handle streaming data?

Data Streams is a raw, low-level stream you write custom consumer code against for real-time processing and replay. Firehose is a fully managed delivery pipeline with no consumer code required — you point it at a destination (S3, Redshift, OpenSearch, or an HTTP endpoint) and it buffers incoming records by a configurable size or time threshold, optionally transforms them via a Lambda function, and delivers batches automatically. Firehose can also read directly from a Kinesis Data Stream as its source.

What's the tradeoff of Firehose's buffering that catches teams off guard?

Firehose is near-real-time, not real-time — data sits in the buffer until the size or time threshold is hit (minimum buffer interval is 60 seconds), so there's an inherent delay before it lands in the destination. Teams expecting sub-second delivery for a live dashboard pick Firehose by mistake instead of consuming a Data Stream directly; Firehose is the right choice for batch-oriented sinks like S3-backed data lakes or Redshift loads, not for use cases needing immediate visibility into each event.