Choose Buffer if you care about…
…moving data without operating a broker. The data path is your bucket: the whole system is a library plus a manifest file on object storage. There is no service to provision or operate. …data transport at S3 prices. Our benchmark moved 30 MiB/s for ~$91/month of object-storage charges. Public calculators price the same throughput at roughly $1,300/month on managed WarpStream and $8,500/month self-hosting Kafka. …highly available pipelines. Producers are stateless: if one dies, another can take over, with no rebalancing protocol. If the downstream database is slow or down, batches accumulate safely in object storage instead of back-pressuring your apps or getting dropped. And zonal producers mean ingest never crosses AZs, resulting in zero cross-AZ transfer fees. …deduplicating data on write. Buffer maintains unique identities for each batch written to a sink that are stable across retries. This makes deduplication on write possible if you need those semantics. …gigabit-scale throughput. Pipelined producers and consumers keep many object-store requests in flight at once. We demonstrated ~1 Gbps of log ingestion into ClickHouse with two nodes, each with 4 vCPUs and 8 GB of RAM. …observability pipelines specifically. An included OpenTelemetry exporter ships MELT data from any OTel Collector through Buffer. A bundled ClickHouse sink can read from Buffer and write to ClickHouse. OpenData Log and Vector can ingest from Buffer natively.How the alternatives stack up
✓ = yes · ~ = partially · ✗ = no| You care about… | Buffer | Kafka (self-hosted / MSK) | WarpStream / AutoMQ / Bufstream | Kinesis | SQS | DIY S3 staging |
|---|---|---|---|---|---|---|
| No broker or service in the data path | ✓ | ✗ | ~ ¹ | ✗ | ✗ | ✓ |
| Cost ≈ S3 requests | ✓ ² | ✗ | ~ ² | ✗ | ✗ | ✓ |
| Stateless producers, zero cross-AZ fees | ✓ | ✗ | ✓ | ~ | ~ | ✓ |
| Deduplication on write | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ ³ |
| Multi-Gbps pipeline throughput | ✓ | ✓ | ✓ | ~ ⁴ | ✗ ⁴ | ~ ³ |
| License | MIT | Apache 2.0 | Proprietary / BSL | Proprietary | Proprietary | n/a |
² At 30 MiB/s: ~$91/mo of S3 charges on Buffer vs. ~$1,300/mo (WarpStream’s calculator) and ~$8,500/mo (2minutestreaming’s Kafka calculator).
³ The closest competitor: producers writing files to a prefix and a consumer listing them. Same architecture and economics, but you’d have to solve the coordination between producer and consumer, write pipelined clients, etc.
⁴ Kinesis meters and throttles per shard, so Gbps means big shard fleets and bills; SQS is per-message with size limits. It’s built for task queues rather than bulk ordered transport. The systems with Buffer’s reliability semantics all put a broker tier in the path; the only alternative at similar cost is building the coordination yourself. If your pipeline has fixed producers and a fixed consumer, which describes many telemetry ingestion paths, the broker isn’t doing necessary work.
The tradeoffs
- Latency is sub-second to seconds: p50 under 0.5s / p99 ~2s at 30 MiB/s, tunable toward a ~50–100ms floor at higher request cost. Millisecond delivery needs a broker.
- Participants are finite: throughput contends on one compare-and-set manifest, so run one producer per zone and a small fixed consumer set.
- Rust only: Buffer is a crate. The OTel exporter covers the most common producer without writing Rust, but there are no Java/Python/Go clients yet. This is the biggest practical limitation.
- At-least-once, per-producer ordering: global total order and transactional produce are non-goals. Exactly-once is an effect of deterministic replay plus an idempotent sink.
Last reviewed July 2026. Please tell us if a claim has gone stale.