1Context and scope
An ingest API publishes events to a stream; a consumer group processes them. The stream keeps events for 30 seconds. If consumers fall behind by more than that, events are deleted before anyone reads them.
2Goals and non-goals
Goals
- 2,000 events a second into 6 partitions with 30 s retention.
- No data lost to retention (critical); success at least 99.9%.
Non-goals
- Sub-second end-to-end latency: a few seconds of lag is fine while nothing is lost.
- Exactly-once side effects in external systems.
- Keeping events for long: a warehouse or object store does that.
3Capacity estimate
Back-of-the-envelope numbers from the template. The simulation uses the same ones, so you can check them against a run.
| Quantity | Value | How |
|---|---|---|
| Produced | 2,000 events/s | |
| Per consumer | ≈ 400 events/s | 2.5 ms each |
| Consumed with 3 consumers | ≈ 1,200/s | falls 800/s behind |
| Time to lose data | < 1 minute | lag passes 30 s of retention |
| Max useful consumers | 6 | one per partition |
4The design
Each component in the template, in the order a request meets it, and why it is there.
- Load balancer and Ingest API
Accepts events and publishes them; the API stays fast because consumers work asynchronously.
- Event stream (6 partitions, 30 s retention)
An ordered log per partition. Order is kept within a partition, and each partition is read by one consumer in the group at a time.
- Consumer group
Processes events. Its throughput must beat the produce rate or lag grows.
5Key concepts
Lag
Lag is how far consumers are behind the newest event. Measure it as time (lag ÷ produce rate) and compare with retention: when lag approaches retention, data is about to be lost.
Partitions cap parallelism
In a consumer group each partition is assigned to one consumer. With 6 partitions, a 7th consumer sits idle. To go wider you must add partitions (which you cannot easily take away).
Retention is a deadline
Events older than retention are deleted whether or not they were read. Longer retention buys time to recover but costs storage.
Retries hold the partition
A consumer that fails on a message retries the same offset; the partition waits. One poison message can stall a partition, so cap retries and park bad messages.
6Alternatives considered
- More partitions allow more consumers but cost ordering across keys and are hard to reduce later.
- Longer retention protects against outages at the price of storage.
7Failure modes and fixes
What goes wrong in the simulation, why, and the change that fixes it. Each one shows up as a finding in the lab; ask “what broke?” to see it, and for the fix when you want it.
Lag grows steadily and events are lost to retention.
Why3 consumers process about 1,200 events a second against 2,000 produced.
FixRun 6 consumers (one per partition). Beyond that, add partitions; raise retention for safety.
8Try it in the lab
Open the template, then type these into the copilot one at a time. The first few are also offered as buttons when the template opens.
set consumers to 6Consumers keep up; lag stays flat and nothing is lost.
set consumers to 12No better than 6: only 6 partitions to share out.
double the partitionsNow more consumers would help.
what broke?The backlog and data loss findings.
9Check yourself
- How do you turn lag in messages into lag in seconds, and why compare it with retention?
- Why do consumers beyond the partition count not help?
- What happens to a partition when one message keeps failing?