1Context and scope
A payment provider calls your webhook endpoint for every payment event. You enqueue each event and a worker charges the customer. The queue delivers at least once: if a worker’s acknowledgement is lost, the message is delivered again and the customer can be charged twice.
2Goals and non-goals
Goals
- 500 webhooks a second.
- Zero duplicate charges (critical).
- Success at least 99%.
Non-goals
- Processing the card itself: the payment provider does that.
- Exactly-once delivery from the provider, which no provider offers. The goal is an exactly-once effect.
- Real-time reporting and reconciliation.
3Capacity estimate
Back-of-the-envelope numbers from the template. The simulation uses the same ones, so you can check them against a run.
| Quantity | Value | How |
|---|---|---|
| Events | 500/s | |
| Lost acknowledgements | 0.2% | about 1 event a second redelivered |
| Duplicate charges without idempotency | ≈ 1 a second | over a day, tens of thousands |
4The design
Each component in the template, in the order a request meets it, and why it is there.
- Load balancer and Webhook API
Receives the provider’s calls, validates them and puts each event on the queue, then answers 200 quickly so the provider does not retry.
- Payment queue (SQS-style)
Buffers events and delivers each to a worker. A message that is not acknowledged within the visibility timeout is delivered again.
- Payment worker
Performs the side effect (the charge). It must be safe to run twice for the same event.
5Key concepts
At-least-once is the norm
Webhook providers and queues retry until they get an acknowledgement. Networks lose acknowledgements, so duplicates are normal, not a bug. Exactly-once delivery is not available; exactly-once effects are.
Idempotency keys
Give every operation a unique key (the provider’s event id). Before doing the side effect, record the key; if it is already there, skip the work and return the stored result. Duplicates become no-ops.
Record before you act
Write the key in the same transaction as the effect, or before calling the external service with the same key, so a crash between the two cannot leave a charge without a record.
Visibility timeout
How long a delivered message stays hidden. Too short and slow workers see duplicates; too long and a crashed worker delays the retry.
6Alternatives considered
- A key registry adds a lookup per message and storage for keys (kept for at least as long as retries can happen).
- At-most-once delivery avoids duplicates by risking lost payments, which is usually worse.
7Failure modes and fixes
What goes wrong in the simulation, why, and the change that fixes it. Each one shows up as a finding in the lab; ask “what broke?” to see it, and for the fix when you want it.
Duplicate charges appear and the critical goal fails.
WhyLost acknowledgements make the queue redeliver events that were already processed, and the worker charges again.
FixTurn on idempotency keys for the worker (a key registry): duplicates are recognised and skipped.
8Try it in the lab
Open the template, then type these into the copilot one at a time. The first few are also offered as buttons when the template opens.
turn on idempotencyDuplicate side effects drop to zero; compare before and after.
what broke?The duplicate side effects finding and its fix.
traffic spike 3xMore events, more redeliveries: the problem scales with traffic.
9Check yourself
- Why can a queue not guarantee exactly-once delivery?
- What should the idempotency key be for a payment webhook, and how long must you keep it?
- Why record the key before (or with) the side effect rather than after?