At a glance
- Did you know
- If a consumer does not finish within the visibility timeout, the message is delivered again.
- Rule of thumb
- Send messages that keep failing to a dead-letter queue instead of retrying forever.
- Watch out
- Redelivery duplicates and a growing backlog.
Problems that start with it
- Payment webhook processor 500 webhooks/s through an at-least-once queue. 0.2% of acknowledgements are lost.
- YouTube end to end 20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.
- Document Q&A on S3 and a vector DB 40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.
- Duplicate charges Payment events go through a queue that delivers at least once. Some acknowledgements are lost, so some customers are charged twice.
- Events that never arrive The API writes the order to the database, then publishes an event. At 10 s the queue is unavailable for 5 seconds: orders are saved but their events vanish.
- Uploads starve questions Questions and document ingestion share one embedding quota. Uploads double and both paths start failing.
- Orders: lost events and double emails The order API writes the order, then publishes an event (dual write). At 10 s the queue is down for 5 seconds. The workers that send confirmation emails lose 2% of their acknowledgements and are not idempotent.
- AI feedback: thumbs-up events go missing Every thumbs-up or thumbs-down on an AI answer is saved, then published as an event for the offline evaluation and training pipeline. At 10 s the queue is down for 5 seconds: the votes are saved but their events vanish, so the next release is evaluated on biased data.
And 6 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.