At a glance
- Did you know
- Queues deliver at least once, so a worker can see the same message twice.
- Rule of thumb
- Make handlers idempotent: record a processed-message id before doing side effects.
- Watch out
- Duplicate side effects, and a backlog when it is slower than producers.
Problems that start with it
- Payment webhook processor 500 webhooks/s through an at-least-once queue. 0.2% of acknowledgements are lost.
- Event pipeline catching up 2,000 events/s into a 6-partition stream with 30 s retention. 3 consumers.
- YouTube end to end 20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.
- Document Q&A on S3 and a vector DB 40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.
- Duplicate charges Payment events go through a queue that delivers at least once. Some acknowledgements are lost, so some customers are charged twice.
- Events that never arrive The API writes the order to the database, then publishes an event. At 10 s the queue is unavailable for 5 seconds: orders are saved but their events vanish.
- Consumers can’t keep up Kafka keeps events for 30 seconds. One consumer processes them, slower than they arrive.
- Too few partitions The topic has 2 partitions. Each partition is read by one consumer at a time, however many you add.
And 20 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.