At a glance
- Did you know
- Kafka keeps order only within a partition, so the partition key decides what stays in sequence.
- Rule of thumb
- Consumers beyond the partition count sit idle: add partitions to scale reads.
- Watch out
- Consumer lag, and data lost once it passes retention.
Problems that start with it
- Event pipeline catching up 2,000 events/s into a 6-partition stream with 30 s retention. 3 consumers.
- YouTube end to end 20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.
- Consumers can’t keep up Kafka keeps events for 30 seconds. One consumer processes them, slower than they arrive.
- Too few partitions The topic has 2 partitions. Each partition is read by one consumer at a time, however many you add.
- Two partitions, slow consumers The topic has 2 partitions and the consumer group processes each event in about 5 ms. Events arrive at 2,000 a second and are kept for 30 seconds.
- News feed: fan-out workers fall behind New posts go into a stream; fan-out workers copy each one into followers’ feeds. One worker can’t keep up, and the stream only keeps 30 seconds.
- Maps ETA: traffic events on two partitions Speed reports from phones feed the ETA model through a topic with 2 partitions. Each partition is read by one consumer at a time.
- Real-time analytics: consumers lag Clickstream events pour into a topic that keeps 30 seconds of data. The analytics consumers process them slower than they arrive.
And 9 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.