Solved design · 11 of 12 · 10 credits, once

Twitter / X timeline

Home timelines from fan-out on write behind a large Redis cache, a ranker on every load, tweets into Kafka, search on an inverted index.

Template
Twitter / X timeline
Level
Advanced
Reading time
18 min
Topics
Home timelineFan-out on writeRedis at scaleRankingSearch indexKafka

In the lab Home timelines from fan-out on write (Manhattan) behind a big Redis cache, a ranker on every load, tweets into Kafka, search on an inverted index. It meets its goals as given (success ≥ 99%, p99 ≤ 500 ms, Apdex ≥ 0.90). Open it to see how it is put together, then push it: 3x traffic, an attack, a flush, a dead replica, and fix what breaks.

1Context and scope

The home timeline is read tens of times for every tweet written, so the classic design precomputes it: a tweet is pushed into each follower’s timeline when it is posted, and loading the timeline is one read plus a ranking pass. Search is a separate path on an inverted index. This design covers home loads, tweets and searches at browser scale, based on Twitter’s engineering write-ups about the timeline, Manhattan and Earlybird.

2Goals and non-goals

Goals

  • 6,000 requests a second: 80% home timeline loads, 10% tweets, 10% searches.
  • Success at least 99%, p99 at most 500 ms, Apdex at least 0.90.
  • Tweets reach timelines and the search index through Kafka with nothing lost.

Non-goals

  • Celebrity fan-out on read (the hybrid for accounts with millions of followers).
  • Trends, ads and direct messages.
  • Media storage and the CDN.

3Capacity estimate

QuantityValueHow
Home loads≈ 4,800/s80% of 6,000
Timeline cache64 GB, 30M timelines, 10 min TTL, 1M ops/sactive accounts are a small share, so most loads hit
Ranker1,024 slots, 96 vCPU, 4 ms CPU4,800 × 4 ms ≈ 19 vCPU busy
Tweets≈ 600/sone MySQL write, one Kafka event each
Fan-out256 workerseach tweet becomes a timeline write per follower plus one index write
Searches≈ 600/s512 slots, 4 ms CPU: 2.4 vCPU busy on 48

The rest of the doc

The design · Key concepts · Alternatives considered · Failure modes and fixes · Try it in the lab · Check yourself · Further reading

Solved design · 10 credits, once

Unlock the Twitter / X timeline design doc

Every component and why it is there, the capacity numbers behind it, what breaks when you push it and how to fix it, and the same system open in the lab to break and fix yourself. Paid once, on your account.