Solved design · 04 of 12 · 10 credits, once

Slack

Messages into sharded MySQL and a Kafka stream, real-time fan-out to channel pub/sub, a message search index, presence heartbeats.

Template
Slack
Level
Advanced
Reading time
18 min
Topics
Real-time messagingFan-outPub/subShardingSearchPresence

In the lab Messages into sharded MySQL and a Kafka stream, real-time fan-out to channel pub/sub, a search index, presence heartbeats. It meets its goals as given (success ≥ 99%, p99 ≤ 500 ms, Apdex ≥ 0.90). Open it to see how it is put together, then push it: 3x traffic, an attack, a flush, a dead replica, and fix what breaks.

1Context and scope

A message sent to a channel has to be stored durably, shown to everyone in the channel within a second, indexed for search and, for people who are away, pushed to their phone. Presence (who is online) is a constant trickle of heartbeats. This design covers sending and reading messages, the real-time fan-out and presence, based on Slack’s engineering posts about its messaging stack and Vitess, scaled to a browser run.

2Goals and non-goals

Goals

  • 5,000 requests a second: 45% history reads, 40% sends, 15% presence heartbeats.
  • Success at least 99%, p99 at most 500 ms, Apdex at least 0.90.
  • Every send is stored and fanned out; redeliveries never duplicate.

Non-goals

  • WebSocket connection handling itself (the pub/sub stands in for the edge servers).
  • Files, calls and apps.
  • Enterprise grid and multi-region.

3Capacity estimate

QuantityValueHow
Sends≈ 2,000/s40% of 5,000: one MySQL write and one Kafka event each
Messages database600 connections, 48 vCPU2,000 × 10 ms ≈ 20 connections busy on writes
Fan-out256 workers at ≈ 3 mseach message: a pub/sub publish, an index write, and a push for 30%
History reads≈ 2,250/sthrough a 32 GB cache of recent messages, 15 min TTL
Presence≈ 750/s5,000 slots at 1 ms: nowhere near its limit
Search index writes≈ 2,000/s256 slots, 3 ms CPU on 32 vCPU

The rest of the doc

The design · Key concepts · Alternatives considered · Failure modes and fixes · Try it in the lab · Check yourself · Further reading

Solved design · 10 credits, once

Unlock the Slack design doc

Every component and why it is there, the capacity numbers behind it, what breaks when you push it and how to fix it, and the same system open in the lab to break and fix yourself. Paid once, on your account.