1Context and scope
A chat product serves long conversations to many people at once. Every message is checked by a moderation model, the conversation is loaded from a store, the prompt is run through prefill (reading the prompt) and decode (generating tokens), and a spend governor keeps a runaway client from burning the budget. Serving is split into separate GPU pools for prefill and decode, with a KV cache tier that lets repeated prefixes skip prefill. This design is based on public LLM serving research (DistServe, Mooncake) and provider documentation, scaled to a browser run; chat requests are judged on full response time.
2Goals and non-goals
Goals
- 300 requests a second: 90% chat messages, 10% history loads.
- Success at least 99%, p99 at most 10 s for the full answer.
- Bots and floods are stopped at the edge; real users are not.
Non-goals
- Training and fine-tuning.
- Billing, plans and sign-up.
- Voice and image inputs.
3Capacity estimate
| Quantity | Value | How |
|---|---|---|
| Chat messages | ≈ 270/s | 90% of 300 |
| Moderation | 600 slots at 60 ms | 270 × 0.06 ≈ 16 slots busy |
| KV cache hits | 60% of prompts | repeated prefixes skip prefill entirely |
| Prefill | ≈ 108/s on 400 slots at 250 ms | 108 × 0.25 ≈ 27 slots busy |
| Decode | ≈ 270/s on 2,400 slots at 2.5 s | 270 × 2.5 ≈ 675 slots busy: decode is where the GPUs go |
| Spend governor | 60,000 calls a minute | 270 × 60 = 16,200: far below the cap at normal traffic |
The rest of the doc
The design · Key concepts · Alternatives considered · Failure modes and fixes · Try it in the lab · Check yourself · Further reading
Unlock the ChatGPT design doc
Every component and why it is there, the capacity numbers behind it, what breaks when you push it and how to fix it, and the same system open in the lab to break and fix yourself. Paid once, on your account.