Problems / Real world, hard

ChatGPT: from launch build to real traffic

A chat product on launch-week hardware: the GPU prefill and decode pools, the KV cache store, moderation and the chat API are an eighth of the size real traffic needs. 300 chats a second arrive with long prompts. Answers queue for 20 seconds and most fail.

Solve it in the labTake it as a mock interview

Unlock it once for 25 credits; everything inside it is free (runs, changes, Submit, results, hints). Sign in with Google to start; new accounts get free credits.

Your goals

Grow it into the real architecture: every goal must hold at 300 requests a second. You may add components, connect them and resize anything.

Test cases

Like a coding problem, a solution has to pass every case, not just the one it was built for. Submit in the lab runs all 6; 2 of them are hidden until then.

  1. As given. The problem’s own traffic, scheduled failures and seed.
  2. Peak hour. 25% more traffic than the problem states, with the same failures.
  3. A slow dependency. The main data store or model answers twice as slowly for part of the run.
  4. A server dies. One replica of the service with the most replicas is killed for part of the run.

The traffic

300 requests a second for 20 seconds: 90% chat (reads), 10% history (reads).

The system

What each component does in this design, and what calls it. The simulator mimics each one from these settings.

ComponentIts job hereSettings
DDoS protectionDDoS protectionTraffic enters here.Calls Bot management.Volumetric floods are absorbed at the network edge, spread over many locations, before they reach you.100000 slots
Bot managementBot managementDDoS protection forwards to it.Calls API gateway.Bot managers score clients by behaviour and fingerprint, not by IP address alone.100000 slots
API gatewayAPI gatewayBot management forwards to it.Calls Spend governor, History API.Gateways centralise auth, rate limits and routing so each service does not reimplement them.50000 slots · 600,000 calls/min
Chat APIApp serviceSpend governor forwards to it.Calls Auth and sessions, Conversation store, Moderation model, KV cache tier (prefix reuse).Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.1 replicas × 500 slots · timeout 30 s
Auth and sessionsAuth / identity serviceChat API calls it.Short-lived access tokens plus refresh tokens limit the damage if a token leaks.250 slots
Conversation storeDocument databaseChat API calls it. History API calls it.Document databases store nested JSON-like records, so one read can fetch a whole aggregate.100 slots
Moderation modelLLM-as-judgeChat API calls it.An LLM judge grades outputs at scale, but it is another model call with its own latency, cost and biases.75 slots
Spend governorSpend governorAPI gateway forwards to it for /chat.Calls Chat API.A spend governor enforces budgets per user, task or team before calls go out.12500 slots · 7,500 calls/min
KV cache tier (prefix reuse)KV cache tierChat API calls it.Calls Prefill GPU pool.Reusing the KV cache of a shared prefix (system prompt, chat history) skips most of the prefill work.500 slots · 60% hit rate
Prefill GPU poolPrefill GPU poolKV cache tier (prefix reuse) calls it.Calls Decode GPU pool.Splitting prefill (reading the prompt) from decode (writing the answer) onto separate GPUs stops long prompts from stalling everyone else’s output.50 slots
Decode GPU poolDecode GPU poolPrefill GPU pool calls it.Decoding is memory-bound: each request holds its KV cache in GPU memory for as long as it generates.300 slots
History APIApp serviceAPI gateway forwards to it for /history.Calls Conversation store.Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.1 replicas × 100 slots · timeout 1.0 s

More real world problems