Build an LLM inference API like the Claude API: the full interview
Design the platform behind an LLM API the way Anthropic, Google and Meta ask it: streamed answers with a fast first token, long-context calls on a cached prefix, per-org rate limits, multi-region GPU fleets, a Batch API, exactly-once usage billing, safety classifiers and a developer console, then survive launch day, a lost GPU region, bots, a jailbreak campaign, a usage-stream outage, a huge batch job and a DDoS attack.
Unlock it once for 25 credits; everything inside it is free (runs, changes, Submit, results, hints). Sign in with Google to start; new accounts get free credits.
What it must do
Messages API: take a prompt (system prompt, tools, conversation) and stream the model’s answer token by token; the first token should arrive within a second for chat.
Long-context agent calls: prompts of tens of thousands of tokens that repeat a large prefix (tools, documents, the conversation so far); reuse the cached prefix instead of recomputing it.
Per-organisation rate limits on requests and tokens a minute, answered with a fast 429 and a retry-after, so no single customer (or launch day) can take the fleet down.
Serve from more than one region and fail over when a GPU cluster fails.
Batch API: accept a file of prompts, return a batch id at once, run it on spare capacity at half price, and put the results in object storage within 24 hours.
Meter usage: every input and output token is billed exactly once, never lost and never double counted.
Safety: classify prompts for injection and policy before they reach a GPU; keep bots, stolen keys and floods out at the edge.
A developer console (a web app served from a CDN) with usage dashboards, and cheap metadata endpoints (count tokens, list models).
Scale
This simulation runs a scaled-down slice: 500 requests a second, of which 250 chat messages, 75 long-context agent calls, 25 Batch API submissions, 75 metadata calls (token counting, model list), 50 console page loads and 25 usage-dashboard reads.
A chat message sends about 3,000 tokens (about three quarters of them a system prompt and earlier turns that the prefix cache usually holds) and streams about 300 tokens back at about 90 tokens a second.
A long-context agent call sends about 40,000 tokens (30,000 of them a cached prefix of tools, documents and history) and streams about 300 tokens back at about 80 tokens a second.
The full-scale numbers in the estimates assume 2 billion model requests a day, averaging 3,000 input and 400 output tokens, served by a 70-billion-parameter-class model on 8-GPU servers.
A new model launch can triple traffic within minutes.
Request types to route: messages (50%, write), agent (15%, write), batch (5%, write), meta (15%, read), console (10%, read), usage (5%, read). Budget: $18,000,000 a month.
Estimate first
Model requests a second, on average, at full scale (req/s)
Output tokens generated a second at full scale (tokens/s)
8-GPU servers needed for decode at a 2× daily peak, if one server sustains 25,000 output tokens a second with continuous batching (servers)
KV-cache memory for one 100,000-token context on a 70B-class model (80 layers, 8 KV heads of 128 dimensions, 16-bit values) (GB)
Share of prefill work saved on agent calls by the prefix cache (40,000-token prompts, 30,000 cached, 90% hit rate) (%)
Chat answers streaming at once in the simulated slice (streams)
Usage records written a second at full scale (records/s)
Monthly cost of the reference design (the simulated slice), at the lab’s default token prices ($3 a million input, $15 a million output, cached input at a tenth) (USD)
The test cases
You start from an empty canvas. When you run the tests, each one simulates your design with its own traffic and failures:
Test
What it does
Passes when
Normal day
The normal day: the traffic in the brief, nothing broken.
Success ≥ 99%; Success ≥ 99% (messages); First token p99 ≤ 1.5 s (messages); First token p99 ≤ 4.0 s (agent); p99 latency ≤ 150 ms (meta); p99 latency ≤ 400 ms (console); Success ≥ 99% (batch); p99 latency ≤ 300 ms (batch)
Launch day: 3× traffic
At 5 s a new model launches: all traffic triples for 8 s, far past what the GPU fleets can serve.
Success ≥ 99% (meta); p99 latency ≤ 300 ms (meta); Success ≥ 99% (batch); First token p99 ≤ 2.5 s (messages); Success ≥ 70%
A GPU region goes dark
At 5 s the primary chat GPU fleet fails (a power event takes out the region’s GPU cluster) and stays down for its restart time; every answer streaming from it is cut off.
Success ≥ 98% (messages); First token p99 ≤ 2.5 s (messages)
Bots on stolen API keys
At 5 s 3,000 requests a second from a botnet arrive at the Messages API for 10 s, sent with leaked and stolen API keys, alongside real traffic.
Attack blocked ≥ 90%; Success ≥ 99%; First token p99 ≤ 1.5 s (messages); First token p99 ≤ 4.0 s (agent)
A jailbreak campaign
At 5 s 400 requests a second of prompt-injection and jailbreak attempts arrive for 10 s, sent with real API keys alongside real traffic.
Attack blocked ≥ 90%; Success ≥ 99%; First token p99 ≤ 1.5 s (messages); First token p99 ≤ 4.0 s (agent)
The usage stream goes down
At 5 s the usage event stream crashes; it comes back after its restart time (10 s unless you set one). While this test runs, brokers redeliver unacknowledged messages after 5 s and 0.2% of acknowledgements are lost, as in production: only idempotent consumers avoid duplicates.
No lost events; No duplicate side effects; Success ≥ 99% (messages)
A runaway agent loop
At 5 s one customer’s agent gets stuck in a loop: long-context agent calls jump 5× for 8 s; every other request type is unchanged.
Success ≥ 99% (messages); First token p99 ≤ 1.5 s (messages)
A huge batch job lands
At 5 s a customer submits a batch of millions of prompts: the batch workers take 10× longer for 10 s.
Success ≥ 99% (batch); p99 latency ≤ 300 ms (batch); First token p99 ≤ 1.5 s (messages); First token p99 ≤ 4.0 s (agent)
DDoS attack
At 5 s 30,000 requests a second of flood traffic arrive for 10 s, alongside real users.
Attack blocked ≥ 95%; Success ≥ 99%; First token p99 ≤ 1.5 s (messages); First token p99 ≤ 4.0 s (agent)
Budget: $18,000,000/month
The normal day's design must cost at most $18,000,000 a month.
Cost ≤ $18,000,000/month
Must include
DDoS protection at the edge. A public API with a free tier is flooded all the time; floods must be absorbed before anything spends compute on them.
An API gateway with API-key auth. Every call carries an API key: authenticate it, find the organisation and its limits, and route by endpoint before any expensive work.
Rate limits (admission control). Requests and tokens a minute per organisation, enforced before a request takes a GPU slot; past the limit answer 429 with a retry-after instead of queueing until everything times out.
A safety classifier before the model. Prompts are classified for injection, jailbreaks and policy before they reach a GPU: a blocked prompt should cost a classifier call, not a model call.
GPU serving fleets. The models run on GPU servers with continuous batching; chat, long-context and batch traffic have different shapes and are served from separate pools.
A queue for the Batch API. Batch jobs are accepted at once and run later on spare capacity; a queue decouples submission from inference.
A usage event stream. Usage is the largest write stream (one record per request with its token counts); billing, dashboards and abuse detection all consume it.
A billing ledger. Invoices are money: usage lands in an append-only ledger keyed by request id, so a replayed event can never be billed twice.
Object storage for files and batch results. Batch input files and results are large blobs that live for days; they belong in object storage, not a database.
A CDN for the console. The console is a web app; its assets and cached pages come from a CDN, never from the API servers.