Design ChatGPT from an empty canvas: streamed answers from a large model with quotas, conversations kept as context, and no outage when the model provider fails.
Unlock it once for 25 credits; everything inside it is free (runs, changes, Submit, results, hints). Sign in with Google to start; new accounts get free credits.
What it must do
Stream the model’s answer token by token; the first token should appear within a second or two.
Keep every conversation and load it as context for the next turn; list past conversations.
Enforce per-user and per-provider quotas, and keep answering when the main model fails or is rate limited.
Avoid paying twice for the same tokens (prompt and prefix caching).
Scale
20 requests a second: 17 chat turns and 3 history loads (about 1.5 million turns a day).
A turn sends about 3,000 tokens (system prompt and earlier turns, 2,500 of them usually a cached prefix) and gets about 250 tokens back, streamed at 80 tokens a second.
The main provider allows 1,500 calls a minute (25 a second); a launch can triple traffic.
Request types to route: chat (85%, write), history (15%, read). Budget: $210,000 a month.
Estimate first
Chat turns a day (million)
Output tokens a day (million)
Answers streaming at once (streams)
Monthly cloud cost of your design (USD)
The test cases
You start from an empty canvas. When you run the tests, each one simulates your design with its own traffic and failures:
Test
What it does
Passes when
Normal day
The normal day: the traffic in the brief, nothing broken.
Success ≥ 99%; First token p99 ≤ 2.0 s; p99 latency ≤ 400 ms (history)
A viral launch
At 5 s a launch goes viral: traffic triples for 8 s, past the main provider’s quota.
Success ≥ 98%; First token p99 ≤ 3.0 s
The model provider goes down
At 5 s the main model endpoint fails and stays down for 10 s; in-flight answers are cut off.
Success ≥ 98%; First token p99 ≤ 3.0 s
Budget: $210,000/month
The normal day's design must cost at most $210,000 a month.
Cost ≤ $210,000/month
Must include
An AI gateway. One door for keys, per-user quotas and routing to the models.
A conversation store. Every turn is saved and the conversation is loaded as context for the next one.
Model serving. The answer comes from a large model with its own quotas and streaming limits.
A prompt cache. Repeated prompts are answered without a model call at all.