Problems / LLM serving, medium
A router splits traffic between a large model and a small one, but it is set to send 90% of chats to the large model, which has 200 concurrent streams. 60 chats arrive a second.
Unlock it once for 15 credits; everything inside it is free (runs, changes, Submit, results, hints). Sign in with Google to start; new accounts get free credits.
Success ≥ 99%.
Like a coding problem, a solution has to pass every case, not just the one it was built for. Submit in the lab runs all 6; 2 of them are hidden until then.
60 requests a second for 30 seconds: 100% request (reads).
What each component does in this design, and what calls it. The simulator mimics each one from these settings.
| Component | Its job here | Settings |
|---|---|---|
| Model routerModel router | Traffic enters here.Calls Large-model path, Small-model path.A router sends easy requests to small, cheap models and hard ones to larger models. | 100000 slots |
| Large-model pathApp service | Model router forwards to it 90% of traffic.Calls Large model.Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s. | 3 replicas × 500 slots · timeout 30 s |
| Large modelLLM endpoint | Large-model path calls it.Latency has two parts: time to the first token, then a steady stream of tokens per second. | 200 streams |
| Small-model pathApp service | Model router forwards to it 10% of traffic.Calls Small model.Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s. | 3 replicas × 500 slots · timeout 30 s |
| Small modelLLM endpoint | Small-model path calls it.Latency has two parts: time to the first token, then a steady stream of tokens per second. | 600 streams |