Problems / Multi-agent and harness, medium
Long-running coding agents run on a durable workflow engine. Every step passes the whole conversation in and out, so the first step records 60 KB and each later one 40 KB more than the last. The engine terminates any run whose history passes 50 MB. Tasks take 50 steps.
Unlock it once for 15 credits; everything inside it is free (runs, changes, Submit, results, hints). Sign in with Google to start; new accounts get free credits.
Success ≥ 95% of tasks, p99 ≤ 90 s.
Like a coding problem, a solution has to pass every case, not just the one it was built for. Submit in the lab runs all 6; 2 of them are hidden until then.
1 requests a second for 30 seconds: 100% task (writes).
What each component does in this design, and what calls it. The simulator mimics each one from these settings.
| Component | Its job here | Settings |
|---|---|---|
| Load balancerLoad balancer | Traffic enters here.Calls Agent API.Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header. | – |
| Agent APIApp service | Load balancer forwards to it.Calls Durable workflow (Temporal-style).Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s. | 3 replicas × 500 slots · timeout 180 s |
| Durable workflow (Temporal-style)Durable workflow engine | Agent API calls it.Calls Model, MCP tool servers.Workflow engines like Temporal persist every step, so long processes survive crashes. | 400 slots · 50 steps |
| ModelLLM endpoint | Durable workflow (Temporal-style) calls it.Latency has two parts: time to the first token, then a steady stream of tokens per second. | 2000 streams |
| MCP tool serversTool / MCP server | Durable workflow (Temporal-style) calls it.Tools give the model actions such as search or database queries; each call is another round trip. | 2000 slots |