At a glance
- Did you know
- An agent loops: think, call a tool, read the result, repeat until done.
- Rule of thumb
- Cap steps and tokens per task; runaway loops are the costliest failure.
- Watch out
- Call amplification; runaway loops burn quota and money.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
- Calls per request (fan-out)
- How many times its outgoing calls run per request: agent steps, jobs, subagents. Load on everything behind it is multiplied by this, and any step failing fails the request.
Problems that start with it
- Agent runaway 15 tasks/s. The agent makes 8 model and tool calls per task; the model allows 4,000 calls a minute and 120 concurrent streams.
- MCP agent platform 4 tasks/s. The agent thinks (model) and acts (tools through an MCP gateway) 6 times per task, with vector memory and results in S3.
- The agent that thinks too much An agent takes 12 model-and-tool steps per task. Steps multiply load, cost and failures.
- Ten steps of MCP tools Agents call MCP tools 10 times per task. A 1% tool error rate becomes a 10% task failure rate.
- Agent: a slow tool and a tight quota An agent takes 6 steps per task at 15 tasks a second. Every step calls a tool server with 8 slots that takes 300 ms, and the model allows 4,000 requests a minute.
- Multi-hop retrieval loops An agentic retriever makes up to 8 search hops per question, and each hop hits the search service.
- An agent that takes 12 steps A support agent takes 12 model-and-tool steps per ticket. Tools fail 1% of the time, and every failure sinks the whole task.
- MCP gateway quota Every tool call goes through an MCP gateway (auth, allow-lists, quotas) limited to 600 calls a minute. Agents make about 18 tool calls a second.
And 45 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.