At a glance
- Did you know
- Agents often repeat the same read-only tool call within and across tasks.
- Rule of thumb
- Cache only read-only tools, and key the cache on normalised arguments.
- Watch out
- Stale results; low hit rate when arguments vary.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
- Hit rate
- Share of calls answered here without calling what sits behind it. Every miss is a call to the next component.
Problems that start with it
- Rerank results never reused A cache in front of the reranker hits 15%, because it is keyed on the raw question text including whitespace and casing.
- Same tool call, every step Agents repeat the same read-only lookups, but the tool-result cache hits 10% because it is keyed on raw arguments with timestamps.
- A new browser for every page Research agents read about 8 web pages per task, 30 tasks a minute. The hosted-browser plan allows 100 sessions at once but only 50 new sessions a minute, and the browser tool opens a fresh session for every page.
- The CRM allows 25 report queries A revenue-ops agent runs CRM reports through an MCP server. The CRM allows 25 report queries at once and rejects the rest immediately, with no queue. Each report takes about 2.4 s, and the agents ask for 12 a second.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.