At a glance
- Did you know
- Third-party APIs have their own rate limits and outages that you cannot fix.
- Rule of thumb
- Always set a timeout, retry with backoff, and cache what you can.
- Watch out
- Quota 429s; outage takes your feature down with it.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- Payment provider outage Checkout calls an external payment API. At 10 s it goes down for 15 seconds.
- Checkout: retries into an outage Checkout calls one payment provider. The API retries failed calls 3 times immediately. At 10 s the provider is down for 15 seconds, and the retries triple the pile-up.
- Shopping agent: payments down A shopping agent checks out through an external payment API that is down from 15 s to 30 s.
- Background agents poll GitHub until throttled Background coding agents open a pull request, then wait for CI by polling GitHub every 10 s, about 12 checks per task. All agents share one GitHub App installation, limited to 15,000 requests an hour (250 a minute). 70 tasks start a minute.
- The CRM allows 25 report queries A revenue-ops agent runs CRM reports through an MCP server. The CRM allows 25 report queries at once and rejects the rest immediately, with no queue. Each report takes about 2.4 s, and the agents ask for 12 a second.
- A filter is not a permission An ops copilot reads public issues (untrusted), queries private repos and the customer database (private data), and posts comments, PRs and webhooks (outbound). That is the lethal trifecta. A prompt-injection classifier that stops 97% of attacks guards it, and attackers file 50 poisoned issues a second for 15 seconds.
- Stripe: from launch build to real traffic A payments API on its first servers: no charge status cache, and the charges API, status API, fraud scoring, ledger and webhook delivery workers at an eighth of their real size. 1,500 requests a second: 70% charges that must never apply twice, 30% status checks. About half of all requests fail.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.