At a glance
- Did you know
- Object stores like S3 keep copies across devices and facilities for very high durability.
- Rule of thumb
- Serve files through a CDN with signed URLs instead of streaming them through your API.
- Watch out
- Request-rate throttling (503s) on hot prefixes.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- YouTube end to end 20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.
- MCP agent platform 4 tasks/s. The agent thinks (model) and acts (tools through an MCP gateway) 6 times per task, with vector memory and results in S3.
- Document Q&A on S3 and a vector DB 40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.
- Ten steps of MCP tools Agents call MCP tools 10 times per task. A 1% tool error rate becomes a 10% task failure rate.
- Uploads starve questions Questions and document ingestion share one embedding quota. Uploads double and both paths start failing.
- Object storage: one hot prefix All images live under one key prefix, which the store limits to about 5,500 reads a second. Reads arrive at 7,000 a second.
- Premiere night: video at three times the traffic Asked at Netflix and YouTube. Every video segment is served by origin servers reading object storage, manifests come from Postgres, and DRM licenses from license servers, all through the same origin fleet. At 6 s a premiere starts and traffic triples for 12 seconds; at 13 s the segment storage is unavailable for 4 seconds.
- File sync: every 4 MB chunk goes through the API Asked at Dropbox and Google (Drive). Uploads and downloads of 4 MB chunks are proxied through the file API, which spends 25 ms of CPU and up to 600 ms per chunk; metadata reads hit Postgres. At 9 s a team shares a large folder and traffic doubles.
And 3 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.