At a glance
- Did you know
- Gateways centralise auth, rate limits and routing so each service does not reimplement them.
- Rule of thumb
- Every request pays the gateway's latency: keep its plugins lean.
- Watch out
- Adds latency to every call; its own quota rejects real users.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- Agent runaway 15 tasks/s. The agent makes 8 model and tool calls per task; the model allows 4,000 calls a minute and 120 concurrent streams.
- YouTube end to end 20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.
- The agent that thinks too much An agent takes 12 model-and-tool steps per task. Steps multiply load, cost and failures.
- Agent: a slow tool and a tight quota An agent takes 6 steps per task at 15 tasks a second. Every step calls a tool server with 8 slots that takes 300 ms, and the model allows 4,000 requests a minute.
- API gateway at its request limit Every request passes an API gateway that handles auth and routing. Its plan allows 90,000 requests a minute; peak traffic is 2,000 a second.
- Java cold starts at launch A Java Lambda (Spring) takes 3 s to initialise. At 30 s a launch email quintuples traffic for 20 s: every request without a warm environment starts a new one and waits for INIT, which needs still more environments.
- One function starves the account Two Lambda functions share an account: the customer API (about 100 ms a call) and a report export (about 4.5 s a call). Exports run 240 a second at month end. Neither has reserved concurrency, and the account limit is 1,000.
- Instagram: from launch build to real traffic Instagram as a small team first shipped it: no timeline or profile cache, no media CDN, and every service an eighth of its real size. Real traffic arrives: 4,000 requests a second, 70% feed loads, 10% profiles, 15% likes and 5% new posts with photos to resize. Feeds time out and photos stall.
And 12 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.