At a glance
- Did you know
- Short-lived access tokens plus refresh tokens limit the damage if a token leaks.
- Rule of thumb
- Verify tokens locally with a public key instead of calling the auth service each time.
- Watch out
- Every request waits on it; a slow token check slows the whole site.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- Every request checks the session Each API request validates its session with an auth service that has 40 slots. At 1,500 req/s it can’t keep up.
- Auth and search on every request Every request checks a session with an auth service (40 slots) and runs a query on a search index (32 slots, 4 ms of CPU each), at 1,500 req/s.
- Public API under abuse: credential stuffing, scrapers and a flood Asked at Stripe, Cloudflare and Okta. A public API with a login endpoint sits behind only a load balancer. From 3 s scrapers send 5,000 requests a second, from 5 s credential stuffing sends 6,000 login attempts a second at the password-hashing service, and at 12 s a 40,000 requests-a-second flood arrives.
- ChatGPT: from launch build to real traffic A chat product on launch-week hardware: the GPU prefill and decode pools, the KV cache store, moderation and the chat API are an eighth of the size real traffic needs. 300 chats a second arrive with long prompts. Answers queue for 20 seconds and most fail.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.