At a glance
- Did you know
- gRPC uses HTTP/2 and Protocol Buffers for compact, typed calls between services.
- Rule of thumb
- Long-lived HTTP/2 connections defeat connection-level balancing: balance per request.
- Watch out
- Deadline propagation; connection reuse limits.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- YouTube end to end 20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.
- A slow dependency drags everyone One downstream service gets slow. API requests pile up waiting on it, and soon even requests that don’t need it fail.
- Snowflake IDs from one service Every new order gets a unique 64-bit ID from a central ID service (Snowflake style). Writes are 1,000 a second; the ID service runs on one vCPU.
- Food delivery: dispatch at dinner rush Orders ramp from 400 to 1,200 a second over the evening. Every order asks the dispatch service to match a courier, and dispatch was sized for lunch.
- Fraud scoring on the checkout path Every checkout is scored by a fraud model before payment: 1,000 checkouts a second, 5 ms of CPU each, on 4 vCPUs.
- Ad bidding: 150 ms or nothing Every page view runs an ad auction, and the exchange drops bids slower than 150 ms. The bidder takes 5 ms of CPU per bid on 4 vCPUs, at 1,200 auctions a second.
- YouTube: from launch build to real traffic YouTube as first built: no media CDN, no metadata cache in front of Vitess MySQL, and the watch, search, comment and upload services, recommendations, view counting and transcoding at an eighth of their real size. 20,000 requests a second arrive, routed by path. Almost everything fails.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.