At a glance
- Did you know
- Search engines build an inverted index: each word points to the documents that contain it.
- Rule of thumb
- Index from a change stream in the background; do not make writes wait for search.
- Watch out
- Query latency spikes under load; stale results.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- YouTube end to end 20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.
- Search falls over Every page calls a search index with 32 query slots and 4 ms of CPU per query on 4 vCPUs, at 1,200 req/s.
- Auth and search on every request Every request checks a session with an auth service (40 slots) and runs a query on a search index (32 slots, 4 ms of CPU each), at 1,500 req/s.
- Search autocomplete on every keystroke Autocomplete queries the search index on every keystroke: 1,500 queries a second, 4 ms of CPU each, on a 4-vCPU index.
- Logging platform: the indexer falls behind Services ship 2,000 log batches a second to a log index with 4 vCPUs, and indexing a batch costs 3 ms of CPU.
- Search autocomplete: 40,000 keystrokes a second under 80 ms Asked at Google and Amazon. Every keystroke runs a prefix query on the full-text search index. Suggestions must come back within 80 ms at p99, and at 8 s a news event raises traffic by half.
- Slack: from launch build to real traffic Slack as first built: channel history read straight from MySQL with no cache, no read replicas, and the message, history and presence services, fan-out workers and search at an eighth of their real size. 5,000 requests a second: 45% reading history, 40% sending messages that fan out to channels, search and push, 15% presence. Messages and history time out.
- Twitter / X timeline: from launch build to real traffic The home timeline as first built: timelines read from Manhattan with no Redis cache, and the timeline API, ranker, tweet API, fan-out workers and search at an eighth of their real size. Real traffic of timeline reads, new tweets and searches arrives, and most requests fail.
And 2 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.