At a glance
- Did you know
- Volumetric floods are absorbed at the network edge, spread over many locations, before they reach you.
- Rule of thumb
- Hide your origin's address so attackers cannot go around the shield.
- Watch out
- Flood gets through when off; scrubbing adds latency.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Attack block rate
- Share of the attacks it covers that it stops. The rest pass through and are served like real users.
- False positives (real users blocked)
- Share of real users it wrongly blocks. Stricter rules raise the block rate and this together.
Problems that start with it
- YouTube end to end 20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.
- The WAF blocks real users A strict WAF rule set blocks 4% of real users as attackers.
- Instagram: from launch build to real traffic Instagram as a small team first shipped it: no timeline or profile cache, no media CDN, and every service an eighth of its real size. Real traffic arrives: 4,000 requests a second, 70% feed loads, 10% profiles, 15% likes and 5% new posts with photos to resize. Feeds time out and photos stall.
- Facebook news feed: from launch build to real traffic The news feed as first built: no feed cache in front of the timeline store, and the feed API, ranking model, social graph and fan-out workers at an eighth of their real size. 5,000 requests a second arrive: 85% feed loads that are ranked on every load, 15% new posts fanned out to followers. Most feed loads fail.
- Netflix: from launch build to real traffic Netflix without Open Connect or EVCache: every request reaches the origin, title metadata is read from the catalog database each time, and the playback, browse, license and recommendation services are an eighth of their real size. 6,000 requests a second: half are playback starts, 45% browsing, 5% DRM licenses. Playback starts fail.
- Slack: from launch build to real traffic Slack as first built: channel history read straight from MySQL with no cache, no read replicas, and the message, history and presence services, fan-out workers and search at an eighth of their real size. 5,000 requests a second: 45% reading history, 40% sending messages that fan out to channels, search and push, 15% presence. Messages and history time out.
- ChatGPT: from launch build to real traffic A chat product on launch-week hardware: the GPU prefill and decode pools, the KV cache store, moderation and the chat API are an eighth of the size real traffic needs. 300 chats a second arrive with long prompts. Answers queue for 20 seconds and most fail.
- Uber: from launch build to real traffic Uber as first shipped: no ETA cache, and the location, ETA and ride services, the geo index, its updaters and the notification workers at an eighth of their real size. 5,000 requests a second: 60% driver location pings, 30% ETA lookups, 10% ride requests. Half of all requests fail.
And 4 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.