At a glance
- Did you know
- Email, SMS and push providers all throttle senders and can delay delivery.
- Rule of thumb
- Queue notifications and send them in the background; never block a user request on them.
- Watch out
- Provider quota; bursts back up.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- Push notifications: the provider throttles you A campaign sends 700 push notifications a second through a push provider whose plan allows 30,000 a minute.
- Chat delivery: 20,000 messages a second and a push outage Asked at Meta (WhatsApp, Messenger) and Slack. Every message is written to Postgres and then pushed to the recipient’s phone in the same request; the app polls the same database to sync, and presence heartbeats are database writes too. At 12 s the push provider is down for 8 seconds, and sending stops.
- Notification burst: a campaign, a quota and a provider outage Asked at Uber, DoorDash and Airbnb. A marketing campaign sends notifications through an API that calls the push provider directly and retries twice. At 6 s the campaign triples the rate for 8 seconds, the provider’s quota is 150,000 a minute, and at 15 s the provider is down for 8 seconds.
- Instagram: from launch build to real traffic Instagram as a small team first shipped it: no timeline or profile cache, no media CDN, and every service an eighth of its real size. Real traffic arrives: 4,000 requests a second, 70% feed loads, 10% profiles, 15% likes and 5% new posts with photos to resize. Feeds time out and photos stall.
- Slack: from launch build to real traffic Slack as first built: channel history read straight from MySQL with no cache, no read replicas, and the message, history and presence services, fan-out workers and search at an eighth of their real size. 5,000 requests a second: 45% reading history, 40% sending messages that fan out to channels, search and push, 15% presence. Messages and history time out.
- Uber: from launch build to real traffic Uber as first shipped: no ETA cache, and the location, ETA and ride services, the geo index, its updaters and the notification workers at an eighth of their real size. 5,000 requests a second: 60% driver location pings, 30% ETA lookups, 10% ride requests. Half of all requests fail.
- Amazon storefront: from launch build to real traffic A storefront as first built: product pages read DynamoDB with no product cache, and the product, cart and checkout APIs, search, recommendations, the orders database, payments and fulfilment workers at an eighth of their real size. Shoppers arrive at real traffic and half of all requests fail, checkout included.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.