At a glance
- Did you know
- Payment providers accept an idempotency key so a retried charge is not taken twice.
- Rule of thumb
- Treat webhooks as the source of truth: they can arrive late, twice or out of order.
- Watch out
- Duplicate charges on retry; declines counted as errors.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- Subscription billing on the first of the month The monthly renewal run charges 300 subscriptions a second through a payment provider that allows 12,000 calls a minute.
- Concert on-sale: six times the traffic, bots and double booking Asked at Ticketmaster and Amazon. Tickets go on sale at 5 s: real fans arrive at six times normal traffic for 15 seconds, and 8,000 requests a second of bots start a second earlier. Everything hits one API and one Postgres database; two fans can hold the same seat.
- Payments: no double charges, no lost ledger entries, an acquirer outage Asked at Stripe and PayPal. The charges API writes the payment, then publishes a ledger event (two separate writes). At 10 s the ledger queue is down for 5 seconds; the ledger worker loses 2% of its acknowledgements and is not idempotent; at 18 s acquirer A is down for 10 seconds.
- Stays search and booking: a slow payment provider takes everything down Asked at Airbnb and Booking.com. One API serves search, listing pages and bookings from one Postgres database, and bookings call the card provider in the same thread pool. At 6 s holiday traffic rises by half; at 10 s the card provider gets eight times slower for 12 seconds, and searches start failing too.
- Stripe: from launch build to real traffic A payments API on its first servers: no charge status cache, and the charges API, status API, fraud scoring, ledger and webhook delivery workers at an eighth of their real size. 1,500 requests a second: 70% charges that must never apply twice, 30% status checks. About half of all requests fail.
- Amazon storefront: from launch build to real traffic A storefront as first built: product pages read DynamoDB with no product cache, and the product, cart and checkout APIs, search, recommendations, the orders database, payments and fulfilment workers at an eighth of their real size. Shoppers arrive at real traffic and half of all requests fail, checkout included.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.