At a glance
- Did you know
- Autoscalers add instances when a metric stays high, but new instances take minutes to be ready.
- Rule of thumb
- Scale ahead of known peaks: reactive scaling lags a sudden spike.
- Watch out
- Serves new capacity 5–8 minutes after a spike starts; a max size below the peak; a health-check grace shorter than the boot.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Instances at start
- Machines taking traffic when the run starts. Slots and vCPU above are per instance.
- Max instances
- The most instances autoscaling may add. Traffic above max instances × per-instance capacity is turned away.
- Scale-out time (ms)
- From deciding to scale out to the new instance taking traffic. During that time the existing instances carry the load.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
- Target CPU
- Average CPU the group aims for. It adds instances after 3 one-minute periods above it and removes them after 15 below.
- Warm pool
- Stopped, pre-initialised instances that take traffic in about 40 s instead of the full boot time. You pay for their disks, not their CPU.
Problems that start with it
- The spike beats the alarm A web fleet runs on 2 EC2 instances in an Auto Scaling group that tracks 50% CPU. At 2 minutes a TV ad triples traffic for 10 minutes. Target tracking needs 3 one-minute periods above target, then each new instance takes about 3 minutes to boot and pass health checks.
- Maximum size below the peak Traffic climbs steadily to 6× over 15 minutes. The Auto Scaling group tracks 50% CPU, which is plenty of time to react, but its maximum size is 3 instances.
- Twenty servers for the work of five After last year’s outage someone set the fleet’s minimum to 20 instances. Traffic needs about 4. The bill shows it.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.