At a glance
- Did you know
- A spend governor enforces budgets per user, task or team before calls go out.
- Rule of thumb
- Warn at a soft limit, then stop at a hard one.
- Watch out
- Budget too low fails real tasks; too high lets loops burn money.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
- Trip at error share
- The breaker opens when this share of the last window of calls failed. Open, it fails fast instead of waiting.
- Cooldown (ms)
- How long the breaker stays open before letting calls through again.
- Window (calls)
- How many recent calls the error share is measured over. Small windows trip on a blip.
Problems that start with it
- Spend governor set too low A spend governor caps model calls at 1,200 a minute to protect the budget, but legitimate traffic is 30 a second.
- ChatGPT: from launch build to real traffic A chat product on launch-week hardware: the GPU prefill and decode pools, the KV cache store, moderation and the chat API are an eighth of the size real traffic needs. 300 chats a second arrive with long prompts. Answers queue for 20 seconds and most fail.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.