At a glance
- Did you know
- GPU servers batch requests together; throughput rises with batch size until memory runs out.
- Rule of thumb
- Long prompts use more KV-cache memory, so fewer requests fit at once.
- Watch out
- KV-cache pressure; queueing when all GPUs are busy.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Problems that start with it
- Self-hosted GPUs at capacity You serve an open-weights model on your own GPU pool with 32 slots. Requests take about a second; traffic is 50 a second.
- GPU autoscaling ceiling The GPU pool autoscales from 2 to 3 nodes of 16 slots each, but traffic needs more than 3 nodes.
- PII redaction on the hot path Every prompt is scanned by a PII redactor with 2 vCPUs and 10 ms of CPU per prompt, at 400 requests a second.
- An LLM judge on every answer Every answer is graded inline by an LLM judge with 20 slots before it returns. The judge is now the slowest part.
- Tracing drops spans Every request writes a trace to an observability service limited to 2,000 spans a minute. At 60 requests a second, traces are rejected and requests fail with them.
- Voice assistant: transcription backs up A voice assistant transcribes speech, answers, and speaks the reply. Speech-to-text has 16 slots at 30 utterances a second.
- Real-time pool starved by batch jobs Most GPUs went to the offline batch queue. The real-time pool that serves live users has 8 slots and a 30-request queue, at 20 requests a second of 900 ms answers.
- GraphRAG traversals melt the CPU Multi-hop questions traverse a knowledge graph: 10 ms of CPU each on 8 vCPUs, at 1,200 questions a second.
And 23 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.