At a glance
- Did you know
- Routing a session back to the replica that holds its prefix can raise KV-cache hit rates from about 20% to 75%.
- Rule of thumb
- Stay sticky until a replica is saturated, then spill to the least busy one.
- Watch out
- Round robin scatters agent steps, so more replicas can mean fewer cache hits.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Session affinity
- Share of requests sent back to the replica that served the same session (or holds the same prefix). 0 is round robin; 100% is fully sticky.
- Prefix-cache hit on the same replica
- How often a request finds its prefix in the KV cache when it lands on the replica that holds it. A hit skips most of the prefill.
- Requests on the shared prefix
- Share of requests that start with one shared prefix (a system prompt and tool schemas). Sticky routing sends all of them to the replica that holds it.
- Spill when queue reaches
- When the sticky replica has this many requests waiting, send the next one to the least busy replica instead. 0 never spills.
Problems that start with it
- More replicas, slower answers Coding agents run 20-step tasks, 1 a second, on 8 self-hosted model replicas with prefix caching. An inference router spreads the calls round robin, so each step of a session lands on a different replica from the last.
- One replica owns the hot prefix A support-agent fleet sends 30 requests a second through a prefix-aware router to 6 model replicas. 85% of requests start with the same 12,000-token system prompt and tool schemas, and the router always sends a prefix to the replica that holds it.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.