At a glance
- Did you know
- A router sends easy requests to small, cheap models and hard ones to larger models.
- Rule of thumb
- Measure quality per route: savings that hurt answers are not savings.
- Watch out
- Misrouting; the cheap path fails and falls back.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
Problems that start with it
- Model routing sends everything to the big model A router splits traffic between a large model and a small one, but it is set to send 90% of chats to the large model, which has 200 concurrent streams. 60 chats arrive a second.
- A/B test of a new model version An experiment splits chats 50/50 between the current model (A) and a new version (B). B was deployed with only 60 concurrent streams; 40 chats arrive a second.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.