At a glance
- Did you know
- Hybrid search combines keyword matching (BM25) with vector similarity.
- Rule of thumb
- Keywords catch exact names and codes that embeddings often miss.
- Watch out
- Two indexes to keep in sync; merge latency.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
- Timeout to dependencies (ms)
- How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
- Error rate
- Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
- Quota (req/min)
- Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
- Calls per request (fan-out)
- How many times its outgoing calls run per request: agent steps, jobs, subagents. Load on everything behind it is multiplied by this, and any step failing fails the request.
Problems that start with it
- Query rewriting on every question Agentic RAG rewrites each question with a small model before retrieval. The rewriter has 8 slots at 80 questions a second.
- Hybrid search saturated Retrieval combines keyword and vector search. The hybrid search service has 40 slots at 1,500 queries a second.
- RAG answers stop during an outage Retrieval works, but the answer model’s provider is down from 15 s to 30 s.
- Too many chunks in the prompt The assistant stuffs 6,000 tokens of chunks into every prompt at 30 questions a second, against an 8-million-token-a-minute quota.
- Corrective RAG evaluator bottleneck Corrective RAG grades retrieved chunks with an evaluator model before answering. The evaluator has 20 slots and grades every question.
- Retrieval traces over quota Each RAG question writes retrieval traces to an observability API capped at 3,000 a minute, at 80 questions a second.
- Multi-hop retrieval loops An agentic retriever makes up to 8 search hops per question, and each hop hits the search service.
- Poisoned web pages in RAG A web-search assistant retrieves pages that contain hidden instructions. 300 poisoned requests a second arrive alongside real users.
And 4 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.