At a glance
- Did you know
- Latency has two parts: time to the first token, then a steady stream of tokens per second.
- Rule of thumb
- Stream answers so they feel fast, and cap output tokens to bound cost.
- Watch out
- Quota 429s, slow first tokens under load, and cost.
Problems that start with it
- RAG support assistant 40 questions/s, 4,000-token prompts, a 6M tokens/min quota and a 25% answer-cache hit rate.
- Agent runaway 15 tasks/s. The agent makes 8 model and tool calls per task; the model allows 4,000 calls a minute and 120 concurrent streams.
- LLM gateway with fallback 40 chats/s through your AI gateway and a prompt cache to one hosted model. At 20 s the provider has a 15 s outage.
- MCP agent platform 4 tasks/s. The agent thinks (model) and acts (tools through an MCP gateway) 6 times per task, with vector memory and results in S3.
- Document Q&A on S3 and a vector DB 40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.
- RAG at twice the traffic Your support assistant doubles to 80 questions a second. The model’s token quota does not.
- Semantic cache that never hits The semantic cache’s similarity threshold is so strict it hits 5% of the time. Nearly every question reaches the model.
- The agent that thinks too much An agent takes 12 model-and-tool steps per task. Steps multiply load, cost and failures.
And 88 more on the problems page.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.