At a glance
- Did you know
- A semantic cache matches questions by meaning using embeddings, not exact text.
- Rule of thumb
- Tune the similarity threshold: too loose returns wrong answers, too strict never hits.
- Watch out
- A low hit rate means every question still reaches the model.
Problems that start with it
- RAG support assistant 40 questions/s, 4,000-token prompts, a 6M tokens/min quota and a 25% answer-cache hit rate.
- Document Q&A on S3 and a vector DB 40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.
- RAG at twice the traffic Your support assistant doubles to 80 questions a second. The model’s token quota does not.
- Semantic cache that never hits The semantic cache’s similarity threshold is so strict it hits 5% of the time. Nearly every question reaches the model.
- Uploads starve questions Questions and document ingestion share one embedding quota. Uploads double and both paths start failing.
- RAG: quota by day, outage by night The support assistant runs at 80 questions a second against a tight token quota, and at 20 s its only model provider goes down for 10 seconds.
- Semantic cache too strict A semantic cache in front of a RAG assistant hits 10%: its similarity threshold is 0.99. The model quota can’t take the rest.
- Support bot launch day A customer-support bot with a semantic cache, retrieval and a hosted model launches to 60 questions a second. The token quota wasn’t raised.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.