At a glance
- Did you know
- Embeddings turn text into vectors where similar meanings sit close together.
- Rule of thumb
- Batch many texts into one call: per-request overhead dominates small inputs.
- Watch out
- Tokens-per-minute quotas return 429s under bursts.
Problems that start with it
- RAG support assistant 40 questions/s, 4,000-token prompts, a 6M tokens/min quota and a 25% answer-cache hit rate.
- Document Q&A on S3 and a vector DB 40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.
- RAG at twice the traffic Your support assistant doubles to 80 questions a second. The model’s token quota does not.
- Semantic cache that never hits The semantic cache’s similarity threshold is so strict it hits 5% of the time. Nearly every question reaches the model.
- Uploads starve questions Questions and document ingestion share one embedding quota. Uploads double and both paths start failing.
- RAG: quota by day, outage by night The support assistant runs at 80 questions a second against a tight token quota, and at 20 s its only model provider goes down for 10 seconds.
- Embedding calls queue up Every question is embedded first, but the embedding deployment allows 8 concurrent calls at 400 questions a second.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.