At a glance
- Did you know
- Approximate indexes like HNSW trade a little recall for much faster search.
- Rule of thumb
- Raw size ≈ vectors × dimensions × 4 bytes. One million 1,536-d vectors is about 6 GB before index overhead.
- Watch out
- An index larger than memory, or queries above its QPS limit.
Problems that start with it
- RAG support assistant 40 questions/s, 4,000-token prompts, a 6M tokens/min quota and a 25% answer-cache hit rate.
- Document Q&A on S3 and a vector DB 40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.
- RAG at twice the traffic Your support assistant doubles to 80 questions a second. The model’s token quota does not.
- Semantic cache that never hits The semantic cache’s similarity threshold is so strict it hits 5% of the time. Nearly every question reaches the model.
- Uploads starve questions Questions and document ingestion share one embedding quota. Uploads double and both paths start failing.
- RAG: quota by day, outage by night The support assistant runs at 80 questions a second against a tight token quota, and at 20 s its only model provider goes down for 10 seconds.
- Vector database: one replica at 600 queries a second A vector database serves nearest-neighbour search for a recommendation feed. One replica answers up to 400 queries a second; traffic is 600.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.