System design interviews now ask how you would serve a model, build retrieval, or keep an agent from running away with the bill. Here you do it hands-on: 133 AI infrastructure problems, plus 13 AI systems problems and guided templates, where an LLM service, a RAG pipeline or an agent platform breaks under real traffic and you fix it.
What you will run into
LLM serving
Token and request quotas, concurrent streams, time to first token, prefill and decode GPU pools, KV cache reuse, speculative decoding, multi-LoRA, provider outages and fallback.
RAG and retrieval
Embedding quotas shared with ingestion, vector indexes that outgrow memory, rerankers, query rewriting, GraphRAG, semantic caching and prompt size.
Agents and tools
Steps that multiply load and cost, MCP tool servers and gateways, sandboxes, human approval queues, checkpoints, memory and tool results worth caching.
Multi-agent and harnesses
Subagent fan-out, lead agents that over-plan, long-running coding harnesses, compaction and shared tools.
Security and governance
Prompt injection in support bots, email and web pages, guardrails that block real users, spend governors and audit logs.
57 AI and agent components back these problems, including semantic cache, embedding model, vector DB, LLM endpoint, AI gateway, prompt cache, reranker, guardrails and more.
Token quota at launchA chat product sends 2,000-token prompts at 50 requests a second. The provider allows 4 million input tokens a minute.EasySolve
Not enough concurrent streamsEach answer streams for about 4 seconds, and the deployment allows 100 concurrent streams. Traffic is 40 requests a second.EasySolve
Provider outage at peakAt 15 s the only model provider goes down for 15 seconds, at 30 chats a second.MediumSolve
Timeouts on long answersAnswers are 800 tokens at 70 tokens a second, about 12 seconds each. The app gives up after 8 seconds.EasySolve
Prompt cache that rarely hitsAn exact-match prompt cache sits in front of the model but hits only 5%: timestamps in the prompt make every prompt unique. The model’s token quota is tight.MediumSolve
Self-hosted GPUs at capacityYou serve an open-weights model on your own GPU pool with 32 slots. Requests take about a second; traffic is 50 a second.EasySolve
GPU autoscaling ceilingThe GPU pool autoscales from 2 to 3 nodes of 16 slots each, but traffic needs more than 3 nodes.MediumSolve
Prefill is the bottleneckServing is split into prefill and decode pools (disaggregated serving). Prompts got longer: the prefill pool can’t keep up, and time to first token suffers.MediumSolve
Decode is the bottleneckDisaggregated serving with a small decode pool: answers got longer and decode slots fill up.MediumSolve
KV cache not reusedA KV-cache tier sits in front of prefill, but only 10% of requests reuse a cached prefix: the system prompt changes per user, so prefixes never match.MediumSolve
Draft model can’t keep upSpeculative decoding speeds up answers, but the single draft-model GPU is now the bottleneck at 150 requests a second.MediumSolve
Too many fine-tunes on one GPUOne multi-LoRA server hosts 200 customer adapters on a shared base model. At 60 requests a second the shared batches overflow.MediumSolve
AI gateway key limitAll traffic goes through an AI gateway whose API key allows 1,500 requests a minute. Traffic is 40 a second.EasySolve
PII redaction on the hot pathEvery prompt is scanned by a PII redactor with 2 vCPUs and 10 ms of CPU per prompt, at 400 requests a second.MediumSolve
An LLM judge on every answerEvery answer is graded inline by an LLM judge with 20 slots before it returns. The judge is now the slowest part.MediumSolve
Tracing drops spansEvery request writes a trace to an observability service limited to 2,000 spans a minute. At 60 requests a second, traces are rejected and requests fail with them.EasySolve
Image generation queueAn image generator takes 4 seconds per image on 40 slots. Users request 15 images a second.EasySolve
Retries into a rate limitThe app retries every failed model call 3 times immediately. When the token quota is hit, retries multiply the 429s.MediumSolve
Real-time pool starved by batch jobsMost GPUs went to the offline batch queue. The real-time pool that serves live users has 8 slots and a 30-request queue, at 20 requests a second of 900 ms answers.MediumSolve
Model routing sends everything to the big modelA router splits traffic between a large model and a small one, but it is set to send 90% of chats to the large model, which has 200 concurrent streams. 60 chats arrive a second.MediumSolve
A/B test of a new model versionAn experiment splits chats 50/50 between the current model (A) and a new version (B). B was deployed with only 60 concurrent streams; 40 chats arrive a second.MediumSolve
More replicas, slower answersCoding agents run 20-step tasks, 1 a second, on 8 self-hosted model replicas with prefix caching. An inference router spreads the calls round robin, so each step of a session lands on a different replica from the last.MediumSolve
One replica owns the hot prefixA support-agent fleet sends 30 requests a second through a prefix-aware router to 6 model replicas. 85% of requests start with the same 12,000-token system prompt and tool schemas, and the router always sends a prefix to the replica that holds it.MediumSolve
A minute to load the weightsA self-hosted 70B model runs on 4 GPU replicas that autoscale to 16. At 20 s traffic triples, from 40 to 120 requests a second. Each new replica must load 140 GB of weights before it serves: 60 s here (2–5 minutes in practice).MediumSolve
Background agents starve the chatOne GPU pool with 64 slots serves interactive chat (30 requests a second) and a background agent that triages every ticket: 10 tasks a second, 8 model calls each. Nothing limits how many background tasks run at once.MediumSolve
The eval run eats the production quotaAn agent product serves 20 chats a second on one provider key with 6 million input tokens a minute. A CI regression gate runs the eval suite on the same key: 2 eval tasks a second, 20 model calls each, 4,000 tokens a call.MediumSolve
Shadowing the new model doubles the loadA fine-tuned candidate model is deployed as an adapter on the production GPU pool, and a traffic mirror copies 100% of live requests to it to compare answers. Production is 60 requests a second on a pool that serves about 80.EasySolve
An AI assistant at 200 chats a second: cost and an outageAsked at OpenAI, Anthropic and AI product companies. Every chat sends a 6,000-token prompt (system prompt, tools and retrieved context) to a frontier model and streams back 400 tokens. At 15 s provider A has a 12-second outage. The token bill alone is several times the budget.HardSolve
RAG and retrieval 22
Query rewriting on every questionAgentic RAG rewrites each question with a small model before retrieval. The rewriter has 8 slots at 80 questions a second.MediumSolve
GraphRAG traversals melt the CPUMulti-hop questions traverse a knowledge graph: 10 ms of CPU each on 8 vCPUs, at 1,200 questions a second.MediumSolve
Hybrid search saturatedRetrieval combines keyword and vector search. The hybrid search service has 40 slots at 1,500 queries a second.EasySolve
Reranker on a single GPUA cross-encoder reranks the top 50 chunks for every question on one GPU with 16 slots, at 300 questions a second.EasySolve
Embedding calls queue upEvery question is embedded first, but the embedding deployment allows 8 concurrent calls at 400 questions a second.EasySolve
Semantic cache too strictA semantic cache in front of a RAG assistant hits 10%: its similarity threshold is 0.99. The model quota can’t take the rest.MediumSolve
Context assembly is slowA context builder stitches retrieved chunks, history and instructions for each question. It has 40 slots at 1,000 questions a second.MediumSolve
OCR before every answerA document assistant OCRs scanned pages on demand: 12 slots, 1.5 seconds per page, 15 pages a second.MediumSolve
Too many chunks in the promptThe assistant stuffs 6,000 tokens of chunks into every prompt at 30 questions a second, against an 8-million-token-a-minute quota.MediumSolve
Corrective RAG evaluator bottleneckCorrective RAG grades retrieved chunks with an evaluator model before answering. The evaluator has 20 slots and grades every question.MediumSolve
Retrieval traces over quotaEach RAG question writes retrieval traces to an observability API capped at 3,000 a minute, at 80 questions a second.EasySolve
Multi-hop retrieval loopsAn agentic retriever makes up to 8 search hops per question, and each hop hits the search service.MediumSolve
Rerank results never reusedA cache in front of the reranker hits 15%, because it is keyed on the raw question text including whitespace and casing.MediumSolve
GraphRAG index shared with ingestionThe GraphRAG index answers questions and absorbs entity extraction from new documents at the same time, at 1,500 operations a second.MediumSolve
Scrapers hammer the RAG APIAt 8 s, 3,000 req/s of scrapers start querying your RAG assistant, eating model capacity meant for users.MediumSolve
Re-indexing 100 million documentsAn ingestion service parses, chunks and embeds documents for search. A re-index of 100 million documents pushes 80 documents a second through it, but it has 20 slots at about half a second per document.MediumSolve
Multi-tenant RAG: the permission filterEach customer may only see its own documents, so every retrieval passes a document-ACL filter after hybrid search. The filter has 2 slots and takes about 60 ms; questions arrive at 60 a second.MediumSolve
Multimodal RAG: the vision encoderQuestions about PDFs with charts and scanned tables send each page image through a vision encoder before retrieval. The encoder has 20 slots and takes about 0.7 s per image; 40 questions arrive a second.MediumSolve
Conversational memory on every turnA chat assistant keeps long conversations within the context window by summarising older turns with a small model before every answer. The summariser has 12 slots at about half a second; 40 turns arrive a second.MediumSolve
Citation checking slows every answerTo cut hallucinations, every answer is checked against the retrieved passages: each cited sentence must be supported. The checker has 16 slots at about 0.6 s; answers arrive at 50 a second.MediumSolve
An agent that takes 12 stepsA support agent takes 12 model-and-tool steps per ticket. Tools fail 1% of the time, and every failure sinks the whole task.EasySolve
MCP gateway quotaEvery tool call goes through an MCP gateway (auth, allow-lists, quotas) limited to 600 calls a minute. Agents make about 18 tool calls a second.EasySolve
One slow tool serverA CRM tool server with 8 slots takes 300 ms per call. Agents call it every step.EasySolve
Code sandboxes run outA data-analysis agent runs Python in sandboxes: 6 sandboxes, about 1.2 s per run, 4 runs per task at 2 tasks a second.MediumSolve
Browser automation bottleneckA web agent drives headless browsers: 16 browsers, about 2.5 s per action, 3 actions per task at 3 tasks a second.MediumSolve
Computer-use desktops are scarceA computer-use agent clicks through a legacy app on virtual desktops: 10 desktops, about 3 s per action, 4 actions per task.MediumSolve
Approvals pile upA finance agent asks a human to approve every payment. Five reviewers take about 20 s each; tasks arrive at 0.5 a second.MediumSolve
Agent memory on a small storeThe agent reads and writes long-term memory every step, on a memory store with 2 slots and 60 ms per call.EasySolve
Same tool call, every stepAgents repeat the same read-only lookups, but the tool-result cache hits 10% because it is keyed on raw arguments with timestamps.MediumSolve
Checkpointing every stepDurable execution saves the agent’s state after every step to a checkpoint store with 10 slots and 50 ms writes.MediumSolve
Tool search on every stepWith 400 tools, the agent searches for the right few each step. The tool-search service has 6 slots.MediumSolve
Compaction calls overflowLong sessions are compacted every step by a summariser with 20 slots, about 600 ms each.MediumSolve
Agents burn the token quotaEach step sends 3,000 tokens of context. Agents make 24 model calls a second against a 3-million-token-a-minute quota.MediumSolve
Model outage mid-taskThe agent’s model provider goes down for 15 seconds. Every task in flight fails at its next step.MediumSolve
Agent model out of streamsAgents hold a model stream for about 2 s per step. The deployment allows 30 streams; agents need about 50.EasySolve
Partner agent goes offlineYour travel agent books hotels through a partner’s agent over A2A. The partner is down from 15 s to 35 s.MediumSolve
Partner agent at capacityA partner agent you delegate to over A2A handles 20 tasks at once, about 2.5 s each. You send it 12 a second.MediumSolve
Workflow engine overloadDeterministic steps run on a workflow engine with 6 slots (200 ms each) between the agent’s decisions.MediumSolve
Loading skills every stepThe agent loads skills (instructions and scripts) from a registry with 4 slots on every step.EasySolve
Agent jobs run twiceAgent jobs are queued and run by workers. When an acknowledgement is lost, the queue redelivers and the job’s side effect (sending an email) happens twice.MediumSolve
Tracing every agent stepEach step writes a trace span to an observability API capped at 1,200 a minute; agents make about 30 steps a second.EasySolve
100,000 agent tasks a minuteAn agent platform runs every task as a 6-step loop. The orchestrator can hold 80 tasks at once; each task takes about 15 seconds, and 10 tasks start every second.MediumSolve
Judging every agent stepTo evaluate agents, an LLM judge scores every step of every trajectory (was the tool call right, did the step help) while tasks run. 8 tasks a second × 6 steps reach a judge with 20 slots at about 0.6 s.MediumSolve
A new browser for every pageResearch agents read about 8 web pages per task, 30 tasks a minute. The hosted-browser plan allows 100 sessions at once but only 50 new sessions a minute, and the browser tool opens a fresh session for every page.MediumSolve
Sandboxes held during approvalsA coding-agent platform runs 7 tasks a second with 3 sandbox runs each, about 3.5 s per run, on a provider that allows 100 sandboxes at once. On 30% of runs the sandbox asks a person to approve a command and stays reserved (and billed) while it waits, about 12 s. Times are scaled: a second here stands for a minute.MediumSolve
Two messages, one threadA chat agent handles 300 turns a second. Each turn reads the thread’s state, runs the agent and writes the state back. Users often send a second message before the first run finishes, so two runs on one thread read the same state and the last write wins.MediumSolve
The prompt cache expires between stepsData-analyst agents send about 82,000 input tokens per step: 26,000 shared (system prompt and tools) plus the task’s growing history. The provider keeps a cached prefix for 5 minutes after its last use and does not count cached tokens against the 3 million tokens a minute quota. Between steps, each agent waits 6–8 minutes for a warehouse query. Times are scaled: a second here stands for a minute.MediumSolve
Ninety-three tools in every promptA coding agent loads the full GitHub MCP toolset, 93 tools and about 55,000 tokens of definitions, into every prompt: 60,000 input tokens per call. This provider counts cached tokens against its 6 million tokens a minute quota, so prompt caching lowers the bill but not the quota.MediumSolve
Online evals silently lose trajectoriesAn agent service samples 5% of its trajectories, 12 a second, into a stream that keeps 20 seconds of data. LLM-judge consumers score them for regressions, but four consumers manage only about 7 a second.MediumSolve
Multi-agent and harness 14
Subagents swamp the search toolA lead agent fans each question out to 8 parallel subagents, and each subagent searches the web. The search tool has 8 slots.MediumSolve
Multi-agent token billEvery subagent sends 3,000 tokens of context. Eight subagents per question multiply token use past the provider quota.MediumSolve
The workflow outgrows its historyLong-running coding agents run on a durable workflow engine. Every step passes the whole conversation in and out, so the first step records 60 KB and each later one 40 KB more than the last. The engine terminates any run whose history passes 50 MB. Tasks take 50 steps.MediumSolve
Retries at every hop of an agent chainA concierge agent calls a booking agent, which calls an airline’s agent over A2A (40 slots, about 4 s a request). Both callers time out after 8 s and retry twice at once. At 15 s bookings jump 2.5× for 10 seconds.MediumSolve
AI security and governance 17
Prompt injection on a support botAt 8 s, attackers send 400 messages a second with hidden instructions (“ignore previous instructions…”). Nothing screens them.MediumSolve
Injected instructions in emailAn email agent reads inbound messages and can call tools. Attackers send 200 emails a second containing instructions to forward data.MediumSolve
Poisoned web pages in RAGA web-search assistant retrieves pages that contain hidden instructions. 300 poisoned requests a second arrive alongside real users.MediumSolve
Malicious issue textA coding harness reads issue descriptions. Attackers file issues with instructions to exfiltrate secrets, 100 a second.MediumSolve
A token exchange on every tool callBefore each tool call, an enterprise MCP agent exchanges the user’s identity for a short-lived, audience-bound token at the company identity provider, then throws the token away. 5 tasks a second × 8 steps reach an identity provider that allows 1,200 token requests a minute.MediumSolve
One tenant’s loop starves everyoneA multi-tenant agent platform shares one provider key of 4,000 requests a minute. Most tenants run 6-step tasks, 5 a second. One tenant, Acme, has a buggy two-agent loop that runs 40 steps per task, 1.5 tasks a second.MediumSolve
A filter is not a permissionAn ops copilot reads public issues (untrusted), queries private repos and the customer database (private data), and posts comments, PRs and webhooks (outbound). That is the lethal trifecta. A prompt-injection classifier that stops 97% of attacks guards it, and attackers file 50 poisoned issues a second for 15 seconds.MediumSolve
AI products 21
AI feedback: thumbs-up events go missingEvery thumbs-up or thumbs-down on an AI answer is saved, then published as an event for the offline evaluation and training pipeline. At 10 s the queue is down for 5 seconds: the votes are saved but their events vanish, so the next release is evaluated on biased data.MediumSolve
Support bot launch dayA customer-support bot with a semantic cache, retrieval and a hosted model launches to 60 questions a second. The token quota wasn’t raised.MediumSolve
IDE code completionsInline completions: 400 requests a second, short outputs, and only 64 streams on the completion model.MediumSolve
AI search engine retrievalAn AI search engine rewrites the query, runs hybrid search and answers. Hybrid search has 30 slots at 1,200 queries a second.MediumSolve
SRE agent runs too longAn on-call SRE agent investigates incidents in 15 steps of model calls and tool queries.MediumSolve
Medical scribe: redaction bottleneckA medical scribe transcribes visits, redacts PII and drafts notes. The redactor has 1 vCPU and 20 ms per note, at 100 notes a second.MediumSolve
Legal research with GraphRAGLegal questions traverse a citation graph: 12 ms of CPU each on 4 vCPUs, at 500 questions a second.MediumSolve
Sales agent reads hostile repliesA sales agent reads inbound replies and updates the CRM. A campaign of replies with injected instructions arrives at 150 a second.MediumSolve
Video generation farmText-to-video takes about 20 s per clip on 40 GPUs; users request 3 clips a second.MediumSolve
Recommendations: the online feature storeEvery recommendation reads user and item features from an online feature store before the ranking model scores candidates. The store holds 6 reads at once at about 3 ms each; 2,500 requests arrive a second.MediumSolve
Background agents poll GitHub until throttledBackground coding agents open a pull request, then wait for CI by polling GitHub every 10 s, about 12 checks per task. All agents share one GitHub App installation, limited to 15,000 requests an hour (250 a minute). 70 tasks start a minute.MediumSolve
The CRM allows 25 report queriesA revenue-ops agent runs CRM reports through an MCP server. The CRM allows 25 report queries at once and rejects the rest immediately, with no queue. Each report takes about 2.4 s, and the agents ask for 12 a second.MediumSolve
Voice agent: the model’s slow tailA voice agent transcribes speech (100–150 ms), asks a model (first token: 250 ms usually, 2.4 s for the slowest 1%) and speaks the answer. The call to the model waits up to 8 s and has nowhere else to go. Callers notice any silence over about a second.MediumSolve
AI systems 13
RAG support assistant40 questions/s, 4,000-token prompts, a 6M tokens/min quota and a 25% answer-cache hit rate.MediumSolve
Agent runaway15 tasks/s. The agent makes 8 model and tool calls per task; the model allows 4,000 calls a minute and 120 concurrent streams.MediumSolve
LLM gateway with fallback40 chats/s through your AI gateway and a prompt cache to one hosted model. At 20 s the provider has a 15 s outage.MediumSolve
MCP agent platform4 tasks/s. The agent thinks (model) and acts (tools through an MCP gateway) 6 times per task, with vector memory and results in S3.MediumSolve
Document Q&A on S3 and a vector DB40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.MediumSolve
RAG at twice the trafficYour support assistant doubles to 80 questions a second. The model’s token quota does not.MediumSolve
Semantic cache that never hitsThe semantic cache’s similarity threshold is so strict it hits 5% of the time. Nearly every question reaches the model.EasySolve
Model provider outageYour chat product depends on one model provider. At 20 s it is down for 15 seconds.MediumSolve
Ten steps of MCP toolsAgents call MCP tools 10 times per task. A 1% tool error rate becomes a 10% task failure rate.MediumSolve
Uploads starve questionsQuestions and document ingestion share one embedding quota. Uploads double and both paths start failing.MediumSolve
RAG: quota by day, outage by nightThe support assistant runs at 80 questions a second against a tight token quota, and at 20 s its only model provider goes down for 10 seconds.MediumSolve
Agent: a slow tool and a tight quotaAn agent takes 6 steps per task at 15 tasks a second. Every step calls a tool server with 8 slots that takes 300 ms, and the model allows 4,000 requests a minute.MediumSolve