Each problem is a working system with something wrong in it: too little capacity, a cache sized badly, retries that amplify an outage, an attack with nothing in front, a model quota too small. Run it, watch what breaks, and fix it until every goal is green. Every problem is tested to fail as given and pass once fixed.
56 easy, 235 medium and 61 hard. Six are free; unlock any other once (Easy 10, Medium 15, Hard 25 credits) and everything inside it is free.
Three learning paths
System design fundamentals
Start here if queues, caches and replicas are words you know but have never watched fail.
Size a service from its traffic: slots, CPU and connection pools.
See what a cache really saves, and what a flush costs the database behind it.
Recognise retry storms, timeouts that are too long, and the fixes for each.
Keep a queue or event stream from falling behind.
Interview practice
End-to-end systems with several goals at once, plus the hard problems from every core topic, the way a design round asks for them, with a mock interviewer that throws curveballs.
Walk a whole design from the edge to the data layer and defend each component.
Estimate capacity and cost out loud, then check the numbers in a run.
Recover from a curveball (a dead instance, a flushed cache, a bot flood) on a running system.
AI infrastructure
For engineers shipping LLM features and agents: provider quotas, time to first token, retrieval, tools and runaway loops.
Explain why replicas cannot lift a provider quota, and what can.
Keep time to first token under a target with caching, routing and fallbacks.
Size retrieval (embeddings, a vector index) and spot when it is the bottleneck.
Put step caps, spend limits and approval gates on agents before they run away.
Kubernetes and cloud
For engineers who run services on Kubernetes and AWS: how pods, probes, autoscalers and rollouts behave under load, and what the bill does.
Explain why autoscaling takes minutes, not seconds, and size the headroom that covers it.
Set readiness, liveness, memory and CPU limits that hold under a spike instead of causing one.
Ship a release with a canary, graceful shutdown and spread replicas, so a bad version or a lost node is a blip.
Read a cloud bill: which meters grow with traffic, and which design choices cut them.
Design a product 56
Problems with a brief: design a known product (YouTube, BookMyShow, S3…), estimate its traffic, storage and cost, build it, and run its test cases (a normal day, a peak, a failure, often an attack, and the budget). The URL shortener is free. Want no brief at all? Build from scratch on an empty canvas.
Build Amazon: the full interviewDesign the Amazon storefront from an empty canvas, as in a senior system-design interview: home page, search, product pages, an always-writeable cart, checkout that never oversells or charges twice, and orders that always reach fulfilment, through Prime Day, failures and attacks.HardSolve
Build YouTube: the full interviewDesign YouTube the way a senior system-design interview expects: resumable uploads and a transcoding pipeline, adaptive playback through a CDN, a sharded metadata store behind a cache, two-stage recommendations, search, comments and view counting, then survive a viral video, a cold CDN, a cold cache, an upload surge, a slow ranker, a stream outage and a DDoS attack.HardSolve
Build Instagram: the full interviewDesign Instagram the way a senior interview expects: uploads processed in the background, posts fanned out into precomputed timelines (with celebrities pulled at read time), a ranked feed with a fallback, stories that expire, hot counters, and media through a CDN.HardSolve
Build an LLM inference API like the Claude API: the full interviewDesign the platform behind an LLM API the way Anthropic, Google and Meta ask it: streamed answers with a fast first token, long-context calls on a cached prefix, per-org rate limits, multi-region GPU fleets, a Batch API, exactly-once usage billing, safety classifiers and a developer console, then survive launch day, a lost GPU region, bots, a jailbreak campaign, a usage-stream outage, a huge batch job and a DDoS attack.HardSolve
Build a URL shortenerDesign bit.ly from an empty canvas: create short links and redirect them, fast, at 2,000 requests a second.EasySolve
Build YouTubeDesign YouTube from an empty canvas: upload and transcode videos, watch them through a CDN, search them and count views, at 2,000 requests a second.HardSolve
Build NetflixDesign Netflix from an empty canvas: a personalised home page, play starts with DRM, video segments from a CDN and progress heartbeats, at 2,500 requests a second.HardSolve
Build SpotifyDesign Spotify from an empty canvas: stream audio through a CDN, search the catalog, open playlists and log every play, at 2,000 requests a second.MediumSolve
Build Twitch live streamingDesign Twitch from an empty canvas: take in broadcasters’ live streams, transcode them, deliver them to viewers through a CDN and run real-time chat, at 2,000 requests a second.HardSolve
Build InstagramDesign Instagram from an empty canvas: post photos, resize them in the background, fan posts out into precomputed timelines, serve feeds and photos, and count likes, at 2,500 requests a second.HardSolve
Build PinterestDesign Pinterest from an empty canvas: an image-heavy home feed through a CDN, pin saves and search, with scrapers kept out, at 2,500 requests a second.MediumSolve
Build TikTok’s For You feedDesign TikTok’s For You feed from an empty canvas: rank candidate videos with a model, play them from a CDN and stream watch events back into the ranker’s features, at 3,000 requests a second.HardSolve
Build PastebinDesign Pastebin from an empty canvas: create pastes and read them back by link, fast, at 1,500 requests a second.EasySolve
Build Google PhotosDesign Google Photos from an empty canvas: upload photos, make thumbnails in the background, browse and search the library, at 2,000 requests a second.MediumSolve
Build BookMyShowDesign BookMyShow from an empty canvas: browse shows and seat maps, hold seats without ever double-booking one, pay, and get a confirmation, even when a blockbuster opens.HardSolve
Build Ticketmaster’s ticket dropDesign a Ticketmaster on-sale from an empty canvas: an event page, ticket holds and checkout that survive the second the sale opens, with scalper bots kept out.HardSolve
Build Amazon checkout and flash saleDesign Amazon’s cart and checkout from an empty canvas: fast product pages, carts that survive, and a checkout that charges once and never loses an order, through a flash sale.HardSolve
Build Airbnb search and bookingDesign Airbnb from an empty canvas: map search, listing pages and bookings that never double-book a home, at 2,000 requests a second.MediumSolve
Build Uber ride matchingDesign Uber’s dispatch from an empty canvas: thousands of driver location updates a second, riders matched to a nearby driver in seconds, and trip state for both phones.HardSolve
Build Swiggy / DoorDash food deliveryDesign a food delivery app from an empty canvas: browse restaurants, order and pay, dispatch a rider, and track the order live, through the dinner rush.MediumSolve
Build a payment system like StripeDesign a Stripe-like payments API from an empty canvas: idempotent charges and refunds on a ledger, card networks that fail, and webhooks that always arrive exactly once in effect.HardSolve
Build a digital wallet like UPI / PaytmDesign a digital wallet from an empty canvas: instant balances, transfers on a ledger, a fraud check before money moves, and notifications.MediumSolve
Build a stock exchangeDesign a stock exchange from an empty canvas: orders through a gateway and a risk check into a matching engine, market data fanned out to every subscriber, all in a few milliseconds.HardSolve
Build a Shopify storefront platformDesign Shopify from an empty canvas: thousands of merchants’ storefronts and product pages served fast, checkout for all of them, and one store’s flash sale kept from hurting the rest.MediumSolve
Build Twitter’s home timelineDesign the Twitter home timeline from an empty canvas: tweets fan out to followers’ timelines, timelines are read from a cache, and tweets are searchable, at 3,000 requests a second.HardSolve
Build the Facebook news feedDesign the Facebook news feed from an empty canvas: retrieve candidates from precomputed feeds, rank them with a model on every load, and take posts and likes, at 3,000 requests a second.HardSolve
Build WhatsApp messagingDesign WhatsApp from an empty canvas: phones connect over WebSockets, messages are stored and then delivered in real time or by push, presence is tracked, and no accepted message is ever lost.HardSolve
Build SlackDesign Slack from an empty canvas: clients connect over WebSockets, messages fan out to channel members in real time, history is scrollable, everything is searchable and presence is live.MediumSolve
Build Zoom video callsDesign Zoom from an empty canvas: signalling over a WebSocket gateway, audio and video through media servers (SFUs), meeting state in a store, and calls that survive losing a media server.HardSolve
Build a notification systemDesign a notification system from an empty canvas: accept requests from other services, queue them, and have workers deliver push, email and SMS through the providers, with dedupe and per-user rate limits.MediumSolve
Build GmailDesign Gmail from an empty canvas: send mail through SMTP workers, receive and store mail with spam filtering, read the inbox from a cache, and search every message.MediumSolve
Build RedditDesign Reddit from an empty canvas: subreddit front pages from a cache, write-heavy vote counters that feed hot rankings, and comment threads.MediumSolve
Build LinkedIn connectionsDesign LinkedIn’s connections from an empty canvas: cached profiles, a connection graph, precomputed “people you may know”, and messaging.MediumSolve
Build Tinder matchingDesign Tinder from an empty canvas: profile cards of people nearby from a geospatial index, write-heavy swipes, match detection and chat.MediumSolve
Build Amazon S3Design Amazon S3 from an empty canvas: PUT, GET and LIST objects at 1,500 requests a second, with the bytes on erasure-coded storage nodes and the metadata in a key-value store.HardSolve
Build Dropbox / Google Drive file syncDesign Dropbox from an empty canvas: chunked uploads to object storage, file metadata in a database, change notifications to the user’s other devices, and downloads, at 2,000 requests a second.HardSolve
Build a distributed key-value store like DynamoDBDesign a DynamoDB-style key-value store from an empty canvas: route each key with consistent hashing to partitioned storage nodes, replicate it, and keep answering in milliseconds when a node dies or slows down.HardSolve
Build an API rate limiter serviceDesign an API rate limiter from an empty canvas: per-client counters in memory in front of an API, adding only milliseconds, and stopping abusive clients before they reach it.MediumSolve
Build a web crawlerDesign a web crawler from an empty canvas: a URL frontier, a Bloom filter that drops seen URLs, polite fetcher workers, a parser and a page store, taking 1,500 discovered URLs a second.MediumSolve
Build search autocomplete / typeaheadDesign search autocomplete from an empty canvas: the top 10 completions for every keystroke within 50 ms at p99, rebuilt offline from a stream of finished searches.MediumSolve
Build Yelp / Google Maps nearby searchDesign nearby search from an empty canvas: find businesses around a location with a geospatial index, show business pages from a cache, and take reviews, at 2,000 requests a second.MediumSolve
Build a gaming leaderboardDesign a live leaderboard from an empty canvas: score updates into a sorted set, top-100 and my-rank reads in milliseconds, and every score persisted to a database.MediumSolve
Build ad click aggregationDesign ad click aggregation from an empty canvas: take 2,850 clicks a second without losing or double-counting one, aggregate them per ad per minute on a stream, and serve advertisers’ reports.HardSolve
Build a metrics and monitoring system like DatadogDesign a Datadog-style monitoring system from an empty canvas: agents push metric batches into an intake, a stream feeds a time-series database, dashboards query it, and alerts page someone.HardSolve
Build Google Docs collaborative editingDesign Google Docs from an empty canvas: many people typing in the same document, every keystroke merged, logged and shown to the others within 100 ms.HardSolve
Build an online judge like LeetCodeDesign LeetCode from an empty canvas: problem pages for everyone, and untrusted code submissions judged in sandboxes, even when a contest starts.MediumSolve
Build a CI/CD system like GitHub ActionsDesign GitHub Actions from an empty canvas: every push starts a pipeline on a shared runner fleet, with artifacts, logs and live status.MediumSolve
Build a distributed job schedulerDesign a cron-as-a-service from an empty canvas: millions of scheduled jobs fired on time, each run at least once and its effect exactly once.MediumSolve
Build a log aggregation pipelineDesign a log pipeline from an empty canvas: agents on every server ship batches, every line is indexed for search within seconds, and nothing is ever lost.MediumSolve
Build web analytics like Google AnalyticsDesign Google Analytics from an empty canvas: page-view beacons from millions of sites at 4,000 a second, aggregated into dashboards within seconds.MediumSolve
Build a ChatGPT-style chat appDesign ChatGPT from an empty canvas: streamed answers from a large model with quotas, conversations kept as context, and no outage when the model provider fails.HardSolve
Build enterprise RAG searchDesign a Glean-style enterprise search from an empty canvas: ingest company documents into a vector index and answer questions with a model, streamed, safely.HardSolve
Build an AI coding agent backendDesign the backend of a Claude Code-style coding agent from an empty canvas: agent turns that call a model, tools and sandboxes step by step, checkpointed, at a sane token bill.HardSolve
Build a meeting scheduler like CalendlyDesign Calendly from an empty canvas: anyone with a link sees a host’s free slots and books one, never twice, with calendars kept in sync.EasySolve
Build a container orchestrator like KubernetesDesign Kubernetes’ control plane from an empty canvas: one API for every change, a consistent store, thousands of watchers, a scheduler, and a heartbeat from every node.HardSolve
Build a container registry like Docker HubDesign Docker Hub from an empty canvas: pushes and pulls of images made of shared, content-addressed layers, through a pull storm when a big cluster scales out.MediumSolve
Scaling 16
One server on launch dayYour startup runs on a single API server behind a load balancer. Launch-day traffic is 1,500 requests a second.EasySolve
Black Friday spikeA shop handles 1,500 req/s comfortably. At 10 s, Black Friday traffic triples for 15 seconds.EasySolve
CPU-bound APIThe API renders pages on the server: 6 ms of CPU per request on 2 vCPUs per replica, at 1,500 req/s.EasySolve
Database out of CPUEvery read and write goes to one Postgres with 2 vCPUs, at 2,000 req/s. There is no cache.MediumSolve
Write-heavy ledgerA ledger service is 60% writes at 1,500 req/s on a 2-vCPU database. Read replicas won’t help writes.MediumSolve
The slow rampTraffic grows from 800 to 2,400 req/s over the run. The API is sized for today.EasySolve
Bursty trafficAverage traffic is 700 req/s, but every 8 seconds a 2-second burst brings 4× that. The API queue is short.MediumSolve
Autoscaling hits its ceilingThe API runs on a container platform that autoscales from 2 to 3 instances. Traffic needs more.MediumSolve
Losing a replica at peakTwo API replicas run at 70% each. At 10 s one of them dies (a bad host), and the other must take everything.MediumSolve
Search falls overEvery page calls a search index with 32 query slots and 4 ms of CPU per query on 4 vCPUs, at 1,200 req/s.MediumSolve
Every request checks the sessionEach API request validates its session with an auth service that has 40 slots. At 1,500 req/s it can’t keep up.EasySolve
Resizing images on uploadEvery upload is resized synchronously by an image service: 40 ms of CPU each on 4 vCPUs, at 300 uploads a second.MediumSolve
Launch day: nothing is sizedThe prototype goes live as it is: one API replica with 2 vCPUs, a 2-vCPU database with a 40-connection pool, and no cache, at 2,500 requests a second.MediumSolve
Auth and search on every requestEvery request checks a session with an auth service (40 slots) and runs a query on a search index (32 slots, 4 ms of CPU each), at 1,500 req/s.MediumSolve
Snowflake IDs from one serviceEvery new order gets a unique 64-bit ID from a central ID service (Snowflake style). Writes are 1,000 a second; the ID service runs on one vCPU.MediumSolve
Cold cache after a deployA 90%-hit cache shields Postgres at 2,500 req/s. At 12 s a deploy flushes the cache, and every read hits the database.MediumSolve
A cache too small5 million products, 2 KB each, but the cache has 512 MB. Popular items fit; most don’t.MediumSolve
TTL too shortThe cache is big enough, but entries expire after 2 seconds, before most are read again.MediumSolve
Flush during a spikeTraffic doubles at 8 s, and at 12 s someone flushes the cache. Two bad things at once.MediumSolve
A cache that is small and forgetful5 million products, 2 KB each, in a 256 MB cache whose entries expire after 10 seconds. The database behind it has 2 vCPUs.MediumSolve
CDN purge at peakAt 12 s a deploy purges the whole CDN while 2,500 requests a second are flowing. Every request now goes to an origin sized for the usual few misses.MediumSolve
Search autocomplete: 40,000 keystrokes a second under 80 msAsked at Google and Amazon. Every keystroke runs a prefix query on the full-text search index. Suggestions must come back within 80 ms at p99, and at 8 s a news event raises traffic by half.HardSolve
Nearby search at lunchtimeAsked at Yelp, Google (Maps) and Uber Eats. Every “near me” search runs a PostGIS radius query, as do place pages and reviews, on one database with read replicas. At 8 s lunchtime doubles traffic for 10 seconds.HardSolve
Databases 13
Slow database rescueThe database answers reads in 9 seconds. The API gives up after 5 seconds and retries twice.EasySolve
URL shortener4,000 req/s, 95% redirects. Popular links are far more popular than the rest.MediumSolve
Key-value store out of partitionsA session store spreads keys over 4 partitions that each handle 400 operations a second. Traffic is 2,500 a second.MediumSolve
The 1.5-second queryA missing index makes every read take 1.5 seconds. The API gives up after 1 second and retries.EasySolve
Connection pool exhaustedThe database allows 10 connections. Each query holds one for about 20 ms, at 1,500 queries a second.MediumSolve
Stale reads from replicasRead replicas lag the primary by 2 seconds. Users edit their profile and immediately see the old one.MediumSolve
Lost updates on a hot rowTwo requests read a counter, both add one, both write it back: one increment disappears. Popular rows are updated constantly.MediumSolve
One viral keyA URL shortener where one link went viral: its partition melts while the rest are idle.MediumSolve
Stale reads and lost updatesA profile service reads from replicas that lag 1.5 seconds and updates popular rows with read-then-write. Users see old profiles, and some edits vanish.MediumSolve
Every checkout takes a distributed lockTo stop double-spending, each checkout takes a lock from a lock service (fencing tokens, leases) before it writes. 800 checkouts a second; the lock service has 2 vCPUs.MediumSolve
Metrics: every scrape lands at onceA metrics system scrapes 400 targets a second on average, but scrapes line up every 15 seconds and 5× as many land at once on a 2-vCPU time-series database.MediumSolve
Object storage: one hot prefixAll images live under one key prefix, which the store limits to about 5,500 reads a second. Reads arrive at 7,000 a second.MediumSolve
Tournament leaderboard: 40,000 score updates a secondAsked at Riot, Epic and Activision. Every score update is an UPDATE in Postgres, and the top-100 and “my rank” pages run ORDER BY on the same table. At 10 s the tournament’s final round doubles traffic.HardSolve
Resilience 13
A rate limit below real trafficA rate limiter in front of the API allows 1,000 req/s. Normal peak traffic is 1,300 req/s, so real users get 429s.EasySolve
Retry stormThe database slows down. The API times out after 300 ms and retries 3 times with no backoff budget, tripling the load.MediumSolve
A slow dependency drags everyoneOne downstream service gets slow. API requests pile up waiting on it, and soon even requests that don’t need it fail.MediumSolve
Timeouts and retries feed each otherThe database answers in 70 ms at the median but 350 ms at p99. The API gives up after 150 ms and retries 3 times with no backoff, so the slow tail turns into extra load.MediumSolve
Checkout: retries into an outageCheckout calls one payment provider. The API retries failed calls 3 times immediately. At 10 s the provider is down for 15 seconds, and the retries triple the pile-up.MediumSolve
Distributed rate limiter: one counter storeA global rate limiter keeps every client’s counter in one Redis node, and every request checks it. The node handles about 1,600 checks a second; traffic is 2,400.MediumSolve
API gateway at its request limitEvery request passes an API gateway that handles auth and routing. Its plan allows 90,000 requests a minute; peak traffic is 2,000 a second.EasySolve
Active-active: one region goes downTwo regions each serve half the traffic at about 60% load. At 10 s the US region fails for 15 seconds and the load balancer sends everything to Europe.MediumSolve
Feature flags fetched on every requestEvery request asks a remote feature-flag service which features are on. The service has 20 slots and answers in about 15 ms, at 2,000 requests a second.EasySolve
Payments: no double charges, no lost ledger entries, an acquirer outageAsked at Stripe and PayPal. The charges API writes the payment, then publishes a ledger event (two separate writes). At 10 s the ledger queue is down for 5 seconds; the ledger worker loses 2% of its acknowledgements and is not idempotent; at 18 s acquirer A is down for 10 seconds.HardSolve
Stays search and booking: a slow payment provider takes everything downAsked at Airbnb and Booking.com. One API serves search, listing pages and bookings from one Postgres database, and bookings call the card provider in the same thread pool. At 6 s holiday traffic rises by half; at 10 s the card provider gets eight times slower for 12 seconds, and searches start failing too.HardSolve
Duplicate chargesPayment events go through a queue that delivers at least once. Some acknowledgements are lost, so some customers are charged twice.MediumSolve
Events that never arriveThe API writes the order to the database, then publishes an event. At 10 s the queue is unavailable for 5 seconds: orders are saved but their events vanish.MediumSolve
Consumers can’t keep upKafka keeps events for 30 seconds. One consumer processes them, slower than they arrive.EasySolve
Too few partitionsThe topic has 2 partitions. Each partition is read by one consumer at a time, however many you add.MediumSolve
Orders: lost events and double emailsThe order API writes the order, then publishes an event (dual write). At 10 s the queue is down for 5 seconds. The workers that send confirmation emails lose 2% of their acknowledgements and are not idempotent.MediumSolve
Two partitions, slow consumersThe topic has 2 partitions and the consumer group processes each event in about 5 ms. Events arrive at 2,000 a second and are kept for 30 seconds.MediumSolve
Real-time analytics: consumers lagClickstream events pour into a topic that keeps 30 seconds of data. The analytics consumers process them slower than they arrive.MediumSolve
CDC sync: the connector can’t keep upDatabase changes are copied to other services by a change-data-capture connector with 8 slots, about 15 ms per change, at 1,200 changes a second.MediumSolve
Ad click aggregator at 20,000 clicks a secondAsked at Meta and Google. Every ad click is written as a row to one Postgres table, and advertisers’ dashboards read counts from the same table. At 8 s a big campaign doubles clicks for 6 seconds. Clicks time out and dashboards stall.HardSolve
Trending videos: top-K at 30,000 views a secondAsked at YouTube and Twitter/X. Each view increments a per-video counter row in a key-value store, and the trending page reads those counters. Views follow a power law, so one viral video hammers one partition; at 10 s a new release doubles views.HardSolve
Notification burst: a campaign, a quota and a provider outageAsked at Uber, DoorDash and Airbnb. A marketing campaign sends notifications through an API that calls the push provider directly and retries twice. At 6 s the campaign triples the rate for 8 seconds, the provider’s quota is 150,000 a minute, and at 15 s the provider is down for 8 seconds.HardSolve
Metrics platform: 40,000 points a second and dashboards on topAsked at Datadog, Google and Netflix. Every metric point is inserted into one column store, and dashboards query the same raw points. At 10 s an incident doubles the points (and everyone opens dashboards).HardSolve
Scheduled jobs: everything fires at the top of the minuteAsked at Airbnb, Netflix and Uber. A scheduler API calls job runners synchronously, and each job takes 400 ms. Cron schedules line up, so at 6 s and 16 s ten times the usual jobs fire for 3 seconds, and at 9 s a runner dies mid-job.HardSolve
Security 8
Edge defence: survive a DDoS1,500 req/s of real users. At 15 s a 20,000 req/s flood hits for 15 s. Nothing stands in front of the load balancer.MediumSolve
A 50,000 req/s floodAt 10 s a volumetric DDoS of 50,000 req/s hits a system with nothing in front of the load balancer.MediumSolve
Bot trafficAt 8 s, 8,000 req/s of bots scrape and spam your API alongside real users.EasySolve
Credential stuffingAttackers replay leaked passwords against your login at 5,000 req/s. Every attempt hits the database.MediumSolve
A sale, and bots join inA flash sale doubles traffic at 10 s. At 8 s, 6,000 req/s of bots start hitting the same API, which has two small replicas and nothing in front of it.MediumSolve
Concert on-sale: six times the traffic, bots and double bookingAsked at Ticketmaster and Amazon. Tickets go on sale at 5 s: real fans arrive at six times normal traffic for 15 seconds, and 8,000 requests a second of bots start a second earlier. Everything hits one API and one Postgres database; two fans can hold the same seat.HardSolve
Public API under abuse: credential stuffing, scrapers and a floodAsked at Stripe, Cloudflare and Okta. A public API with a login endpoint sits behind only a load balancer. From 3 s scrapers send 5,000 requests a second, from 5 s credential stuffing sends 6,000 login attempts a second at the password-hashing service, and at 12 s a 40,000 requests-a-second flood arrives.HardSolve
AI systems 13
RAG support assistant40 questions/s, 4,000-token prompts, a 6M tokens/min quota and a 25% answer-cache hit rate.MediumSolve
Agent runaway15 tasks/s. The agent makes 8 model and tool calls per task; the model allows 4,000 calls a minute and 120 concurrent streams.MediumSolve
LLM gateway with fallback40 chats/s through your AI gateway and a prompt cache to one hosted model. At 20 s the provider has a 15 s outage.MediumSolve
MCP agent platform4 tasks/s. The agent thinks (model) and acts (tools through an MCP gateway) 6 times per task, with vector memory and results in S3.MediumSolve
Document Q&A on S3 and a vector DB40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.MediumSolve
RAG at twice the trafficYour support assistant doubles to 80 questions a second. The model’s token quota does not.MediumSolve
Semantic cache that never hitsThe semantic cache’s similarity threshold is so strict it hits 5% of the time. Nearly every question reaches the model.EasySolve
Model provider outageYour chat product depends on one model provider. At 20 s it is down for 15 seconds.MediumSolve
Ten steps of MCP toolsAgents call MCP tools 10 times per task. A 1% tool error rate becomes a 10% task failure rate.MediumSolve
Uploads starve questionsQuestions and document ingestion share one embedding quota. Uploads double and both paths start failing.MediumSolve
RAG: quota by day, outage by nightThe support assistant runs at 80 questions a second against a tight token quota, and at 20 s its only model provider goes down for 10 seconds.MediumSolve
Agent: a slow tool and a tight quotaAn agent takes 6 steps per task at 15 tasks a second. Every step calls a tool server with 8 slots that takes 300 ms, and the model allows 4,000 requests a minute.MediumSolve
Real world 52
YouTube end to end20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.HardSolve
Concert ticket dropTickets go on sale at 10 s: traffic jumps 5× for 10 seconds, mostly people refreshing the same few pages.MediumSolve
Breaking newsA news article goes viral: traffic ramps to 4× over the run. Pages are cacheable at the edge, but the CDN is tiny.MediumSolve
Chat history on one databaseA messaging app stores every message and reads conversation history from one database: 70% reads at 2,000 req/s.MediumSolve
Game leaderboardMillions of players check the leaderboard after every match; the top of the board is read constantly.MediumSolve
Core banking at month endMonth-end batch traffic doubles writes at 10 s. The database runs hot and crashes under CPU pressure.MediumSolve
Flash-sale shopping cartsDuring a flash sale everyone updates the same few product rows (stock counts). Updates are read-then-write.MediumSolve
Leaderboard: one hot key, one shardOne viral link gets most of the reads. The cache in front of the store hits only 60%, and its single shard handles at most 1,500 operations a second.MediumSolve
Breaking news on a tiny CDNA story goes viral: traffic ramps to 4×. The CDN has 256 MB of edge storage for half a million pages, and the origin API renders pages at 6 ms of CPU each on 2 vCPUs per replica.MediumSolve
WhatsApp: WebSocket gateways at capacityEvery message and receipt passes a WebSocket gateway that holds the open connections. Messages arrive at 2,200 a second; the gateway has one vCPU at 0.6 ms a message.MediumSolve
News feed: fan-out workers fall behindNew posts go into a stream; fan-out workers copy each one into followers’ feeds. One worker can’t keep up, and the stream only keeps 30 seconds.MediumSolve
Instagram Stories: views reach the originStories are photos and short videos viewed millions of times, then they expire. The CDN is too small, so most views go back to the origin.MediumSolve
Dropbox: chunk metadata on one databaseFiles sync in 4 MB chunks, and every chunk writes a metadata row. Sync traffic is 70% writes at 1,200 a second on a 2-vCPU database.MediumSolve
Web crawler: fetchers wait on slow sitesA crawler fetches 300 pages a second. A fetch takes about 300 ms (slow sites, politeness delays), and there are 60 fetcher slots.MediumSolve
Netflix: a new season at midnightA new season drops: traffic jumps 4× at 10 s for 15 seconds. The CDN is undersized, so the origin sees a large share of the spike.MediumSolve
Spotify: the playlist everyone playsToday’s Top Hits is read by everyone, and all of it comes from one cache node that handles 1,500 operations a second. Reads arrive at 2,700 a second.MediumSolve
Live streaming: transcoders fall behindEach live segment is transcoded into several renditions: about 1.5 s per segment, 20 segments a second, on 24 transcoder slots.MediumSolve
Uber: driver location updatesDrivers send their location every few seconds: 2,500 updates and lookups a second into a geospatial index with 4 vCPUs and 2 ms of CPU each.MediumSolve
Food delivery: dispatch at dinner rushOrders ramp from 400 to 1,200 a second over the evening. Every order asks the dispatch service to match a courier, and dispatch was sized for lunch.MediumSolve
Hotel booking: two guests, one roomTwo guests check the last room, both see it free, both book it. Popular hotels sell out on the same rooms at once.MediumSolve
Fraud scoring on the checkout pathEvery checkout is scored by a fraud model before payment: 1,000 checkouts a second, 5 ms of CPU each, on 4 vCPUs.MediumSolve
Ad bidding: 150 ms or nothingEvery page view runs an ad auction, and the exchange drops bids slower than 150 ms. The bidder takes 5 ms of CPU per bid on 4 vCPUs, at 1,200 auctions a second.MediumSolve
Ride dispatch: 30,000 location pings a secondAsked at Uber and Lyft. Drivers ping their location every few seconds, riders ask for ETAs and request rides, and all of it goes through one API to one PostGIS database. At 10 s a concert lets out and traffic rises 80%.HardSolve
Chat delivery: 20,000 messages a second and a push outageAsked at Meta (WhatsApp, Messenger) and Slack. Every message is written to Postgres and then pushed to the recipient’s phone in the same request; the app polls the same database to sync, and presence heartbeats are database writes too. At 12 s the push provider is down for 8 seconds, and sending stops.HardSolve
Real-time document editing: an all-hands documentAsked at Google (Docs) and Figma. Every keystroke is a transaction that locks the document row in Postgres, cursors are database writes, and opening a document replays it from the database. At 8 s a company all-hands starts and traffic triples for 10 seconds.HardSolve
Premiere night: video at three times the trafficAsked at Netflix and YouTube. Every video segment is served by origin servers reading object storage, manifests come from Postgres, and DRM licenses from license servers, all through the same origin fleet. At 6 s a premiere starts and traffic triples for 12 seconds; at 13 s the segment storage is unavailable for 4 seconds.HardSolve
Market open: six times the traffic in one secondAsked at Robinhood and Coinbase. Apps poll quotes from Postgres, place orders through the same API (which calls the exchange and writes positions in the same request), and read portfolios from the same database. At 5 s the market opens and traffic jumps six-fold for 10 seconds.HardSolve
Live comments on a celebrity streamAsked at Meta (Facebook Live) and Twitch. Thirty thousand viewers a second poll a comments API that reads Postgres; posting writes to the same table. At 8 s a celebrity joins and traffic triples for 10 seconds.HardSolve
File sync: every 4 MB chunk goes through the APIAsked at Dropbox and Google (Drive). Uploads and downloads of 4 MB chunks are proxied through the file API, which spends 25 ms of CPU and up to 600 ms per chunk; metadata reads hit Postgres. At 9 s a team shares a large folder and traffic doubles.HardSolve
Instagram: from launch build to real trafficInstagram as a small team first shipped it: no timeline or profile cache, no media CDN, and every service an eighth of its real size. Real traffic arrives: 4,000 requests a second, 70% feed loads, 10% profiles, 15% likes and 5% new posts with photos to resize. Feeds time out and photos stall.HardSolve
Facebook news feed: from launch build to real trafficThe news feed as first built: no feed cache in front of the timeline store, and the feed API, ranking model, social graph and fan-out workers at an eighth of their real size. 5,000 requests a second arrive: 85% feed loads that are ranked on every load, 15% new posts fanned out to followers. Most feed loads fail.HardSolve
Netflix: from launch build to real trafficNetflix without Open Connect or EVCache: every request reaches the origin, title metadata is read from the catalog database each time, and the playback, browse, license and recommendation services are an eighth of their real size. 6,000 requests a second: half are playback starts, 45% browsing, 5% DRM licenses. Playback starts fail.HardSolve
Slack: from launch build to real trafficSlack as first built: channel history read straight from MySQL with no cache, no read replicas, and the message, history and presence services, fan-out workers and search at an eighth of their real size. 5,000 requests a second: 45% reading history, 40% sending messages that fan out to channels, search and push, 15% presence. Messages and history time out.HardSolve
ChatGPT: from launch build to real trafficA chat product on launch-week hardware: the GPU prefill and decode pools, the KV cache store, moderation and the chat API are an eighth of the size real traffic needs. 300 chats a second arrive with long prompts. Answers queue for 20 seconds and most fail.HardSolve
Deep research: from launch build to real trafficA multi-agent research product on its first servers: the lead agent fans each question out to sub-agents that search and browse the web, and the browser tool, sub-agent pool and model capacity are an eighth of their real size. Questions arrive one a second and each takes minutes. One in ten fails.MediumSolve
Uber: from launch build to real trafficUber as first shipped: no ETA cache, and the location, ETA and ride services, the geo index, its updaters and the notification workers at an eighth of their real size. 5,000 requests a second: 60% driver location pings, 30% ETA lookups, 10% ride requests. Half of all requests fail.HardSolve
Stripe: from launch build to real trafficA payments API on its first servers: no charge status cache, and the charges API, status API, fraud scoring, ledger and webhook delivery workers at an eighth of their real size. 1,500 requests a second: 70% charges that must never apply twice, 30% status checks. About half of all requests fail.HardSolve
Twitch: from launch build to real trafficLive streaming as first built: no manifest cache in front of the live streams catalog, and the playback and chat APIs, chat fan-out, live ingest and transcoders at an eighth of their real size. Real viewers arrive, watching and chatting, and three in four requests fail.HardSolve
Twitter / X timeline: from launch build to real trafficThe home timeline as first built: timelines read from Manhattan with no Redis cache, and the timeline API, ranker, tweet API, fan-out workers and search at an eighth of their real size. Real traffic of timeline reads, new tweets and searches arrives, and most requests fail.HardSolve
Amazon storefront: from launch build to real trafficA storefront as first built: product pages read DynamoDB with no product cache, and the product, cart and checkout APIs, search, recommendations, the orders database, payments and fulfilment workers at an eighth of their real size. Shoppers arrive at real traffic and half of all requests fail, checkout included.HardSolve
Coding agent platform: the model provider goes downA coding agent platform: each task is a loop of model calls, tool calls, code runs in a sandbox and checkpoints, minutes long. Twenty seconds in, the model provider has an outage for 30 seconds. Every step that needs the model fails, and with it the whole task: two in three tasks are lost.MediumSolve
YouTube: from launch build to real trafficYouTube as first built: no media CDN, no metadata cache in front of Vitess MySQL, and the watch, search, comment and upload services, recommendations, view counting and transcoding at an eighth of their real size. 20,000 requests a second arrive, routed by path. Almost everything fails.HardSolve
LLM serving 29
Token quota at launchA chat product sends 2,000-token prompts at 50 requests a second. The provider allows 4 million input tokens a minute.EasySolve
Not enough concurrent streamsEach answer streams for about 4 seconds, and the deployment allows 100 concurrent streams. Traffic is 40 requests a second.EasySolve
Provider outage at peakAt 15 s the only model provider goes down for 15 seconds, at 30 chats a second.MediumSolve
Timeouts on long answersAnswers are 800 tokens at 70 tokens a second, about 12 seconds each. The app gives up after 8 seconds.EasySolve
Prompt cache that rarely hitsAn exact-match prompt cache sits in front of the model but hits only 5%: timestamps in the prompt make every prompt unique. The model’s token quota is tight.MediumSolve
Self-hosted GPUs at capacityYou serve an open-weights model on your own GPU pool with 32 slots. Requests take about a second; traffic is 50 a second.EasySolve
GPU autoscaling ceilingThe GPU pool autoscales from 2 to 3 nodes of 16 slots each, but traffic needs more than 3 nodes.MediumSolve
Prefill is the bottleneckServing is split into prefill and decode pools (disaggregated serving). Prompts got longer: the prefill pool can’t keep up, and time to first token suffers.MediumSolve
Decode is the bottleneckDisaggregated serving with a small decode pool: answers got longer and decode slots fill up.MediumSolve
KV cache not reusedA KV-cache tier sits in front of prefill, but only 10% of requests reuse a cached prefix: the system prompt changes per user, so prefixes never match.MediumSolve
Draft model can’t keep upSpeculative decoding speeds up answers, but the single draft-model GPU is now the bottleneck at 150 requests a second.MediumSolve
Too many fine-tunes on one GPUOne multi-LoRA server hosts 200 customer adapters on a shared base model. At 60 requests a second the shared batches overflow.MediumSolve
AI gateway key limitAll traffic goes through an AI gateway whose API key allows 1,500 requests a minute. Traffic is 40 a second.EasySolve
PII redaction on the hot pathEvery prompt is scanned by a PII redactor with 2 vCPUs and 10 ms of CPU per prompt, at 400 requests a second.MediumSolve
An LLM judge on every answerEvery answer is graded inline by an LLM judge with 20 slots before it returns. The judge is now the slowest part.MediumSolve
Tracing drops spansEvery request writes a trace to an observability service limited to 2,000 spans a minute. At 60 requests a second, traces are rejected and requests fail with them.EasySolve
Image generation queueAn image generator takes 4 seconds per image on 40 slots. Users request 15 images a second.EasySolve
Retries into a rate limitThe app retries every failed model call 3 times immediately. When the token quota is hit, retries multiply the 429s.MediumSolve
Real-time pool starved by batch jobsMost GPUs went to the offline batch queue. The real-time pool that serves live users has 8 slots and a 30-request queue, at 20 requests a second of 900 ms answers.MediumSolve
Model routing sends everything to the big modelA router splits traffic between a large model and a small one, but it is set to send 90% of chats to the large model, which has 200 concurrent streams. 60 chats arrive a second.MediumSolve
A/B test of a new model versionAn experiment splits chats 50/50 between the current model (A) and a new version (B). B was deployed with only 60 concurrent streams; 40 chats arrive a second.MediumSolve
More replicas, slower answersCoding agents run 20-step tasks, 1 a second, on 8 self-hosted model replicas with prefix caching. An inference router spreads the calls round robin, so each step of a session lands on a different replica from the last.MediumSolve
One replica owns the hot prefixA support-agent fleet sends 30 requests a second through a prefix-aware router to 6 model replicas. 85% of requests start with the same 12,000-token system prompt and tool schemas, and the router always sends a prefix to the replica that holds it.MediumSolve
A minute to load the weightsA self-hosted 70B model runs on 4 GPU replicas that autoscale to 16. At 20 s traffic triples, from 40 to 120 requests a second. Each new replica must load 140 GB of weights before it serves: 60 s here (2–5 minutes in practice).MediumSolve
Background agents starve the chatOne GPU pool with 64 slots serves interactive chat (30 requests a second) and a background agent that triages every ticket: 10 tasks a second, 8 model calls each. Nothing limits how many background tasks run at once.MediumSolve
The eval run eats the production quotaAn agent product serves 20 chats a second on one provider key with 6 million input tokens a minute. A CI regression gate runs the eval suite on the same key: 2 eval tasks a second, 20 model calls each, 4,000 tokens a call.MediumSolve
Shadowing the new model doubles the loadA fine-tuned candidate model is deployed as an adapter on the production GPU pool, and a traffic mirror copies 100% of live requests to it to compare answers. Production is 60 requests a second on a pool that serves about 80.EasySolve
An AI assistant at 200 chats a second: cost and an outageAsked at OpenAI, Anthropic and AI product companies. Every chat sends a 6,000-token prompt (system prompt, tools and retrieved context) to a frontier model and streams back 400 tokens. At 15 s provider A has a 12-second outage. The token bill alone is several times the budget.HardSolve
RAG and retrieval 22
Query rewriting on every questionAgentic RAG rewrites each question with a small model before retrieval. The rewriter has 8 slots at 80 questions a second.MediumSolve
GraphRAG traversals melt the CPUMulti-hop questions traverse a knowledge graph: 10 ms of CPU each on 8 vCPUs, at 1,200 questions a second.MediumSolve
Hybrid search saturatedRetrieval combines keyword and vector search. The hybrid search service has 40 slots at 1,500 queries a second.EasySolve
Reranker on a single GPUA cross-encoder reranks the top 50 chunks for every question on one GPU with 16 slots, at 300 questions a second.EasySolve
Embedding calls queue upEvery question is embedded first, but the embedding deployment allows 8 concurrent calls at 400 questions a second.EasySolve
Semantic cache too strictA semantic cache in front of a RAG assistant hits 10%: its similarity threshold is 0.99. The model quota can’t take the rest.MediumSolve
Context assembly is slowA context builder stitches retrieved chunks, history and instructions for each question. It has 40 slots at 1,000 questions a second.MediumSolve
OCR before every answerA document assistant OCRs scanned pages on demand: 12 slots, 1.5 seconds per page, 15 pages a second.MediumSolve
Too many chunks in the promptThe assistant stuffs 6,000 tokens of chunks into every prompt at 30 questions a second, against an 8-million-token-a-minute quota.MediumSolve
Corrective RAG evaluator bottleneckCorrective RAG grades retrieved chunks with an evaluator model before answering. The evaluator has 20 slots and grades every question.MediumSolve
Retrieval traces over quotaEach RAG question writes retrieval traces to an observability API capped at 3,000 a minute, at 80 questions a second.EasySolve
Multi-hop retrieval loopsAn agentic retriever makes up to 8 search hops per question, and each hop hits the search service.MediumSolve
Rerank results never reusedA cache in front of the reranker hits 15%, because it is keyed on the raw question text including whitespace and casing.MediumSolve
GraphRAG index shared with ingestionThe GraphRAG index answers questions and absorbs entity extraction from new documents at the same time, at 1,500 operations a second.MediumSolve
Scrapers hammer the RAG APIAt 8 s, 3,000 req/s of scrapers start querying your RAG assistant, eating model capacity meant for users.MediumSolve
Re-indexing 100 million documentsAn ingestion service parses, chunks and embeds documents for search. A re-index of 100 million documents pushes 80 documents a second through it, but it has 20 slots at about half a second per document.MediumSolve
Multi-tenant RAG: the permission filterEach customer may only see its own documents, so every retrieval passes a document-ACL filter after hybrid search. The filter has 2 slots and takes about 60 ms; questions arrive at 60 a second.MediumSolve
Multimodal RAG: the vision encoderQuestions about PDFs with charts and scanned tables send each page image through a vision encoder before retrieval. The encoder has 20 slots and takes about 0.7 s per image; 40 questions arrive a second.MediumSolve
Conversational memory on every turnA chat assistant keeps long conversations within the context window by summarising older turns with a small model before every answer. The summariser has 12 slots at about half a second; 40 turns arrive a second.MediumSolve
Citation checking slows every answerTo cut hallucinations, every answer is checked against the retrieved passages: each cited sentence must be supported. The checker has 16 slots at about 0.6 s; answers arrive at 50 a second.MediumSolve
An agent that takes 12 stepsA support agent takes 12 model-and-tool steps per ticket. Tools fail 1% of the time, and every failure sinks the whole task.EasySolve
MCP gateway quotaEvery tool call goes through an MCP gateway (auth, allow-lists, quotas) limited to 600 calls a minute. Agents make about 18 tool calls a second.EasySolve
One slow tool serverA CRM tool server with 8 slots takes 300 ms per call. Agents call it every step.EasySolve
Code sandboxes run outA data-analysis agent runs Python in sandboxes: 6 sandboxes, about 1.2 s per run, 4 runs per task at 2 tasks a second.MediumSolve
Browser automation bottleneckA web agent drives headless browsers: 16 browsers, about 2.5 s per action, 3 actions per task at 3 tasks a second.MediumSolve
Computer-use desktops are scarceA computer-use agent clicks through a legacy app on virtual desktops: 10 desktops, about 3 s per action, 4 actions per task.MediumSolve
Approvals pile upA finance agent asks a human to approve every payment. Five reviewers take about 20 s each; tasks arrive at 0.5 a second.MediumSolve
Agent memory on a small storeThe agent reads and writes long-term memory every step, on a memory store with 2 slots and 60 ms per call.EasySolve
Same tool call, every stepAgents repeat the same read-only lookups, but the tool-result cache hits 10% because it is keyed on raw arguments with timestamps.MediumSolve
Checkpointing every stepDurable execution saves the agent’s state after every step to a checkpoint store with 10 slots and 50 ms writes.MediumSolve
Tool search on every stepWith 400 tools, the agent searches for the right few each step. The tool-search service has 6 slots.MediumSolve
Compaction calls overflowLong sessions are compacted every step by a summariser with 20 slots, about 600 ms each.MediumSolve
Agents burn the token quotaEach step sends 3,000 tokens of context. Agents make 24 model calls a second against a 3-million-token-a-minute quota.MediumSolve
Model outage mid-taskThe agent’s model provider goes down for 15 seconds. Every task in flight fails at its next step.MediumSolve
Agent model out of streamsAgents hold a model stream for about 2 s per step. The deployment allows 30 streams; agents need about 50.EasySolve
Partner agent goes offlineYour travel agent books hotels through a partner’s agent over A2A. The partner is down from 15 s to 35 s.MediumSolve
Partner agent at capacityA partner agent you delegate to over A2A handles 20 tasks at once, about 2.5 s each. You send it 12 a second.MediumSolve
Workflow engine overloadDeterministic steps run on a workflow engine with 6 slots (200 ms each) between the agent’s decisions.MediumSolve
Loading skills every stepThe agent loads skills (instructions and scripts) from a registry with 4 slots on every step.EasySolve
Agent jobs run twiceAgent jobs are queued and run by workers. When an acknowledgement is lost, the queue redelivers and the job’s side effect (sending an email) happens twice.MediumSolve
Tracing every agent stepEach step writes a trace span to an observability API capped at 1,200 a minute; agents make about 30 steps a second.EasySolve
100,000 agent tasks a minuteAn agent platform runs every task as a 6-step loop. The orchestrator can hold 80 tasks at once; each task takes about 15 seconds, and 10 tasks start every second.MediumSolve
Judging every agent stepTo evaluate agents, an LLM judge scores every step of every trajectory (was the tool call right, did the step help) while tasks run. 8 tasks a second × 6 steps reach a judge with 20 slots at about 0.6 s.MediumSolve
A new browser for every pageResearch agents read about 8 web pages per task, 30 tasks a minute. The hosted-browser plan allows 100 sessions at once but only 50 new sessions a minute, and the browser tool opens a fresh session for every page.MediumSolve
Sandboxes held during approvalsA coding-agent platform runs 7 tasks a second with 3 sandbox runs each, about 3.5 s per run, on a provider that allows 100 sandboxes at once. On 30% of runs the sandbox asks a person to approve a command and stays reserved (and billed) while it waits, about 12 s. Times are scaled: a second here stands for a minute.MediumSolve
Two messages, one threadA chat agent handles 300 turns a second. Each turn reads the thread’s state, runs the agent and writes the state back. Users often send a second message before the first run finishes, so two runs on one thread read the same state and the last write wins.MediumSolve
The prompt cache expires between stepsData-analyst agents send about 82,000 input tokens per step: 26,000 shared (system prompt and tools) plus the task’s growing history. The provider keeps a cached prefix for 5 minutes after its last use and does not count cached tokens against the 3 million tokens a minute quota. Between steps, each agent waits 6–8 minutes for a warehouse query. Times are scaled: a second here stands for a minute.MediumSolve
Ninety-three tools in every promptA coding agent loads the full GitHub MCP toolset, 93 tools and about 55,000 tokens of definitions, into every prompt: 60,000 input tokens per call. This provider counts cached tokens against its 6 million tokens a minute quota, so prompt caching lowers the bill but not the quota.MediumSolve
Online evals silently lose trajectoriesAn agent service samples 5% of its trajectories, 12 a second, into a stream that keeps 20 seconds of data. LLM-judge consumers score them for regressions, but four consumers manage only about 7 a second.MediumSolve
Multi-agent and harness 14
Subagents swamp the search toolA lead agent fans each question out to 8 parallel subagents, and each subagent searches the web. The search tool has 8 slots.MediumSolve
Multi-agent token billEvery subagent sends 3,000 tokens of context. Eight subagents per question multiply token use past the provider quota.MediumSolve
The workflow outgrows its historyLong-running coding agents run on a durable workflow engine. Every step passes the whole conversation in and out, so the first step records 60 KB and each later one 40 KB more than the last. The engine terminates any run whose history passes 50 MB. Tasks take 50 steps.MediumSolve
Retries at every hop of an agent chainA concierge agent calls a booking agent, which calls an airline’s agent over A2A (40 slots, about 4 s a request). Both callers time out after 8 s and retry twice at once. At 15 s bookings jump 2.5× for 10 seconds.MediumSolve
AI security and governance 17
Prompt injection on a support botAt 8 s, attackers send 400 messages a second with hidden instructions (“ignore previous instructions…”). Nothing screens them.MediumSolve
Injected instructions in emailAn email agent reads inbound messages and can call tools. Attackers send 200 emails a second containing instructions to forward data.MediumSolve
Poisoned web pages in RAGA web-search assistant retrieves pages that contain hidden instructions. 300 poisoned requests a second arrive alongside real users.MediumSolve
Malicious issue textA coding harness reads issue descriptions. Attackers file issues with instructions to exfiltrate secrets, 100 a second.MediumSolve
A token exchange on every tool callBefore each tool call, an enterprise MCP agent exchanges the user’s identity for a short-lived, audience-bound token at the company identity provider, then throws the token away. 5 tasks a second × 8 steps reach an identity provider that allows 1,200 token requests a minute.MediumSolve
One tenant’s loop starves everyoneA multi-tenant agent platform shares one provider key of 4,000 requests a minute. Most tenants run 6-step tasks, 5 a second. One tenant, Acme, has a buggy two-agent loop that runs 40 steps per task, 1.5 tasks a second.MediumSolve
A filter is not a permissionAn ops copilot reads public issues (untrusted), queries private repos and the customer database (private data), and posts comments, PRs and webhooks (outbound). That is the lethal trifecta. A prompt-injection classifier that stops 97% of attacks guards it, and attackers file 50 poisoned issues a second for 15 seconds.MediumSolve
AI products 21
AI feedback: thumbs-up events go missingEvery thumbs-up or thumbs-down on an AI answer is saved, then published as an event for the offline evaluation and training pipeline. At 10 s the queue is down for 5 seconds: the votes are saved but their events vanish, so the next release is evaluated on biased data.MediumSolve
Support bot launch dayA customer-support bot with a semantic cache, retrieval and a hosted model launches to 60 questions a second. The token quota wasn’t raised.MediumSolve
IDE code completionsInline completions: 400 requests a second, short outputs, and only 64 streams on the completion model.MediumSolve
AI search engine retrievalAn AI search engine rewrites the query, runs hybrid search and answers. Hybrid search has 30 slots at 1,200 queries a second.MediumSolve
SRE agent runs too longAn on-call SRE agent investigates incidents in 15 steps of model calls and tool queries.MediumSolve
Medical scribe: redaction bottleneckA medical scribe transcribes visits, redacts PII and drafts notes. The redactor has 1 vCPU and 20 ms per note, at 100 notes a second.MediumSolve
Legal research with GraphRAGLegal questions traverse a citation graph: 12 ms of CPU each on 4 vCPUs, at 500 questions a second.MediumSolve
Sales agent reads hostile repliesA sales agent reads inbound replies and updates the CRM. A campaign of replies with injected instructions arrives at 150 a second.MediumSolve
Video generation farmText-to-video takes about 20 s per clip on 40 GPUs; users request 3 clips a second.MediumSolve
Recommendations: the online feature storeEvery recommendation reads user and item features from an online feature store before the ranking model scores candidates. The store holds 6 reads at once at about 3 ms each; 2,500 requests arrive a second.MediumSolve
Background agents poll GitHub until throttledBackground coding agents open a pull request, then wait for CI by polling GitHub every 10 s, about 12 checks per task. All agents share one GitHub App installation, limited to 15,000 requests an hour (250 a minute). 70 tasks start a minute.MediumSolve
The CRM allows 25 report queriesA revenue-ops agent runs CRM reports through an MCP server. The CRM allows 25 report queries at once and rejects the rest immediately, with no queue. Each report takes about 2.4 s, and the agents ask for 12 a second.MediumSolve
Voice agent: the model’s slow tailA voice agent transcribes speech (100–150 ms), asks a model (first token: 250 ms usually, 2.4 s for the slowest 1%) and speaks the answer. The call to the model waits up to 8 s and has nowhere else to go. Callers notice any silence over about a second.MediumSolve
Kubernetes and containers 15
Traffic before the pod is readyThe API scales on CPU from 6 pods. Each pod takes 20 s to warm up after its container starts, and there is no readiness probe. At 40 s traffic jumps 2.5×.EasySolve
The health check restarts busy podsEight pods, each limited to 1 vCPU, can absorb a 2.2× surge by queueing requests for a second or two. Their liveness probe has the default 1 s timeout and is answered by the same busy request threads.MediumSolve
OOMKilled at peakEach pod uses 300 MB plus 8 MB for every request it holds (about 250 ms each, waiting on other services), under a 512 MB memory limit. Normal traffic is fine; at 40 s a 2.6× surge arrives.MediumSolve
Throttled by the CPU limitEach pod needs about 0.6 vCPU at normal traffic but has a 0.5 vCPU limit. The nodes have idle cores; the pods cannot use them.MediumSolve
The autoscaler cannot outrun the spikeThe API scales on CPU from 4 pods, and new pods need a node that takes about a minute to start. At 30 s a flash sale brings 3× traffic all at once.MediumSolve
The autoscaler hits its ceilingThe API scales on CPU between 4 and 8 pods, each limited to 1 vCPU. At 30 s afternoon traffic triples, which needs about 10.EasySolve
Pods with nowhere to runThe autoscaler may run up to 30 pods, but the cluster has 2 nodes and no node autoscaling. A pod requests 1 vCPU and 1 GB. Traffic climbs steadily to 4× over four minutes.EasySolve
A 4 GB imageThe API image is 4 GB. Traffic climbs steadily to 4× over four minutes, so the cluster keeps adding nodes, and each new node pulls the image before its first pod starts.MediumSolve
Requests dropped on every deployA routine release rolls out at 30 s. The app exits as soon as it gets SIGTERM, while the load balancer is still sending it requests.EasySolve
A release that never startsAt 30 s a release ships with a missing config value: the new pods start but the app never comes up. There is no readiness probe.MediumSolve
A buggy release reaches every podAt 30 s a release ships that fails 30% of requests. It passes its readiness probe, so the rolling update replaces every pod.MediumSolve
All pods on one nodeTo save money the scheduler packs pods onto as few nodes as possible. At 30 s the node running them fails.MediumSolve
A zone goes downThe cluster’s nodes all sit in one availability zone. At 30 s the zone has an outage.MediumSolve
Six minutes without a nodeThree pods run on three nodes, each limited to 1 vCPU and busy most of the time. At 30 s one node dies; Kubernetes replaces its pod only after 50 s plus the 5-minute toleration.MediumSolve
A rolling update at the busy hourTo finish deploys quickly, the update strategy lets half the pods be unavailable and adds none extra (maxUnavailable 50%, maxSurge 0). A release goes out at 30 s, at 170 requests a second.MediumSolve
Cloud and cost 5
The spike beats the alarmA web fleet runs on 2 EC2 instances in an Auto Scaling group that tracks 50% CPU. At 2 minutes a TV ad triples traffic for 10 minutes. Target tracking needs 3 one-minute periods above target, then each new instance takes about 3 minutes to boot and pass health checks.MediumSolve
Maximum size below the peakTraffic climbs steadily to 6× over 15 minutes. The Auto Scaling group tracks 50% CPU, which is plenty of time to react, but its maximum size is 3 instances.EasySolve
Java cold starts at launchA Java Lambda (Spring) takes 3 s to initialise. At 30 s a launch email quintuples traffic for 20 s: every request without a warm environment starts a new one and waits for INIT, which needs still more environments.MediumSolve
One function starves the accountTwo Lambda functions share an account: the customer API (about 100 ms a call) and a report export (about 4.5 s a call). Exports run 240 a second at month end. Neither has reserved concurrency, and the account limit is 1,000.MediumSolve