Nothing here runs real software. Each component mimics how the real thing behaves from the numbers you set: capacity, latency, queues, CPU, quotas, errors, caching, blocking and fan-out. That is what lets you test any design at any scale in seconds.
Open any component for what it is, how it breaks and its settings. Lessons with a live break-and-fix exercise are being added: 2 credits each (3 for AI, agent and security components), once, and the core components free.
No plans. Every component is available to everyone; you pay its price in credits when you place it. Inside a problem you have opened, placing components is free; on the blank canvas you pay each one's price. New accounts start with 100 free credits when they sign in with Google.
Spreads requests across replicas.
Did you know Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header.
Rule of thumb Least-connections beats round-robin when request times vary a lot.
Breaks when Forwards everything; the tier behind it is what breaks.
edge zone · About it →Token bucket in front of a service.
Did you know Most limiters use a token bucket: tokens refill at the rate, and the bucket size is the burst you allow.
Rule of thumb Return 429 with a Retry-After header so well-behaved clients back off.
Breaks when Rejects real users when the limit sits below peak traffic.
edge zone · About it →Serves cacheable reads at the edge; storage, TTL and popularity decide the hit rate.
Did you know A CDN keeps copies at edge locations near users, so a cache hit skips the trip to your origin.
Rule of thumb Purge by path or tag. Purging everything sends every request to the origin at once.
Breaks when Miss storms after a purge; origin overload.
edge zone · About it →Replicas with slots, CPU, timeouts, retries.
Did you know Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.
Rule of thumb Keep retries under about 10% of calls, or a slow dependency turns into a retry storm.
Breaks when Slot and queue exhaustion; retries amplify a slow dependency; crash on CPU.
app zone · About it →Consumes a queue or stream.
Did you know Queues deliver at least once, so a worker can see the same message twice.
Rule of thumb Make handlers idempotent: record a processed-message id before doing side effects.
Breaks when Duplicate side effects without idempotency; backlog when slower than producers.
app zone · About it →Routing, auth checks and per-key limits at the door.
Did you know Gateways centralise auth, rate limits and routing so each service does not reimplement them.
Rule of thumb Every request pays the gateway's latency: keep its plugins lean.
Breaks when Adds latency to every call; its own quota rejects real users.
edge zone · About it →Resolves names; failover and geo routing.
Did you know DNS answers are cached for their TTL, so a change can take that long to reach everyone.
Rule of thumb Lower the TTL a day before a planned failover.
Breaks when Slow resolution; wrong region during failover.
internet zone · About it →Sidecar proxy: retries, mTLS and tracing between services.
Did you know A mesh runs a sidecar proxy beside each service to handle mTLS, retries and telemetry.
Rule of thumb Retries in the mesh and in the app multiply: configure them in one place.
Breaks when Sidecar latency on every hop; retry storms.
app zone · About it →Fails fast when a dependency keeps failing, then retries after a cooldown.
Did you know A breaker trips after too many failures and fails fast, giving the dependency room to recover.
Rule of thumb Pair it with a fallback, such as cached data, so users still see something useful.
Breaks when Trips too early (flapping) or too late (cascade).
app zone · Lesson · 2 cr →Token checks, sessions and sign-in.
Did you know Short-lived access tokens plus refresh tokens limit the damage if a token leaks.
Rule of thumb Verify tokens locally with a public key instead of calling the auth service each time.
Breaks when Every request waits on it; a slow token check slows the whole site.
app zone · About it →A third-party API with its own latency, quota and outages.
Did you know Third-party APIs have their own rate limits and outages that you cannot fix.
Rule of thumb Always set a timeout, retry with backoff, and cache what you can.
Breaks when Quota 429s; outage takes your feature down with it.
internet zone · About it →Card authorisations; slow, must never run twice.
Did you know Payment providers accept an idempotency key so a retried charge is not taken twice.
Rule of thumb Treat webhooks as the source of truth: they can arrive late, twice or out of order.
Breaks when Duplicate charges on retry; declines counted as errors.
internet zone · About it →Push, email and SMS fan-out.
Did you know Email, SMS and push providers all throttle senders and can delay delivery.
Rule of thumb Queue notifications and send them in the background; never block a user request on them.
Breaks when Provider quota; bursts back up.
app zone · About it →Long-lived connections for realtime updates.
Did you know WebSockets hold a connection per client, so memory grows with connected users, not requests.
Rule of thumb Add jitter to client reconnects to avoid a storm after every deploy.
Breaks when Connection limit per node; reconnect storms.
edge zone · About it →Runs jobs on a timer; each job calls other components.
Did you know Cron jobs tend to start on the hour, so many systems see a spike at :00.
Rule of thumb Add jitter to schedules and make every job safe to run twice.
Breaks when Thundering herd at the top of the hour; missed runs.
app zone · About it →Config and flags read on every request.
Did you know Flags let you ship code dark and turn it on gradually for a share of users.
Rule of thumb Evaluate flags locally from cached rules, not with a network call per request.
Breaks when A slow flag lookup slows everything; a bad flag rollout.
app zone · About it →Nginx-style front door: TLS termination, compression, routing.
Did you know Reverse proxies like Nginx terminate TLS, compress responses and route to backends.
Rule of thumb Reuse upstream connections with keep-alive; a new connection per request wastes time.
Breaks when Connection limits; a single proxy becomes the bottleneck.
edge zone · About it →AWS Lambda: an environment per concurrent request, cold starts, a shared account limit, billed per GB-second.
Did you know Serverless functions scale per request, but a cold start adds latency to the first call.
Rule of thumb Watch the account concurrency limit: past it, requests are throttled.
Breaks when Cold starts on bursts (seconds for Java); 429s at the account's 1,000 concurrency; one function starving the rest; costly at steady load.
app zone · About it →Small code at the CDN edge: rewrites, auth, A/B.
Did you know Edge functions run in many locations close to users, often in lightweight isolates.
Rule of thumb Keep them small: a call back to a distant database erases the latency win.
Breaks when CPU time limit per request; no local state.
edge zone · About it →One query fans out to many services (federation).
Did you know GraphQL lets a client ask for exactly the fields it needs in one request.
Rule of thumb Limit query depth and cost; one nested query can fan out into thousands of lookups.
Breaks when N+1 fan-out; one slow resolver slows the whole query.
app zone · About it →Binary RPC between internal services.
Did you know gRPC uses HTTP/2 and Protocol Buffers for compact, typed calls between services.
Rule of thumb Long-lived HTTP/2 connections defeat connection-level balancing: balance per request.
Breaks when Deadline propagation; connection reuse limits.
app zone · About it →Composes several backend calls into one screen-shaped response.
Did you know A backend-for-frontend shapes data for one client, such as the mobile app.
Rule of thumb Call downstream services in parallel; serial calls add their latencies together.
Breaks when Sequential calls add up; it inherits every downstream failure.
app zone · About it →Resizes and converts uploads on demand.
Did you know Resizing images is CPU-heavy, so it is usually done once and cached.
Rule of thumb Generate common sizes on upload, or cache on-the-fly results behind a CDN.
Breaks when CPU-bound; large uploads block workers.
app zone · About it →Long jobs that turn uploads into streamable renditions.
Did you know Transcoding turns one video into several resolutions for adaptive streaming.
Rule of thumb Run it as queued background jobs; it can take longer than the video itself.
Breaks when Minutes per job; backlog grows faster than it drains.
app zone · About it →Kubernetes cluster: pods on nodes, probes, the pod and cluster autoscalers, rolling updates.
Did you know Kubernetes places pods on nodes by their CPU and memory requests, and only replaces a lost node's pods after about 5 minutes.
Rule of thumb Give every pod a readiness probe, requests, a memory limit that covers the peak, and graceful shutdown.
Breaks when Traffic before pods are ready; probes restarting busy pods; OOMKills; Pending pods; autoscaling minutes late; all pods on one node.
app zone · About it →Stores and serves container images (Docker Hub, ECR): manifests from a database, layers from object storage.
Did you know An image is a manifest plus layers; a node pulls only the layers it does not have, so slim, shared base layers pull fastest.
Rule of thumb Pull from a registry in the same region (ECR) or a pull-through cache, never anonymously from Docker Hub at scale.
Breaks when Pull storms when many nodes start at once; Docker Hub's anonymous limit (100 pulls per 6 h); huge images slow every scale-out.
app zone · About it →The way into a Kubernetes cluster: routes by host and path to Services, terminates TLS.
Did you know An ingress controller is a reverse proxy (NGINX, Envoy, an ALB) that routes requests from outside to Services by host and path.
Rule of thumb Run it on several replicas spread across nodes; it is the front door for every Service behind it.
Breaks when Too few replicas for the connections; 502s while pods roll out; one ingress for everything is a single choke point.
edge zone · About it →EC2 Auto Scaling: VMs that track a CPU target, judged over 1-minute periods; new instances need about 3 minutes.
Did you know Autoscalers add instances when a metric stays high, but new instances take minutes to be ready.
Rule of thumb Scale ahead of known peaks: reactive scaling lags a sudden spike.
Breaks when Serves new capacity 5–8 minutes after a spike starts; a max size below the peak; a health-check grace shorter than the boot.
app zone · About it →Where services find each other (Consul, Eureka).
Did you know A registry tracks which instances are healthy so callers can find them.
Rule of thumb Cache lookups in the client so a registry outage does not stop all traffic.
Breaks when Stale entries route to dead instances; registry outage.
app zone · About it →Runs heavy jobs off the request path.
Did you know Batch jobs process large data sets on a schedule rather than per request.
Rule of thumb Checkpoint progress so a failed job resumes instead of starting over.
Breaks when Jobs pile up; retries re-run side effects.
app zone · About it →Who is online and typing: heartbeats in, status out (Slack, WhatsApp).
Did you know Presence is a heartbeat: each client pings every few seconds, and a status flips to away when the pings stop.
Rule of thumb Fan presence out through pub/sub by channel, never by polling every user.
Breaks when Heartbeat storms after a reconnect; stale status when it falls behind.
app zone · About it →Takes a broadcaster’s stream (RTMP/SRT), hands it to transcoding and storage.
Did you know Broadcasters push RTMP or SRT to an ingest point near them; from there the stream is transcoded into renditions and packaged into segments.
Rule of thumb Put ingest points in several regions so the first hop is short; segments then travel over the CDN.
Breaks when Few concurrent streams per node; a slow transcoder backs up every broadcaster.
edge zone · About it →Forwards each caller’s audio and video to the others in a call (Zoom, Meet).
Did you know An SFU receives each caller’s stream once and forwards it to every other participant, so outbound bandwidth grows with the square of the call size.
Rule of thumb Cap participants per server and cascade big meetings across several SFUs.
Breaks when Bandwidth and CPU per participant; one server per big meeting runs out.
app zone · About it →Unique, time-ordered IDs without a database round trip (short links, tweets).
Did you know A Snowflake ID packs a timestamp, a machine number and a sequence into 64 bits, so every server mints unique IDs without asking a database.
Rule of thumb Give each generator its own machine number; never let two share one.
Breaks when Clock skew; one generator becomes a bottleneck.
app zone · About it →Pairs orders or riders with drivers in sequence, one market or area at a time.
Did you know An order book or dispatch queue is processed in sequence per market, which is why one market cannot simply be spread over many CPUs.
Rule of thumb Partition by market or area, and keep each partition’s queue short.
Breaks when Single-threaded per market: a hot market queues up.
app zone · About it →Merges concurrent edits to one document and broadcasts them (Google Docs, Figma).
Did you know Operational transforms or CRDTs let several people edit one document at once and converge on the same result.
Rule of thumb Route every editor of one document to the same server, and snapshot the document so joins are fast.
Breaks when One busy document pins one server; a slow merge delays everyone.
app zone · About it →Sends each key to its shard; adding a shard moves only a slice of keys.
Did you know Consistent hashing puts keys and shards on a ring, so adding a shard moves only about 1/N of the keys.
Rule of thumb Use virtual nodes so shards get even slices of the ring.
Breaks when A hot key still lands on one shard; rebalancing traffic.
app zone · About it →Holds a launch-day crowd in a queue and lets people in at a rate the site can take.
Did you know A waiting room admits visitors at a fixed rate and holds the rest in a queue, so a ticket drop cannot overload checkout.
Rule of thumb Set the admit rate from what checkout can take, not from the crowd size.
Breaks when Admits too fast and the site falls over; too slow and people leave.
edge zone · About it →Memory, TTL and key popularity decide the hit rate.
Did you know An in-memory cache like Redis usually answers in well under a millisecond inside one region.
Rule of thumb A cold cache sends every read to the database: warm it before traffic arrives.
Breaks when Cold-cache miss storm; hot keys.
data zone · About it →Connections, CPU, replicas, contracts.
Did you know Postgres runs one process per connection, which is why poolers like PgBouncer sit in front of busy databases.
Rule of thumb Read replicas scale reads, not writes, and can serve slightly stale data.
Breaks when Pool exhaustion, CPU crash, lost updates, stale replica reads.
data zone · About it →Partitions with per-partition limits.
Did you know Key-value stores spread keys over partitions; one very popular key can overload a single partition.
Rule of thumb Cache hot keys or add a random suffix to spread them across partitions.
Breaks when Hot partition despite low average load.
data zone · About it →At-least-once delivery, visibility timeout.
Did you know If a consumer does not finish within the visibility timeout, the message is delivered again.
Rule of thumb Send messages that keep failing to a dead-letter queue instead of retrying forever.
Breaks when Redelivery and dead letters; visibility-timeout duplicates.
data zone · About it →Ordered partitions, consumer groups, retention.
Did you know Kafka keeps order only within a partition, so the partition key decides what stays in sequence.
Rule of thumb Consumers beyond the partition count sit idle: add partitions to scale reads.
Breaks when Consumer lag; data lost past retention.
data zone · About it →Full-text search with shards and refresh delay.
Did you know Search engines build an inverted index: each word points to the documents that contain it.
Rule of thumb Index from a change stream in the background; do not make writes wait for search.
Breaks when Query latency spikes under load; stale results.
data zone · About it →Blobs and files; high throughput, higher latency.
Did you know Object stores like S3 keep copies across devices and facilities for very high durability.
Rule of thumb Serve files through a CDN with signed URLs instead of streaming them through your API.
Breaks when Request-rate throttling (503s) on hot prefixes.
data zone · About it →JSON documents, flexible schema, single-document atomicity.
Did you know Document databases store nested JSON-like records, so one read can fetch a whole aggregate.
Rule of thumb Model around your queries; unbounded arrays inside one document become slow reads.
Breaks when Index write amplification; large documents slow every read.
data zone · About it →Metrics and events by time; downsampling and retention.
Did you know Time-series databases compress by time and downsample old data to keep storage small.
Rule of thumb High-cardinality tags, like user ids, explode the number of series.
Breaks when Cardinality explosion; ingest falls behind.
data zone · About it →Relationships and traversals.
Did you know Graph databases follow relationships directly, so multi-hop queries avoid large joins.
Rule of thumb Traversals through super-nodes with millions of edges are the usual slow query.
Breaks when Deep traversals burn CPU; supernodes.
data zone · About it →Analytics queries on big tables, limited query slots.
Did you know Warehouses store data by column, so analytical queries read only the columns they need.
Rule of thumb Keep dashboards off the production database: copy the data to the warehouse.
Breaks when Queueing on slots; dashboards go stale.
data zone · About it →Coordination service (etcd, ZooKeeper): leases and leader election.
Did you know A distributed lock needs a lease that expires, or a crashed holder blocks everyone.
Rule of thumb Use a fencing token so a holder whose lease expired cannot still write.
Breaks when Lock contention serialises work; lease expiry during GC pauses.
data zone · About it →Fan-out messaging to many subscribers.
Did you know Pub/sub delivers each message to every subscriber, so producers never know who listens.
Rule of thumb Give each subscriber its own retry and dead-letter handling.
Breaks when Slow subscriber backs up the topic; no ordering.
data zone · About it →Where poison messages go after too many retries.
Did you know A dead-letter queue holds messages that failed too often, so they stop blocking the rest.
Rule of thumb Alert when it grows, and keep a way to replay messages once the bug is fixed.
Breaks when Fills silently; nobody re-drives it.
data zone · About it →Streams database changes to other systems.
Did you know Change data capture reads the database's write-ahead log to stream every change.
Rule of thumb Watch replication-slot lag: an unread slot makes the database keep old logs and fill its disk.
Breaks when Lag during bulk writes; schema changes break the stream.
data zone · About it →Stateful processing over streams (Flink, Kafka Streams).
Did you know Stream processors like Flink compute over moving windows of events in real time.
Rule of thumb Plan for late events with watermarks and an allowed lateness.
Breaks when Checkpoint stalls; state grows without TTL.
data zone · About it →Scheduled batch loads into the warehouse.
Did you know ETL extracts data, transforms it and loads it into a warehouse, usually in batches.
Rule of thumb Make each run rerunnable for a given date so backfills are safe.
Breaks when Runs overlap when one is late; partial loads.
data zone · About it →Fast aggregations over wide tables (ClickHouse-style).
Did you know Columnar storage compresses well because each column holds similar values.
Rule of thumb Load in batches; many tiny single-row writes are slow.
Breaks when Many small inserts; concurrent heavy queries.
data zone · About it →Append-only, verifiable records for money and audits.
Did you know Ledgers are append-only: a correction is a new entry, never an edit.
Rule of thumb Use double-entry records so every movement balances to zero.
Breaks when Write serialisation; slow proofs under load.
data zone · About it →Fast store for login sessions and carts.
Did you know A shared session store lets any app server handle a user's next request.
Rule of thumb Keep sessions small and give them a TTL; they are read on every request.
Breaks when Every request reads it; an outage logs everyone out.
data zone · About it →Nearby searches and map tiles.
Did you know Geo indexes split the map into cells, such as geohashes, to find nearby points fast.
Rule of thumb Dense cities overload single cells: use smaller cells where points cluster.
Breaks when Dense areas are hot cells; large radius queries.
data zone · About it →Snapshots on a schedule; restores take minutes.
Did you know A backup only counts once you have restored from it successfully.
Rule of thumb Test restores regularly and keep a copy in another region or account.
Breaks when Snapshot I/O slows the primary; untested restores.
data zone · About it →Ships writes to another region over a slow link.
Did you know Asynchronous replication is fast but can lose the last few writes if the primary fails.
Rule of thumb Monitor replication lag: it is the window of data you could lose.
Breaks when Lag during write bursts; split brain on failover.
data zone · About it →Raw disks that hold object chunks, split and coded across nodes (how S3 is built).
Did you know Object stores split each object into data and parity chunks (for example 6+3) across nodes, so any three nodes can fail without losing data.
Rule of thumb Keep metadata separate from the bytes: a key-value store for where chunks live, storage nodes for the chunks.
Breaks when A lost node means rebuilds; small objects waste IOPS.
data zone · About it →Answers “have I seen this?” in memory, with rare false positives (crawlers, usernames).
Did you know A Bloom filter answers "definitely not seen" or "probably seen" from a few bits per item, with no false negatives.
Rule of thumb Size it for the item count you expect: 10 bits per item keeps false positives near 1%.
Breaks when Fills up and false positives climb; cannot delete.
data zone · About it →Ranked scores in memory: leaderboards, top-N, rate windows.
Did you know A sorted set keeps members ordered by score, so rank and top-N reads cost log n.
Rule of thumb Shard leaderboards by region or bucket; one global key on one node is a hot spot.
Breaks when One huge key on one shard; memory for millions of members.
data zone · About it →Keys and credentials fetched at start-up or per call.
Did you know A secrets manager stores keys encrypted and can rotate them on a schedule.
Rule of thumb Cache secrets in memory for a few minutes; fetching per request hits API limits.
Breaks when Per-call fetches hit its quota; outage blocks deploys.
app zone · About it →Absorbs floods at the edge before they reach you.
Did you know Volumetric floods are absorbed at the network edge, spread over many locations, before they reach you.
Rule of thumb Hide your origin's address so attackers cannot go around the shield.
Breaks when Flood gets through when off; scrubbing adds latency.
edge zone · About it →Blocks injection, scraping and known-bad requests by rule.
Did you know A WAF inspects HTTP requests for attacks such as SQL injection and cross-site scripting.
Rule of thumb Run new rules in log-only mode first to see false positives before blocking.
Breaks when Rules off means attacks pass; strict rules block real users.
edge zone · About it →Tells humans from bots with fingerprints and challenges.
Did you know Bot managers score clients by behaviour and fingerprint, not by IP address alone.
Rule of thumb Challenge suspicious clients instead of blocking them, to spare real users.
Breaks when Bots scrape or stuff credentials; challenges annoy real users.
edge zone · About it →Rejects requests without a valid token before they reach the app.
Did you know Checking credentials at the edge stops credential stuffing before it reaches the app.
Rule of thumb Limit login attempts per account and per IP, and offer MFA.
Breaks when Credential stuffing without lockout; token checks add latency.
edge zone · About it →Mutual TLS between services: only trusted peers connect.
Did you know Mutual TLS makes both sides prove identity with certificates, not just the server.
Rule of thumb Automate certificate rotation; expired certificates are a classic outage.
Breaks when Handshake CPU under load; expired certificates.
app zone · About it →Collects security events for detection and audits.
Did you know A SIEM collects logs from everywhere and correlates them to spot attacks.
Rule of thumb Alert on a few high-signal rules; noisy alerts get ignored.
Breaks when Ingest lag hides an attack; log volume cost.
data zone · About it →OIDC / SAML sign-in for people and services.
Did you know Identity providers handle sign-in with standards like OpenID Connect and SAML.
Rule of thumb Cache the provider's signing keys; checking a token should not need a network call.
Breaks when Login storms after an outage; token refresh floods.
app zone · About it →Encrypts and decrypts with managed keys.
Did you know A KMS keeps master keys in hardware and encrypts your data keys for you (envelope encryption).
Rule of thumb Encrypt locally with a data key; calling KMS for every record hits its rate limits.
Breaks when Per-call quota; envelope encryption done per request.
app zone · About it →Scans content for secrets and personal data before it leaves.
Did you know Data loss prevention scans content for patterns such as card numbers and national IDs.
Rule of thumb Scan in the background where you can; inline scanning adds latency to every request.
Breaks when Scan latency; false positives block real content.
app zone · About it →Signature and anomaly detection on traffic.
Did you know An IDS alerts on suspicious traffic; an IPS sits inline and blocks it.
Rule of thumb Tune signatures to your own traffic, or false positives block real users.
Breaks when Misses novel attacks; alert fatigue.
edge zone · About it →Every request authenticated and authorised, no network trust.
Did you know Zero trust checks identity and device on every request instead of trusting the network.
Rule of thumb Roll it out one application at a time, starting with internal admin tools.
Breaks when Policy evaluation latency; a misrule locks everyone out.
edge zone · About it →Issues and rotates short-lived certificates.
Did you know A certificate authority signs the certificates that prove a service's identity.
Rule of thumb Prefer short-lived certificates with automatic renewal.
Breaks when Rotation storms; an expired root breaks everything.
app zone · About it →Scores payments and sign-ups for risk.
Did you know Fraud models score each transaction in milliseconds from behaviour and history.
Rule of thumb Decide in advance what happens when scoring times out: allow or decline.
Breaks when Model latency on the checkout path; false declines cost sales.
app zone · About it →Issues the keys a player needs to decrypt protected video (Widevine, FairPlay).
Did you know Protected video is encrypted once; the player fetches a short-lived key from a license server before it can play.
Rule of thumb Cache license responses per device for a few minutes so a restart does not hit the server again.
Breaks when Every playback start waits on it; a key-server outage stops all new plays.
app zone · About it →Every outbound call from agents and sandboxes goes through an allowlist; credentials are added outside the agent.
Did you know An egress proxy lets agents reach only allowlisted destinations and injects credentials so the agent never holds them.
Rule of thumb Allowlist destinations per tool; deny everything else by default.
Breaks when Blocks real destinations missing from the allowlist; a chokepoint when undersized.
app zone · About it →Holds each user’s delegated tokens and reuses a short-lived one until it nears expiry.
Did you know A token broker exchanges a user’s identity for short-lived tool tokens and reuses each one until it nears expiry.
Rule of thumb Cache tokens per user, audience and scope, and refresh them once, ahead of expiry.
Breaks when Without reuse, every tool call mints a token at the identity provider.
app zone · About it →Answer cache: a hit skips the model.
Did you know A semantic cache matches questions by meaning using embeddings, not exact text.
Rule of thumb Tune the similarity threshold: too loose returns wrong answers, too strict never hits.
Breaks when Low hit rate means every question reaches the model.
ai zone · About it →Tokens per request, TPM quota.
Did you know Embeddings turn text into vectors where similar meanings sit close together.
Rule of thumb Batch many texts into one call: per-request overhead dominates small inputs.
Breaks when Token quota 429s under bursts.
ai zone · About it →Index memory, query latency, QPS.
Did you know Approximate indexes like HNSW trade a little recall for much faster search.
Rule of thumb Raw size ≈ vectors × dimensions × 4 bytes. One million 1,536-d vectors is about 6 GB before index overhead.
Breaks when Index larger than memory; QPS limit.
ai zone · About it →Time to first token, tokens/s, quotas.
Did you know Latency has two parts: time to the first token, then a steady stream of tokens per second.
Rule of thumb Stream answers so they feel fast, and cap output tokens to bound cost.
Breaks when Quota 429s; time to first token under load; cost.
ai zone · About it →One door to every model: keys, quotas, caching, fallback.
Did you know An AI gateway sits in front of model providers to add keys, quotas, logging and fallbacks.
Rule of thumb Track spend per team or feature at the gateway, not after the bill arrives.
Breaks when Gateway quota; added latency per call.
ai zone · About it →Exact-match cache for repeated prompts.
Did you know Providers can cache a long repeated prompt prefix, cutting cost and time to first token.
Rule of thumb Put stable content first (instructions, documents) and the changing question last.
Breaks when Low hit rate on personalised prompts.
ai zone · About it →Cross-encoder that reorders retrieved chunks.
Did you know A reranker reads each retrieved passage together with the question and reorders by relevance.
Rule of thumb Retrieve broadly, rerank, then send only the top few passages to the model.
Breaks when Adds 50–200 ms per query; GPU concurrency limit.
ai zone · About it →Input and output checks: injection, PII, policy.
Did you know Guardrails check inputs and outputs for unsafe content, prompt injection or leaked data.
Rule of thumb Run cheap checks first and model-based checks only when they are needed.
Breaks when Latency on every call; misses injections at low detection.
ai zone · About it →Checks model output against a schema; retries on failure.
Did you know Validating model output against a schema catches malformed answers before users see them.
Rule of thumb On failure, retry once with the error message, then fall back.
Breaks when Retry amplification when the invalid-output rate is high.
ai zone · About it →Sends easy requests to a cheap model, hard ones to a big one.
Did you know A router sends easy requests to small, cheap models and hard ones to larger models.
Rule of thumb Measure quality per route: savings that hurt answers are not savings.
Breaks when Misrouting; the cheap path fails and falls back.
ai zone · About it →Collects requests and runs them later at a discount.
Did you know Batch APIs trade latency for price: results arrive later but cost less.
Rule of thumb Use it for evals, backfills and nightly jobs, never for a user waiting on a reply.
Breaks when Deadline misses when the batch window is long.
ai zone · About it →Your own GPUs with continuous batching.
Did you know GPU servers batch requests together; throughput rises with batch size until memory runs out.
Rule of thumb Long prompts use more KV-cache memory, so fewer requests fit at once.
Breaks when KV-cache pressure; queueing when all GPUs are busy.
ai zone · About it →Parses and chunks documents before embedding.
Did you know Ingestion splits documents into chunks before they are embedded for retrieval.
Rule of thumb Chunk on headings and paragraphs with a little overlap, not on fixed character counts.
Breaks when Backlog on large files; poisoned content.
ai zone · About it →Keyword and vector retrieval merged with reciprocal rank fusion.
Did you know Hybrid search combines keyword matching (BM25) with vector similarity.
Rule of thumb Keywords catch exact names and codes that embeddings often miss.
Breaks when Two indexes to keep in sync; merge latency.
ai zone · About it →Assembles retrieved chunks, memory and instructions into the prompt.
Did you know Context building assembles retrieved passages, history and instructions into the prompt.
Rule of thumb More context is not always better: irrelevant passages lower quality and raise cost.
Breaks when Context overflow; stale chunks; prompt-injection carried in.
ai zone · About it →Samples answers and scores quality, drift and cost.
Did you know Online evals score a sample of live answers to catch quality drops early.
Rule of thumb Score a small sample; grading every answer can double model spend.
Breaks when Scoring cost at high sample rates; silent quality drops.
ai zone · About it →Streams audio into transcripts.
Did you know Streaming speech recognition returns partial words while the user is still talking.
Rule of thumb Send audio in small chunks to keep the transcript close to real time.
Breaks when Concurrency caps; latency on long utterances.
ai zone · About it →Turns answers into audio, often streamed.
Did you know Speech synthesis can start playing before the whole sentence is generated.
Rule of thumb Cache common phrases; synthesising the same greeting again is wasted money.
Breaks when Time to first audio; character quotas.
ai zone · About it →Diffusion model behind a queue.
Did you know Image generation takes seconds per image and runs on GPUs.
Rule of thumb Run it in the background and show progress instead of holding a request open.
Breaks when Seconds per image; GPU queueing; content filters.
ai zone · About it →Extracts text and tables from scans and PDFs.
Did you know OCR turns scanned pages into text; layout and tables are the hard part.
Rule of thumb Process pages in parallel and cache results by document hash.
Breaks when Page-rate quotas; large files time out.
ai zone · About it →Trains an adapter on your data; hours per job.
Did you know Fine-tuning adapts a model to your examples; it changes style and format more than knowledge.
Rule of thumb Improve prompts and retrieval first, and fine-tune when they plateau.
Breaks when GPU queue; data drift makes it stale.
ai zone · About it →Scores candidates for a feed, recommendations or search (a learned ranker on CPUs or GPUs).
Did you know Feeds rank a few hundred candidates per load with a learned model; retrieval narrows millions of posts first.
Rule of thumb Keep ranking off the write path: rank at read time from a precomputed candidate list.
Breaks when Called on every feed load: CPU-bound and on the hot path; a slow ranker slows every feed.
ai zone · About it →GPUs that only process prompts (prefill), split from decoding; sets time to first token.
Did you know Splitting prefill (reading the prompt) from decode (writing the answer) onto separate GPUs stops long prompts from stalling everyone else’s output.
Rule of thumb Size prefill for prompt tokens per second and decode for output tokens per second; they scale differently.
Breaks when Long prompts queue for prefill and time to first token climbs.
ai zone · About it →GPUs that only generate output tokens, batched; sets tokens per second.
Did you know Decoding is memory-bound: each request holds its KV cache in GPU memory for as long as it generates.
Rule of thumb Watch KV-cache memory, not just GPU utilisation; it sets how many requests decode at once.
Breaks when KV-cache memory fills; batches shrink and output slows.
ai zone · About it →Keeps attention KV blocks in CPU memory or SSD so repeated prefixes skip prefill.
Did you know Reusing the KV cache of a shared prefix (system prompt, chat history) skips most of the prefill work.
Rule of thumb Put the stable part of the prompt first so more requests share a prefix.
Breaks when Low hit rate when prompts do not share prefixes; transfer time on misses.
ai zone · About it →A small draft model proposes tokens the big model verifies in one pass.
Did you know A small draft model guesses several tokens; the large model checks them in one pass and keeps the ones it agrees with.
Rule of thumb It only pays off when most drafted tokens are accepted; measure the acceptance rate.
Breaks when Low acceptance rate wastes GPU time; draft model becomes the bottleneck.
ai zone · About it →Serves many fine-tuned adapters on one base model in shared batches.
Did you know Multi-LoRA serving runs one base model for many fine-tuned adapters, batching requests for different adapters together.
Rule of thumb Cap adapters per batch; swapping adapters in and out of GPU memory costs time.
Breaks when Adapter swaps and too many adapters per batch slow everyone.
ai zone · About it →A small model rewrites or expands questions before retrieval.
Did you know Rewriting a vague question into a precise search query often improves retrieval more than a bigger model does.
Rule of thumb Use a small, fast model; it runs on every question.
Breaks when Adds a model call to every question; quota and latency.
ai zone · About it →A knowledge graph built from documents for multi-hop questions.
Did you know GraphRAG builds a knowledge graph from documents so multi-hop questions can follow relationships.
Rule of thumb Build cost is high: refresh incrementally and keep vector search for simple lookups.
Breaks when Graph traversals are CPU-heavy; expensive to rebuild.
ai zone · About it →A model that grades answers for quality and safety (online evals).
Did you know An LLM judge grades outputs at scale, but it is another model call with its own latency, cost and biases.
Rule of thumb Judge a sample asynchronously instead of every answer inline.
Breaks when Doubles model calls; slow judges back up the pipeline.
ai zone · About it →Collects spans for every model and tool call (LLM observability).
Did you know Agent traces record every model and tool call with inputs, outputs, tokens and timing.
Rule of thumb Sample traces at high volume, but keep every failed one.
Breaks when Ingest limits drop traces exactly when you need them.
data zone · About it →Screens user input and retrieved text for instructions that try to hijack the model.
Did you know Prompt injection hides instructions in input or retrieved content; no filter catches all of them.
Rule of thumb Combine a filter with design: keep untrusted data out of the control flow and limit what tools can do.
Breaks when Misses novel injections; false positives block real users.
ai zone · About it →Removes personal data from prompts and logs before they leave your system.
Did you know Redacting personal data before it reaches a model or a log reduces privacy and compliance risk.
Rule of thumb Redact before logging too; traces are a common leak.
Breaks when CPU-bound on long prompts; adds latency to every call.
ai zone · About it →Picks the model replica that already holds a session’s prefix in its KV cache, or the least busy one.
Did you know Routing a session back to the replica that holds its prefix can raise KV-cache hit rates from about 20% to 75%.
Rule of thumb Stay sticky until a replica is saturated, then spill to the least busy one.
Breaks when Round robin scatters sessions and wastes the cache; pure stickiness overloads one replica.
ai zone · About it →Copies a share of live requests to a candidate model and discards its answers.
Did you know A shadow deploy copies live requests to a candidate model and compares its answers without showing them to users.
Rule of thumb Mirror a small sample, and run the candidate on its own capacity.
Breaks when A 100% mirror on shared capacity doubles the load; every mirrored call costs money.
ai zone · About it →Plans and loops: each request makes several model and tool calls.
Did you know An agent loops: think, call a tool, read the result, repeat until done.
Rule of thumb Cap steps and tokens per task; runaway loops are the costliest failure.
Breaks when Call amplification; runaway loops burn quota and money.
ai zone · About it →Executes tool calls for agents; may call other systems.
Did you know Tools give the model actions such as search or database queries; each call is another round trip.
Rule of thumb Return short, structured results; long tool output eats the context window.
Breaks when Tool outage; retried side effects.
app zone · About it →Auth, allow-lists and quotas for tool servers.
Did you know MCP (Model Context Protocol) is an open standard for connecting models to tools and data.
Rule of thumb Allow-list the tools each agent may call, and log every call.
Breaks when Scope creep; quota rejects agent calls.
edge zone · About it →Isolated containers that run agent-written code.
Did you know Sandboxes run model-written code in an isolated container with limited network and time.
Rule of thumb Set CPU, memory and wall-clock limits; generated code can loop forever.
Breaks when Cold starts; sandbox queueing; timeouts.
app zone · About it →A person approves risky actions; the agent waits.
Did you know Human-in-the-loop review holds risky actions until a person approves them.
Rule of thumb Ask only for irreversible or costly actions, or reviewers start rubber-stamping.
Breaks when Minutes-long waits; deadline misses; rejected actions.
app zone · About it →Short- and long-term memory reads and writes.
Did you know Agent memory keeps facts across sessions so the agent does not start from scratch.
Rule of thumb Store short summaries rather than raw transcripts, and let memories expire.
Breaks when Write amplification; poisoned memories.
data zone · About it →Another organisation's agent, reached over the agent-to-agent protocol.
Did you know Agent-to-agent protocols let agents from different teams or vendors hand work to each other.
Rule of thumb Treat another agent like an external API: timeouts, retries and a budget per call.
Breaks when Cascading delegation; remote quota; partial failure.
internet zone · About it →Long-running agent tasks that survive restarts (Temporal-style).
Did you know Workflow engines like Temporal persist every step, so long processes survive crashes.
Rule of thumb Keep each activity idempotent: the engine may retry it.
Breaks when History grows; replay cost; stuck workflows.
app zone · About it →Hands each task to the specialist agent that fits it.
Did you know An agent router picks which specialised agent should handle a request.
Rule of thumb Log every routing decision; a wrong route looks like a bad answer.
Breaks when Misrouted tasks bounce; every hop adds a model call.
ai zone · About it →Versioned tool and skill definitions agents load at run time.
Did you know A skill registry lists reusable instructions and tools an agent can load when needed.
Rule of thumb Load skills on demand so unused ones do not fill the context window.
Breaks when Unpinned skills change behaviour; registry outage.
app zone · About it →Headless browsers the agent drives to read sites.
Did you know Browser tools let agents read and act on web pages, which is slow and brittle.
Rule of thumb Use an API when one exists and keep the browser as the last resort.
Breaks when Seconds per page; bot blocks; memory per browser.
internet zone · About it →Compacts long histories into short memories with a model call.
Did you know Summarising older turns keeps long sessions inside the context window.
Rule of thumb Keep key facts and decisions word for word; summaries lose detail.
Breaks when Lossy summaries; summarisation cost grows with activity.
ai zone · About it →Token and dollar budgets per task; cuts off runaway agents.
Did you know A spend governor enforces budgets per user, task or team before calls go out.
Rule of thumb Warn at a soft limit, then stop at a hard one.
Breaks when Budget too low fails real tasks; too high lets loops burn money.
app zone · About it →Buffers agent work so bursts drain at a steady rate.
Did you know Queuing agent tasks smooths bursts and caps how many run at once.
Rule of thumb Give each task a deadline so stale work does not run hours later.
Breaks when Tasks wait minutes; retries re-run tool side effects.
data zone · About it →A lead agent fans work out to parallel subagents, each with its own context.
Did you know Multi-agent research systems fan a task out to parallel subagents and use many times more tokens than a single agent.
Rule of thumb Use them for broad, parallel work; give each subagent a clear goal, tools and output format.
Breaks when Token use multiplies with subagents; tools saturate under the fan-out.
ai zone · About it →Summarises old turns and tool results so long sessions fit the context window.
Did you know Compaction summarises a long context so the agent can keep going without hitting the window limit.
Rule of thumb Keep decisions, open tasks and file paths verbatim; summarise the rest.
Breaks when A model call per compaction; loses details it should keep.
ai zone · About it →Reuses results of identical, read-only tool calls across steps and agents.
Did you know Agents often repeat the same read-only tool call within and across tasks.
Rule of thumb Cache only read-only tools, and key the cache on normalised arguments.
Breaks when Stale results; low hit rate when arguments vary.
data zone · About it →Durable execution: saves agent state after each step so work resumes after a crash.
Did you know Durable execution saves state after each step so a crash resumes instead of starting over.
Rule of thumb Checkpoint before side effects, and make side effects idempotent.
Breaks when Write volume on every step; large states are slow to save.
data zone · About it →A virtual desktop an agent drives with screenshots, clicks and typing.
Did you know Computer-use agents act through screenshots and clicks, which takes seconds per action.
Rule of thumb Prefer APIs and MCP tools; use the desktop only when nothing else exists.
Breaks when Slow (seconds per action); few concurrent desktops; flaky UIs.
app zone · About it →Checks every tool call against permissions and data-flow rules before it runs.
Did you know A policy engine checks each tool call against what this agent, user and data may do.
Rule of thumb Deny by default for side-effecting tools; log every decision.
Breaks when Over-strict policies stall agents; policy store becomes a hot path.
app zone · About it →Runs a long-lived agent loop: context, tools, checkpoints and retries around the model.
Did you know A harness is everything around the model in an agent: context management, tools, checkpoints, retries and evaluation.
Rule of thumb Externalise progress to durable notes so a fresh session can pick up where the last one stopped.
Breaks when Too many turns per task; state lost without checkpoints.
ai zone · About it →Finds the few tools a step needs instead of loading every definition into context.
Did you know Loading hundreds of tool definitions into every prompt wastes context and confuses the model.
Rule of thumb Search for the few relevant tools per step and load only those.
Breaks when Misses the right tool; another hop per step.
ai zone · About it →