Design Kubernetes’ control plane from an empty canvas: one API for every change, a consistent store, thousands of watchers, a scheduler, and a heartbeat from every node.
Unlock it once for 25 credits; everything inside it is free (runs, changes, Submit, results, hints). Sign in with Google to start; new accounts get free credits.
What it must do
Take changes (deploy this, scale that) through one API, and keep the desired and actual state of every object in a consistent store.
Let thousands of node agents and controllers list and watch objects, and see every change in order.
Schedule each new pod onto a node that has room for it.
Take a status heartbeat from every node, so lost nodes are noticed.
Scale
2,500 requests a second: 1,900 lists and watches from node agents and controllers, 500 node heartbeats, 100 pod creations.
5,000 nodes each sending a heartbeat every 10 s is the 500 writes a second; every write must survive a server dying.
Watches outnumber writes about 4 to 1: serve them from a cache of the store, never from the store itself.
Request types to route: list (76%, read), heartbeat (20%, write), create (4%, write). Budget: $3,000 a month.
Estimate first
Node heartbeats a second (5,000 nodes, every 10 s) (writes/s)
Lists and watches for every write (× writes)
Pods created a day at 100 a second (million)
Monthly cloud cost of your design (USD)
The test cases
You start from an empty canvas. When you run the tests, each one simulates your design with its own traffic and failures:
Test
What it does
Passes when
Normal day
The normal day: the traffic in the brief, nothing broken.
Success ≥ 99.9%; p99 latency ≤ 300 ms (list); p99 latency ≤ 800 ms (heartbeat)
A cluster-wide rollout
At 5 s every team deploys at once: traffic triples for 8 s, 300 pod creations a second.
Success ≥ 99%; p99 latency ≤ 1.5 s (create)
A server dies
At 4 s one server of the first app service on the request path dies for 10 s.
Success ≥ 99%
Database 4× slower
At 5 s the main database on the request path answers 4× slower for 8 s.
Success ≥ 99%; p99 latency ≤ 800 ms (list)
Budget: $3,000/month
The normal day's design must cost at most $3,000 a month.
Cost ≤ $3,000/month
Must include
A consistent store (etcd). Every object’s desired and actual state lives in one strongly consistent store; everything else is a cache of it.
A watch cache. Lists and watches vastly outnumber writes; the API servers answer them from memory, not from the store.
A scheduler. New pods are placed off the request path: the API records them, a scheduler picks a node and writes the binding.