Container platform: how it works, with a live simulation

At a glance

Did you know: Kubernetes places pods on nodes by their CPU and memory requests, and only replaces a lost node's pods after about 5 minutes.
Rule of thumb: Give every pod a readiness probe, requests, a memory limit that covers the peak, and graceful shutdown.
Watch out: Traffic before pods are ready; probes restarting busy pods; OOMKills; Pending pods; autoscaling minutes late; all pods on one node.

Its settings in ScaleMySystems

What you can change on it in the lab, and what each number does.

Concurrency (slots): Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
Queue capacity: Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
CPU per request (ms): CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
Latency (p50) (ms): Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
Latency (p99) (ms): The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
Timeout to dependencies (ms): How long it waits for each call it makes before giving up. Set it above the dependency’s p99 but low enough that a dead dependency does not hold every slot.
Error rate: Share of calls that fail on their own (bugs, declines, flaky tools). An agent that calls it on every step multiplies this.
Quota (req/min): Calls allowed per minute (an API key or provider limit). Calls over it get 429s. Needed = calls per second × 60, with headroom.
Pods: Pods at the start (the Deployment's replicas). Slots, memory and CPU above are per pod.
Autoscaler min pods: The pod autoscaler never goes below this. A floor sized for the peak serves a spike at once.
Autoscaler max pods: The pod autoscaler never goes above this, however busy the pods are.
Autoscaler CPU target: Average CPU, as a share of each pod's request, the autoscaler aims for. Lower leaves headroom for a spike. It checks every 15 s and waits 5 minutes before scaling down.
CPU request (vCPU): CPU reserved for each pod on its node. Decides how many pods fit on a node, and what the autoscaler measures against.
CPU limit (vCPU): Above this a pod is throttled in 100 ms slices, even when its node has idle cores. Empty = no limit.
Memory request (MB): Memory reserved for each pod on its node.
Memory limit (MB): Above this the pod is OOMKilled (exit 137) and restarted after a back-off that doubles from 10 s to 5 minutes. Empty = no limit.
Nodes: Worker nodes at the start. Pods are placed on them by their requests.
Max nodes (cluster autoscaler): The cluster autoscaler adds nodes, up to this, when pods are Pending. A new node takes about a minute. Empty = no node autoscaling.
Node vCPU (vCPU): Allocatable CPU per node.
Node memory (GB): Allocatable memory per node.
Image size (MB): Each node pulls the image once (about 100 MB/s from a registry in the region) before its first pod starts.
App start time (ms): From the container starting to the app being able to serve.

Problems that start with it

Autoscaling hits its ceiling The API runs on a container platform that autoscales from 2 to 3 instances. Traffic needs more.
Traffic before the pod is ready The API scales on CPU from 6 pods. Each pod takes 20 s to warm up after its container starts, and there is no readiness probe. At 40 s traffic jumps 2.5×.
The health check restarts busy pods Eight pods, each limited to 1 vCPU, can absorb a 2.2× surge by queueing requests for a second or two. Their liveness probe has the default 1 s timeout and is answered by the same busy request threads.
OOMKilled at peak Each pod uses 300 MB plus 8 MB for every request it holds (about 250 ms each, waiting on other services), under a 512 MB memory limit. Normal traffic is fine; at 40 s a 2.6× surge arrives.
Throttled by the CPU limit Each pod needs about 0.6 vCPU at normal traffic but has a 0.5 vCPU limit. The nodes have idle cores; the pods cannot use them.
The autoscaler cannot outrun the spike The API scales on CPU from 4 pods, and new pods need a node that takes about a minute to start. At 30 s a flash sale brings 3× traffic all at once.
The autoscaler hits its ceiling The API scales on CPU between 4 and 8 pods, each limited to 1 vCPU. At 30 s afternoon traffic triples, which needs about 10.
Pods with nowhere to run The autoscaler may run up to 30 pods, but the cluster has 2 nodes and no node autoscaling. A pod requests 1 vCPU and 1 GB. Traffic climbs steadily to 4× over four minutes.

And 8 more on the problems page.

Full lesson coming soon

Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.