At a glance
- Did you know
- WebSockets hold a connection per client, so memory grows with connected users, not requests.
- Rule of thumb
- Add jitter to client reconnects to avoid a storm after every deploy.
- Watch out
- Connection limit per node; reconnect storms.
Its settings in ScaleMySystems
What you can change on it in the lab, and what each number does.
- Concurrency (slots)
- Calls it can work on at once (per instance when autoscaling). Needed slots ≈ calls per second × seconds each takes. Extra calls wait in the queue, then are rejected.
- Queue capacity
- Calls allowed to wait for a free slot. A bigger queue absorbs bursts but adds waiting time; 0 rejects straight away.
- vCPU
- CPU cores available. Needed ≈ calls per second × CPU per request ÷ 1000. Above about 90% busy it slows down and may crash.
- CPU per request (ms)
- CPU time each call burns. Part of what the component is; make the code cheaper to lower it.
- Latency (p50) (ms)
- Typical time one call takes here, before any calls it makes itself. Slots are held for this long.
- Latency (p99) (ms)
- The slowest 1% of calls. Callers must set their timeout above this or they cut off healthy calls.
Problems that start with it
- WhatsApp: WebSocket gateways at capacity Every message and receipt passes a WebSocket gateway that holds the open connections. Messages arrive at 2,200 a second; the gateway has one vCPU at 0.6 ms a message.
- Voice agent: the model’s slow tail A voice agent transcribes speech (100–150 ms), asks a model (first token: 250 ms usually, 2.4 s for the slowest 1%) and speaks the answer. The call to the model waits up to 8 s and has nowhere else to go. Callers notice any silence over about a second.
Full lesson coming soon
Lessons are being added one component at a time: how it works, its settings, a live exercise to break and fix it, interview questions and a short check. See the Circuit breaker lesson for what one looks like.