Problems / Resilience, hard

Payments: no double charges, no lost ledger entries, an acquirer outage

Asked at Stripe and PayPal. The charges API writes the payment, then publishes a ledger event (two separate writes). At 10 s the ledger queue is down for 5 seconds; the ledger worker loses 2% of its acknowledgements and is not idempotent; at 18 s acquirer A is down for 10 seconds.

Solve it in the labTake it as a mock interview

Unlock it once for 25 credits; everything inside it is free (runs, changes, Submit, results, hints). Sign in with Google to start; new accounts get free credits.

Your goals

Success ≥ 99%, no duplicate side effects, no lost ledger events, for at most $1,100,000 a month (card fees included).

Test cases

Like a coding problem, a solution has to pass every case, not just the one it was built for. Submit in the lab runs all 4; 1 of them are hidden until then.

  1. As given. The problem’s own traffic, scheduled failures and seed.
  2. Peak hour. 25% more traffic than the problem states, with the same failures.
  3. A slow dependency. The main data store or model answers twice as slowly for part of the run.

The traffic

800 requests a second for 20 seconds: 100% charge (writes).

The system

What each component does in this design, and what calls it. The simulator mimics each one from these settings.

ComponentIts job hereSettings
Load balancerLoad balancerTraffic enters here.Calls Charges API.Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header.–
Charges APIApp serviceLoad balancer forwards to it.Calls Payments (Postgres), Card network (acquirer A), Ledger events (publishes).Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.4 replicas × 100 slots · timeout 9.0 s
Payments (Postgres)SQL databaseCharges API calls it.Postgres runs one process per connection, which is why poolers like PgBouncer sit in front of busy databases.8 vCPU · 400 connections
Ledger eventsMessage queueCharges API publishes to it.Calls Ledger and receipt worker.If a consumer does not finish within the visibility timeout, the message is delivered again.–
Ledger and receipt workerWorkerLedger events consumes from it.Queues deliver at least once, so a worker can see the same message twice.64 concurrent jobs
Card network (acquirer A)Payment gatewayCharges API calls it.Payment providers accept an idempotency key so a retried charge is not taken twice.500 slots

More resilience problems