Capacity that follows your traffic, not your fears

Every system we deliver can scale horizontally — running multiple instances of a service behind a load balancer, with traffic spread evenly across them. How the instance count changes is a choice between two methods, and most platforms use both: autoscaling for the workhorse services, manual scaling for the few components where capacity should be a deliberate decision.

Autoscaling is a feedback loop. You define a floor, a ceiling, and a target — for example: minimum 2 instances, maximum 20, target CPU utilization 65%. The autoscaler continuously compares actual utilization across all running instances against the target and adjusts the count to close the gap.

• Scale up: when average utilization stays above target for the evaluation window (typically 60 seconds), instances are added — aggressively if the gap is large, one at a time if it is small.
• Scale down: when utilization stays below target for a longer window (typically 5–10 minutes), instances are removed gradually. Down-scaling is deliberately slower than up-scaling: flapping is worse than briefly over-provisioned.
• New instances only receive traffic after passing health checks, so a scaling event can never route requests to a cold or broken instance.
• Both boundaries are hard guarantees: the count never drops below the floor (resilience) and never exceeds the ceiling (cost protection).

A worked example: a service runs 4 instances at 65% target CPU. A marketing campaign drives traffic up and CPU hits 90%. The autoscaler computes the desired count — roughly currentCount × (actual ÷ target), so 4 × (90 ÷ 65) ≈ 6 — and adds two instances. Three minutes later the campaign spike ends; utilization falls to 40%, and over the next ten minutes the fleet steps back down to 4.

CPU is the default scaling signal because it is universal, but it is not always the honest one. We pick the metric that actually predicts when your workload degrades:

Horizontal scaling adds more instances; vertical scaling makes each instance bigger (more CPU/RAM). They solve different problems and we use both:

Rule of thumb: scale stateless things out, scale stateful things up (and replicate them for availability). A platform that can only scale vertically has a ceiling and a single point of failure built into its architecture — one of the first things we fix in takeover audits.

Autoscaling is an infrastructure feature with application prerequisites. Before we enable it on any service, we verify the code is actually safe to run N times in parallel:

• Statelessness: no session data or uploaded files on local disk — state lives in the database, cache, or object storage, so any instance can serve any request.
• Connection discipline: database connections are pooled and bounded per instance, so 20 instances cannot exhaust the database. Where needed we add a shared pooler (e.g. PgBouncer).
• Graceful shutdown: instances finish in-flight requests and drain connections when terminated during scale-down — no user ever sees a dropped request because the fleet shrank.
• Warm-up behavior: services that need caches primed or JIT warm-up report "not ready" until they actually are, so the load balancer does not send traffic to a cold instance.
• Idempotent background jobs: workers can be killed mid-job and re-run safely, because scale-down will eventually interrupt something.

Autoscaling changes your bill from a flat line into a curve that follows usage — that is the point. Each instance is billed only while it runs, which is why clients moving from worst-case provisioning typically see 25–40% lower compute spend at equal or better performance.

• Maximum-instance ceilings cap the worst-case spend of any single service — a bug or an attack cannot scale you into a five-figure surprise.
• Budget alerts fire at 50/80/100% of expected monthly spend, routed to both teams.
• Anomaly detection flags scaling patterns that deviate from history — the 3 a.m. scale-up that has no matching traffic is investigated, not just paid for.
• Every scaling event is logged and reviewable: when, why, which metric, and what it cost.

Configuration is theory; load is proof. Before launch — and before every known peak season — we run load tests that push the system past its expected maximum and watch the autoscaler respond. A game day simulates the real event end to end: traffic ramps, instance failures injected mid-spike, dashboards watched by the same people who will be on call.

The Cartful engagement is the reference case: three full-scale game days at 25× baseline traffic before Black Friday, each one finding and fixing a real bottleneck. The actual event peaked at 22× and was, by design, uneventful.