// In depth
Why deploys are a reliability feature #
Most production incidents are self-inflicted: a deploy went wrong. Teams respond by deploying less often, which makes each deploy bigger and riskier — a spiral that ends with quarterly midnight releases everyone fears. We engineer the opposite loop: deploys so safe and reversible that shipping small changes daily is the low-risk option.
| Symptom | What it tells you |
|---|---|
| Releases scheduled after midnight | The process cannot tolerate users being present |
| A "maintenance page" exists | Downtime is designed in, not designed out |
| Rollback is a multi-hour procedure | Every deploy is a one-way door under pressure |
| Deploys batched monthly or quarterly | Each release carries weeks of compounded risk |
How a rolling deploy works #
The default strategy on every platform we deliver is the rolling deploy. The fleet is replaced gradually, and at every moment during the deploy, full capacity is serving traffic:
- New instances running the new version start alongside the old ones.
- Each new instance must pass health checks — liveness, readiness, and application-level smoke checks — before the load balancer sends it a single request.
- Old instances are drained: they stop receiving new requests, finish in-flight ones (bounded by a grace period), then terminate.
- The rollout proceeds in waves (e.g. 25% at a time). A health-check failure at any wave halts the deploy automatically with the healthy majority still serving.
The result: users experience nothing. No dropped requests, no error blip, no maintenance window. Deploys at 2 p.m. on a Tuesday are not bravado — they are the safest time, because the whole team is awake and watching.
Rolling vs. blue-green vs. canary #
| Strategy | How it works | When we use it |
|---|---|---|
| Rolling | Replace the fleet in waves behind the load balancer. | The default — right for the vast majority of changes. |
| Blue-green | A complete second environment is built; traffic switches over atomically and can switch back the same way. | High-stakes releases, major framework upgrades, and changes that are hard to run mixed-version. |
| Canary | The new version receives a small slice of real traffic (e.g. 5%) while metrics are compared against the old version, then is promoted or rolled back. | Risky business-logic changes, performance-sensitive paths, anything touching payments. |
NoteStrategies compose: a canary stage in front of a rolling deploy is our standard pipeline for payment and authentication services.
Database migrations without downtime #
The hardest part of zero-downtime delivery is not the application — it is the database. During a rolling deploy, old and new code run simultaneously against the same schema, so every migration must be safe for both versions. We use the expand-and-contract pattern:
- Expand: add the new column/table/index alongside the old structure. Old code ignores it; new code can use it. Deploy this first, alone.
- Migrate: backfill data in batches sized to never lock the table or saturate I/O — large backfills run as monitored background jobs.
- Switch: deploy application code that reads and writes the new structure.
- Contract: only when no code references the old structure — verified, not assumed — remove it in a later release.
This turns one risky migration into three boring ones. It costs a little discipline and buys the ability to ship schema changes on a Tuesday afternoon, which compounds into faster product development all year.
Rollback mechanics #
Every deploy is reversible because every artifact is immutable and retained: rolling back means redeploying the previous image, which follows the exact same health-checked rollout as any deploy. One command, under a minute to begin serving the old version, no "rebuild the old commit and hope."
- Automatic rollback: if error rates or health checks breach thresholds during the rollout window, the pipeline reverts without waiting for a human.
- Schema-safe by construction: because migrations follow expand-and-contract, the previous app version always runs correctly against the current schema.
- Feature flags handle the cases code rollback cannot: a misbehaving feature is switched off in seconds without any deploy at all.
What gates a production deploy #
A deploy only reaches the rolling stage after the full pipeline passes — the same gates on every change, with no manual bypass path:
| Stage | What it verifies | Typical duration |
|---|---|---|
| Build & unit tests | Code compiles, logic is correct | 2–4 min |
| Static analysis & dependency scan | No known-vulnerable packages, no obvious defects | 1–2 min |
| Integration tests | Services work together against real datastores | 3–6 min |
| Preview environment | Humans review the actual feature (see Preview Environments) | until approved |
| Canary (where configured) | Real traffic agrees with the test suite | 10–30 min |
| Rolling deploy | Health-checked waves to 100% | 3–8 min |
Median commit-to-production across client platforms is 14 minutes. Speed is a safety feature: when shipping a fix takes minutes, hotfixes go through the pipeline instead of around it.
Application considerations #
- Mixed-version tolerance: during a rollout, v2.41 and v2.40 serve simultaneously — API contracts and message formats must be backward compatible for one version step.
- Session continuity: sessions live in a shared store, never in instance memory, so users do not get logged out when the instance serving them is replaced.
- Long-running work: jobs and websocket connections either survive instance replacement (checkpointing, reconnect logic) or drain within the grace period.
- Idempotent startup: instances may start and stop frequently; startup routines (migrations check, cache warm) must be safe to run concurrently.