Incident response & escalation
What happens when something breaks, minute by minute.
Detection & first response #
Most incidents are detected by monitoring before any human notices — alerting pages the on-call engineer directly, with a median first response under 15 minutes around the clock. You can also raise incidents yourself via the emergency line in your runbook; never feel you are "bothering" anyone, that is what the line is for.
During an incident #
- A single incident commander owns the response — you always know who is driving.
- Status updates every 30 minutes for P1s, even if the update is "still investigating."
- Mitigation first, root cause later: we restore service, then understand it.
- Your stakeholders get plain-language updates they can forward upward.
After: the postmortem #
Every P1 and P2 produces a written postmortem within five business days: timeline, root cause, impact, and the action items that prevent recurrence — each with an owner and date. Blameless inside our team, transparent toward you. The same incident paging twice is treated as a process failure.
Severity, by example #
| Scenario | Severity | Why |
|---|---|---|
| Checkout failing for all users | P1 | Revenue path down — full response, 24/7 |
| Login slow (8s) but succeeding | P2 | Badly degraded, workaround exists (patience) |
| CSV export broken, UI fine | P3 | Single function, limited users affected |
| Dashboard chart mislabeled | P4 | Cosmetic — fixed in normal sprint flow |
| Disk trending toward full in ~3 days | Internal ticket | Leading indicator — fixed before users could notice |
Severity describes user impact, never effort or fault. A one-line config mistake that takes checkout down is a P1; a heroic three-day fix for a cosmetic bug is still a P4.
Maintained by the delivery team · updated quarterly