// In depth
The three pillars, and what each answers #
| Pillar | Question it answers | How we implement it |
|---|---|---|
| Metrics | Is the system healthy? Is it trending toward unhealthy? | Time-series for every service: traffic, errors, latency, saturation — retained 13 months for seasonality |
| Logs | What exactly happened? | Structured (JSON) logs, centralized, searchable, with request IDs linking every line to a request |
| Traces | Where did the time go? | Distributed tracing across service boundaries — one slow checkout decomposed into its 14 spans |
The pillars are linked, which is what makes them useful at 3 a.m.: an alert (metric) links to the affected traces, each trace links to its logs. Five minutes from "something is slow" to "this query on this table after this deploy" — without anyone grepping through SSH sessions.
SLOs: deciding what "healthy" means #
A service-level objective is a measurable promise: "99.95% of checkout requests succeed in under 500ms, measured monthly." SLOs turn the vague goal of "reliability" into engineering math, and they come with a budget — the 0.05% you are allowed to fail:
- Error budget: 99.95% over 30 days allows ~21 minutes of full downtime, or proportionally more partial degradation. The budget is spent deliberately, not discovered retroactively.
- Budget burning fast → alerts fire and risky changes pause. Budget healthy → ship aggressively. The SLO arbitrates the speed-vs-safety argument with data.
- SLOs are set per user journey (checkout, login, search), not per server — users do not care about your CPU.
Worked example: a 99.95% SLO with 64% of the monthly error budget remaining means the system failed ~7.5 minutes of the allowed 21 so far. That single number tells both teams whether this is a normal month or a problem month.
Alerting philosophy: page on symptoms, ticket the causes #
Alert fatigue is how real incidents get missed: when the pager cries wolf nightly, humans stop believing it. Our alerting rule is strict — a page means a human must act now because users are affected or about to be. Everything else is a ticket.
| Signal | Action | Example |
|---|---|---|
| SLO burn rate critical | Page immediately, 24/7 | Checkout error rate will exhaust the monthly budget in 4 hours |
| User-facing symptom | Page immediately | p95 latency 3× baseline on the login journey |
| Leading indicator | Ticket + business-hours review | Disk 75% full, certificate expiring in 14 days |
| Informational | Dashboard only — never an alert | CPU briefly high, single pod restarted |
NoteFleet-wide last quarter: 3 pages fired, 3 were actionable, 0 were noise. A quiet pager that you trust completely is the goal — and it is measurable.
Dashboards humans actually read #
Every system gets one primary dashboard built top-down: the SLO and user-journey health at the top, the four golden signals (traffic, errors, latency, saturation) per service below, and capacity/cost trends at the bottom. The test for every panel: would someone act differently because of it? Panels that exist "for completeness" are deleted — a dashboard nobody can read in 30 seconds protects nobody.
Clients get the same dashboards we use, not an export. When the quarterly review says 99.97% availability, you can see the same graph the number came from — verifiable claims are the whole point of measuring.
On-call: how 24/7 actually works #
- Follow-the-sun rotation across our regions — nobody is paged at their 3 a.m. as a lifestyle.
- Median first response under 15 minutes for P1s (currently 11 — see the changelog), with a secondary escalation if the primary does not acknowledge in 5.
- Every page links to the runbook for that alert: symptoms, diagnosis steps, safe mitigations. New on-call engineers shadow for two rotations before taking the pager.
- Pages are reviewed weekly: anything noisy is fixed or deleted, anything missing a runbook gets one. The pager is a product we maintain.
Incidents become improvements: the postmortem loop #
Every P1/P2 produces a blameless postmortem within five business days: timeline, contributing causes, user impact, and actions that prevent recurrence — each with an owner and a date, tracked like any other engineering work. "The same incident never pages twice" is the standard we hold the process to; when it fails, that is itself a postmortem finding.
Clients see every postmortem affecting their systems, unredacted. Vendors who hide their incident reviews are asking you to assume the lesson was learned — we would rather show you.