Designing calm on-call systems

Context

Most on-call pain is not about traffic spikes. It is about unclear ownership and alerts that fire without actionability.

Calm systems are designed before incidents happen: you define who responds, what signal matters, and what fallback behavior is acceptable.

Three operational rules

First, alerts should map to user-facing impact, not internal noise. If a signal does not imply action, it should not page.

Second, every high-severity alert must include a first-step runbook link. The first five minutes decide the full incident curve.

Third, prefer graceful degradation over binary failure. A partially available product preserves trust while teams recover.

type ServiceHealth = {
  service: string;
  latencyP95Ms: number;
  errorRate: number;
};

export function shouldPageOnCall(input: ServiceHealth) {
  const highLatency = input.latencyP95Ms > 450;
  const highErrors = input.errorRate > 0.02;
  return highLatency && highErrors;
}

Execution pattern

Run a monthly alert quality review and remove or downgrade noisy checks.

Introduce game days for top failure modes and log what surprised the team. Those surprises become next quarter reliability work.