Skip to main content

Incident Postmortems

Postmortems are immutable historical records except for clearly marked corrections. A current audit of the index does not revalidate every external fact in an incident report.

Logโ€‹

DateIncidentSeverityAudit state
2026-03-14RunPod image regressionCriticalGPU-owned evidence excluded from the 2026-07-23 audit

When one is requiredโ€‹

  • any critical incident;
  • user-impacting data loss or a credible near miss;
  • a security or privacy incident;
  • a multi-user outage lasting more than 15 minutes;
  • a partial/failed deployment that changes production behavior unexpectedly.

Processโ€‹

  1. Mitigate first and preserve timestamps, revisions, IDs, and logs.
  2. Copy the template.
  3. Separate observed facts from hypotheses.
  4. Identify system conditions, not an individual to blame.
  5. Give every action item an owner, due date, and verifiable completion test.
  6. Have a second responder review it and add it to this table.

Related: Runbooks and Observability.