Incident Runbooks
Use the runbook matching the first confirmed failing boundary. Preserve job documents and structured errors; avoid manual state edits that erase evidence or bypass idempotent cleanup.
| Runbook | Use when | Scope state |
|---|---|---|
| Job Stuck in Running | a job does not reach a terminal state | control-plane steps are current |
| Functions Deploy Failed | a Firebase Functions release fails | verified 2026-07-23 |
| Admin Panel Cannot Sign In | admin, docs, or launchpad authorization fails | verified 2026-07-23 |
| Enable Firebase App Check | planning or rolling out callable attestation | follow phased metrics gate |
| RunPod API Errors | the external provider boundary is confirmed failing | excluded from this audit; reverify first |
First five minutes
- Establish impact, start time, and last known-good deployment.
- Capture a job ID, UID, request ID, or function revision.
- Inspect data and logs without editing production state.
- Identify client, auth, callable, Storage, Firestore, task, or provider as the first failing boundary.
- Communicate the scope and chosen runbook.
Severity
| Level | Meaning | Response |
|---|---|---|
| Critical | broad upload/report outage, security incident, or data loss | immediate coordinated response |
| High | material subset affected or release/operations blocked | begin within one hour |
| Medium | individual users or a contained workflow with workaround | same-day triage |
| Low | cosmetic or non-user-facing | normal prioritization |
Severity is based on measured impact, not the component name.
Writing a runbook
Include observable symptoms, ownership/scope, read-only diagnosis, reversible mitigation, durable resolution, verification, and postmortem criteria. Commands must identify the repository and project. Planning values and external service details need an as-of date and source.
Related: Debugging, Observability, and Postmortems.
Was this page helpful?