Engineering runbooks. Each one is the on-call's playbook for a specific alert — symptom, diagnose, mitigate, verify, escalate.
Each runbook is the on-call’s playbook for a single named alert. Format is consistent: symptom, most-likely-causes, diagnose, mitigate, verify-resolved, escalate.
Watchtower alerts (Phase 1 — CMMD coverage)
Section titled “Watchtower alerts (Phase 1 — CMMD coverage)”These runbooks are the targets for runbook_url annotations on Watchtower alerts. Every alert in the Phase 1 alert rules should link here.
App + infrastructure
Section titled “App + infrastructure”- CMMD app down — public HTTP endpoint not responding or returning sustained 5xx
- Neon Postgres degraded — DB queries failing or pool exhausted
- Upstash Redis degraded — Redis ops failing (app falls back gracefully — P1)
- Stripe webhook failures — billing state desync risk (P0)
- Watchtower can’t scrape CMMD /metrics — visibility loss without customer impact (P1)
AI cost + provider health
Section titled “AI cost + provider health”- AI provider monthly budget breached — cost runaway (P0 pre-revenue)
- AI provider daily spend burn — burning 5× normal rate — catch before P0
- Upstream LLM provider degraded — provider showing failures or status-page incident
Legacy / general
Section titled “Legacy / general”- Production incident response — P0/P1 general triage
- Database migration deploy
- CI pipeline failure
- Cloudflare Pages
- Agent Browser release
When you write a new runbook
Section titled “When you write a new runbook”Match the existing pattern: one frontmatter, a metadata table at the top (severity, owner, alert source, time estimate, required access), then symptom → causes → diagnose → mitigate → verify → escalate. Keep it short — if a step takes more than 3 sentences to explain, link to a deeper doc and stay terse here.