Runbook — Production Incident Response

Field	Value
Severity	P0 / P1
Owner	@aphaiboon
Last verified working	2026-05-16
Estimated time	15 min (P0 detection → mitigation start)
Required access	Railway, Neon, PostHog, GitHub, Slack #incidents

When to use this runbook

• App shows 5xx on a customer-facing endpoint for >2 minutes
• PostHog alert fires on error-rate spike
• Customer reports the app is down via support@cmmd.ai or Slack
• Status page shows a degradation

If the symptom is something specific (DB migration failed, CI runner offline) — use the more specific runbook below first, fall back to this one if it doesn’t fit.

Prerequisites

• Railway dashboard access (production project)
• Neon dashboard access (production DB)
• PostHog access (Adoption Pulse + error dashboard)
• GitHub access (CMMD-Center org)
• Slack #incidents channel
• Your laptop, your terminal, and a clear head

Steps

Step 1 — Acknowledge in #incidents (within 5 min of paging)

Post in Slack #incidents:

🚨 P0 incident — investigating
What I see: [one sentence — what's broken]
Started looking: HH:MM UTC
I'll update every 10 min.

Set a 10-minute timer on your phone or laptop. Cadence over content.

Step 2 — Confirm scope (≤5 min)

Open in parallel browser tabs:

1. Railway → production project → recent deploys + logs
2. PostHog → error-rate dashboard
3. Neon → connection pool + slow query
4. The actual app at cmmd.ai — does it work for you?

Decide severity:

Symptom	Severity
App down or 5xx on >50% of requests	P0
Specific feature broken for many users	P1
Edge case, single-user, workaround exists	P2

Post severity to #incidents immediately. Don’t downgrade until you’re certain.

Step 3 — Form a hypothesis

# What deployed recently?
gh api repos/CMMD-Center/cmmd/commits?per_page=10 | jq '.[] | {sha:.sha[:8], msg:.commit.message[:80], when:.commit.author.date}'

# Was the deploy successful?
# Check Railway → Deployments → recent rows

Most outages = recent deploy. If the last deploy was within the incident window, that’s your prime suspect.

Step 4 — Mitigate (preserve evidence)

Option A — Rollback the last deploy

# In Railway dashboard → production → Deployments → click previous successful → Rollback
# OR via CLI
railway rollback --service cmmd-app --to <previous-deploy-id>

Wait 2 min. Re-check the app.

Option B — Disable a feature flag

# Admin → Feature Flags → toggle the new flag OFF for all orgs
# OR via DB if admin UI is broken:
psql $DATABASE_URL -c "UPDATE feature_flags SET enabled_globally=false WHERE key='<flag-key>';"

Option C — Scale up

If it’s a load issue (CPU/memory/connection pool spike):

# Railway → bump replicas
# Neon → bump compute size if connection pool exhausted

Option D — Kill a stuck migration

If a Railway deploy is stuck on migrations:

# Find the migration that's hung
psql $DATABASE_URL -c "SELECT * FROM pg_stat_activity WHERE state != 'idle';"

# Cancel the bad query
psql $DATABASE_URL -c "SELECT pg_cancel_backend(<pid>);"

# If migration is corrupting state, see Database Migration Runbook

Step 5 — Verify (don’t skip)

After mitigation, confirm the fix:

• App loads at cmmd.ai
• Healthy auth flow (try logging in)
• Sample Sidekick chat succeeds
• Error rate trending down in PostHog
• No unexpected 5xx in Railway logs

A vibe check is not a verification. Use specific checks.

Step 6 — Resolve

Post in #incidents:

✅ Resolved at HH:MM UTC
What happened: [one sentence]
What I did: [one sentence]
Customer impact: [duration + scope]
Postmortem doc by: [tomorrow EOD]

Update status page if used.

Step 7 — Postmortem (within 48 hours)

Use the Incident Postmortem template. Add page under Incidents. Update FM-XXX index.

Required action item: add at least one guard (ESLint rule, CI check, or test) that prevents this class of incident from recurring. Without a guard, the same incident WILL come back.

Rollback

If your mitigation makes things worse:

• Rollback rollback: if you rolled back to a bad version, roll forward to the deploy before that
• Feature flag re-enable: if disabling a flag broke a different feature, re-enable
• Stop and reassess: if every action is making things worse, pause for 60 seconds and re-read the situation. Don’t keep poking.

Troubleshooting

Problem: “I can’t tell what’s broken from the logs”

Cause: log.error calls missing stack: (CLAUDE.md rule 4). Fix in the moment: SSH to a Railway shell, run console.trace() manually if you can reproduce. Long-term fix: Add the stack field to the offending log call.

Problem: “Railway deploy stuck pending”

Cause: Self-hosted CI runner down (we don’t use ubuntu-latest as fallback). Fix: Check runner-health-check.yml. Restart runner if needed.

Problem: “Neon shows DB at 100% connection pool”

Cause: Connection leak in recent code. Mitigate: Bump compute size temporarily. Fix: Find the leak in the recent diff.

Problem: “Sidekick responses are timing out”

Cause: Anthropic/Apex upstream issue. Mitigate: Check provider status. If Anthropic is down, communicate degradation; we don’t have automatic failover yet (see ADR-0005).

After the procedure

• Post in #incidents: “Resolved — full timeline + link to postmortem (TBD)”
• File postmortem within 48 hours
• Add guard (ESLint / CI / test) — required, not optional
• Update this runbook with anything new learned
• Customer notification within 24h if customer-impacting

Related runbooks

• Database Migration Deploy — DB-specific issues
• CI Pipeline Failure — runner / deploy infra
• On-call — rotation, severity, paging