Runbook — CMMD app down
cmmd.ai HTTP endpoint not responding or returning sustained 5xx. First mitigation step is rollback via Railway if the spike correlates with a recent deploy.
| Field | Value |
|---|---|
| Severity | P0 |
| Owner | @aphaiboon |
| Alert source | Watchtower — probe_success{job="cmmd-app", product="cmmd-app"} == 0 for 2m |
| Estimated time | 10–20 min |
| Required access | Railway, GitHub, PostHog |
Symptom
Section titled “Symptom”watchtower.cmmd.ai shows the CMMD app tile red. ntfy push to phone says cmmd-app-down. External Cronitor probe may also be alerting via SMS.
Most likely causes (ranked by frequency)
Section titled “Most likely causes (ranked by frequency)”- Bad deploy — most common. The blackbox probe started failing within minutes of a
mainmerge. - Neon database degraded — app starts returning 5xx because all queries time out. See neon-db-down.
- Railway platform issue — confirm via status pages tile on Watchtower home (
status-pagesrow → Railway). - Memory leak / OOM — Railway shows the service in restart loop.
Diagnose
Section titled “Diagnose”- Open Watchtower home → check whether Railway status page tile is also red. If yes → upstream platform, jump to step “Escalate to Railway”.
- Open Railway dashboard for the CMMD production service → check current deploy status + recent deploy times.
- Pull recent logs from Railway: filter for
level=errorin the last 30 min. - Open PostHog → check error-rate dashboard.
Mitigate
Section titled “Mitigate”If it correlates with a recent deploy
Section titled “If it correlates with a recent deploy”- Railway → CMMD service → Deployments → Rollback to last known-good build.
- Verify within 2 min:
curl -i https://cmmd.ai/healthreturns 200. - Post in Slack
#alertsconfirming rollback. - Open a follow-up issue to investigate the broken build before re-deploying.
If no recent deploy
Section titled “If no recent deploy”- Check
Watchtower → Infra critical → Neon. If Neon is red, follow neon-db-down. - Railway → check service health → restart the container if it’s been wedged.
- If memory pressure visible → bump the service’s memory limit one tier in Railway.
Verify resolved
Section titled “Verify resolved”curl -i https://cmmd.ai/healthreturns 200 for 3 minutes in a row- Watchtower
cmmd-apptile green for 3 minutes - Error-rate dashboard back below normal
Escalate
Section titled “Escalate”- Railway platform issue → open ticket via Railway support, post in Slack
#alerts - Sustained 30+ min downtime → notify in
#incidentschannel + customer comm plan
Related
Section titled “Related”- runbook-production-incident-response
- runbook-neon-db-down
- runbook-cmmd-metrics-scrape-down (only the metrics scrape is down, not the app)