Runbook — Neon Postgres degraded

Field	Value
Severity	P0
Owner	@aphaiboon
Alert source	Watchtower — cmmd_database_healthy == 0 for 1m OR pg_up{datname="cmmd"} == 0
Estimated time	5–30 min
Required access	Neon dashboard, Railway, SQL access via psql

Symptom

Watchtower → Infra critical → Neon DB tile red. App returns 5xx on most routes. PostHog error rate spikes.

Most likely causes

1. Connection pool exhausted — usually a runaway worker or stuck transactions. Look at pg_stat_activity.
2. Neon platform incident — check the Neon status page tile on Watchtower home.
3. Failed migration — recently merged migration locked a hot table or never completed.
4. Compute auto-scaled down to zero (Neon free tier behavior on idle). Wakes on next query but causes 30s+ first-request latency.

Diagnose

1. Open Neon dashboard → check compute state + active connections graph.

2. Check the Neon status page tile on Watchtower home.

3. From any developer machine with access:

   -- Active connections by application
   SELECT application_name, state, count(*)
   FROM pg_stat_activity
   WHERE datname = 'cmmd'
   GROUP BY 1, 2;
   
   -- Long-running queries (suspect)
   SELECT pid, now() - query_start AS duration, state, query
   FROM pg_stat_activity
   WHERE datname = 'cmmd'
     AND state != 'idle'
     AND now() - query_start > interval '30 seconds'
   ORDER BY duration DESC;

4. If a single query is dominating: capture its text, then SELECT pg_terminate_backend(<pid>);

Mitigate

Connection storm

1. Identify the worst offender from pg_stat_activity query above.
2. Terminate the bad backend(s).
3. Watch the pool: active connections should drop within seconds.
4. If a CMMD background worker is the source: pause it in Railway until you can ship a fix.

Neon platform incident

1. Confirm on status.neon.tech (or Neon status page tile in Watchtower home).
2. Inform in #alerts — “Neon platform incident, customer impact: <scope>”.
3. There is no failover (single Neon project). Wait for Neon to restore. Capture timestamps + customer impact for postmortem.

Failed migration

1. Railway → check the latest deploy. If migration was the cause, the Railway deploy log will show the migration error.
2. Roll back the deploy (which re-deploys with the prior migration head).
3. Open an issue to fix the migration before re-attempting.

Verify resolved

• cmmd_database_healthy == 1 for 3 minutes
• curl -i https://cmmd.ai/health returns 200 with DB-backed payload populated

Escalate

• Neon dashboard shows account-level issues → email Neon support
• More than 15 min downtime → notify customers via status.cmmd.ai

Related

• runbook-cmmd-app-down
• runbook-database-migration-deploy