On-call

Who’s on the pager, what counts as a page, how to respond.

Current rotation

Week	Primary	Backup
(rotation to be filled in as team grows)	@aphaiboon	—

For now: @aphaiboon is primary on-call for all severities. As we add engineers, rotation becomes 1-week shifts.

Severity definitions

Sev	Definition	Response time
P0 — Critical	App down or data loss in progress, >1 customer affected	<15 min — drop everything
P1 — High	Feature broken for many users, no data loss	<1 hour
P2 — Medium	Feature broken for some users, workaround exists	<1 business day
P3 — Low	Cosmetic, single-user, edge case	Next sprint

A P0 is rare. A P1 is news. A P2 is a Jira ticket. A P3 is a backlog item.

What pages the on-call

• PostHog alert — error rate spike on /api/sidekick/chat or auth endpoints (TODO: configure)
• Railway alert — app restart loop, OOM, deploy failure (TODO: configure)
• Neon alert — DB at >80% connection pool, slow query >5s sustained (TODO: configure)
• Status page — customer-reported outage at status.cmmd.ai (TODO: launch status page)
• Manual — engineer escalates from Slack #incidents

The on-call playbook

When you’re paged:

1. Acknowledge in Slack #incidents within 5 min (“I’m on it”)
2. Assess — is it P0/P1/P2? Look at the dashboard.
3. Communicate — post in #incidents: what’s broken, what’s the impact, what’s your next step
4. Mitigate — use a runbook if one exists. If not, work from first principles.
5. Resolve — confirm with a specific check (not vibes), post “Resolved at HH:MM”
6. Postmortem — within 48 hours for any P0 or P1, using the Postmortem template, filed in Incidents

Runbooks you need handy

• Production Incident Response — the primary “what do I do” runbook
• Database Migration Deploy — when the issue is migration-related
• CI Pipeline Failure — when CI is the blocker

Tools you need handy

• Railway dashboard
• Neon dashboard
• PostHog dashboard
• Status page admin
• This wiki (Runbooks, Incidents)
• The Sidekick (use it — ask “what changed in the last 24h that could cause X”)

Handing off

End of your week → handoff in Slack #oncall with:

• Active incidents (none / list)
• Action items from your week
• Anything strange you noticed but didn’t fully investigate

Related

• Runbooks — operational procedures
• Incidents — 47 FM postmortems (search past incidents before declaring a new one)
• Security Posture — for security-classed incidents
• Vulnerability Disclosure Policy — when someone reports a vuln externally