Runbook — Stripe webhook failures

Field	Value
Severity	P0 (billing state desynchronization → revenue loss + compliance risk)
Owner	@aphaiboon
Alert source	Watchtower — rate(cmmd_webhooks_received_total{provider="stripe", status="failed"}[5m]) > 0.1 OR absent(cmmd_webhook_last_received_seconds{provider="stripe"}) for 30m
Estimated time	15–45 min
Required access	Stripe dashboard, Railway, CMMD AdminApp

Symptom

ntfy push: “Stripe webhook failures spiking” or “No Stripe webhooks received in 30 min”. App may show subscription state that doesn’t match Stripe.

Why P0

Stripe webhooks drive subscription lifecycle in CMMD (created, updated, canceled, payment failures). When they’re failing:

• New customers may not get correct entitlements
• Cancellations may not propagate (customers retain access they shouldn’t have)
• Failed payments aren’t reflected in dunning UI
• Revenue reporting becomes inaccurate

Most likely causes

1. Webhook endpoint returning 5xx — recent CMMD deploy broke the route. Stripe retries with exponential backoff.
2. Signing secret mismatch — STRIPE_WEBHOOK_SECRET rotated in Stripe dashboard but not updated in Railway env.
3. Stripe disabled the endpoint — after too many failures Stripe auto-disables. Stripe dashboard → Developers → Webhooks shows status.
4. Network / Cloudflare WAF blocking — Stripe’s IPs hit a WAF rule. Unlikely but possible.

Diagnose

1. Stripe dashboard → Developers → Webhooks → click the CMMD endpoint → check recent delivery attempts.

2. Railway → CMMD logs filter [STRIPE-WEBHOOK] for the last 1 hour. Look for the actual error.

3. Check signing secret matches:

   # Stripe dashboard endpoint setting → reveal "Signing secret"
   # vs Railway env STRIPE_WEBHOOK_SECRET

4. From CMMD AdminApp → query recent webhook events:

   SELECT event_type, status, COUNT(*), MAX(created_at) as last_seen
   FROM stripe_webhook_events  -- or whatever the table is named
   WHERE created_at >= NOW() - INTERVAL '1 hour'
   GROUP BY 1, 2;

Mitigate

Endpoint returning 5xx (most common)

1. Identify which event_type is failing. Drill into log error.
2. If a recent deploy broke handler logic: Railway rollback.
3. If the issue is a specific event handler: ship a fix.
4. After fix is live: Stripe dashboard → resend the failed events from “Recent deliveries” view.

Signing secret mismatch

1. Stripe dashboard → reveal current signing secret.
2. Railway → CMMD env → update STRIPE_WEBHOOK_SECRET → redeploy.
3. Stripe dashboard → resend recent failed events.

Endpoint disabled by Stripe

1. Stripe dashboard → Webhooks → CMMD endpoint → re-enable.
2. Fix the root cause first (whatever caused Stripe to disable it).
3. Resend recent failed events.

Verify resolved

• Stripe dashboard “Recent deliveries” shows 200 OK for new events
• rate(cmmd_webhooks_received_total{provider="stripe"}[5m]) > 0 (events flowing again)
• cmmd_webhooks_received_total{status="failed"} rate dropped to 0

Backfill missed events

If endpoint was down for >1 hour, replay events:

1. Stripe dashboard → Events → filter by time window → identify subscription state changes.
2. Resend each event individually from Stripe UI (preserves signing semantics).
3. Cross-reference with CMMD’s subscriptions table to confirm sync.

Escalate

• Sustained 30+ min webhook downtime → manually reconcile high-value subscriptions before they cause customer-facing issues
• Recurring secret-mismatch incidents → add a CI check that compares Stripe API’d secret vs Railway env

Related

• runbook-cmmd-app-down (if endpoint 500s are part of broader app-down event)
• runbook-production-incident-response