Runbook — Upstash Redis degraded

Field	Value
Severity	P1
Owner	@aphaiboon
Alert source	Watchtower — cmmd_redis_connected == 0 for 5m
Estimated time	5–15 min
Required access	Upstash dashboard, Railway

Symptom

Watchtower → CMMD overall → request rate still flowing, but cmmd_redis_connected tile flips to 0. App functions but session validation + rate limiting are degraded.

Why this is P1 not P0

Per server/redis-client.ts + AGENTS.md, CMMD gracefully falls back to in-memory stores for sessions, rate limiting, and caching when Redis is unavailable. The app does not crash. But:

• Session state isn’t shared across Railway replicas → users may see inconsistent state under load
• Rate limiting is per-replica → an attacker could bypass it by hitting different replicas
• Sidekick session persistence may be impacted

Most likely causes

1. Upstash quota exhausted — free tier 30MB limit. Check cmmd_redis_memory_bytes if exposed.
2. Network blip between Railway and Upstash — usually self-heals in <60s.
3. Upstash platform incident — check Upstash status (no public status page tile yet; check status.upstash.com manually).
4. Stale REDIS_URL env var — recently rotated and Railway env wasn’t updated.

Diagnose

1. Upstash dashboard → check database state, memory usage, recent ops.

2. Railway → CMMD service env → confirm REDIS_URL is set + not expired.

3. From Railway shell:

   redis-cli -u "$REDIS_URL" ping
   # Expect: PONG

Mitigate

Memory quota

1. Upstash dashboard → flush or shrink the keyspace if low-value cache entries dominate.
2. If quota exhausted often → upgrade Upstash tier or evict more aggressively in code.

Stale URL

1. Upstash → copy the current connection URL.
2. Railway → CMMD service → env → update REDIS_URL → redeploy.

Upstash incident

1. Wait for Upstash to restore.
2. CMMD continues functioning (degraded) during the outage.

Verify resolved

• cmmd_redis_connected == 1 for 2 minutes
• redis-cli -u $REDIS_URL ping returns PONG
• No new error logs of pattern [REDIS] in Railway

Escalate

• Sustained 30+ min outage with customer-visible impact → notify in #alerts and consider customer comm
• Upstash account-level issue → contact Upstash support

Related

• runbook-cmmd-app-down
• AGENTS.md “Redis optional” note (app gracefully falls back)