Runbook — Upstream LLM provider degraded

Field	Value
Severity	varies — P0 for critical-tier providers (Anthropic, OpenAI, Apex), P1 for others
Owner	@aphaiboon
Alert source	Watchtower — probe_success{job="llm-...-list", provider=...} == 0 for 3m OR probe_success{job="status-statuspage-io", service=...} == 0 for 5m
Estimated time	5–20 min
Required access	Provider status pages, Railway, CMMD AdminApp

Symptom

Watchtower → LLM providers row shows one tile red. ntfy push: “Anthropic provider degraded”.

How CMMD handles provider failures (already)

server/services/ai-gateway/AIGateway.ts implements provider fallback via ai-gateway-fallback-ops-alert.ts. When the primary provider fails, the gateway retries on the next provider in the fallback chain. So most provider degradations are invisible to customers — they just get slower responses while the gateway falls back.

This runbook is for the cases where:

• The primary provider AND its fallbacks are all degraded simultaneously
• The provider is up but returning garbage (200 + empty content)
• Cost is spiking due to fallback retries

Diagnose

1. Open Watchtower home → identify which provider is red.

2. Click into the corresponding status-page tile (Row 5) — does the provider itself acknowledge the incident?

3. Open CMMD AdminApp → Reporting → recent AI gateway fallback events for that provider.

4. Check ai_agent_usage_logs for the last 30 min — what’s the success rate by provider?

   SELECT provider, model, success, COUNT(*)
   FROM ai_agent_usage_logs
   WHERE created_at >= NOW() - INTERVAL '30 minutes'
   GROUP BY 1, 2, 3
   ORDER BY 1, 2;

Mitigate

Single provider degraded, fallbacks healthy → no action

AIGateway is already falling back. Acknowledge the alert. Watch for cost-burn (fallbacks usually use cheaper or different models).

Multiple providers degraded simultaneously

Check status.cloud.google.com (Google Cloud), status.cloudflare.com, etc — sometimes a backbone outage takes down multiple providers at once. If so, wait it out + post in #alerts.

Single provider, no public status-page acknowledgement, but our probes fail

1. Spot-check with curl:

   curl -sS https://api.<provider>.com/v1/models \
     -H "Authorization: Bearer <test_key>"

2. If our key is rate-limited / suspended → handle as separate auth issue.

3. If the response is malformed (200 but no model list) → file a ticket with the provider + rely on AIGateway fallback.

Cost spike from fallback retries

If fallback to a more expensive provider is driving up cost, consider temporarily disabling that provider in CMMD’s AI provider config (Admin UI or env override) until upstream recovers.

Verify resolved

• Probe probe_success{job="llm-..."} returns 1 for 5 minutes
• Provider status-page tile back to green
• ai_agent_usage_logs success rate above 95% for the last 15 min

Escalate

• Provider gives no ETA + business-critical feature degraded → notify customers via status.cmmd.ai
• Cost runaway from fallback retries → see runbook-ai-provider-daily-spend-burn

Related

• runbook-ai-provider-daily-spend-burn
• server/services/ai-gateway/AIGateway.ts — fallback logic
• server/services/ai-gateway/ai-gateway-fallback-ops-alert.ts — alerting on fallback events