Runbook — AI provider monthly budget breached

Field	Value
Severity	P0 (cost runaway can be existential pre-revenue)
Owner	@aphaiboon
Alert source	Watchtower — cmmd_provider_monthly_cost_usd / cmmd_provider_monthly_budget_usd > 1.0
Estimated time	30 min
Required access	CMMD AdminApp → AI usage, Railway, provider dashboards

Symptom

ntfy push: “Anthropic budget exceeded (105%)” or similar per provider. Watchtower → CMMD overall → Today’s AI spend tile may also be in burn-rate alert.

Why P0

Pre-revenue, every dollar spent on LLM upstream is money out the door without revenue offset. A runaway can be existential. Treat as P0 even when there’s no customer impact.

Most likely causes (ranked)

1. Retry storm — a worker is retrying failed LLM calls without backoff. Check logs for repeating error patterns.
2. Bug increasing call rate — a recent change unintentionally calls an LLM more often per user action (e.g. removed a cache, broke a deduper).
3. Abuse — a user or external actor is driving traffic deliberately. Check per-user breakdown in CMMD AdminApp.
4. Real growth + stale budget — actual usage scaled past the configured cap. Decision: raise the budget, throttle, or reprice.
5. Provider raised prices — verify against the provider’s pricing page; CMMD’s CostCalculator constants may be stale.

Diagnose

1. Open CMMD AdminApp → Reporting → AI usage by provider (last 24h vs last 7d).

2. From SQL access, drill into the suspicious provider:

   SELECT feature, user_id, provider, model,
          COUNT(*) as requests,
          SUM(total_tokens) as tokens,
          SUM(cost_cents)/100.0 as cost_usd
   FROM ai_agent_usage_logs
   WHERE created_at >= NOW() - INTERVAL '24 hours'
     AND provider = '<provider>'
   GROUP BY 1, 2, 3, 4
   ORDER BY cost_usd DESC
   LIMIT 20;

3. Identify the top spender — usually one feature or one user dominates.

4. Cross-reference recent commits on main for that feature.

Mitigate

Retry storm

1. Identify the worker firing repeated calls.
2. Pause the worker in Railway (or feature-flag it off if there’s a feature_flags row).
3. Open an issue to add proper backoff before re-enabling.

Bug causing call rate spike

1. Roll back the offending deploy via Railway.
2. Open an issue for the fix.

Abuse

1. Identify the user from ai_agent_usage_logs.user_id.
2. Admin-disable the account if clearly abusive.
3. Add per-user rate limit if not already in place.

Real growth

1. Raise the budget via AI_PROVIDER_MONTHLY_BUDGET_OVERRIDES env var on Railway.
2. Notify in #alerts of the new ceiling.
3. Open an issue to think about pricing/throttling for sustained scale.

Verify resolved

• cmmd_provider_monthly_cost_usd / cmmd_provider_monthly_budget_usd < 0.95 for 10 minutes (or budget raised so the ratio is back below 1.0)
• Cause identified + recorded in postmortem doc

Escalate

• If spend is doubling every hour → cut off the offending feature flag immediately, even before full diagnosis
• If sustained for 1h+ with no clear cause → escalate to incident response

Related

• runbook-ai-provider-daily-spend-burn
• runbook-llm-provider-degraded (retry storms often coincide with provider degradation)