Runbook — AI provider daily spend burn

Field	Value
Severity	P1 (catches problems before they’re P0 monthly-budget)
Owner	@aphaiboon
Alert source	Watchtower — sum by (provider) (rate(cmmd_ai_cost_usd_total[1h])) * 24 > 5 * (trailing 7d avg) for 30m
Estimated time	15 min
Required access	CMMD AdminApp, Railway

Symptom

ntfy push: “DeepSeek (or other) burning at 23× normal — $42/day projected vs $1.80/day baseline”.

Most likely causes

Same as runbook-ai-provider-budget-breached, but earlier in the timeline. Investigation is identical — the difference is you have more headroom to fix without breaching budget.

Diagnose

1. Open CMMD AdminApp → Reporting → AI cost-per-hour chart for the offending provider.

2. Confirm the spike is real (not a Prometheus blip — wait 5 min, re-check).

3. SQL drill-down:

   SELECT date_trunc('hour', created_at) as hour,
          feature,
          COUNT(*) as requests,
          SUM(cost_cents)/100.0 as cost_usd
   FROM ai_agent_usage_logs
   WHERE created_at >= NOW() - INTERVAL '4 hours'
     AND provider = '<provider>'
   GROUP BY 1, 2
   ORDER BY hour DESC, cost_usd DESC;

4. Look for: (a) sudden new feature with high spend, (b) one user dominating, (c) a worker that started firing more often.

Mitigate

This is an early-warning alert — you have time to fix the root cause rather than slap an emergency throttle.

1. Identify the root cause via the diagnosis SQL above.
2. Open a Slack thread in #alerts with the cause + a plan: “fix in code”, “raise budget”, or “throttle the feature”.
3. Ship the fix or budget change.
4. Cancel the snooze on the alert once spend rate drops back below 2× normal.

Verify resolved

• Burn-rate query drops back below 5× for 30 min
• Daily projected spend within configured budget

Escalate

• If you cannot identify the cause within 1 hour → ack the alert + open an incident channel
• If burn rate continues to climb → treat as full budget breach (see runbook-ai-provider-budget-breached)

Related

• runbook-ai-provider-budget-breached