Runbook — Watchtower can't scrape CMMD /metrics

Field	Value
Severity	P1 (visibility loss, not customer impact)
Owner	@aphaiboon
Alert source	Watchtower — up{job="cmmd-app-metrics"} == 0 for 5m OR absent(cmmd_product_ai_cost_24h_cents) for 10m
Estimated time	5–15 min
Required access	Watchtower (Coolify env), Railway (CMMD env), Prometheus UI

Symptom

ntfy push: “Watchtower scrape of cmmd.ai/metrics is down”. CMMD app itself may be perfectly healthy (Watchtower → CMMD app probe tile still green); only the metrics scrape is broken.

Why this matters

Without /metrics, all of Watchtower’s CMMD-internal tiles go blank:

• Today’s AI spend
• Per-provider usage gauges
• Database health gauge
• HTTP request rate
• Active users

We lose visibility into cost-burn (the P0-class signal). The app still serves customers fine.

Most likely causes

1. Token mismatch — CMMD_METRICS_TOKEN in watchtower’s Coolify env doesn’t match PROMETHEUS_BEARER_TOKEN in CMMD’s Railway env. Probably the value drifted (rotated on one side).
2. CMMD app down — the /metrics endpoint can’t respond because the whole app isn’t responding. In that case the cmmd-app probe tile is also red → see runbook-cmmd-app-down first.
3. Network path broken — Cloudflare blocking the bearer-auth request, or Coolify can’t reach cmmd.ai
4. Token unset on the watchtower side — CMMD_METRICS_TOKEN not set on Coolify; Prometheus targets page shows 401 unauthorized for the scrape.

Diagnose

1. Open Watchtower → Connections → Prometheus → /targets → find cmmd-app-metrics. Check the error message.

2. If “401 unauthorized”: token mismatch.

3. If “context deadline exceeded”: network path.

4. Manually curl from your machine:

   # No bearer — should 401
   curl -i https://cmmd.ai/metrics
   
   # With current token (the working one)
   curl -i -H "Authorization: Bearer <CMMD_PROMETHEUS_BEARER_TOKEN>" https://cmmd.ai/metrics
   # Expect: 200 + prom-format text

Mitigate

Token mismatch

1. CMMD → Railway → env → reveal PROMETHEUS_BEARER_TOKEN. (Or PROMETHEUS_SCRAPE_TOKEN — server/observability/prometheus.ts accepts either.)
2. Watchtower → Coolify → env → set CMMD_METRICS_TOKEN to that value.
3. Restart watchtower’s prometheus container: docker compose restart prometheus (or via Coolify UI).
4. Watch the /targets page — within 30s the cmmd-app-metrics target should flip from DOWN to UP.

Token unset

Same as above — set the env var fresh on Coolify side.

Network path broken

1. From inside the Coolify host: curl -i https://cmmd.ai/metrics — does it reach cmmd.ai?
2. If the same curl works from your laptop but not from Coolify → outbound block on Coolify side. Investigate networking.

CMMD app down

Follow runbook-cmmd-app-down first. The metrics scrape will recover automatically once the app is back.

Verify resolved

• Watchtower → Prometheus → /targets → cmmd-app-metrics shows UP
• Query in Prometheus: cmmd_product_ai_requests_24h_total returns data
• Watchtower home dashboard tiles populated

Escalate

• If the token CAN’T be found in either env → rotate: generate a new strong token, set in both Railway + Coolify, restart both
• If you suspect a leaked token → rotate immediately

Related

• runbook-cmmd-app-down
• server/observability/prometheus.ts — token-checking source