ADR-0003 — Self-Hosted CI Runners Only

Field	Value
Status	🟢 Accepted
Date	2026-03
Deciders	@aphaiboon
Related incidents	FM-006 (OOM during TS compilation), FM-046 (Deleted scheduled producer)

Context

CMMD CI runs typecheck, lint, unit tests, integration tests, E2E (Playwright), and production build. The E2E suite alone exceeds GitHub-hosted runner memory limits intermittently — tsc + Vite + Playwright + Postgres on a 7GB GitHub-hosted runner OOMs unpredictably (see FM-006).

Two additional concerns:

1. Cost. With multiple parallel CI jobs on every PR + push to main + staging, GitHub-hosted minutes burn fast (a single full-CI cycle on ubuntu-latest is ~20–30 minutes × 4–8 jobs). At scale this is a meaningful spend with poor predictability.
2. Environment fidelity. Production runs on Railway with specific Node + system library versions. GitHub runners get whatever Ubuntu LTS image is current. Subtle library-version mismatches caused at least one production-vs-CI divergence.

Self-hosted runners solve both: persistent machines with predictable resources, controlled images, and zero per-minute cost.

Decision

All CI jobs runs-on: [self-hosted, <label>]. NEVER ubuntu-latest.

Labels and their workloads

Label	Workload	Resource profile
validation	Typecheck, lint, unit tests, build	8GB RAM minimum
e2e	Playwright suite + Postgres + dev server	16GB RAM, headed Chromium
agent-worker	AI-assisted background jobs (CI-time codemods, etc.)	Variable
studio	Specialized one-off workloads (e.g. design-system regenerate)	As needed

cmmd-center org note (2026-05-19): the same rule extends to all cmmd-center/* repos. Org Actions billing is gated; ubuntu-latest is silently rejected. See Self-hosted runners only — cmmd-center.

Alternatives considered

Option A — Keep ubuntu-latest, increase parallelism

• ✅ Zero infra to manage
• ❌ OOM failures unpredictable on E2E
• ❌ Cost scales with team size
• ❌ Environment drift from Railway

Option B — BuildJet / Namespace / similar managed runner SaaS

• ✅ Bigger runners, still managed
• ❌ Cost premium over self-hosted (~2–3× GitHub-hosted at our load)
• ❌ Same env-drift concern (their images, not ours)
• Rejected: Self-host is cheaper once we own a machine, and we needed env fidelity anyway

Option C — Hybrid (self-hosted for E2E only, GitHub-hosted for the rest)

• ✅ Smaller blast radius
• ❌ Two CI systems to maintain
• ❌ Cost reduction is partial
• Rejected: Operational complexity not worth the partial benefit

Consequences

Positive

• ✅ Zero OOM on E2E since switchover
• ✅ Predictable CI time per job (no noisy-neighbor on shared GitHub-hosted)
• ✅ Cost stable as PR volume grows
• ✅ Environment matches Railway closely

Negative / trade-offs

• ⚠️ Operational burden: runner machines need updates, restarts, monitoring (see Runbook — CI Pipeline Failure)
• ⚠️ If runners go offline, CI queues forever — every job uses runs-on: [self-hosted, <label>] so no fallback to GitHub-hosted exists
• ⚠️ Scaling for traffic spikes requires manual provisioning (no elastic scale)

Risks

• 🔴 Single point of failure when only one runner per label. Mitigation: at least 2 runners per label in production, with runner-health-check.yml scheduled probe
• 🟡 Runner ghost-processes accumulating — mitigation: pkill -f cleanup in parallel-worker-validation.md rule

Implementation plan

• Provision first self-hosted runner on dedicated machine
• Migrate all workflows from runs-on: ubuntu-latest to runs-on: [self-hosted, validation]
• Add e2e runner with 16GB + Chromium
• Document in CLAUDE.md as rule #9: NON-NEGOTIABLE
• Runner health-check workflow .github/workflows/runner-health-check.yml
• Add second runner per label for redundancy (in progress)

Review

Next review: When team grows past ~10 engineers OR when runner operational burden exceeds 2 hours/week.

References

• Runbook — CI Pipeline Failure
• FM-006 — OOM during TS compilation
• FM-046 — Deleted scheduled producer
• Engineering Principles → The guard, not the rule
• AGENTS.md — “Queued forever” troubleshooting section
• .github/workflows/ci-cd.yml, runner-health-check.yml