Runbook — CI Pipeline Failure

Field	Value
Severity	P1 (deploys + merges blocked)
Owner	@aphaiboon
Last verified working	2026-05-16
Estimated time	10–30 min
Required access	GitHub org admin, Railway, self-hosted runner machine

When to use this runbook

• CI checks stay “Queued” forever on a PR
• “CI Complete — Waiting for status to be reported” persists >5 min
• Multiple workflows fail with no obvious cause
• A workflow that just worked yesterday is now red

If the failure looks like app code is broken — that’s not this runbook. Read the actual test failure and fix the code.

This runbook is for CI infrastructure problems.

Prerequisites

• GitHub admin access to CMMD-Center org
• SSH access to self-hosted runner machine(s)
• Railway dashboard access
• Slack #incidents channel

Steps

Step 1 — Identify the failure mode

# Check runner status in GitHub
gh api orgs/CMMD-Center/actions/runners | jq '.runners[] | {name:.name, status:.status, labels:[.labels[].name]}'
# Expected: all runners show "online", labels include "validation", "e2e", etc.

What you see	Likely cause	Jump to
All runners “offline”	Runner machine down	Step 2
Runners online but workflows queued	Label mismatch or capacity	Step 3
Workflows fail at a specific step	Image / dependency issue	Step 4
ci-complete job pending forever	Required check labels misaligned	Step 5

Step 2 — Runner machine down

# SSH to runner host
ssh runner@<runner-host>

# Check process
systemctl status actions.runner.CMMD-Center.<name>.service
# or
ps aux | grep Runner.Listener

# Restart if stopped
sudo systemctl restart actions.runner.CMMD-Center.<name>.service

# Verify in GitHub
gh api orgs/CMMD-Center/actions/runners | jq '.runners[] | select(.name=="<name>") | .status'

If the runner host itself is offline, see the host’s hosting provider dashboard.

Long-term: at least 2 runners per label (validation, e2e) for redundancy.

Step 3 — Workflows queued (runners online, no pickup)

Check label match. The CI/CD Pipeline workflow assigns:

runs-on: [self-hosted, validation]

GitHub matches runners that have both labels. If you accidentally removed the validation label from a runner, jobs queue.

# Add missing labels via GitHub UI:
# Org Settings → Actions → Runners → click runner → Add label "validation"

Capacity issue: if you have 1 runner and 5 PRs push simultaneously, jobs queue. Adding a second runner per label fixes this.

Step 4 — Workflows fail at a specific step

Common: npm ci failing

Cause: Cache corruption or upstream package issue. Fix: Don’t add bare npm ci — use .github/scripts/install-deps-cached.sh (CLAUDE.md rule, 2026-03-27).

Common: typecheck OOM

Cause: Default Node memory limit hit. Fix: Ensure NODE_OPTIONS=--max-old-space-size=8192 is set for TypeScript steps.

Common: Playwright browsers missing

Cause: Runner image doesn’t have browsers installed. Fix:

# On runner host
npx playwright install --with-deps chromium

Or update the runner provisioning script to include this.

Common: Pre-push hook failing locally but not for others

Cause: Local environment drift (different Node version, etc.). Fix: nvm use 20 (per .node-version).

Step 5 — ci-complete aggregate check pending forever

The ci-complete job needs: all gates. Until ALL earlier jobs finish (or runners pick them up), GitHub treats the aggregate required check as pending.

If nothing is executing, runner capacity is the issue — see Step 2/3.

If individual jobs failed, fix the underlying failure first. ci-complete won’t run until its dependencies succeed.

Step 6 — Verify

After mitigation:

• All runners show “online” via gh api
• Test PR with a trivial change can pass CI end-to-end
• ci-complete reports green on at least one PR

Rollback

CI doesn’t really “roll back.” If a recent CI config change broke things, revert that PR:

gh pr revert <bad-pr-number>

If a workflow file change broke CI, edit the workflow file directly in .github/workflows/ and push a fix.

Troubleshooting

Problem: “GitHub says workflow is required but it never runs”

Cause: Workflow file is broken / has a syntax error. Fix: gh api repos/CMMD-Center/cmmd/actions/runs?per_page=5 | jq — check for workflow_dispatch failures. Validate YAML syntax.

Problem: “Self-hosted runner shows online but never picks up jobs”

Cause: Runner has wrong labels OR workflow uses different labels. Fix: Compare runner labels vs runs-on in the workflow. They must match exactly (both must be present on the runner).

Problem: “CI passes locally but fails on Railway deploy”

Cause: Railway is a deploy, not CI. If you mean npm run db:migrate is failing on Railway, see Database Migration Deploy Runbook.

Problem: “Pre-push hook is too slow for cloud agents”

Per AGENTS.md, use SKIP_BUILD=true git push to skip the production build step, or git push --no-verify to skip all hooks. Only for cloud-agent contexts — don’t use on human pushes without team agreement.

After the procedure

• Post in #deploys or #incidents: “CI restored at HH:MM UTC — root cause: [X]”
• If runner went down, add monitoring or redundancy (single point of failure)
• If a runbook step was wrong, fix this page
• If a new failure mode appeared, add it under Troubleshooting

Related

• ADR-0003 — Self-Hosted CI Runners Only
• AGENTS.md “Queued forever” section
• .github/workflows/ci-cd.yml, runner-health-check.yml