Apex — Operations Runbook

Service inventory

Sentinel (Mac Studio)

Service	Managed by	Health check
Ollama	LaunchAgent com.apex.ollama	curl http://127.0.0.1:11434/api/tags
LiteLLM	LaunchAgent com.cmmd.litellm-apex	curl http://127.0.0.1:4000/health
cloudflared (apex tunnel)	LaunchAgent com.cmmd.cloudflared-apex	pgrep -lf cloudflared
node_exporter	LaunchAgent com.prometheus.node_exporter	curl http://127.0.0.1:9100/metrics

Ryzen

Service	Managed by	Health check
Ollama	systemd ollama.service	systemctl is-active ollama
LiteLLM	systemd litellm-apex.service	systemctl is-active litellm-apex
cloudflared (lab-apex tunnel)	systemd cloudflared-lab-apex.service	systemctl is-active cloudflared-lab-apex
node_exporter	systemd node_exporter.service	curl http://127.0.0.1:9100/metrics

Common operations

Restart a service

Sentinel:

ssh mac-studio 'launchctl kickstart -k gui/$(id -u)/com.cmmd.litellm-apex'

Ryzen:

ssh ryzen 'sudo systemctl restart litellm-apex'

Pull a new model

# Sentinel
ssh mac-studio 'ollama pull <model>'

# Ryzen
ssh ryzen 'ollama pull <model>'

After pulling, add the alias to LiteLLM config and restart LiteLLM. Source of truth is cmmd-center/apex/<host>/litellm/config.yaml and cmmd-center/apex/models/registry.yml.

Re-deploy all Apex configs from source-of-truth

cd ~/projects/apex
./scripts/deploy-mac-studio.sh
./scripts/deploy-ryzen.sh

Phase C will run these automatically on merge to cmmd-center/apex main.

Verify endpoints

export APEX_API_KEY="$(security find-generic-password -s apex-api.cmmd.ai -a apex-core -w)"

# apex-core
curl -s -m 30 https://apex-api.cmmd.ai/v1/messages \
  -H "x-api-key: $APEX_API_KEY" -H "anthropic-version: 2023-06-01" \
  -d '{"model":"apex-core","max_tokens":10,"messages":[{"role":"user","content":"hi"}]}'

# lab-flash
curl -s -m 30 https://lab-apex-api.cmmd.ai/v1/messages \
  -H "x-api-key: $APEX_API_KEY" -H "anthropic-version: 2023-06-01" \
  -d '{"model":"lab-flash","max_tokens":10,"messages":[{"role":"user","content":"hi"}]}'

Monitoring

Live dashboard: https://monitor.aphaiboon.com → Services → Apex API & Lab API — LiteLLM Monitoring.

Panels include:

• Service up/down per tier
• In-flight requests
• Request rate (req/s)
• Latency p50/p95/p99 by model
• Proxy overhead (p95)
• Input/output tokens by model
• Per-output-token speed

Prometheus scrapes both LiteLLM instances every 15s at <tailnet-ip>:4000/metrics/.

Troubleshooting

Endpoint returns 502

Cloudflare reached the tunnel but LiteLLM not responding on :4000. Check the service:

• Sentinel: ssh mac-studio 'launchctl list | grep litellm; tail -20 /tmp/litellm-apex.log'
• Ryzen: ssh ryzen 'sudo journalctl -u litellm-apex --since "5 min ago" -n 30'

Endpoint returns 401

Wrong/missing master key. Verify via:

security find-generic-password -s apex-api.cmmd.ai -a apex-core -w

Compare with on-host:

ssh mac-studio 'cat ~/.apex/litellm/.master_key'

If they differ, the keychain version is canonical — push to host.

Endpoint returns model-not-found

LiteLLM config doesn’t have that alias OR Ollama doesn’t have the model. Check both:

ssh mac-studio 'cat ~/.apex/litellm/config.yaml && ollama list'

Inference is unexpectedly slow

Model may have been swapped out by Ollama’s KEEP_ALIVE timeout. First call after idle = model load (slow). Subsequent calls = warm.

If slowness persists with warm model, check load on the host:

ssh mac-studio 'top -l 1 -n 0 | grep -E "Load|PhysMem"'

Cloudflare tunnel offline

Tunnel daemon stopped on host. Check:

ssh mac-studio 'launchctl list | grep cloudflared-apex'  # should show non-zero PID
ssh ryzen 'sudo systemctl status cloudflared-lab-apex'

Restart via the LaunchAgent / systemd unit.

Capacity

Host	Total RAM	Reserved for model	Concurrent requests @ 8K ctx
Sentinel (M1 Ultra 128 GB)	128 GB	81 GB (apex-core) + ~10 GB overhead	6-10 native, 4 concurrent slots
Ryzen (32 GB + 8 GB GPU)	32 GB	~23 GB (both lab models swapping)	4-6 native

For higher concurrency, add more nodes or increase OLLAMA_NUM_PARALLEL. Production scaling is Phase E (not yet planned).

Related

• Architecture
• Models
• Apex repo