Forge — Provider Egress Proxy

Architecture, 2026-05-15 incident postmortem, and operational runbook. User VMs never see raw provider keys; master mints per-user, time-bounded proxy tokens.

Status (as of 2026-05-15): All 5 routable upstreams are proxied through master in prod. Zero raw sk-* keys reach user VMs.

Provider	User-VM client	Proxy route	Status
Anthropic	Claude Code CLI (honors `ANTHROPIC_BASE_URL`)	`/api/provider-egress/anthropic/*`	✅ Verified end-to-end in prod 2026-05-15
OpenAI	Codex CLI (honors `~/.codex/config.toml` only — NOT `OPENAI_BASE_URL` env)	`/api/provider-egress/openai/*`	✅ Verified end-to-end in prod 2026-05-15
DeepSeek	Claude Code via Apex model profile	`/api/provider-egress/apex/anthropic/*`	✅ Architecturally proxied since PR #310
Kimi	Claude Code via Apex model profile	`/api/provider-egress/apex/anthropic/*`	✅ Same as DeepSeek
OpenCode	OpenCode CLI (Anthropic SDK, honors `ANTHROPIC_BASE_URL`)	`/api/provider-egress/anthropic/*`	✅ Routes through Anthropic egress
Cursor	Cursor CLI	—	⚠️ Not investigated end-to-end

Owner: @aphaiboon · Last verified in prod: 2026-05-15 17:09 UTC via PR #323 deploy.

Why this exists

User VMs run on shared Fly infrastructure (forge-cloud-users app) and execute untrusted code on behalf of the user — agent CLIs, project scripts, user-submitted shell commands. Forge holds paid upstream API keys (Anthropic, OpenAI, etc.) and is on the hook for the bill plus rate limits. The user VM must never see those raw keys. If it did:

A compromised user VM could exfiltrate the key
Workload from one user could impersonate another at the upstream
We lose per-user attribution / rate limiting / revocation

The egress proxy keeps the real keys on master (one fly machine, controlled deploy, audited code) and mints per-user, time-bounded, provider-scoped tokens that user VMs use in place of the raw keys. Master validates the token, swaps in the real upstream key, and streams the response back.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ forge-cloud-master (fly app, 1 always-on machine)               │
│   ─ holds real ANTHROPIC_API_KEY, OPENAI_API_KEY,               │
│     DEEPSEEK_API_KEY, KIMI_API_KEY, OPENCODE_API_KEY            │
│     as Fly secrets                                              │
│   ─ holds FORGE_PPROXY_HMAC_SECRET for signing proxy tokens     │
│   ─ runs Dockerfile (master image, no provider CLIs)            │
│                                                                 │
│   Routes:                                                       │
│     ├─ POST /api/provider-egress/tokens (programmatic)          │
│     ├─ * /api/provider-egress/anthropic/* → api.anthropic.com   │
│     ├─ * /api/provider-egress/openai/*    → api.openai.com      │
│     └─ * /api/provider-egress/apex/anthropic/*→ Apex upstreams  │
│                                                  (DeepSeek,     │
│                                                   Kimi,         │
│                                                   CMMD-managed) │
│                                ↑                                │
│   apexAnthropicProxyHandler resolves the upstream key           │
│   dynamically based on the token's model claim.                 │
└─────────────────────────────────────────────────────────────────┘
                │ spawns + holds FLY_IMAGE env
                ▼
┌─────────────────────────────────────────────────────────────────┐
│ forge-cloud-users (fly app, N machines spawned per-user)        │
│   ─ image: registry.fly.io/forge-cloud-master:user-vm-<SHA>     │
│   ─ built from Dockerfile.user-vm (CLI-rich)                    │
│   ─ env from machineEnv() ONLY — no app-level secrets           │
│                                                                 │
│   Per-VM env wired by WorkspaceOrchestrator.machineEnv():       │
│     ├─ ANTHROPIC_API_KEY = forge_proxy_<JWT>                    │
│     ├─ ANTHROPIC_BASE_URL = master.fly.dev/api/.../anthropic    │
│     ├─ OPENAI_API_KEY    = forge_proxy_<JWT>                    │
│     ├─ OPENAI_BASE_URL   = master.fly.dev/api/.../openai/v1     │
│     └─ FORGE_PROXY_HMAC_SECRET = (WS secret, master ws auth)    │
│                                                                 │
│   NO direct env vars for DeepSeek/Kimi/OpenCode/Cursor —        │
│   those CLIs/SDKs read ANTHROPIC_BASE_URL (Claude Code does     │
│   this for Apex models too) and inherit the Anthropic proxy.    │
│                                                                 │
│   forge-entrypoint.sh writes ~/.codex/config.toml when          │
│   OPENAI_BASE_URL is set (Codex CLI ignores OPENAI_BASE_URL     │
│   env var on its own).                                          │
└─────────────────────────────────────────────────────────────────┘

How DeepSeek/Kimi traffic actually flows

This is non-obvious — the same ANTHROPIC_BASE_URL env var serves multiple upstream providers:

User selects an Apex profile (e.g., “Apex Flash” which maps to a DeepSeek upstream)
Forge mints a proxy token with provider: "apex" and models: ["apex-flash"]
Forge tells Claude Code in the user VM to use that token + an Apex-scoped route variant
Claude Code sends an Anthropic-shape request to /api/provider-egress/apex/anthropic/* carrying the token
apexAnthropicProxyHandler verifies the token, reads the model claim, calls resolveApexApiKey() to pick the right upstream key (DeepSeek for apex-flash, Kimi for apex-coder-fast, etc.), rewrites the URL to the upstream’s Anthropic-compatible endpoint, forwards the request
User VM only ever sees the scoped proxy token

This is why master needs DEEPSEEK_API_KEY, KIMI_API_KEY etc. as Fly secrets but the user VM env is clean — same route, different upstreams resolved server-side from the model claim.

Token format

Signed JWT-style token. Structure:

forge_proxy_<base64url(claims)>.<base64url(hmac-sha256 sig)>
claims: {
  userId, provider, aud: "forge-provider-egress",
  iat, exp, jti, vmId?, threadId?, sessionId?, models?
}

Signing secret: FORGE_PPROXY_HMAC_SECRET on master. NEVER forwarded to user VMs.
Default TTL: 30 minutes (PROVIDER_PROXY_MAX_AGE_SECONDS)
Provider scoping: Token’s provider claim must match the route. An anthropic token sent to /openai/* returns 401.
Model scoping (Apex only today): Apex tokens carry a models: ["apex-flash"] claim. The handler uses this to pick the right upstream key.
No revocation list yet — there’s a createProviderProxyRevocationStore factory but no revocation events are persisted. Future hardening.

Why two secrets, not one

FORGE_PROXY_HMAC_SECRET (WS) must be on user VMs so they can sign WS-auth headers when calling master. If the same secret signed egress tokens, a compromised user VM could mint tokens scoped to OTHER users’ IDs and bill those users’ rate limits. Splitting the secrets isolates the trust domains:

WS secret: lives in user VMs, validates VM→master WS upgrades only.
Pproxy secret: lives in master only, signs egress tokens.

The boot guard (in WorkspaceOrchestrator.ts) refuses to start master if FORGE_PPROXY_HMAC_SECRET is unset while an upstream key is present.

The 2026-05-15 incident — what happened

TL;DR: A chain of three independent bugs combined to take prod down. Each one looked locally correct but composed catastrophically. Fix took ~6 PRs over ~10 hours.

Bug #1: Dev-script ordering

scripts/dev-fly-master.sh sourced .env.local AFTER the if [ "$FORGE_LOCAL_DOCKER_MASTER" = "true" ] conditional. Result: variable empty when checked, master image never rebuilt, dev kept running stale pre-egress-proxy code that forwarded raw provider keys to user VMs.

Fix: PR #312 moved .env.local source above the conditional.

Bug #2: Deploy workflow only built one image

.github/workflows/deploy-forge-cloud.yml built one image from Dockerfile and reused it for both apps. PR #310 hardened the master image by removing provider CLIs — correct for master. But user VMs shared the same image, so they also lost their CLIs.

Fix: PR #314/#315 added a second docker/build-push-action step that builds Dockerfile.user-vm and pushes it to registry.fly.io/forge-cloud-master:user-vm-<SHA>.

Bug #3: Fly app-level secrets overlay machineEnv()

Even after the orchestrator stopped forwarding raw keys in machineEnv(), user VMs still had raw keys in env. Fly Machines API overlays app-level secrets (set via flyctl secrets set ... -a forge-cloud-users) into every spawned machine’s env at runtime — AFTER our createMachine returns.

Fix: PR #317 added a deploy-time CI step that calls flyctl secrets list -a forge-cloud-users and fails the deploy if any of ANTHROPIC_API_KEY, OPENAI_API_KEY, DEEPSEEK_API_KEY, KIMI_API_KEY, CURSOR_API_KEY, OPENCODE_API_KEY appear.

Bug #4 (latent): Boot guard silent fallback

The guard fired only when both secrets were missing. In prod, the WS secret was set but the pproxy secret was not. Guard passed, pproxyHmacSecret fell back to "", machineEnv() saw an empty secret and skipped minting a proxy token.

Fix: PR #317 tightened the guard to if (!config.forgePproxyHmacSecret). Now master refuses to boot in proxy mode without an explicit pproxy secret.

Bug #5: Codex CLI ignores OPENAI_BASE_URL

PR #318 wired the OpenAI egress proxy server-side. Tests passed. Post-deploy QA: Codex still 401’d. The CLI was ignoring OPENAI_BASE_URL. It only honors base_url set in a named-provider block in ~/.codex/config.toml.

Verified empirically with a local echo server (15-min experiment). Confirmed schema:

model_provider = "forge"
[model_providers.forge]
name = "Forge proxy"
base_url = "https://forge-cloud-master.fly.dev/api/provider-egress/openai/v1"
env_key = "OPENAI_API_KEY"
wire_api = "responses"

Fix: PR #323 added a block to scripts/forge-entrypoint.sh that writes this config.toml at user-VM startup, gated on OPENAI_BASE_URL being set.

Operational runbook

Verify the proxy is engaged for a given user VM

VM=$(fly machines list -a forge-cloud-users --json | jq -r '.[0].id')
fly ssh console --machine "$VM" -a forge-cloud-users -C "env" \
  | grep -E "^(ANTHROPIC|OPENAI|DEEPSEEK|KIMI|OPENCODE|CURSOR)" | sort

Expect:

ANTHROPIC_API_KEY=forge_proxy_*
OPENAI_API_KEY=forge_proxy_*
ANTHROPIC_BASE_URL=https://forge-cloud-master.fly.dev/api/provider-egress/anthropic
OPENAI_BASE_URL=https://forge-cloud-master.fly.dev/api/provider-egress/openai/v1
No DEEPSEEK_API_KEY, KIMI_API_KEY, OPENCODE_API_KEY, CURSOR_API_KEY

Plus the Codex config:

fly ssh console --machine "$VM" -a forge-cloud-users -C "cat /home/forge/.codex/config.toml"

Force a fresh user VM (after secret rotation or env change)

fly machines list -a forge-cloud-users --json \
  | jq -r '.[].id' \
  | xargs -I{} fly machines destroy {} -a forge-cloud-users --force

Add an upstream provider key on master

fly secrets set ANTHROPIC_API_KEY="sk-ant-..." -a forge-cloud-master

DO NOT set it on forge-cloud-users. The CI assertion in deploy-forge-cloud.yml will fail the next deploy.

Rotate FORGE_PPROXY_HMAC_SECRET

Generate new: openssl rand -hex 32
fly secrets set FORGE_PPROXY_HMAC_SECRET="<new>" -a forge-cloud-master
Destroy all user VMs (their existing tokens were signed with the old secret)
Next user request → fresh VM with fresh token signed by the new secret

Lessons learned

What unit tests can and cannot catch

Caught by unit tests	Required prod-only QA
Token signing/verification	Fly app-level secret overlay
Header swap in proxy handler	Codex CLI’s actual config.toml requirement
`*.localhost` loopback detection	Codex CLI version compatibility
`machineEnv()` doesn’t include raw keys	Provider CLIs actually using OPENAI_BASE_URL env
Boot guard fires on misconfig	Real upstream returning real responses
Deploy workflow YAML shape	Fly Machines spawn race conditions

The pattern that bit us repeatedly: code was unit-correct AND prod-broken simultaneously. The integration boundary (between our orchestrator and Fly, between machineEnv and the real CLI binaries in the user VM) is where the bugs lived.

TDD where you can verify, experiments where you can’t

For PR #323 (config.toml), the order that worked:

Local echo-server experiment first — 15-min prove the config.toml schema actually works with the live Codex CLI binary
Write TDD tests for the entrypoint behavior using the empirically-verified schema
Land + verify in prod with a fresh user VM

For PR #318 (OpenAI server-side proxy), the order that DIDN’T work:

Plan based on web research that said OPENAI_BASE_URL would work
Wrote tests + implementation against that assumption
Deployed
Discovered in prod that Codex ignored the env var
Had to write PR #323 to fix the client-side gap

Cost of skipping the empirical step: one bad deploy, one rollback, one extra PR. Cheap experiments earlier prevent expensive PRs later.

Open follow-ups

Tier 1 (security / coverage gaps)

Cursor egress — never investigated. Needs the same echo-server experiment treatment.
DeepSeek + Kimi + OpenCode turn-level verification — architecture says they’re proxied; haven’t sent a turn through them today specifically.
WS upgrade for Codex /v1/responses — performance only, not security.

Tier 2 (operational)

Token revocation persistence — createProviderProxyRevocationStore exists but no events persisted.
Model scope constraints on Anthropic + OpenAI tokens — currently only Apex tokens carry model claims.
Master HTTP 412 cold-start race — Fly Machines API race; master self-heals via retry.

Tier 3 (process)

“Before shipping a provider proxy” checklist — empirical verification of client CLI’s config requirements
Prod smoke after deploy automated check