Codex as a function in the incident loop

YAML that shells out to codex exec is one way to run Codex in CI. A product that needs parallel hypotheses, typed output, and a hard merge boundary needs the agent to be a function in a larger loop.

Triage Copilot is that loop. A storefront emits an ErrorEvent. A heartbeat claims it. Three named Codex sub-agents investigate in parallel. Zod rejects malformed output. Losing traces stay on disk. The winner becomes a draft PR. Humans merge.

The SDK is for when Codex is a step, not the pipeline.

The incident loop

The public shop produces realistic storefront errors. The private triage app owns Clerk, Prisma, Codex orchestration, MCP servers, and GitHub. Shared domain contracts live in a workspace package. The two apps do not share auth, database, or the Codex client.

The shared unit of automation is the heartbeat. Cron, a Trigger now button, and fixable injected-bug automation all call the same function. Locally that is a watcher. Hosted it is a Vercel Cron route with a bearer secret.

Three hypotheses, one event

One ErrorEvent is not one agent run. The orchestrator (apps/triage/src/triage/orchestrator.ts) starts three threads with fixed roles:

The role names are UI labels, database values, prompt keys, and grader inputs. They are not free-form personas.

Each thread gets the same sandbox and the same two MCP servers. error-stream is for persisted ErrorEvents. repo-search is for read-only repo context. Network and web search are off. Approval is never inside workspace-write.

const thread = codex.startThread({
  model,
  workingDirectory: path.resolve(workingDirectory),
  sandboxMode: "workspace-write",
  approvalPolicy: "never",
  networkAccessEnabled: false,
  webSearchEnabled: false,
  modelReasoningEffort: "high",
});

const turn = await thread.run(prompt, { outputSchema });

The Codex client does not take an API key in this path. The spawned codex CLI uses local Codex auth. MCP servers are attached on the Codex config, not invented inside the prompt.

Parallelism is ordinary Promise.all over the three roles. Fan-out is three concurrent Cloud attempts, labeled in metadata as codex cloud exec --attempts 3.

Structured output is a gate, not a nicety

Each sub-agent must return JSON that satisfies apps/triage/src/triage/schemas.ts. A complete investigation has a root cause, a failing test, a unified-diff patch, a risk level, and a confidence score. A failed investigation has a failure reason and missing-evidence list. There is no third shape.

Malformed output never reaches Prisma or GitHub. It becomes a failed SubAgentRun column, not a blank UI. Losing hypotheses stay persisted. They are evidence.

The live heartbeat calls gradeInvestigations() when exactly three sub-agent runs exist and OPENAI_API_KEY is available. The verdict can name a completed run by ID or role; a missing key, grader error, or unknown or failed winner falls back to selectHeartbeatWinner(), which prefers Recent change hunter and then the highest confidence score. Dispatch and promotion stay deterministic when the model grader is unavailable.

Codex never merges

The GitHub boundary is apps/triage/src/triage/github.ts. Codex produces reviewable patch text. The app validates paths against the workspace, refuses protected source branches, and opens a draft PR.

The GitHub App is scoped to pull-requests: write. It cannot push to main, merge, or update protected branches. Runtime GH_APP_* credentials are never used for developer pushes.

If GitHub App env is valid, Octokit creates a head branch through Git Data APIs and opens the draft. If it is not, a fallback uses a temp clone, applies the patch, and runs gh pr create --draft. Either path records opened or pending. Humans merge or reject.

What the SDK is for here

The companion post on how codex-action sandboxes Codex in GitHub Actions covers runner security: drop-sudo, the responses-api proxy, and treating the action as the last step in a job. This post is the other half. The product loop owns dispatch, schema, persistence, and the merge boundary. The GitHub Action owns how a runner talks to Codex. They are not substitutes.

A safe automation shape

The sequence in Triage Copilot is narrow and reversible:

1. Persist the ErrorEvent.
2. Claim it in the heartbeat.
3. Start three threads with explicit roles, sandbox, and MCP servers.
4. Parse each final response through Zod.
5. Persist every SubAgentRun, including failures and losers.
6. Select a winner.
7. Open a draft PR, or record why it stayed pending.
8. Stop. A human reviews.

The sub-agent prompt forbids GitHub calls, branch pushes, merges, and claiming the patch was applied. Candidate patches must be a strict unified diff. The agent writes text. The app applies policy.

Takeaways