Eval gate split — implementation plan

Status: In progress · Author: Claude (NOS-216 orchestrator) · Date: 2026-07-23 · Repo: north-os
Related: Linear NOS-216; design spec 2026-07-23-eval-gate-split-design.html

Order of work

Measure the five cheap batteries. Run each locally, record cost, wall time and today's score. This decides which can run on a hosted runner and which need the corpus, and gives the PR gate a known-green starting point.
Battery report contract. scripts/eval-battery-report.ts — BatteryReport/EvalBaseline types plus pure buildBaseline, compareToBaseline, formatRatchet. Pure and DB-free so it unit-tests without a runner.
--report / --only on eval-agent-quality.ts. Per-case verdicts to JSON (a case passes only if it passed every run) and a case filter for the confirmation re-run. No change to how any case is judged.
scripts/eval-ratchet.ts. Check mode and --record mode, with the confirmation re-run and the refusal to lower the bar without --allow-regression --reason. Scripts: eval:ratchet, eval:baseline:record.
Unit tests. eval-battery-report.test.ts + eval-ratchet.test.ts.
Runner setup. deploy/ci-runner/{install.sh,north-os-ci-runner.service,README.md}. Committed and documented, never a manual folk step.
Workflow rewrite. changes / cheap-gate / corpus-gate / eval-gate plus .github/scripts/eval-changed-batteries.sh.
Deploy gate. deploy/preprod/deploy.sh step 4d (ratchet) and 4e (opt-in trajectory smoke), EVAL_CORPUS_DATABASE_URL in env.preprod.example, README runbook.
Record the baseline. One full eval:baseline:record run; commit the JSON.
Gates, PR, CI green, merge.

Files

Path	Change
`.github/workflows/eval-gate.yml`	rewritten — four jobs (shared edit point: keep additions on their own lines, rebase before final gates)
`.github/scripts/eval-changed-batteries.sh`	new — per-battery path filter
`deploy/ci-runner/install.sh`	new — idempotent runner install
`deploy/ci-runner/north-os-ci-runner.service`	new — systemd user unit template
`deploy/ci-runner/README.md`	new — runbook, security posture, uninstall
`deploy/preprod/deploy.sh`	steps 4d + 4e before the fleet restart
`deploy/preprod/env.preprod.example`	`EVAL_CORPUS_DATABASE_URL`
`deploy/preprod/README.md`	gate documentation
`packages/agent-runtime/scripts/eval-battery-report.ts`	new — report/baseline contract + pure comparison
`packages/agent-runtime/scripts/eval-ratchet.ts`	new — the gate runner
`packages/agent-runtime/scripts/eval-agent-quality.ts`	`--report`, `--only`
`packages/agent-runtime/eval/baselines/agent-quality.json`	new — the committed contract
`package.json` (root)	`eval:gate:cheap` extracted, `eval:gate` composed from it, `eval:ratchet` (shared edit point)

Verification

bun x vitest run scripts/eval-battery-report.test.ts scripts/eval-ratchet.test.ts — the ratchet's decision logic.
bash -n on all three shell files; the filter script exercised against a real polaris-only commit (must select nothing) and against this branch (must select everything).
The five cheap batteries run locally, scores in the PR body.
eval:ratchet run end-to-end against the freshly recorded baseline — must be green, and must be green while the battery's own absolute gate is red.
bun run lint, bun run typecheck, bun --filter=@workspace/agent-runtime test.
The workflow proves itself on its own PR run.

Risks

Runner registration needs repo admin. Minting the token is the one step the orchestrator cannot perform; it is handed to the human with the exact command. Until the runner is online and EVAL_SELF_HOSTED_RUNNER=true, the corpus batteries skip loudly — never silently, never hanging.
Judge flicker reddening a deploy. Mitigated by the confirmation re-run; a case must fail twice.
Deploy time and cost. The ratchet adds ~20 min and ~$15–30 per preprod deploy. Preprod deploys are morning-only and infrequent; EVAL_GATE_SKIP=1 exists for incidents.
Shared edit point. eval-gate.yml and the root package.json are also being edited by the parallel Mail-RAG / learning-loop factory. Rebase on origin/main before the final gates and re-run them.