Eval gate split — cheap batteries per PR, agent-quality ratchet at the deploy — design spec

Problem (verified 2026-07-23)

The job that runs on pull_request executed only eval:router, behind a paths filter scoped to router files. The full eval:gate chain was workflow_dispatch-only and self-hosted, and gh api repos/north-os/north-os/actions/runners returns total_count: 0 — no runner is registered, so that job would sit unassigned even if dispatched. Retrieval, memory-calibration and memory-recall therefore gated nothing. Every "eval:gate green" claim in the wave-1 PRs came from an orchestrator running the chain by hand in its worktree: good discipline, zero enforcement. NOS-209 rewrote the vector-retrieval SQL under RLS and nothing in CI protected it from silent regression.

Two facts decide the whole design. First, the five "cheap" batteries are not uniform — they split cleanly by what infrastructure they can physically reach:

Correction to the batch premise: eval:memory-calibration was assumed to be DB-backed. It is not. Its gate mode imports classifyMemoryDisposition directly, deliberately avoiding the memory index (which would pull in @workspace/db); the DB-touching discard probe is opt-in behind --probe and is not in the chain. That moves a whole battery from "can never gate a PR" to "gates every PR for free".

Second, eval:retrieval and eval:memory-recall read the live ingested corpus and seeded memories that exist only in DEFAULT_ORG on the dev box's Postgres. There is no fixture and no dump that recreates it — it is real ingested HSE documents with real embeddings. A GitHub-hosted runner cannot run them at all. This is an infrastructure problem, not a workflow-wiring one, and no amount of YAML fixes it.

Third, eval:agent-quality is a different animal: full agent turns on Opus plus an Opus judge per case, ~$15–30 and ~20 minutes per run — and it is red on main at its NOS-183 level. An absolute 90% gate that is already red gates nothing: it is either permanently ignored or permanently blocking.

Design

1 · Register the dev box as a self-hosted runner

The unlock is infrastructure. deploy/ci-runner/ commits the whole setup — install.sh (download, configure with --replace, install a systemd user unit, enable linger), the unit file, and a runbook — so the runner is reproducible rather than a folk memory. Labels: self-hosted,linux,x64,eval-corpus,north-dev-box.

A user unit rather than the runner's own svc.sh: no root-owned service files, correct HOME/PATH for the box's pinned bun, and no chance of entangling with the system north-os-preprod@ fleet units. The runner runs as kwiss, never root. The single privileged step is sudo loginctl enable-linger.

Security posture: a self-hosted runner executes PR code. north-os/north-os is private, so only collaborators can open a PR — the fork-PR attack that makes self-hosted runners dangerous on public repos does not apply. Standing rule: never make this repo public while a runner is registered. The runner is also forbidden from starting, stopping or restarting the shared Postgres (5440) or Redis (6390); nothing in the job or the unit references them.

2 · Four jobs, split by what can physically run where

• changes — always runs, ~10s, no secrets. Decides which batteries the diff can affect.
• cheap-gate — ubuntu-latest. eval:coverage, eval:memory-calibration, eval:router, each step-gated on its own path filter.
• corpus-gate — [self-hosted, eval-corpus]. eval:retrieval, eval:memory-recall.
• eval-gate — always runs, aggregates, and is the only job that belongs in the "Protect main" ruleset as a required check.

3 · Cost scoping without the required-check trap

A PR that only touches apps/polaris (Swift), apps/polaris-win, docs/ or a marketing surface must burn zero eval spend. The obvious mechanism — a workflow-level paths: filter — is wrong twice over: it is workflow-wide (one filter for five batteries with different inputs), and a skipped path-filtered job never reports a conclusion, so if it were a required check the PR would block forever on a check that will never run.

Instead, .github/scripts/eval-changed-batteries.sh diffs against the PR merge-base and emits one boolean per battery plus two job rollups; jobs and steps consume those. The always-running eval-gate job reports on their behalf: skipped ⇒ success, failure|cancelled ⇒ failure. Exactly one required check, and it always reports.

Hand-rolled rather than a marketplace paths-filter action: this is a confidential repo with a self-hosted runner in the matrix, and the supply-chain surface of a third-party action is not worth twenty lines of grep. Any change to the gate itself (the workflow, the filter, bun.lock, the root package.json, or the shared eval helpers) selects every battery, so the gate cannot be weakened without running under its own new rules.

4 · The runner-not-registered trap

A job whose runs-on label has no online runner does not fail — it queues, for up to 24 hours, holding the PR open. So corpus-gate is additionally gated on the repo variable EVAL_SELF_HOSTED_RUNNER. Until it is true, the corpus batteries skip cleanly instead of hanging every PR. When they are skipped and the diff touched retrieval/memory paths, eval-gate emits a warning annotation and a summary line saying so out loud — green must never be mistaken for covered.

5 · agent-quality as a deploy gate, ratcheted

eval:agent-quality gates the preprod deploy, inside deploy/preprod/deploy.sh (step 4d) — the existing enforcing mechanism, not the deploy-preprod skill, which is instruction rather than enforcement. It sits immediately before systemctl restart: a failure means the fleet was never touched.

It runs as a ratchet. eval-ratchet.ts runs the battery with its absolute gate neutralised (--gate 0), compares the per-case verdicts against a committed baseline, and fails only when a case that passed at the baseline now fails. Consequences: the gate is ON today at the NOS-183 level, the score can only go up, and fixing NOS-183 stays separate, unblocked work.

• Per-case, not aggregate. An aggregate rate lets a fixed case pay for a broken one. The baseline records the passing set by name.
• Missing cases block. Deleting or renaming a case must not be a silent way to move the bar.
• Confirmation re-run. LLM judges flicker. A suspected regression is re-run (--only the suspects) and only fails the deploy if it fails twice — the same confirmation discipline the battery already applies to individual claims.
• Re-recording is deliberate and obvious. A separate script (eval:baseline:record), a mandatory --reason, refusal to lower the bar without --allow-regression, and a committed JSON file whose diff a reviewer reads. A gate run can never rewrite its own baseline.

The confirmation re-run is load-bearing, not theoretical — measured on the very first end-to-end gate run. The baseline recorded 4/8. The verification run also scored 4/8, but a different four: grounded-regulatory-agreement (a baseline pass) failed, and bait-default-notice-days (a baseline failure) passed. The ratchet re-ran the single suspect, it passed with "25 claims, all supported, cited", and the gate went green at 5/8. Without that re-run the first real deploy would have gone red on judge flicker alone — the fastest possible way to teach everyone to set EVAL_GATE_SKIP=1.

The honest reading: this battery flickers by about ±1 case. The confirmation squares the odds of a spurious red, which is the right trade at ~$20 per extra single-case run. It also argues against reflexively re-recording every improvement: bait-default-notice-days has now been seen both passing and failing, so promoting it into the defended set would import that flicker into the gate. Lock in wins that hold, not wins that happened.

Which corpus: the ratchet runs against EVAL_CORPUS_DATABASE_URL — the dev app database, not preprod's. The golden cases assert over the live closing-binder corpus that lives there and the baseline was recorded against it; preprod's DB is a manually-refreshed clone (refresh-db.sh) that drifts, and comparing a score to a baseline measured on a different corpus is meaningless. Exported before the call, because dotenv never overrides an already-exported variable (verified). The work is read-only: an in-memory checkpointer, read-only tools.

If EVAL_CORPUS_DATABASE_URL is absent the deploy stops — a gate that cannot run must be red, which is the whole lesson of this issue. EVAL_GATE_SKIP=1 is the documented emergency override, because a deploy path with no escape hatch is a hostage situation during an incident.

6 · Trajectories: considered, opt-in

eval:trajectories --smoke is wired at the same point behind PREPROD_EVAL_TRAJECTORIES=1, off by default. Three reasons it is not automatic: cost ($1–5 per agent run × 9 scenarios, plus a Sonnet simulator and an Opus judge per case — potentially more than the rest of the deploy combined); it mutates matter/mail fixture state rather than reading; and it already carries its own graduation history (eval/trajectories), which is the honest home for its trend — a second ratchet on top would fight it.

Alternatives rejected

• Build a corpus fixture so the DB-backed batteries run on hosted CI. The corpus is real ingested HSE documents with real embeddings; a fixture that preserved retrieval behaviour would be a project in itself, and a confidential one to store. Registering the box that already has it is cheaper and more honest.
• Enable corpus-gate on pull_request without the variable guard. With no runner, every PR queues for 24h. The guard is what makes "not registered yet" a skip instead of an outage.
• Make cheap-gate/corpus-gate the required checks directly. Path-filtered jobs that skip never report; every unrelated PR would block forever. Hence the always-running aggregator.
• Workflow-level paths: filter (the status quo). One filter for five batteries with different inputs, and it skips the whole workflow — the required-check trap again.
• Absolute threshold for agent-quality (status quo, 0.9). Already red; gates nothing. Would have to wait on NOS-183, which is exactly the coupling the ratchet removes.
• Aggregate-rate ratchet instead of per-case. Lets a newly-fixed case pay for a newly-broken one, and hides which behaviour changed.
• Run agent-quality per PR. ~$15–30 and ~20 minutes per PR. It is the one battery whose value does not justify per-PR cost; once per deploy is the right cadence.
• Ratchet against the preprod DB. Stale clone; a corpus refresh would read as an agent regression.
• Put the gate in the deploy-preprod skill. A skill is instruction, not enforcement — the precise failure this issue exists to fix.

Gates

• All five cheap batteries green locally before the split lands (measured above).
• eval-ratchet unit tests: regression detection, missing-case detection, improvement reporting, red-battery-still-green-gate, confirmation-rerun merge, arg parsing.
• The workflow proves itself on its own PR: eval-gate green, and changes selecting every battery (the PR edits gate-defining paths).
• A committed eval/baselines/agent-quality.json recorded from a real run, with its reason.
• bun run lint, bun run typecheck, @workspace/agent-runtime tests.

Follow-ups (named, not silently dropped)

• Register the runner + gh variable set EVAL_SELF_HOSTED_RUNNER --body true, then add eval-gate to the "Protect main" ruleset as a required check. Until that check is required, the gate reports but does not block.
• NOS-183 remains the real fix. Each case it recovers should be locked in with a deliberate eval:baseline:record.
• Adding a second battery to the ratchet is a BATTERIES entry plus a --report flag on that battery.