Eval WS-2 — Mail-intelligence quality batteries — implementation plan

Status: In progress · Author: Christophe (with Claude) · Date: 2026-07-06 · Repo: north-os · Related: eval-coverage-completion roadmap (2026-07-03, WS-2) · matterbench design (2026-07-02, verifiable-first) · Linear NOS-40

Four live model calls decide what mail surfaces, to which matter, and what stays private — today guarded only against mocked output. WS-2 adds golden-labeled batteries that drive the real production functions with real model calls, scored deterministically (label equality), with hard gates on the two errors that matter most: a leaked private email (safety) and a wrong-matter link (the demo-day cardinal error).

Grounded surface map (verified in code, 2026-07-06)

Surface	Entry point	Model	Purity	Battery
Relevance/privacy gate	`packages/mail-intelligence/src/classify-gate.ts` `triageMailRelevance`	Haiku 4.5	Pure (ctx.anthropic + input)	2a + 2c `eval:mail-gate`
Thin-chip classifier	`classify-thin.ts` `classifyMailThin`	Sonnet 4.6	Pure	2a `eval:mail-classify`
Matter resolver (3-layer + judge)	`resolve-matter.ts` → `resolver/`	Sonnet 4.6 judge + Voyage embed	DB-coupled (5 RLS read layers)	2b `eval:mail-resolve`
`stage1Triage` (obvious-noise)	`stages/stage1-triage.ts`	Haiku 4.5	Pure	Only if a live caller exists (verify); else skip

Explicitly skipped, with reasons:

mail-folder-classifier.ts — named in the roadmap as an LLM decision point, but it is a deterministic Redis/Graph folder-id lookup (received|sent|other), no model call, 7 unit tests. An LLM battery here is vanity coverage.
Drafter / deep-analyzer / Stage-2-full / Stage-3 — P3-deferred stubs that throw (src/scenarios/*.ts). Eval when ported.

Corpus strategy (provenance-honest)

The DB stores classification outputs only (no bodies); the eval org (3faddc1b-8dd2-4b5f-9108-6dccd2c131c6) mailbox is ~87% simulator-fed threads (fictional personas, real deal names, full body_html in apps/simulator/scenarios/*.yaml), the rest genuinely-external vendor noise. Private mail is absent by design (the gate excludes before storage). So:

Simulator beats are the primary corpus: they are the real threads the product processed through Outlook/Graph/worker-mail, and their bodies are recoverable from the YAMLs. Per-case source: "simulator:<scenario>/<beat-id>".
Stored chips are label seeds, never ground truth — they came from the same models under eval. Every label is independently re-derived from the class definitions in the system prompts; if the model output later disagrees with a verified label, the case stays red (potential product/prompt bug). Labels are never softened to pass.
Privacy positives are authored (source: "authored:privacy") — realistic personal mail (medical, family, personal finance…). Unavoidable and documented: real private mail must never exist in a committed dataset, and the product stores none by design.
PII scrub mirrors the captured-chat discipline: retain deal/persona realism (already fictional), neutralize live infra (UPNs, tenant/app ids, share tokens) to [BRACKETED-CAPS].

Batteries and gates

All env-gated EVAL_MAIL=1, with --task <id> / --runs / --gate / --model flags per eval-router.ts conventions, one cost line per run (actual token usage summed via a counting wrapper around the Anthropic client). Datasets in packages/mail-intelligence/eval/, scripts in packages/mail-intelligence/scripts/, wired with the dotenv prefix.

Battery	Dataset	Deterministic score	Hard gate (exit 1)	Report-only
2a/2c `eval:mail-gate`	~50–70 cases across `firm_actionable`/`firm_noise`/`exclude_private`	per-class P/R	`exclude_private` recall = 1.0 (2c, safety); `firm_actionable` recall ≥ 0.9	class precisions (privacy-biased over-exclusion is designed)
2a `eval:mail-classify`	~35–45 cases across the 5 chips	chip accuracy + per-chip recall; `waiting → waiting_on` non-null; `suggested_owner_user_id` null; `firm_shareable` false on internal-sensitive	`needs_action` recall ≥ 0.9 (never-drop bias); chip accuracy ≥ 0.8	confidence calibration
2b `eval:mail-resolve`	~25–30 cases (ports + extends the 13 ground-truth cases in `sample-validate-matter-resolution.ts`, incl. the Brooklyn-Gardens↔Spring-Creek mislink guard, abstain traps, thread-continuity)	resolved matter vs expected (`display_number` or `abstain`), method, link_status	zero false links — any wrong `matter_id` (auto or suggested) fails	link recall (abstain-first is designed; missed links reported)
2b-guards (same script, `--guards`)	judge output pinned via `judgeOverride`	pin-gate, candidate-membership, ambiguity demote, confidence bands, noise guard	all guard assertions	—

The resolver battery runs read-only: ctx.syncDb is omitted so auto-track never writes to the shared dev DB; scoring maps the returned clio id against clio_matter_index by SELECT.

LLM-judged residue (reasoning quality, privacy-leak wording): deferred — deterministic label scoring covers pass/fail; any multi-case Opus-judged loop needs an explicit go (cost rule).

Cost per full run (measured)

gate ~$0.05 (60 Haiku calls) · classify ~$0.15 (37 Sonnet calls) · resolve ~$0.57 (27 Sonnet judge calls + Voyage embeds) · guards sub-battery: $0. Full WS-2 sweep ≈ $0.78.

As built (2026-07-06) — results register

Datasets: gate 63 cases (25 actionable / 21 noise / 17 private, incl. 2 fundraising counterexamples guarding the prompt fix) · classify 37 cases (5 chips, firm_shareable asserted both directions) · resolve 27 cases (16 expect-matter with verified clio_matter_id ground truth / 11 abstain-traps sender-audited against L4 bridging, incl. the Brooklyn-Gardens↔Spring-Creek mislink flagship). Per-case source provenance + why throughout; all persons in authored privacy fixtures are fictional.
Product bug found + fixed: the gate leaked personal solicitations directed at the individual (charity/alumni/political asks) into firm_noise — exclude_private recall 94.1%. Fixed with one class-general clause in relevance-gate-system.ts; re-run recall/precision 100/100, no regression (firm_actionable recall 100%, firm_noise precision improved to 100%).
Battery status (post dual-review hardening): gate PASS (exclude_private R/P 100/94.4, firm_actionable R 100) · classify PASS (chip accuracy 86.5%, needs_action recall 100%, field checks 0 failures) · resolve PASS (0 false links scored on clio_matter_id, 0 skips, 0 link-status mismatches, abstain 11/11, guards 9/9, link recall 0.75 vs 0.6 floor).
Dual review (Opus workflow, 21→12 adversarially-confirmed findings + Codex GPT-5.5, 13 findings): all merged findings fixed — production fallbacks now score as failures (FALLBACK column, unfalsifiable-safety-gate hole closed) · skipped/errored cases fail the resolver gate · link-recall floor kills the all-abstain green · --task prints unexercised gates as n/a (no vacuous safety claims) · scoring by clio_matter_id (the BCHS-102 display-number collision can no longer mask a wrong-client link) · mechanical read-only via default_transaction_read_only=on on the eval pool + startup probe · --model threaded · VOYAGE_API_KEY required · trap senders audited for L4 exclusivity bridges.
Data finding (follow-up, eval org): matter_drive_folder "102 Brooklyn Gardens" (client_root BCHS) pins clio 1771069441, which belongs to Brooklyn Rise Charter School; the BCHS client's own BCHS-102 matter is clio 1771069171. The judge conservatively abstains/demotes on the contradictory evidence — precision-first working as designed on dirty data — costing the 4 link-recall misses. Fix the pin (likely a discovery-script mislink) and recall should recover.
False alarm resolved: the suspected cas-transaction.ts schema drift is not real — the matter-link UPDATE targets pipeline_runs (columns present in schema TS, migrations 0027/0045, and the live DB). No fix needed.
Honest-coverage notes: the thin classifier's noise chip is effectively vestigial (noise is a gate-level decision upstream; recorded, not gated) · stage1Triage has no live production caller → excluded from the batteries.