Eval agent-quality grounding fix — implementation plan

Status: In progress
Author: OpenAI Codex
Date: 2026-07-20
Repo: north-os
Related: design spec; NOS-183

Goal: preserve real search and document-read provenance through the legal-assistant graph without exposing coverage artifacts to the model, then re-measure the unchanged agent-quality gate.
Architecture: graph state preserves validated search/email touch artifacts and a minimal document-read artifact containing only checker-required identity and range provenance. The model receives a sanitized projection with no artifact field.
Tech stack: Bun, TypeScript, LangChain ToolMessage, LangGraph, Vitest.

Output structure

packages/agent-runtime/src/agents/legal-assistant/graph.ts — preserve validated touch artifacts and minimal document-read artifacts through capping; omit all artifacts at the model boundary.
packages/agent-runtime/src/agents/legal-assistant/graph.test.ts — lock returned-state, checkpoint, transcript, and model-projection behavior through the real graph.
docs/superpowers/specs/2026-07-20-eval-agent-quality-grounding-design.html and this plan — decision record, measured outcome, and Defect 2 disposition.
docs/superpowers/STATUS.md and docs/superpowers/index.html — generated only by status.ts/build-site.ts.

Implementation tasks

Tasks 0–5 record shipped Defect 1. Tasks 6–8 extend the same state/model-view split to document reads without changing the checker.

Task 0 — preflight status and baseline

Before production-code changes, run bun docs/superpowers/status.ts set eval-agent-quality-grounding in-progress, bun docs/superpowers/status.ts linear eval-agent-quality-grounding NOS-183, and bun docs/superpowers/build-site.ts.
Run the metered gate once through bunx dotenv with the repository .env.local and .env. Record passed cases, failed cases, aggregate rate, and non-fatal infrastructure errors in this plan's Outcome section.

EVAL_AGENT=1 bunx dotenv \
  -e /home/kwiss/workspace/north-os/.env.local \
  -e /home/kwiss/workspace/north-os/.env -- \
  bun --filter=@workspace/agent-runtime run eval:agent-quality

Observed baseline: 0/8 (0.0%) on 2026-07-20; every case failed. Non-fatal product_event insert errors were logged and did not abort the run.

Task 1 — establish the red regression

Test: packages/agent-runtime/src/agents/legal-assistant/graph.test.ts

Replace the existing “strips coverage artifacts” expectation with the split contract. Emit a real content_and_artifact search-shaped result carrying ordered resultIds: ["result_2", "result_1"].
Assert exact touch-artifact preservation in returned graph state, a MemorySaver checkpoint reload, and readTranscript(result). Assert the message captured in the second model call retains its excerpt content but has no artifact.
From packages/agent-runtime, run bunx vitest run src/agents/legal-assistant/graph.test.ts. Expected: FAIL because the returned-state, checkpoint, and transcript artifacts are undefined; the model-side no-artifact assertion remains green.

Task 2 — preserve state, sanitize model input

Modify: packages/agent-runtime/src/agents/legal-assistant/graph.ts

In capToolResultWithCoverage, retain the validated artifact only for non-document touch events (search/email_thread). Keep the existing minimal document-read metadata projection and document-artifact stripping unchanged.
Add a model-boundary projection that returns non-tool and artifact-free messages unchanged and rebuilds artifact-bearing ToolMessages with every field present on the capped message except artifact.
Pass only the projected assembled.messages to modelWithTools.invoke. Do not mutate AgentState or checkpoint messages.
Re-run bunx vitest run src/agents/legal-assistant/graph.test.ts from packages/agent-runtime. Expected: PASS.

Task 3 — package verification and post-fix measurement

Run bun --filter=@workspace/agent-runtime typecheck.
Run bun --filter=@workspace/agent-runtime lint.
Confirm packages/agent-runtime/scripts/eval-agent-quality.ts has no diff.
Run the metered gate once after the focused/static checks. Record the exact before/after delta in the Outcome section; do not loop.

Task 4 — Defect 2 disposition

Inspect ranked titles in the first post-fix HFA search_knowledge call. Treat lol-clause-body as a live golden reference, not a deterministic test.
This one-fix plan does not authorize retrieval code changes without a new deterministic ordering fixture that fails first. If no single existing ranking seam is both proven and narrowly testable, defer Defect 2 and do not spend the reserved third metered run.
If deferred, record in the Outcome section: failing query; wrong top title; intended title; observed rank or explicit absence at title leg, fused pool, best-extent recovery, and reranker where available; next stage to instrument; and acceptance criterion.

Scoped follow-up if deferred: trace the HFA source across title search, RRF, best-extent recovery, and Voyage rerank; add a deterministic named-instrument ordering fixture; then introduce title/entity-aware scoring only at the stage proven responsible. Acceptance: a named instrument's own source ranks ahead of same-topic clauses in other instruments without query-specific literals or checker changes.

Task 5 — outcome record, adversarial review, and delivery

Add an Outcome section to this plan with baseline/post-fix counts, exact delta, checker-file unchanged confirmation, and the Defect 2 implemented/deferred decision plus required handoff evidence.
Review the diff against the spec: exact artifact preservation, no model leakage, no checker edits, no text-parsed provenance, and no unplanned retrieval work. Fix high-confidence findings and rerun affected checks.
Keep spec and plan status in-progress while the change is in an open PR; the merge owner flips them to implemented when the feature lands on main.
Stage each changed path by name. Commit as fix(agent-runtime): preserve search provenance in transcripts with Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>.
Push fix/eval-agent-quality-grounding and open a PR referencing NOS-183. Include before/after gate counts and an explicit “checker guards unchanged” statement.

Task 6 — establish the document-read red regression

Test: packages/agent-runtime/src/agents/legal-assistant/graph.test.ts

Change “checkpoints a minimal validated <resourceType> projection” to expect a direct-read artifact containing exactly resourceType, resourceKey, title, totalChars, and the cap-aware bodyServedRange. Keep the existing metadata assertion.
Add a real graph regression that performs a structured search_knowledge followed by a deep read_knowledge_document. Reload MemorySaver state, call readTranscript(checkpoint.values), and assert the read result carries only common identity fields, coverageMode: "deep-read", and merged readEvidence[].bodyServedRange entries.
Pass that transcript to the exported unchanged validateGroundingToolContract and expect no failure, proving its internal sourceBearingRead accepts the projection. Assert the model call receives the read text but no artifact.
Run bunx vitest run src/agents/legal-assistant/graph.test.ts -t "document-read|minimal validated" from packages/agent-runtime. Expected: FAIL because document artifacts are currently undefined in state and transcript.

Task 7 — retain the minimal document-read state artifact

Modify: packages/agent-runtime/src/agents/legal-assistant/graph.ts

Define a private document-read artifact projection. Common fields are resourceType, resourceKey, title, and totalChars. A direct projection adds only the existing cap-aware servedRange as bodyServedRange. A deep projection adds only coverageMode: "deep-read" and merged source bodyServedRange entries.
Construct the artifact only from the structurally validated DocumentCoverageEvent. Do not copy source version, manifests, markup IDs, content maps, revision-ledger evidence, or document text.
Attach the minimal projection to the capped state ToolMessage. Leave modelViewWithoutToolArtifacts unchanged so every model-bound message still omits its artifact.
Re-run the focused red tests. Expected: PASS.

Task 8 — verify, measure, diagnose, and deliver the follow-up

Run the agent-runtime graph/package tests and record the exact pass count, then run agent-runtime typecheck and lint.
Confirm packages/agent-runtime/scripts/eval-agent-quality.ts has no diff, then run the metered gate once against the root environment. Compare against the shipped 2/8 baseline.
Diagnose each remaining failure as document-read provenance, retrieval precision, genuine absence, or another observed path. Do not alter retrieval or checker behavior in this cut.
Append a fresh Outcome section with case-by-case results and remaining-gap evidence, keep status in progress unless the gate is fully green, and regenerate the docs site with bun docs/superpowers/build-site.ts.
Stage named paths, commit with the required Claude Opus 4.8 attribution trailer, push fix/eval-agent-quality-docread-projection, and open a NOS-183 PR.

Verification matrix

Behavior: returned graph state, checkpoint reload, and readTranscript preserve complete validated search touch artifacts and minimal source-bearing document-read artifacts; model-facing messages omit every artifact.
Regression: focused graph tests fail before and pass after through the real cap/tool-node path; the unchanged grounding contract accepts the checkpoint-reloaded deep-read projection.
Static: agent-runtime typecheck and lint pass.
End to end: the post-fix agent-quality run and one confirmatory run report exact deltas from the shipped 2/8 Defect 1 baseline.
Anti-cheat: no diff in eval-agent-quality.ts; duplicate-title, deep-read, and provenance guards are unchanged.

Outcome — 2026-07-20

Before: 0/8 (0.0%). All eight cases failed before production-code changes.
After Defect 1: 2/8 (25.0%), a +2-case / +25-point improvement. grounded-lol-bond-mortgage passed with seven supported, cited claims; bait-default-notice-days passed with nine supported claims.
Scope and remaining document-read gap: this cut fixes structured provenance only for search-excerpt grounding, which is why 2/8 cases now pass. The remaining cases ground through read_knowledge_document. Document coverage artifacts from that tool are deliberately still omitted from checkpoint state by this cut, so those transcripts cannot satisfy sourceBearingRead. Closing these failures requires extending the state/model-view split with a durable, checker-consumable document-read projection while continuing to omit artifacts from model-facing messages.
Defect 2: deferred. The named HFA query passed after Defect 1, so the previously observed wrong-document result (“Kingsbrook Estates – Environmental Indemnity Agreement” instead of the Kingsbrook HFA Bond Mortgage §8.22) did not reproduce in the post-fix full graph run. The gate does not emit title-leg, fused-pool, recovery, or reranker ranks, so those stages are recorded as not observed. If the miss recurs, instrument those four stages for the exact agent query and add a deterministic ordering fixture before changing ranking. No third metered run was spent.
Anti-cheat: packages/agent-runtime/scripts/eval-agent-quality.ts was not edited. Duplicate-title, incomplete-deep-read, source-bearing-read, and provenance guards were not weakened.
Verification: focused graph suite 109/109; agent-runtime typecheck passed; agent-runtime lint exited 0 with pre-existing warnings.

Outcome — 2026-07-20 document-read projection follow-up

Baseline: shipped Defect 1 measured 2/8 (25.0%): grounded-lol-bond-mortgage and bait-default-notice-days passed.
After: the first post-change run measured 2/8 (25.0%); the bounded confirmatory run measured 1/8 (12.5%). grounded-board-resolution, previously structurally unable to pass, passed both post-change runs. Aggregate movement was masked by independent answer, retrieval, routing, and judge variance in cases outside this projection seam.
Target-path result: grounded-board-resolution moved from fail to stable pass. grounded-promissory-note now gets past sourceBearingRead in both runs—the checker emitted its later whole-document-completion failure, not the source-bearing-coverage failure—but the complete-read predicate still failed.
Anti-cheat: packages/agent-runtime/scripts/eval-agent-quality.ts has no diff. sourceBearingRead, structuredSearchDocuments, duplicate-title, incomplete-deep-read, whole-document, and provenance guards are unchanged.
Verification: focused graph suite 110/110; the red subset failed 4/4 before implementation and passed 4/4 after; the expanded projection subset passed 11/11; agent-runtime typecheck passed; agent-runtime lint exited 0 with 165 pre-existing warnings. Both metered runs logged non-fatal product_event insert errors.

Case	Defect 1 baseline	Post-change	Confirm	Diagnosis
`grounded-lol-bond-mortgage`	PASS	FAIL	FAIL	Different path, not document provenance. The first run rejected tool narration; the confirm found no admissible search evidence. This previously passing search-only case exposes answer/retrieval variance and remains Defect 2 territory.
`grounded-promissory-note`	FAIL	FAIL	FAIL	The read artifact is now source-bearing. The later whole-document predicate remained false in both runs, so the observed gap is incomplete range/required-anchor/title completion within the genuine deep read, not missing checkpoint provenance.
`grounded-board-resolution`	FAIL	PASS	PASS	Fixed by this cut. Structured search followed by the retained deep-read projection produced supported, cited claims in both runs.
`grounded-regulatory-agreement`	FAIL	FAIL	FAIL	Search ran but produced no admissible grounding evidence. The gate output cannot distinguish genuine corpus absence from retrieval precision/empty-result behavior; no document read reached this projection path.
`bait-section-810`	FAIL	FAIL	FAIL	`search_knowledge` ran, but its result did not satisfy unchanged structured-search provenance. This is a search-result path; the bounded output does not distinguish a zero-result response from another structured-result mismatch.
`bait-dollar-cap`	FAIL	FAIL	FAIL	The agent routed through `search_matters` and, on confirm, unauthorized `search_matter_documents` calls instead of beginning with `search_knowledge`. This is tool routing/auth context, not document-read provenance.
`bait-liquidated-damages`	FAIL	FAIL	FAIL	The first run used matter tools without `search_knowledge`; the confirm reached `search_knowledge` only after an unauthorized matter-document call and still lacked structured search provenance. This is routing plus search-result quality, not the read projection.
`bait-default-notice-days`	PASS	PASS	FAIL	Structured provenance remained admissible. The confirm instead rejected a specific unsupported “immediate Event of Default” claim, showing answer/judge variance outside the projection seam.