Eval agent-quality grounding fix — design spec

Status: In progress
Author: OpenAI Codex
Date: 2026-07-20
Repo: north-os
Related: NOS-183; docs/reviews/2026-07-20-eval-agent-quality-harness-bug.md

Problem

The original eval:agent-quality gate was 0/8. Defect 1 now preserves structured search_knowledge provenance and moved main to 2/8, but capToolResultWithCoverage still drops every document-read artifact after deriving metadata.documentReadCoverage. readTranscript therefore receives no checker-consumable artifact for read_knowledge_document, and sourceBearingRead must reject otherwise genuine deep reads.

A second, independent retrieval-quality failure may remain after plumbing is fixed: a named-instrument query can rank a same-topic Environmental Indemnity Agreement above the HFA Bond Mortgage itself.

Goals and invariants

• Preserve validated touch-coverage artifacts (search and email_thread) in graph state and checkpoints after result capping, including the exact meta.resultIds array.
• Retain a durable, minimal document-read artifact projection in graph state: only the identity, title, total length, read mode when semantically required, and source ranges consumed by sourceBearingRead.
• Every model-facing ToolMessage omits its artifact field; the stored message retains it as evaluator/runtime metadata, not prompt content.
• Make readTranscript(state).results[n].artifact expose the real structured search or document-read provenance emitted by the tool-node path.
• Keep structuredSearchDocuments, sourceBearingRead, duplicate-title checks, deep-read completeness checks, and provenance checks unchanged.
• Use the shipped Defect 1 result, 2/8, as the baseline, then measure the metered gate once after the document-read projection change. A confirmatory run is allowed only if the first post-change run is inconclusive.

Non-goals

• No checker relaxation, text-derived fake IDs, fixture-only fallback, or special casing of the HFA case.
• No redesign of coverage-ledger persistence or the context cleaner.
• No broad retrieval rewrite. Defect 2 proceeds only if the post-fix measurement and a targeted retrieval trace identify a small ranking seam with deterministic regression coverage.

Root-cause decision

Alternatives rejected

Document-read projection

The checkpoint artifact is derived only from the validated DocumentCoverageEvent already emitted by the real tool. Its common shape is resourceType, resourceKey, title, and totalChars. For read_knowledge_document, the checker compares resourceKey to the call's document_id and the title to prior structured search provenance.

• Direct read: add only bodyServedRange, using the existing cap-aware DocumentReadCoverageProjection.servedRange so the checkpoint never claims bytes removed by result capping.
• Deep read: add coverageMode: "deep-read" and readEvidence entries containing only merged bodyServedRange values. Keeping the mode preserves the unchanged incomplete-deep-read guard; keeping the ranges preserves whole-document coverage checks.
• Deliberately omitted: sourceVersion, manifests, markup IDs, content maps, revision-ledger evidence, and full document text. Those fields are not parsed by sourceBearingRead; omitting them bounds checkpoint growth.

The projection survives returned state, checkpoint serialization, and readTranscript. modelViewWithoutToolArtifacts continues to clone artifact-bearing tool messages without the artifact before model invocation.

Data flow

searchKnowledgeTool / readKnowledgeDocumentTool
  → coverageToolResult(text, validated coverage event)
  → tool.invoke(call): ToolMessage { content, artifact }
  → capToolResultWithCoverage:
      search/email → capped ToolMessage { content, validated artifact }
      document     → capped ToolMessage { content, minimal artifact, metadata? }
  → AgentState/checkpoint
      ├─→ readTranscript → checker-consumable structured provenance
      └─→ model-facing projection → ToolMessage without artifact → chat model

The model projection retains content, tool identity, status, provider metadata, and document-read projection metadata. Only artifact is omitted.

Regression coverage

• A legal-assistant graph test emits a search-shaped touch-coverage event with known IDs and asserts the complete touch artifact after capping in returned graph state and after a checkpoint persist/reload round trip.
• The same graph test passes the returned state through readTranscript, asserts exact artifact equality including ordered meta.resultIds, and asserts the second model call receives no artifact field.
• The test caps/rebuilds through the real tool node, proving the failure seam rather than only testing a manually constructed ToolMessage.
• A document-read graph regression performs structured search followed by a deep read_knowledge_document, reloads the checkpoint, passes it through readTranscript, and proves the unchanged grounding contract accepts the minimal source-bearing artifact.
• The document regression also asserts exact minimal shape and verifies the model-facing read result has no artifact.
• Focused verification: graph test, agent-runtime typecheck, agent-runtime lint, then one post-fix metered agent-quality run.

Defect 2 decision gate

After Defect 1, record which cases pass and inspect the ranked candidates from the first HFA search_knowledge call. Defect 2 remains present when the named HFA Bond Mortgage does not rank ahead of same-topic candidates from other instruments, whether it is absent or merely lower-ranked. Treat the existing lol-clause-body case as a live golden reference, not a deterministic test. Implement only if a trace identifies one narrow title, fused-pool, best-extent, or reranker seam and a new deterministic fixture reproduces the ordering failure before code changes. Otherwise, defer Defect 2 as a separate retrieval-quality change and preserve the failing query, wrong and intended titles, observed candidate ranks (including explicit absent markers), stage to probe, and acceptance criterion (“the named instrument's own source ranks ahead of same-topic clauses from other instruments”).

Risks

• Artifact leakage to the model: prevented by a dedicated model-view projection and locked by the graph test.
• Other ToolMessage fields lost at the model boundary: the model-facing clone retains every field present on the capped state message except artifact; the regression asserts model content remains intact.
• False green: the gate guards remain byte-for-byte unchanged; pass-rate movement must come from structured provenance reaching the checker.
• Metered variance: report exact before/after counts and avoid repeated runs.

Acceptance criteria

1. Graph-produced search results retain complete validated touch artifacts, and graph-produced document reads retain only the minimal checker-consumable projection, after capping in returned state, after checkpoint persist/reload, and in readTranscript.
2. Direct-read ranges are cap-aware; deep-read projections retain coverageMode plus source ranges sufficient for the unchanged completeness guard. No full document text or unused document metadata enters the artifact.
3. Every model-facing ToolMessage omits the artifact field.
4. The focused regression test, agent-runtime typecheck, and agent-runtime lint pass.
5. The post-fix gate is measured and compared with the 2/8 Defect 1 baseline.
6. No line in packages/agent-runtime/scripts/eval-agent-quality.ts changes. Any independently proven checker defect requires a separate scoped change and may not weaken duplicate-title, deep-read, or provenance guards.