Artifact figure provenance - implementation plan

Status: Implemented - runtime gate, shared eval helper, regression tests, Codex review, Claude review, and live production evals passed on 2026-06-23
Author: Codex
Date: 2026-06-17
Repo: north-os
Related: docs/superpowers/specs/2026-06-17-artifact-figure-provenance-design.html, docs/superpowers/plans/2026-06-10-rag-eval-harness.html, packages/agent-runtime/scripts/eval-agent-tools.ts, packages/agent-runtime/src/agents/legal-assistant/graph.ts

Scope: Implement deterministic runtime gating for figure-bearing artifact writes and keep the production eval assertion on the same helper code. No database schema or migration work is expected. Do not edit existing migrations. Do not run db:push.

Goal

Close the spec-only gap by enforcing this invariant in the legal-assistant runtime: every dollar amount, percentage, day/month/year period, section reference, and 4+-digit number written into a figure-bearing artifact must already appear in a prior tool result or in a human message. If the figure is absent, the graph returns a synthetic error ToolMessage for that tool call and does not invoke the write tool.

Architecture

Add a shared helper at packages/agent-runtime/src/artifacts/figure-provenance.ts that owns figure extraction, digit normalization, haystack construction, and unsourced-figure detection.
Refactor packages/agent-runtime/scripts/eval-agent-tools.ts to import that helper instead of maintaining private FIGURE_PATTERNS, extractFigures, and figureSourced logic.
Intercept selected artifact tool calls inside packages/agent-runtime/src/agents/legal-assistant/graph.ts, before tool.invoke(call). This is the only place with both pending tool args and prior conversation messages.
Start with create_spreadsheet as the hard regression target. Include registry entries for create_tracker where initial items carry sourced values. Treat tabular_review carefully: its spreadsheet cells are produced inside the tool from reviewed documents, so runtime gating of LLM-supplied args should only scan fields that can actually contain model-supplied artifact values. If there are no such fields for tabular_review, document it as intentionally not pre-invocation-scanned and rely on its extraction/page-citation path.
Preserve Anthropic/LangGraph pairing: every blocked tool_call still receives exactly one ToolMessage with status: "error", tool_call_id, and name.

Implementation note, 2026-06-23: create_spreadsheet and create_tracker are pre-invocation scanned. tabular_review is intentionally omitted from the runtime registry because its current schema accepts source file ids, columns, and questions; final cells are produced inside the tool from reviewed documents rather than model-supplied artifact rows. This follows the plan's safe-scope decision and was reviewed as non-blocking by Claude.

Non-goals

No prose-answer validation. The scope is artifact write arguments and the existing production eval assertion.
No automatic substitution of "correct" values. The guard refuses, then the model must source the figure through a tool or write a blank / "not in retrieved documents" value.
No database changes, RLS changes, migrations, or provisioning changes.
No broad artifact/toolbelt refactor beyond the graph seam and shared helper.

Increment 1 - Shared provenance helper

Files: create packages/agent-runtime/src/artifacts/figure-provenance.ts and packages/agent-runtime/src/artifacts/figure-provenance.test.ts.

[x] Move the eval's figure regexes into the helper. Export FIGURE_PATTERNS, extractFigures(text), figureSourced(figure, haystack), and unsourcedFigures(argsText, haystackText).
[x] Keep digit normalization behavior compatible with the eval: $1,500,000, 1,500,000, and 1500000 must match the same source.
[x] Deduplicate returned figures while preserving useful message order.
[x] Add unit coverage for sourced money, unsourced money, digit-normalized matches, percentages, day/month/year periods, section references, 4+-digit numbers, and ignored small integers/status text.
[x] Add explicit tests that a tool-result text from run_calculation or compute_dates sources a derived number, while the same number absent from the haystack is refused.

Green check: bun --filter=@workspace/agent-runtime test -- src/artifacts/figure-provenance.test.ts.

Increment 2 - Refactor the production eval to use the helper

Files: modify packages/agent-runtime/scripts/eval-agent-tools.ts. Modify exports only if the script import path needs a public barrel; prefer a relative source import from the script to avoid expanding the package API unnecessarily.

[x] Delete the script-local FIGURE_PATTERNS, extractFigures, digits, and figureSourced implementations.
[x] Replace the assertion loop with unsourcedFigures(call.args, hay.text).
[x] Keep the assertion wording stable enough that existing eval failure output remains readable.
[x] Run a low-cost script check that does not require live model calls if available; otherwise rely on typecheck plus the targeted live eval in Increment 5.

Green check: bun --filter=@workspace/agent-runtime typecheck.

Increment 3 - Add the legal-assistant runtime gate

Files: modify packages/agent-runtime/src/agents/legal-assistant/graph.ts; optionally add a tiny local helper near the existing context-budget helpers if it does not warrant a new graph-specific file.

[x] Add a figure-bearing registry that maps tool name to a function returning only the model-supplied artifact data to scan. For create_spreadsheet, scan sheets[*].rows, not filenames or column headers.
[x] Build the haystack from prior ToolMessage text and HumanMessage text in state.messages. Exclude the pending tool call's own args and exclude assistant prose.
[x] Before invoking a registered tool, stringify the selected arg slice, compute unsourced figures, and if any exist, push a synthetic error ToolMessage instead of calling the tool.
[x] Make the refusal text explicit: name the tool, list the first few unsourced figures, and tell the agent to retrieve/read/calculate the value first or leave the cell blank / "not in retrieved documents".
[x] Keep unknown-tool, thrown-tool, and result-budget behavior unchanged.
[x] Do not wrap GraphInterrupt handling differently. The guard runs before invocation, so confirmed writes and interrupts continue to work for sourced artifacts.

Green check: covered by Increment 4 graph tests, then bun --filter=@workspace/agent-runtime typecheck.

Increment 4 - Graph regression tests

Files: modify packages/agent-runtime/src/agents/legal-assistant/graph.test.ts.

[x] Add a fake create_spreadsheet tool that records whether it was invoked. Use the existing scripted chat model pattern.
[x] Test blocked write: human asks for a value without providing it; model calls create_spreadsheet with 1500000; result contains a ToolMessage with status: "error", the fake tool is not invoked, and the graph reaches the next assistant turn without crashing.
[x] Test sourced write: prior ToolMessage or HumanMessage contains $1,500,000; model calls create_spreadsheet with 1500000; the fake tool is invoked normally.
[x] Test computed write: prior tool result contains a calculation result; the matching derived figure passes.
[x] Test non-registered tools are unaffected.
[x] If create_tracker is included in the registry, add one focused test for an unsourced 4+-digit deadline/value in an item field.

Green check: bun --filter=@workspace/agent-runtime test -- src/agents/legal-assistant/graph.test.ts src/artifacts/figure-provenance.test.ts.

Increment 5 - Production eval regression

Files: no code changes expected unless the eval output reveals a true runtime bug.

[x] Run the two production tasks named in the spec with the real eval harness and the dev eval org.
[x] Confirm loan-figures-not-invented no longer persists a fabricated amount. A successful outcome can be a sourced artifact, a blank/"not in retrieved documents" artifact, or a final answer explaining that the figure was not in retrieved documents, as long as the provenance assertion passes.
[x] Confirm spreadsheet-cure-periods still passes, proving sourced numeric artifact cells are not blocked.
[x] Keep the honest-eval rule: do not soften the golden task to make the gate pass.

Eval evidence, 2026-06-23: bun --filter=@workspace/agent-runtime run eval:agent-production -- --task loan-figures-not-invented passed 100%; bun --filter=@workspace/agent-runtime run eval:agent-production -- --task spreadsheet-cure-periods passed 100%.

Commands:

bun --filter=@workspace/agent-runtime run eval:agent-production -- --task loan-figures-not-invented
bun --filter=@workspace/agent-runtime run eval:agent-production -- --task spreadsheet-cure-periods

Increment 6 - Final verification

[x] Run bun --filter=@workspace/agent-runtime test -- src/artifacts/figure-provenance.test.ts src/agents/legal-assistant/graph.test.ts.
[x] Run bun --filter=@workspace/agent-runtime typecheck.
[x] Run root bun run typecheck and bun run lint before commit, per repo rules.
[x] If the helper is exported from the package barrel, verify no unintended public API churn in downstream packages.
[x] Do not create or edit migrations. If any migration file changes, stop and investigate because this plan does not need DB work.

Verification evidence, 2026-06-23: focused agent-runtime tests passed after rebase (24 tests); bun --filter=@workspace/agent-runtime typecheck, root bun run typecheck, root bun run lint, and git diff --check passed. No DB or migration files changed.

Acceptance criteria

Unsourced figures in create_spreadsheet rows are blocked before confirmArtifact or persistArtifact can run.
Sourced figures from prior tool results and human messages pass, including digit-normalized formatting differences.
The production eval and runtime gate share the same extraction/source-checking helper.
The existing graph guarantees remain intact: one ToolMessage per tool_call, unknown tool handling still works, unexpected tool throws still become synthetic errors, and context-budget truncation still runs after tool results are assembled.
loan-figures-not-invented and spreadsheet-cure-periods are both green under the production eval harness.

Risks and decisions to resolve during implementation

Tabular review pre-invocation scope: tabular_review does not accept final spreadsheet cell values from the model; it extracts them inside the tool. Do not add a broad JSON scan that blocks filenames, prompts, or file ids. Either scan no fields for this tool in v1 or add only a narrowly justified field slice.
Header/filename false positives: scan row/cell/item values only. Headers like "Original Principal" and filenames with dates should not trip the gate.
Assistant-message exclusion: assistant prose is not a source. Only human text and tool results count, so the model cannot launder a number by saying it before writing it.
Derived figures: require run_calculation / compute_dates output before artifact write. Refusal is the intended behavior for head-computed values.
False positives are safer than false negatives: an oddly formatted real figure may need one extra tool call; an invented legal figure must not leave the chat as an artifact.

Out of scope follow-ups

Extend the registry to additional future artifact writers such as template-fill or chronology builders once their arg shapes exist.
Add UI messaging for blocked write attempts. The runtime error message is sufficient for v1 because the model receives it in the normal tool loop.
Generalize provenance to file/document ids; the eval already has a separate assertion, but this plan is figure-only.