MatterBench — Agentic Legal Benchmark — design spec

0. Decisions locked (brainstorm 2026-07-01/02)

1. Thesis

An open benchmark for agentic legal work with verifiable-first scoring. Three claims, each a direct answer to a failure of the existing benchmarks:

1. Real work. Tasks mirror what attorneys at a working firm actually do on matters — sourced from HSE practice patterns (NY affordable-housing real estate for v1), a corpus that grows as North operates.
2. Agentic. The unit of evaluation is a matter environment plus a tool-using agent trajectory, not a prompt/response pair. Multi-document, multi-step, real file formats.
3. Verified, not vibed. Scores come primarily from deterministic verifiers with binary, enumerable invariants. A model cannot rubric-charm its way past a broken document or a fabricated citation.

2. Anatomy — pyramid on shared matter-worlds

A matter-world is a self-contained fictional matter: a document corpus in real formats, a cast of parties, a factual timeline, and a set of gold artifacts. Authoring a world is expensive (days); deriving task instances from an existing world is cheap. So worlds are shared and instances tiered:

The tier structure is also the publication story: a model that aces Tier 1 and collapses at Tier 3 is exactly the finding that makes the benchmark worth citing.

2.1 Splits

• Public split — 10–15 synthetic worlds (v1: 4–6) with all task instances, gold data, verifiers, and harness. Fully open.
• Private held-out split — 2–3 worlds derived from anonymized real HSE material. Never released. Used to (a) detect training-set contamination of the public split (public score ≫ held-out score is the alarm), (b) validate that public rankings transfer to genuinely unseen work.

2.2 Real formats, real formatting complexity — a hard requirement

World documents are authored as the artifacts they claim to be: .docx with Word paragraph styles, multi-level list numbering, field-code cross-references, tables, headers/footers, and where relevant a tracked-changes history; PDFs (some scanned-image pages in later versions); .eml email threads with attachments. Never markdown pretending to be a lease. The layout-preservation verifiers (§3) are only meaningful when there is real layout to break — this is what keeps the "clean edit" claim honest.

3. Scoring — two-layer architecture (the core)

3.1 Layer 1 — deterministic verifiers

Every task declares a set of invariants checked by code. Each verifier returns an enumerated list of violations, never a scalar vibe. The catalog:

3.2 Layer 2 — rubric judge (bounded residue)

Only qualities that genuinely cannot be programmatic: legal judgment, advice quality, tone, organization. Severity-weighted rubrics authored per task family, judged by LLM, calibrated against the attorney-sampled 20% (judge/attorney agreement is measured and published). The judge's contribution to an instance score is bounded (target: ≤30%), so verifier performance dominates.

3.3 Instance score

score = hard_gates(critical verifiers all pass ? 1 : 0)
        × ( w_v · verifier_score + w_j · judge_score )      // w_j ≤ 0.3

Reported per tier, per task family, and per verifier class — the per-verifier breakdown ("model X fails layout preservation 40% of the time") is the actionable output, for the leaderboard and for North's own agents alike.

4. Task families

Eight families, each declaring which verifiers apply. All are recognizable attorney work:

1. Clean document editing & redlining — clean-edit, cross-ref, trajectory
2. Clause / data extraction — exact extraction, trajectory
3. Chronology construction — chronology F1, citation existence, trajectory
4. Conflict review (lease vs regulatory agreement, OA vs term sheet) — citation existence, exact extraction, judge
5. Drafting from template (estoppels, notices, amendments) — clean-edit, cross-ref, exact extraction, judge
6. Defined-terms & cross-ref hygiene — cross-ref (pure)
7. Email triage — checklist match, exact extraction (deadlines), judge
8. Closing checklist assembly — checklist match, citation existence, date-math

v1 ships 4 families: 1, 2, 3, 5 — they exercise every verifier class including the flagship, and map directly onto what North's own agents do today.

5. Harness architecture

Standalone repo, TypeScript + Bun (matches the North stack; verifier code may shell out where a mature library wins). Components:

matterbench/
  worlds/<world-id>/          # documents (docx/pdf/eml), world.json (cast, timeline)
  tasks/<task-id>/            # task.json: brief, world ref, tier, family,
                              #   verifier config, gold data, rubric
  packages/
    harness/                  # task loader, run orchestration, sandbox, reporting
    reference-agent/          # fixed scaffold: the agent loop + toolset
    adapters/                 # anthropic / openai / google model adapters
    verifiers/                # one package per verifier class
    fixtures/                 # known-good + known-broken artifacts for verifier tests
  results/                    # versioned run outputs (JSON), report generator (HTML)

• Reference agent — a deliberately plain tool-loop (no vendor-specific tricks): list_documents, read_document, search_matter, edit_document (structured edit operations against the docx, not raw XML), draft_document, submit. The scaffold is fixed and versioned; only the model swaps. Runs sandboxed against a copy of the world, budgeted (max steps / tokens per instance, published with results).
• Model adapters — thin: messages in, tool calls out. Anthropic, OpenAI, Google at v1.
• Verifier runner — executes the task's declared verifiers against the final artifacts + the recorded trajectory; emits enumerated violations.
• Report — JSON per run (machine) + HTML summary (human): per-tier, per-family, per-verifier-class breakdowns, hard-gate failure list with document diffs.

5.1 The clean-edit verifier — build first

OOXML diffing is the hardest single piece of engineering in the project and also the moat. Approach: parse both docx packages; align w:p/w:r trees; classify each delta as (a) inside an authorized edit span, (b) formatting-preserving whitespace/rsid noise (ignored), or (c) violation — with violations reported as human-readable statements ("numbering of §7 list changed from legal to decimal"). It gets built and unit-tested against fixtures before any world authoring, to prove the concept and de-risk the schedule.

5.2 North internal wiring

North's own agents are evaluated by running them as an alternative scaffold against the same tasks and verifiers (internal-only results; the public leaderboard stays models-in-reference-agent). This is the internal quality gate: a release of agent-runtime that regresses the clean-edit pass rate is visible before it ships.

6. World authoring pipeline

1. Pattern harvest — from HSE interviews, the real-estate clause ontology, and (with care) real matter shapes: what documents exist on a matter of type X, what tasks recur, where the traps are (mismatched defined terms, stale cross-refs, conflicting dates between lease and regulatory agreement).
2. World bible — fictional parties, property, deal economics, timeline; internally consistent numbers (rents, dates, escalations) generated from a seed so gold answers are derivable, not hand-maintained.
3. Document generation — templates + the world bible produce the docx/pdf/eml corpus; deliberate injected defects (a broken cross-ref, a conflicting date) become task targets with known gold answers.
4. Task + verifier config derivation — atomic tasks largely derive mechanically from the bible; composite/capstone tasks are hand-authored.
5. Attorney sample review — ~20% of tasks/rubrics reviewed by HSE attorneys; disagreements fix the task or the rubric, and the review rate is published.

Because worlds are seeded-synthetic, new worlds (and refreshed releases) are cheap to mint as HSE content grows — this is the renewable-data answer to contamination.

7. Releases & contamination

• Versioned releases — v2026.3 style; each release may add worlds/tasks and retire saturated ones. Results are always reported against a named release.
• Contamination watch — held-out split scored on every leaderboard run; a public-vs-held-out gap beyond threshold flags the public split for refresh.
• Synthetic-at-source — no world text exists anywhere on the public internet before release, and post-release contamination only helps a model memorize a retired version.

8. v1 scope

• Clean-edit verifier + fixtures + tests (first)
• Remaining verifier classes: cross-ref, citation existence, exact extraction, chronology, trajectory, date-math
• Harness + reference agent + 3 model adapters
• 4–6 synthetic worlds, 4 task families (edit/redline, extraction, chronology, drafting), ~90 instances across tiers (≈60 / 24 / 4–6)
• 1 private held-out world (anonymized-real)
• Attorney sample review of ~20%
• JSON + HTML reporting; first published results for Claude / GPT / Gemini frontier models

Out of v1: leaderboard website, OCR/scanned-PDF tasks, open agent track, task families 4/6/7/8, jurisdictions beyond NY affordable-housing RE.

9. Verification — how we prove the benchmark itself is right

• Verifiers are unit-tested against fixture pairs: known-good artifacts must pass with zero violations; known-broken artifacts (one per invariant) must produce exactly the expected violation. This is part of the public credibility story: the benchmark's own correctness is verified.
• Gold-answer consistency — gold data derives from the world bible seed; a consistency checker asserts every gold answer is reproducible from the bible (no hand-drift).
• Judge calibration — LLM-judge vs attorney agreement on the sampled 20%, measured and published; a rubric whose agreement is poor gets rewritten or the quality moves to a verifier.
• Reference-agent sanity — a scripted "oracle agent" that replays gold trajectories must score ~100%; a null agent must score ~0. Both run in CI.

10. Risks & open questions

• OOXML diff complexity — mitigated by building it first; if a week of effort shows it's a tarpit, fallback is a constrained edit API (agent submits structured operations; verifier checks the operations rather than raw XML) — weaker claim, still honest.
• Synthetic realism — synthetic worlds risk being "too clean". Mitigation: injected messiness patterns harvested from real matters (inconsistent naming, scanned junk pages, duplicate versions), and the attorney sample review as the realism check.
• HSE consent for the held-out split — anonymized-real material needs explicit HSE sign-off (Russ). Needed before the held-out world is built; public v1 does not block on it.
• Naming — "MatterBench" is a working title; check collisions before publishing.
• License — dataset + harness license (Apache-2 code / CC-BY-SA data is the default posture) to be decided before release.