Knowledge retrieval precision — design spec

Status: In progress
Author: OpenAI Codex
Date: 2026-07-20
Repo: north-os
Related: NOS-183 Defect 2; docs/reviews/2026-07-20-eval-agent-quality-harness-bug.md

Problem

search_knowledge ranks same-topic clauses from the wrong legal instrument above the named instrument. The reproduced query “What does the HFA bond mortgage say about limitation on liability?” returns Environmental Indemnity Agreement excerpts and does not return an HFA Bond Mortgage in the first 84 diversified results. The target source exists. Its best vector chunk is rank 229, while the title leg returns no candidates because a long natural-language query does not meet the current trigram operator thresholds.

This is candidate recall plus final-ranking precision, not corpus absence, provenance plumbing, or answer-grounding policy.

Measured baseline

Goals and invariants

• When a query explicitly names a legal instrument, admit title-matching sources to the rerank pool and rank the named instrument above a same-topic different instrument.
• Use document identity only as a conditional relevance signal. Queries without a recognized instrument mention retain their current ordering.
• If no candidate title matches the named qualifiers, preserve the reranker’s order rather than promoting a merely same-type document.
• Preserve authorization filters, RLS, source status checks, parent expansion, best-extent recovery, matter provenance, source diversity, score values, and honest-absence behavior.
• Keep the change schema-free and local to @workspace/knowledge; the agent tool continues to consume the shared retrieval service unchanged.

Non-goals

• No database schema, index, migration, document classification, ingestion, or packages/db change.
• No eval-checker relaxation and no HFA-specific title special case.
• No replacement of vector search, lexical search, RRF, Voyage reranking, or matter-provenance ranking.
• No claim that filename identity proves the requested clause exists; the selected chunk still comes from best-extent recovery and semantic reranking.

Approaches considered

Design

1. Recognize title intent

A pure helper recognizes a bounded, longest-first vocabulary of common legal instrument phrases, including bond mortgages, promissory notes, operating agreements, loan and regulatory agreements, leases, guaranties, term sheets, opinions, certificates, resolutions, and by-laws. It normalizes punctuation and derives title qualifiers from words surrounding the instrument phrase after removing question and topic boilerplate. The helper returns null when no instrument phrase is present.

The vocabulary is type-level retrieval policy, not corpus-specific IDs or filenames. Longest-first matching prevents “mortgage” from swallowing “bond mortgage” and “agreement” from swallowing a more specific agreement type.

2. Focus the existing title leg

For a recognized intent, the title leg searches with the compact title-intent phrase instead of the full question. That compact phrase satisfies the existing trigram title predicate without changing the title leg for unnamed topical queries.

3. Reserve title candidates and recover their best extents

RRF remains the main candidate source, but top title hits are deduplicated and reserved after the fused cutoff so a strong document-identity signal cannot be erased merely because it appears in one retrieval leg. Named title sources are prioritized in the existing bounded best-extent redirect. This lets the document’s clause body compete rather than returning only its first or definitions chunk. Cache replay retains the complete expanded pool by using the same revised bound on cold and warm paths.

4. Apply a conditional stable title boost

After Voyage reranking and matter-provenance ranking, a stable partition promotes only candidates whose display title contains the recognized instrument phrase and, when qualifiers were detected, overlaps those qualifiers. Relative order inside the promoted and remaining groups is unchanged. If no title satisfies the full intent, the function returns the reranker order unchanged. The boost therefore cannot turn a named-but-absent “Acme bond mortgage” request into a confident result from another bond mortgage solely because the document type matches.

Data flow

query
  → optional named-instrument intent
  → lexical + vector + focused title legs
  → RRF cutoff + deduplicated reserved title hits
  → parent expansion + title-source best-extent recovery
  → Voyage rerank
  → matter-provenance stable partition
  → named-instrument stable partition (conditional)
  → source diversity + top-k

Failure behavior

• No recognized instrument: no title-intent boost and no changed final ordering.
• Recognized instrument but no matching title candidate: no final boost; semantic relevance remains authoritative.
• Matching title but no useful body chunk: the existing best-extent and reranker behavior decide whether it survives; title matching does not synthesize content.
• Reranker disabled: deterministic candidate order still receives the same conditional title policy.

Verification

• Pure regression fixtures mirror HFA Bond Mortgage versus Environmental Indemnity Agreement and prove conditionality, qualifier absence, stable ordering, and no row loss.
• Title-query tests prove longest-specific phrase selection and compact intent extraction for HFA, Brooklyn Community Housing Services, and Kingsbrook Estates queries.
• Preprod rank probe reports the same three named queries before and after.
• @workspace/knowledge, @workspace/agent-runtime, deterministic eval:retrieval, root typecheck, and root lint remain green.

Acceptance criteria

1. The HFA Bond Mortgage appears at rank 1 for the reproduced limitation-on-liability query.
2. The named promissory-note and operating-agreement controls remain rank 1.
3. An unnamed topical query receives byte-for-byte-equivalent ordering from the pure final-ranking helper.
4. A named qualifier with no title match does not promote a different same-type instrument.
5. No schema, migration, eval-checker, or agent-runtime ranking fork is introduced.