fix: Title/entity-aware knowledge retrieval ranking — implementation plan

Status: In progress
Author: OpenAI Codex
Date: 2026-07-20
Repo: north-os
Related: design spec; NOS-183 Defect 2

Summary

Repair named-instrument retrieval at the shared @workspace/knowledge ranking seam. A compact title intent makes the existing title leg retrieve the named document, reserved title candidates and best-extent recovery keep its operative body in the pool, and a conditional stable boost makes explicit document identity outrank a same-topic different instrument. No agent-runtime fork, schema change, or checker relaxation.

Requirements trace

ID	Requirement	Verification
R1	Named instrument’s own document outranks a same-topic different instrument.	HFA fixture plus preprod rank probe.
R2	Unnamed topical-query ordering does not change.	Pure helper identity test and deterministic retrieval eval.
R3	No exact-title match preserves semantic order and honest absence.	Qualifier-miss regression test.
R4	Shared retrieval path serves `search_knowledge` without tool-specific ranking.	Knowledge and agent-runtime package tests.
R5	No schema, migration, DB package, or eval-checker change.	Changed-file review.

Key technical decisions

Use a pure, bounded legal-instrument vocabulary with longest-first phrase matching. This is deterministic and testable; no model or persisted entity extraction is introduced.
Qualifiers are derived from query context around the instrument phrase. Final boosting requires an instrument title match and qualifier overlap when qualifiers exist.
Reserve a bounded number of title hits outside the RRF cutoff. A one-leg title hit must not lose solely because two content legs fill the fused pool.
Prioritize named title sources in bounded best-extent recovery so the reranker compares operative text, not only a filename or first chunk.
Apply the title stable partition after matter provenance and before source diversity. Explicit document identity wins between source groups; existing relative order remains within each group.

Implementation units

U1. Named-instrument intent and stable title ranking

Goal: Add a pure policy module that recognizes named legal instruments, builds a compact title query, and conditionally promotes matching source titles without dropping or duplicating hits.

Requirements: R1, R2, R3.

Dependencies: None.

Files:

Create packages/knowledge/src/retrieval/instrument-title-boost.ts.
Create packages/knowledge/test/retrieval/instrument-title-boost.test.ts.

Approach: Normalize punctuation and case, match the longest supported instrument phrase, derive meaningful surrounding qualifier tokens, and return a focused title query. The ranking helper performs a stable partition only when at least one candidate title satisfies the instrument plus qualifier intent. It returns a copy in original order for unnamed queries and named qualifier misses.

Patterns to follow: Generic stable-partition shape and invariants in packages/knowledge/src/retrieval/matter-boost.ts and packages/knowledge/test/retrieval/matter-boost.test.ts.

Test scenarios:

HFA Bond Mortgage ranks above Environmental Indemnity Agreement while preserving order among mortgage versions and among nonmatches.
Brooklyn Community Housing Services promissory-note and Kingsbrook Estates operating-agreement queries produce focused intents with their qualifiers.
An unnamed limitation-of-liability topical query is an identity transformation.
An “Acme” named instrument with no Acme title match does not promote another same-type document.
Longest-specific phrase wins over a shorter nested type; punctuation and case do not change detection.
No rows are lost or duplicated.

Verification: Pure tests establish deterministic query classification and ordering invariants.

U2. Candidate recall and shared retrieval integration

Goal: Feed title intent through the existing title leg, reserve its strongest candidates, recover their best body extents, and apply the final conditional ranking in searchKnowledgeWithAuth.

Requirements: R1, R2, R3, R4, R5.

Dependencies: U1.

Files:

Modify packages/knowledge/src/retrieval/title.ts.
Modify packages/knowledge/src/handlers/retrieval.ts.
Modify packages/knowledge/test/retrieval/instrument-title-boost.test.ts if integration-discovered boundaries require additional pure coverage.

Approach: Use the compact title query only when present, append deduplicated title hits after the fused cutoff, expand the revised bounded pool, and put named title source IDs first in best-extent recovery. Increase the shared cold/warm candidate cap by exactly the reserved title bound. Apply the pure title boost to the full post-rerank pool after matter provenance and before source diversity. Unnamed topical queries retain the prior title query and fused candidate construction.

Patterns to follow: Existing candidate constants, cache replay bound, TOC redirect deduplication, matter boost, and expanded-by-ID metadata projection in packages/knowledge/src/handlers/retrieval.ts.

Test scenarios:

Named title hits outside the fused cutoff remain eligible for expansion and reranking.
Duplicate chunk IDs from title and fused legs appear once.
Named title sources receive best-extent recovery before generic top-fused sources.
Cold and warm paths share the revised maximum pool bound.
No recognized title intent leaves final ordering unchanged.

Verification: Package tests pass and the preprod HFA target enters and wins the final result set.

U3. Measurement and regression gates

Goal: Prove the defect is repaired without weakening broader retrieval quality.

Requirements: R1–R5.

Dependencies: U1, U2.

Files: No permanent evaluator file unless the existing deterministic suite needs a new observable contract case. The preprod probe remains outside the repository.

Approach: Re-run the identical three-query preprod rank probe at k=84, then run the existing deterministic retrieval evaluation and both affected package suites. Finish with repository typecheck and lint. Review the changed-file set to confirm no DB, migration, checker, or agent-runtime ranking changes.

Test scenarios:

HFA Bond Mortgage moves from beyond rank 84 to rank 1.
Named promissory note remains rank 1.
Named operating agreement remains rank 1.
Deterministic eval:retrieval meets its existing gate with no topical-query regression.

Verification: Record before/after ranks, package test outcomes, eval success rate, typecheck, and lint in the PR.

System-wide impact

Knowledge service: slightly larger rerank pool only for title hits; bounded by ten reserved candidates and existing redirect caps.
Agent runtime: behavior improves through the existing package API; formatting, provenance, absence handling, and tool schema remain unchanged.
Cache: cold and warm paths replay the same complete candidate sequence under a revised shared bound.
Database: existing filtered title query only; no writes beyond normal retrieval audit telemetry and no DDL.

Risks and mitigations

Risk	Mitigation
Same-type but wrong-entity source is over-promoted.	Require qualifier overlap when a named qualifier is present; no-match is identity.
Title-only chunk wins without operative content.	Prioritize its source in best-extent recovery and retain semantic reranking.
Candidate growth changes cache-hit behavior.	Use one constant-derived maximum for cold construction and warm replay.
Fixed vocabulary becomes an accidental document taxonomy.	Keep it limited to common instrument phrases and independent of persisted `doc_type`; expand only with measured misses.

Outcome

Ship only if the reproduced HFA query ranks an HFA Bond Mortgage first, the two already-correct named-instrument controls remain first, deterministic retrieval evaluation does not regress, all requested package tests pass, and root typecheck/lint pass.