Dynamic context cleaner — design spec

Status: Planned (design — no implementation yet)
Author: Claude (Fable), for Christophe
Date: 2026-07-09
Repo: north-os (packages/agent-runtime; packages/chat-runtime; packages/db; new apps/worker-context)
Related:

• packages/agent-runtime/src/agents/legal-assistant/graph.ts:62-66 — today's guardrail constants (PER_RESULT_CHAR_CAP, CONVERSATION_TOOL_CHAR_BUDGET, MODEL_HISTORY_CHAR_BUDGET) this spec upgrades
• Commit 01e01dfb — long-document pagination (read_document offsets + tracked-change ledger); pagination is what makes "condense, re-read on demand" cheap
• Big-doc subagent read — design spec dated 2026-07-09, in flight on a sibling worktree, not yet on main (cited by topic, not path) — subagent fan-out reads keep bulk out of context up front; this spec condenses whatever still lands. Complementary, not competing.
• OpenRouter multi-model — design spec dated 2026-07-09, in flight on a sibling worktree, not yet on main (cited by topic, not path) — supplies the cheap digest-model tier
• docs/superpowers/specs/2026-06-27-chat-sharing-design.html §7.1 — the cross-process Redis turn lock (packages/chat-runtime/src/turn-lock.ts) the cleaner respects
• apps/worker-memory — the post-turn background-mining worker this one is modelled on (packages/chat-runtime/src/run.ts:1425 debounce-enqueue)

Problem

North's agent threads die of context bloat, and the current defenses are blunt instruments.

Today's mechanics, all in graph.ts (the one choke point every tool reply and every model call passes through):

1. PER_RESULT_CHAR_CAP = 60_000 — any single tool result over ~15k tokens is truncated with a "re-call with an offset" notice.
2. CONVERSATION_TOOL_CHAR_BUDGET = 600_000 — once the thread's accumulated tool output passes ~150k tokens, every further tool reply is hard-cut to its first 2,000 characters. The notice tells the model to "work with what is already in the conversation, or tell the user a fresh, narrower chat is needed." For a lawyer mid-matter, that is a dead chat.
3. MODEL_HISTORY_CHAR_BUDGET = 320_000 — at prompt-build time, compactMessagesForModel (graph.ts:209-250) drops the oldest messages wholesale, replacing them with 300-character snippets capped at 12k chars total. Whatever legally material fact lived in message 3 — a defined term, a tracked-change ledger, the client's stated position — survives as at most 300 characters of prefix.

These caps exist because a live eval run blew the API's 1M-token window ("prompt is too long: 1,234,743 tokens" — a crashed turn). They stop the crash. They do not solve the problem, because they are blind: they cut by position and size, never by value. Three failure modes follow:

• Threads end early. The 600k tool budget is consumed fastest by exactly the work lawyers value most — multi-document reads. One 190k-char lease read via paginated read_document is ~4 pages ≈ 230k chars of context, over a third of the thread's lifetime budget, even though after the agent has extracted what it needed those pages are almost pure dead weight.
• Old facts vanish. Drop-oldest compaction discards the most confirmed content (early user instructions, established facts) to protect the most speculative (recent tool noise).
• Every turn pays for the junk until the cliff. Duplicate tool results, superseded search lists, error/retry pairs, and consumed pagination pages ride in the prompt on every model call — latency and token cost on each turn — until the budget cliff amputates them along with everything else.

The product bar is explicit: North's harness must be decisively better than Claude Cowork's ("at least 150% of Cowork"). Cowork-class harnesses do window-slide compaction — the blind mechanism we already have. Beating it means semantic, provenance-preserving junk removal that runs in parallel with the session: longer effective threads, faster and cheaper turns, and zero loss of legally material facts.

Goals

1. Longer effective threads — a heavy multi-document thread should survive several times more turns before any budget cliff.
2. Faster + cheaper turns — prompt size (and thus latency + cost) drops on threads that have accumulated bulk.
3. No loss of legally material facts — proven by evals, not asserted. Anything condensed is re-readable on demand via recorded provenance.
4. Cleaning is parallel to the session — the expensive judgment (LLM digesting) never adds latency to a live turn.

Non-goals

• Replacing the hard caps. They stay as the last-resort backstop; the cleaner's job is to make them rarely fire across turns.
• Intra-turn bloat. The motivating crash was a single-turn fan-out (a dozen reads in one turn); this design is structurally cross-turn — the recency wall protects everything at/after the latest user message, and digests are loaded once at turn start. Within a single turn the blind caps remain the only protection, deliberately. (Intra-turn relief is the sibling big-doc-subagent-read spec's territory: keep the bulk out of the main context in the first place.)
• Cross-thread or firm-wide summarization (that is the knowledge/memory layer's job).
• Cleaning the chat_message business record or anything the user sees. The cleaner operates only on what the model sees.

1. What counts as junk — the taxonomy

Concrete, in priority order. Categories 1-4 are deterministically detectable (no model judgment); category 5 needs an LLM.

Never junk — deterministic keep-rules

Hard rules evaluated before any LLM sees anything, implemented as pure functions with unit tests. A message matching any keep-rule is untouchable regardless of what the digest model thinks:

• User (HumanMessage) messages. Never condensed, never dropped by the cleaner.
• Tracked-change / comment ledgers. The revision ledger rides the offset=0 page of a read_document (commit 01e01dfb made this an invariant — "never lose the tracked-change ledger"). The offset=0 page of any document carrying tracked changes or comments is kept verbatim; only its follow-on pages are condensable.
• Confirmation / interrupt tool messages. ask_user, document-write and mail-send interrupts — runChat enforces resume toolCallId matching; these messages are load-bearing for paused threads and are never touched.
• Messages with an empty tool_call_id. graph.ts emits synthetic error ToolMessages with tool_call_id: call.id ?? "" in three places; multiple ""-keyed messages can coexist in one thread, so they are excluded from digesting and substitution entirely (they are tiny error strings anyway). The §3.2 uniqueness assumption holds for non-empty ids only.
• The recency window. Nothing at or after the latest HumanMessage, plus the last RECENCY_KEEP_EXCHANGES (default 3) assistant/tool exchanges before it, is ever substituted. The active turn is structurally unreachable (§4).
• Anything referenced by the latest turns. If a file_id, matter, or tool result is named in the recency window, its source messages are kept raw this pass (re-evaluated next pass).
• save_memory results, matter-context blocks, and skill loads (load_skill output is methodology the agent is mid-following).

The figure-provenance constraint (load-bearing)

graph.ts blocks artifact tools (create_spreadsheet, create_tracker) whose figures appear in no prior tool result or human message (figureProvenanceHaystack, graph.ts:123-132). That haystack is built from state.messages — the raw checkpoint messages. Because this design never mutates the checkpoint (§3), the guard keeps scanning the raw record and mechanically keeps passing; an architecture that rewrote checkpoint state would instead silently block legitimate figures whose source got condensed.

Honest limit: the guard's meaning weakens under cleaning. Its purpose is to check the model's figures against what was retrieved — but on a cleaned thread the model's view (digests) is a subset of the haystack (raw), so a figure the model misremembers that happens to occur anywhere inside an unseen 190k-char lease still passes. Mitigations in v1: every digest carries figures verbatim (§2) so cited figures are usually in the model's view too; the eval battery (§4.6) includes artifact-figure probes on condensed threads. Whether the guard should additionally require figures to be sourced in the assembled (model-visible) view — forcing a re-read before citing an un-digested figure — is an open question (§6.3): it strictly strengthens sourcing but adds re-read latency to artifact turns.

2. Condense, don't delete — the digest contract

Junk is never deleted; it is replaced in the model's view by a structured digest that carries provenance sufficient to re-read the original in one tool call. The pagination params from 01e01dfb make that re-read cheap and exact.

Digest format (the ToolMessage content the model sees in place of the bulk):

[Condensed by North — original preserved and re-readable.]
tool: read_document
source: file_id=cd_9f2… "Ground Lease — 445 W 35th" · characters 58,000–116,000 of 190,412 (page 2 of 4)
digest:
  - Article 8 (Assignment): consent required, not to be unreasonably withheld; carve-out for affiliate transfers §8.3
  - Rent schedule Exhibit B: base $1,250,000/yr, 2.5% annual escalator from year 6
  - Defined terms introduced: "Qualified Transferee" (§8.2), "Stabilization Date" (§1.44)
figures verbatim: $1,250,000; 2.5%; §8.2; §8.3; §1.44
re-read: read_document(file_id="cd_9f2…", offset=58000)

Contract rules:

• Provenance is machine-actionable: tool name + the tool's exact document/offset args (file_id+offset, document_id+text_offset, or drive_document_id+text_offset per §1's key mapping) so the agent re-reads with one call when a digest turns out to be insufficient. The digest text explicitly says the original is one call away — the model is never told to "start a fresh chat". The re-read promise is made only for idempotent document reads (§1 cat. 5 caveat); time-varying sources stay raw in v1.
• Figures verbatim: every number, date, dollar amount, section reference, and party name extracted into the digest appears exactly as in the source, so downstream artifact provenance and citation both survive.
• Structured, not prose-soup: parties / dates / amounts / obligations / defined terms / cross-references — the fields lawyers actually come back for.
• Bounded: DIGEST_CHAR_CAP (default 4,000 chars ≈ 1k tokens). A 58k-char page condenses ~14:1.
• Hash-gated: each digest stores sha256(original content); substitution happens only on hash match (§3.3). A digest can never silently misrepresent content it wasn't computed from.
• Deterministic categories 2-4 get deterministic stubs (no LLM): a duplicated earlier result becomes [Identical result re-read later in this conversation (tool_call_id=…) — see that occurrence for the full content.] (latest stays raw, per §1 cat. 2 direction); a retried error becomes [Transient error, retried successfully below.]; a superseded search list keeps its hit ids verbatim and warns that re-running returns current data (§1 cat. 3).

3. Architecture — the core decision

Three candidate shapes were on the table:

(a) Synchronous prompt-time overlay only. Extend compactMessagesForModel: at every llmCall, assemble a condensed view; checkpointer untouched. Safest — and insufficient alone. The deterministic categories (2-4) fit here because they're microsecond-cheap. But category 1 and 5 bulk needs an LLM to condense well; doing that inside the turn adds seconds of latency to exactly the turns that are already slow, or you skip the LLM and you've rebuilt the same blind truncation we have. It also re-does work every turn (nothing is remembered between calls). (a) alone fails the "parallel to the session" requirement outright.

(b) Background worker rewriting checkpointer state between turns. True parallel cleaning, and the prompt path stays dumb. Rejected as the primary mechanism:

• PostgresSaver checkpoints are keyed by thread_id alone, no org column, no version discipline for external writers (checkpointer.ts:12-19 carries an explicit multi-tenant warning). Rewriting means depending on LangGraph's internal checkpoint/writes serialization format — one @langchain/langgraph upgrade away from corrupting every live thread.
• It races live turns. Safe rewriting requires holding the same Redis turn lock chat turns hold (turn-lock.ts) for the whole rewrite — which blocks user turns, the opposite of "parallel".
• It destroys the lossless record. Once rewritten, the original bytes are gone from the execution state; the figure-provenance haystack (§1) breaks; un-cleaning (kill switch, bug recovery) is impossible.

(c) Hybrid — background digest computation + synchronous overlay substitution. ← Chosen.

A dedicated background worker computes digests off-turn into a side table; prompt assembly consults that table synchronously (one indexed read per turn) and substitutes digests into the model's view. The checkpointer is never written by the cleaner — it remains the immutable, lossless record.

This is the only shape that delivers all four goals at once: the expensive judgment runs genuinely in parallel with the session (Christophe's "run en parallèle des sessions"); the turn path adds one cheap DB read; a dropped digest table or kill switch reverts behavior instantly; and the raw record stays intact for provenance, audit, and re-reads.

3.1 Components

                    turn N ends
packages/chat-runtime/run.ts ──debounce-enqueue──▶  BullMQ context.digest queue
   (fire-and-forget, mirrors                          │
    memory-synthesis at run.ts:1425)                  ▼
                                            apps/worker-context  (new, port 4011)
                                              1. skip if turn lock held (non-blocking peek)
                                              2. read raw messages via checkpointer (read-only)
                                              3. deterministic keep-rules + junk taxonomy
                                              4. LLM digests for category 1 & 5 bulk
                                              5. INSERT INTO context_digest (org-scoped, RLS)
                    turn N+1 starts
packages/chat-runtime/run.ts ── loads digests for thread (1 query) ──▶ configurable.context_digests
packages/agent-runtime llmCall ── assembleModelView(state.messages, digests) ──▶ model
                                   (substitution + deterministic inline cleaning;
                                    checkpointer state untouched)

3.2 The side table

New table context_digest (in packages/db/src/schema/, generated via bun db:generate, RLS via pgPolicy() + .enableRLS() on app.organization_id like every org-scoped table):

Unique on (thread_id, tool_call_id) — one digest per tool result; re-digestion upserts.

3.3 Assembly-time substitution (synchronous, cheap)

compactMessagesForModel is refactored into packages/agent-runtime/src/context/assemble.ts as assembleModelView(messages, digests, opts):

1. Recency wall: compute the untouchable suffix (§1 keep-rules). Nothing at/after the latest HumanMessage or in the last RECENCY_KEEP_EXCHANGES exchanges is eligible.
2. Digest substitution: for each eligible ToolMessage with a context_digest row whose content_sha256 matches the live content — substitute. Hash mismatch ⇒ keep raw, emit a digest_stale metric (should never happen; checkpoints are append-only).
3. Deterministic inline cleaning (runs even with an empty digest table — this is Phase 1 and needs no worker): exact-duplicate dedupe (cat. 2), error/retry collapse (cat. 4). Pure functions, microseconds.
4. History-compaction backstop last: the current MODEL_HISTORY_CHAR_BUDGET compaction runs after substitution inside assembleModelView — now over a much smaller haystack, so it rarely fires. (The per-result / budget caps in capToolResult are not assembly-time: they run in toolNode when a fresh result is created, and the capped content is what gets checkpointed — they are untouched by assembly ordering. Their accounting changes below.) Constants stay as-is in this spec.

Digests reach llmCall via configurable.context_digests, loaded once per turn in chat-runtime/run.ts (exactly how memory_block and firm_skills ride today) — the graph stays free of DB reads, and the map is stable across a turn's tool loop (digests only cover pre-turn messages, so staleness within a turn is impossible).

The budget refund — required for Goal 1, shipped in v1

Without this, the cleaner would not touch the spec's headline failure. The CONVERSATION_TOOL_CHAR_BUDGET cliff is what kills heavy-read threads ("a dead chat"), and its usedBudget is summed from raw ToolMessage chars in toolNode (graph.ts:424-429). Digest substitution changes none of those inputs — with raw-only accounting the cliff fires at exactly the same turn as baseline, and Goal 1 / the ≥3× thread-length target would be unreachable. So once digests exist (Phase 3):

• usedBudget counts effective chars: for each prior ToolMessage, digest_chars when a valid (hash-matched) digest exists in the turn's context_digests map, else raw chars. toolNode reads the same once-per-turn map llmCall does.
• A separate, much higher raw ceiling — CHECKPOINT_RAW_CHAR_CEILING (default 2,000,000 raw chars of accumulated tool output) — above which the old raw-accounting behavior resumes. This bounds checkpoint growth and worst-case raw assembly, the legitimate concern that raw accounting was protecting.
• Caveat stated plainly: once the (now much later) cliff does fire, capToolResult amputates new results to 2,000 chars at write time — that content never reaches the checkpoint and can never be digested or recovered. The refund defers this; it does not abolish it.

3.4 The worker (apps/worker-context)

New dedicated worker app — the "new background domain = new worker" rule; the clicky/desktop stack is not touched. Port 4011 (next after meetings' 4010). Chassis boilerplate mirrors apps/worker-memory/src/index.ts; digest logic lives in packages/agent-runtime/src/context/ (it must parse graph messages — same reasoning that put memory mining next to the runtime), with the queue in @workspace/agent-runtime/context-queue mirroring memory-queue.

Per job (payload: { threadId } — org resolved server-side):

1. Trigger check: skip threads below CONTEXT_CLEAN_TRIGGER_CHARS (default 120,000 raw tool-output chars). The count rides the enqueue payload — run.ts already walks the turn's messages and can carry the running total — so a below-trigger job dies without touching the checkpoint. Most threads never need cleaning; the queue must stay near-free for them.
2. Turn-lock peek: read the thread's Redis turn-lock key without acquiring it (via a read-only isTurnInFlight helper exported from turn-lock.ts). Held ⇒ a turn is in flight ⇒ requeue with delay. Best-effort only (§4.1) — safety never depends on it; it just avoids wasting digests on a thread that's about to grow.
3. Read messages via the checkpointer (read-only, app_role), apply keep-rules, classify junk.
4. Emit deterministic stubs (cat. 2-4) directly; batch category-1/5 bulk into digest LLM calls — pages of the same document digest in one call.
5. Upsert context_digest rows.

Enqueue trigger: post-turn, debounced, fire-and-forget from run.ts — a queue hiccup must never touch the chat turn (verbatim posture of the memory-synthesis enqueue at run.ts:1425-1439).

3.5 Digest model + cost

The digester is a cheap fast model — Haiku-class today via createModel overrides, and the natural first consumer of the cheap tier in the OpenRouter multi-model spec (in flight; this spec depends only on "a cheap model is injectable", not on OpenRouter landing first). Digesting is summarization-shaped work with deterministic guardrails around it — precisely the tier's sweet spot. Confidential-data posture is unchanged: same provider boundary as the rest of the agent runtime; nothing leaves the existing model path.

Cost envelope: a 58k-char page ≈ 15k input tokens ≈ ~$0.02 at Haiku-class pricing. A heavy thread (say 20 condensable pages) digests once for ~$0.40 and then saves ~55k chars × remaining turns of prompt every turn thereafter — the digest pays for itself in one or two turns of a long thread. Controls:

• CONTEXT_DIGEST_MAX_CALLS_PER_JOB (default 12) — a runaway thread digests incrementally across passes, not in one burst.
• CONTEXT_DIGEST_DAILY_BUDGET_USD per org (default $5) — worker-side ledger check against cost_usd, job skips (with metric) when exhausted.
• Content-hash reuse: before calling the LLM, look up an existing digest with the same content_sha256 within the org (same page read in two threads ⇒ one digest). Cross-thread reuse is an index lookup, not a design change.

4. Safety rails

1. Never clean the active turn. The structural guarantees are checkpoint immutability + hash gating + assembly's recency wall (the active turn and trailing exchanges are ineligible regardless of what's in the digest table, with the full keep-rule set re-evaluated at assembly time against live messages). The worker's turn-lock peek is a best-effort scheduling optimization on top, not a guarantee: turn-lock.ts degrades soft (no Redis key when REDIS_URL is unset or Redis errors mid-acquire), and headless automation runs drive the graph without taking the Redis lock — a mid-turn worker pass is therefore possible and harmless by construction (it only inserts digest rows for pre-existing messages; a turn loads digests once at start, so mid-turn inserts are invisible until the next turn).
2. Deterministic keep-rules before any LLM judgment. §1's never-junk list is code, unit-tested, and runs first in both the worker and the inline cleaner. The digest model is only ever shown content that already passed the rules — it can produce a bad digest, but it can never select the wrong target.
3. Checkpointer is never written. The raw record is lossless; figure-provenance scanning is unaffected; recovery from any cleaner bug is "delete digest rows".
4. Hash-gated substitution. A digest applies only to byte-identical original content (§3.3).
5. Kill switch. CONTEXT_CLEANER_DISABLED=1 — the worker consumes-and-drops jobs, and assembleModelView skips substitution and inline cleaning, reverting byte-for-byte to today's compactMessagesForModel behavior. One env var, no deploy ordering.
6. Eval gate, extended. bun run eval:gate (retrieval + router + memory-recall) gains an eval:context-cleaner battery in packages/agent-runtime: golden long-thread fixtures → clean → assert (i) legally-material probe questions still answered correctly from the condensed view (memory-recall style), (ii) the figure-provenance guard still passes for artifact calls citing condensed sources, (iii) interrupt/resume on a cleaned thread still matches toolCallId, (iv) chars-saved ≥ threshold on the heavy fixtures. Existing batteries must not regress. Cost stays in eval:gate's "~cents" class — the fixtures replay against fake models except a small real-digest smoke set.
7. Observability, not vibes. Via @workspace/observability (incCounter / observeHistogram / setGauge). Honest current state: agent-runtime / chat-runtime emit no metrics today, and only worker-chassis apps serve /metrics — so Phase 0 adds the dependency, a llm_call_duration histogram around llmCall, and a minimal metrics endpoint on apps/web (workers get /metrics from the chassis for free). One registry gap to close: observeHistogram uses a single global millisecond bucket array topping out at 60,000 — prompt-char observations (120k–1M+) would all land in +Inf and the §5 median would be unmeasurable. Phase 0 adds an optional per-metric buckets parameter (additive; the shared default array untouched) and registers char-scale buckets for context_prompt_chars. Metric set: context_prompt_chars{stage=raw|assembled} (histogram — the headline before/after), context_digest_substitutions_total, context_chars_saved_total, context_digest_calls_total / context_digest_cost_usd_total, context_digest_stale_total (expect 0), context_clean_skipped_total{reason=turn_lock|below_trigger|budget}, and the latency delta read off llm_call_duration.

5. Metrics — what "150% of Cowork" means, measurably

Baseline numbers come from Phase 0 instrumentation before any cleaning ships — the before/after is measured, not estimated.

6. Open questions (decisions Christophe owns)

1. Digest visibility in the UI — should the web thread view show a subtle "context condensed" marker on turns where substitution was active? (Transparency vs noise; no UI work is in this spec's phases.)
2. Per-firm opt-out — is the env kill switch enough for launch, or does HSE want a firm-level setting from day one?
3. Assembled-view figure sourcing — should the figure-provenance guard additionally require artifact figures to be sourced in the model-visible (assembled) view, forcing a re-read before citing un-digested figures (§1)? Strictly stronger sourcing vs added re-read latency on artifact turns. v1 keeps the raw haystack + eval probes; decide with Phase 4 data.