Meeting lifecycle recovery — design spec

Status: Planned
Author: Christophe
Date: 2026-07-21
Repo: north-os (packages/meetings, apps/worker-meetings, apps/web/lib/meetings)
Related: WS-4 of 2026-07-21-meeting-production-readiness · review docs/reviews/2026-07-21-meeting-system-review.md (findings 4, 9, 15, 16 + the best-effort-emit caveat).

Problem

The transcription lifecycle has two unrecoverable stuck states and a fire-and-forget event. After the pending→transcribing CAS (process.ts:68-83), a worker crash (OOM decrypting a large blob, deploy SIGKILL) leaves the row transcribing forever: BullMQ redelivers, the handler throws CasConflictError, and the consumer treats it as a benign race and ack-drops (consumer.ts:36-49). A failed bus.emit after a durable insert strands the row pending forever. And source.meeting.transcribed is emitted best-effort with no outbox (process.ts:130-151) — fine for the summary consumer today, load-bearing the moment a second consumer (RAG, WS-12) depends on it. The summary path already has the missing mechanism (15-min stale-generating reclaim, runtime.ts:386-395) — the asymmetry is the gap.

Design

• Owner-lease claims. Add claimed_by / claimed_at to the transcription claim. Redelivery acknowledges only a fresh claim owned by another live worker; a transcribing row whose lease exceeds lock+grace is reclaimable under CAS back to pending. (Revised 2026-07-22 — see Revision below: no new columns; updated_at is the lease clock.)
• Stale-transcribing sweep. A scheduler loop (mirror of summary-backfill) releases leases past grace to pending and emits an operator signal. markFailed's silent catch (process.ts:224-227) gains a real log.
• Pending-rescue sweep. A row durably committed but never dispatched (emit failed / web crash between insert and emit) is re-driven with a fresh deliveryJobId. The web route returns 202 once the row commits, even if the immediate emit fails.
• Retryable classification fix. AudioFetchError (transient R2 5xx) joins the release-for-retry class instead of terminal-on-first-attempt (errors.ts:61 vs process.ts:167-193).
• Worker exit semantics. Non-zero exit on the fatal-exception path (so systemd Restart=on-failure restarts), plus an unhandledRejection hook (worker-meetings/src/index.ts:196-204).
• Durable transcribed-event (WS-12 prerequisite). Make source.meeting.transcribed at-least-once: re-emit-from-ready on a sweep so a lost emit is recovered. Consumers must be idempotent on meetingId. (Revised 2026-07-22 — see Revision below: seam documented, blanket sweep not built in WS-4.)

Revision — 2026-07-22 (as implemented, NOS-193)

The per-workstream plan in Linear NOS-193 was adversarially verified against main and refined two of the design bullets above. The original text is kept for the record; this section is what shipped.

• Lease clock = updated_at, no new columns. meeting.updated_at has $onUpdate(now()) and every meeting write goes through Drizzle .update(); the only writes during the transcribing span are the claim itself and the terminal transition, so the timestamp set at the pending→transcribing claim is a stable lease start — exactly the property the summary path's stale-generating reclaim already relies on. Owner identity is already recorded per transition in meeting_audit.actor, so claimed_by/claimed_at add a migration without adding recovery power. No migration; scheduler_role grants unchanged. A DB guard test pins the $onUpdate lease assumption. Redelivery semantics are unchanged (CAS conflicts still ack-drop); recovery is owned entirely by the sweep. Poisoned rows are bounded by an audit-count reclaim cap (N reclaims, then quarantine to failed with a distinct reason — still an operator-visible terminal state).
• Pending-rescue is dispatch-touch gated. Each stale pending row is claimed for dispatch by a CAS updated_at bump before its re-emit, so re-drives are grace-bounded and deduplicated across worker replicas; re-emits pass a fresh deliveryJobId (the plain meetingId job id would dedup against the retained original).
• Transcribed-event durability is a documented seam, not a WS-4 sweep. The only live consumer (summary) is already re-driven by the summary-backfill sweep; a second blanket re-emit-from-ready has no delivery ledger and would only re-hit claimSummary's ack-drop. WS-12's RAG consumer gets a natural ledger for free (ready meetings with no chunks) and must (a) be idempotent on meetingId and (b) own its re-drive sweep against that ledger. The at-least-once guarantee for the transcribed event therefore lands with WS-12, on WS-4's mechanics.
• Fidelity caveat. languageHint is event-only (not persisted on meeting), so recovery re-emits carry languageHint: null; providers auto-detect.

Invariant & acceptance

No meeting is ever stuck with no operator signal, and every ready meeting's transcribed-event is eventually delivered at least once (the second clause lands with WS-12 — see Revision; WS-4 ships the mechanics and the seam). Acceptance: a crash-after-CAS integration test (kill between the CAS and the catch, assert the row returns to pending and re-transcribes), a lost-emit test (assert pending-rescue re-drives), and an at-least-once test for the transcribed event (the last with WS-12).