Add checkpoint capture: whole-session segmented memories

[jamby77]

Problem

After #8 moved capture to SessionEnd, a latent cap surfaced: selectTranscript keeps only 8,000 chars, so a long session now captures ~5% of itself into a single memory. Distinct topics (PR review, debugging, a design discussion) collapse into one truncated summary, which also hurts recall — near-duplicate/bunched content always scores "low" at the confidence gate.

Change

A new Stop hook checkpoints the transcript incrementally: every ~8,000 chars it queues one segment to the existing ingest queue and advances a /tmp checkpoint file — doing zero model work. SessionEnd flushes only the post-checkpoint tail and spawns the existing detached drain. A long session becomes many distinct, sequential segments instead of one husk. The drain is unchanged (Track 1's empty-summary guard still applies); session consolidation and provider-aware thresholds are deliberately deferred.

Implementation (7 TDD tasks)

• nextChunk — pure decision (in turns, resuming in bytes), table-tested with no fs/Valkey/model.
• Checkpoint file I/O — atomic temp+rename; missing/corrupt → zero checkpoint.
• Shared parseTranscriptLine + byte-offset parseTurnsFrom (replaces the private parser in session-end, de-duplicated).
• The Stop hook — stat-gate → parse new bytes → nextChunk → queue + advance to endByte (never EOF).
• session-end flushes only the tail from the checkpoint offset, tags segment, cleans up.
• Registration across index.ts, register-hooks.ts, install-hooks.sh.
• Integration test — a seeded transcript yields sequential segments [0,1,2,…] + a tail against real Valkey.

Install-path refactor

Registering a fifth hook surfaced drift the branch inherited: all three install paths kept their own copy of the hook map, next to a literal Registered 5 hooks count and a hand-aligned summary listing — so adding a hook meant remembering several places.

src/hook-spec.ts is now the single typed source of truth (HookEvent/HookSpec/HookEntry/HookCommand — no more Record<string, unknown[]> and no casts). All three paths derive from HOOK_SPECS: index.ts and register-hooks.ts their settings map, hook count and summary listing (plus, in index.ts, the hook entries of BINARIES), and install-hooks.sh via requireing the module from its embedded Bun snippets. isOwnHookEntry replaces substring matching on "betterdb"/BIN_DIR when identifying our own entries to replace.

install-hooks.sh also gains two fixes. It wrote { ...existing, hooks }, replacing the entire hooks block — installing silently deleted every hook the plugin did not write, third-party entries on unregistered events included. It now merges per event and drops only its own entries, like the other two paths, and reinstalling stays idempotent. Its verification step counted Object.keys(settings.hooks), which reported success even with a hook missing; it now checks each spec's binary is present and exits non-zero naming what is absent.

The hook-migration strip is deleted (module + 89-line test file). It only ever covered Stop, and Stop is registered again, so both install paths' merges already replace the legacy session-end entry before the strip could. Verified rather than assumed: replaying a v0.4.x settings.json (legacy session-end on Stop, a third-party Stop hook, an unrelated PreCompact hook) through both paths with and without the strip produces byte-identical output — stale entry gone, third-party hooks and unrelated keys preserved.

Review & verification

Each task passed an independent spec+quality review (three needed a fix loop: a plan off-by-one, a missing byte-offset test, a dead tail-flush path — all fixed). A whole-branch review verified all four load-bearing invariants end-to-end — no model work on hooks, checkpoint advances to endByte not EOF, segment continuity across Stop firings + SessionEnd, tail parses from the offset — verdict ready to merge.

bun run typecheck clean · bun test tests/unit 164/164 · bun test tests/integration 0 fail (checkpoint test runs unskipped).

Build script

bun run build:hooks was referenced by CLAUDE.md and install-hooks.sh but not defined in package.json — the documented installer was broken independent of this branch. It is now scripts/build-hooks.ts, which compiles each hook entry point to dist/hooks/ and derives its build list from HOOK_SPECS, so a new hook needs no separate build wiring. The optional provider openai (an undeclared dynamic import) is marked --external so --compile does not fail on it. Verified: all five binaries compile and exit 0 on the empty-stdin hook contract, and the working tree stays clean (*.bun-build temp files are swept and ignored).

Stack

Top of a 2-PR stack, based on #8. Its diff is scoped to the checkpoint work only; merge #8 first.

---

Note

Medium Risk
Changes memory capture semantics and Claude settings hook merging; mistakes could duplicate or drop transcript segments or remove third-party hooks, though isOwnHookEntry and tests target those cases.

Overview
Long sessions no longer collapse into a single ~8K-character memory at SessionEnd. A new Stop hook (stop-checkpoint) incrementally queues ~8K transcript segments to Valkey with no LLM work, tracking progress in a /tmp checkpoint file (byte offset + segment index).

SessionEnd now resumes from that checkpoint, chunks any remaining tail the same way (instead of one selectTranscript cap), tags each queue item with segment, spawns the drain when Stop already queued work, and cleans up the checkpoint. Shared checkpoint.ts (nextChunk, planTailSegments, parseTurnsFrom) and parseTranscriptLine in transcript.ts replace the private parser that lived in session-end.

Hook registration is centralized in hook-spec.ts (HOOK_SPECS, buildHookMap, isOwnHookEntry); install/register scripts and index.ts derive the fifth hook from that list. hook-migration.ts is removed. build:hooks compiles all hook specs to dist/hooks/.

^{Reviewed by Cursor Bugbot for commit a1be843. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Duplicate queue on checkpoint failure

Medium Severity

The Stop hook pushes a segment to Valkey before advancing the on-disk checkpoint. If writeCheckpoint fails after a successful pushIngestQueue, the checkpoint byte offset and segment counter stay unchanged, so the next Stop run re-enqueues the same transcript chunk under the same segment index.

 

^{Reviewed by Cursor Bugbot for commit d0310ba. Configure here.}

[jamby77]

Acknowledged, but working as intended — not fixing.

The push-before-checkpoint ordering is deliberate. It yields at-least-once: if writeCheckpoint fails after a successful pushIngestQueue, the next Stop re-enqueues the same chunk, and the worst case is one duplicate near-identical memory. Reversing it to checkpoint-then-push would yield at-most-once: a failed push after an advanced checkpoint would lose that segment permanently, with nothing to detect it.

This plugin exists precisely to stop silently losing session content — the bug it was written to fix stored empty husks on the success path for eight days. Given that, duplicating beats dropping, so the current order is the safe one.

Scope of the window is also narrow: it requires the /tmp temp-file rename to fail after a successful RPUSH. The checkpoint write is already atomic (temp + rename), runHook exits 0 either way, and the drain's isEmptySummary guard independently drops anything that summarizes to nothing.

Noting for the record that this is the third independent review to raise it, which says the intent isn't legible enough in the code — that's a fair signal even though the behavior stands.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit a1be843. Configure here.}

[cursor]

Valkey failure skips queued drain

Medium Severity

When getValkeyClient fails, SessionEnd cleans up and returns without spawning a drain. The empty-tail path already drains if checkpoint.segment > 0 because Stop never does, but the Valkey failure path ignores that and can leave previously queued segments unsummarized until some later session happens to drain.

 

^{Reviewed by Cursor Bugbot for commit a1be843. Configure here.}

[cursor]

Binary hooks miss ownership match

Medium Severity

isOwnHookEntry recognizes checkout .ts sources from HOOK_SPECS, but install-hooks.sh registers compiled binaries without those filenames. Ownership then depends on the current distDir or a betterdb path substring. After the switch from overwriting the hooks block to merging, a moved or renamed checkout without betterdb in the path leaves stale binary entries beside the new ones.

 

^{Reviewed by Cursor Bugbot for commit a1be843. Configure here.}

[KIvanow]

Approve. Builds cleanly on #8 (merge #8 first). The checkpoint design is careful — byte-offset checkpoints with atomic temp+rename, multi-byte-safe slicing at newline boundaries, segment continuity across Stop→SessionEnd, and dedup gating on the event-file fallback. Good test coverage, and the install-path refactor to a single HOOK_SPECS source of truth is a nice cleanup.

Verified locally on this branch (contains #8): bun run typecheck clean · bun test tests/unit 164/164 · build:hooks compiles the hook binaries and sweeps its temp files.

Two low-severity hook-matching edge cases, neither blocking:

• src/hook-spec.ts isOwnHookEntry (source-name match) — the doc comment calls this "deliberately conservative — a false positive deletes a third party's hook," but matching on generic source filenames like pre-tool.ts / post-tool.ts is arguably the opposite: a third-party hook command that happens to reference such a path would be silently deleted on install. Consider requiring a path marker and a source-name match, or namespacing the filenames (e.g. betterdb-post-tool.ts).

• src/hook-spec.ts isOwnHookEntry (cross-install-path gap) — installed-binary entries are recognized only via the caller's markers (BIN_DIR/distDir/"betterdb"), because the binary command is .../session-end and does not contain session-end.ts. So an entry written by install-hooks.sh (binaries under a project path with no "betterdb" in it) won't be recognized by a later index.ts CLI install (marker ~/.betterdb/bin) — it survives the merge and you get two hooks on the same event. Edge case (mixing install mechanisms), but eliminating exactly this drift was the goal of the refactor. Worth a test covering the install-path → CLI-install upgrade.

Also see the note left on #8 about processIngestQueue dropping already-popped items on a mid-batch failure — pre-existing, but segmented capture here makes it materially more likely to lose a session's later segments.