feat(decisions): structured ledger + deterministic render for decisions/pitfalls

[dean0x]

Summary

Reworks the decisions/pitfalls system so decisions.md / pitfalls.md become a deterministic, active-only render of a new committed anchored ledger (.devflow/decisions/decisions-ledger.jsonl) — eliminating the prior dual-source-of-truth (parallel .md + .jsonl writes bridged by a brittle regex, which had already drifted). Also tightens the Dream creation bar (fewer, higher-quality entries), makes removal recoverable ("retire", not delete), bounds the observation log via rotation, and makes the Dream pipeline commit its own maintenance with recognizable commits.

What changed

Data model (two-file split)

• NEW decisions-ledger.jsonl — committed, anchored rows only (incl. Deprecated/Superseded/Retired); the render source of truth + recoverability.
• decisions-log.jsonl (raw observations) + NEW decisions-log.archive.jsonl — stay gitignored.
• decisions.md / pitfalls.md — committed, rendered from the ledger (active entries only — one tier).

Creation bar (Dream) — abstain-by-default, negative examples, ADR-XOR-PF (one incident → an ADR or a PF, never both), dedup-before-create. confidence is now honest LLM metadata, not a gate (the dead ≥0.65 gate + 0.95 default are gone).

Ops — new assign-anchor (numbering owned by the ledger: max+1 over all anchored incl. Retired, ADR/PF independent, 3-digit pad), retire-anchor (flips decisions_status, recoverable), rotate-observations (archives observing rows >30d). decisions-append is removed (hard cut, no shim).

Render + format — pure render-decisions.cjs (render / --check) + a shared decisions-format.cjs imported by both the renderer and the writer so the byte-compat format can't drift; locking consolidated into mkdir-lock.cjs.

Migration — decisions-ledger-unify-v1 (per-project) backfills the ledger from existing .md + log, preserving every body verbatim (raw_body), synthesizing entries whose source obs is missing, marking hand-deleted anchors Retired (numbers reserved, never resurrected), and preserving inline amendments. Idempotent + crash-safe (always reconciles .md from the ledger). sync-devflow-gitignore-v3 pushes the gitignore change to existing projects.

Dream auto-commit — dream-commit helper writes chore(dream): <action> commits with Dream-Task: / Dream-Session: / Co-Authored-By: trailers, staging ONLY allowed .devflow maintenance paths (never git add -A), skipping cleanly mid-rebase/merge/cherry-pick/detached-HEAD and when clean. Default ON via .devflow/dream/config.json autoCommit; reported by devflow decisions --status.

Breaking changes / rollout

• One-tier render: Deprecated/Superseded/Retired entries no longer appear in the rendered .md (they live in the committed ledger, recoverable).
• Gitignore change (all users): existing projects begin tracking decisions-ledger.jsonl on next devflow init; the unify migration then materializes it and re-renders the .md.
• decisions-append removed — zero callers remain.
• Takes effect after npm run build + devflow init (PF-007).

Verification

• Build clean, 1792 tests green (render goldens, ledger ops, migration golden, dream-commit safety, schema back-compat, perf ratio tests).
• Quality gates: Validator PASS, Scrutinizer PASS (fixed an empty-TL;DR byte drift + TS/CJS lock-path parity), Evaluator ALIGNED (26/26 acceptance criteria), QA Tester PASS (14/14 AC groups end-to-end).
• Dogfood (migration run on a copy of this repo's own corpus): 29-row ledger (26 active incl. synthesized ADR-001 + 3 retired hand-deletions); .md bodies byte-identical except the repopulated TL;DR Key line; ADR-002/PF-003/PF-005 Retired + absent from render; ADR-016 amendment preserved.

🤖 Generated with Claude Code

[dean0x]

TypeScript: Dead and Divergent Status Enum

Lines 13 & 54: DecisionsEntryStatus (line 13) is dead code with zero usages AND diverges from decisions_status field (line 54).

Problem:

• DecisionsEntryStatus defines: 'Accepted' | 'Active' | 'Deprecated' | 'Superseded' | 'Unknown'
• LearningObservation.decisions_status defines: 'Accepted' | 'Active' | 'Deprecated' | 'Superseded' | 'Retired'

The type has a phantom 'Unknown' and is missing the load-bearing 'Retired' status used by the renderer, retire-anchor, and migrations.

Fix: Delete the dead type, or consolidate to one source of truth:

export type DecisionsEntryStatus = 'Accepted' | 'Active' | 'Deprecated' | 'Superseded' | 'Retired';

Reviewers: TypeScript (90%), Architecture (90%), Consistency (90%), Regression
Confidence: 90% | Category: Blocking

---

PR #241 Review — Architecture, TypeScript, Consistency, Regression

[dean0x]

Architecture: Committed Ledger Leaks Observation Metadata

Line 687 (assign-anchor): Ledger rows copy the entire observation record via Object.assign({}, aaObs, { anchor_id, decisions_status }).

Problem: The renderer only reads 8 fields (type, anchor_id, decisions_status, raw_body, details, pattern, date, id). The other 6 (evidence, confidence, quality_ok, count, first_seen, last_seen) are render-irrelevant lifecycle telemetry.

Live inspection: 28 of 29 ledger rows carry these extra fields. This:

• Bloats diffs/PRs with volatile observation metadata
• Leaks evidence snippets into committed history
• Invites future code to couple the renderer to lifecycle fields
• Erodes the "single-source-of-truth" boundary

Fix: Project to render-relevant fields only before atomic append. Create a small toLedgerRow helper in decisions-format.cjs that whitelists the 8 fields. The migration's synthesized path already does this (9 clean keys) — make add-path match.

Reviewer: Architecture | Confidence: 90% | Category: Blocking

---

PR #241 Review — Architecture

[dean0x]

Reliability: CPU-Pegging Busy-Wait on Lock Contention

Lines 40–45: Fallback for unavailable Atomics.wait is a busy-wait spin loop:

const end = Date.now() + 50;
while (Date.now() < end) { /* spin */ }\n```

**Problem**: With a 30s timeout, a contended lock burns a full CPU core for up to 30 seconds during a background Dream run. Violates reliability principle "minimize work in hot paths." The TS counterpart (`mkdir-lock.ts:40`) correctly uses `setTimeout` (truly idle).

**Fix**: Hoist buffer, drop spin, use idle sleep:
```js
// module scope: allocate once\nlet waitBuf = null;\ntry { waitBuf = new Int32Array(new SharedArrayBuffer(4)); } catch { waitBuf = null; }\n\n// in loop: use idle sleep on fallback\nif (waitBuf) {\n  Atomics.wait(waitBuf, 0, 0, 50);\n} else {\n  require('child_process').execSync('sleep 0.05', { stdio: 'ignore' });\n}\n```

**Reviewer**: Reliability | **Confidence**: 90% | **Category**: Blocking

---
*PR #241 Review — Reliability*

[dean0x]

Reliability: Missing Precondition Assertions on Source-of-Truth Write

Lines 641–717 (assign-anchor): No assertions for core invariants before atomic ledger write:

1. No duplicate-anchor check: nextAnchorFromLedger could return a colliding ID (malformed anchor_id, regex mis-parse); op would write duplicate ADR-NNN.
2. No already-anchored check: Re-anchoring an obs silently creates a second ledger row for the same observation.

Impact: Silent corruption of the committed source-of-truth ledger. Violation invisible until a human notices duplicate ADR numbers.

Fix: Add cheap preconditions inside the lock, before atomic write:
js\nif (aaLedgerRows.some(r => r.anchor_id === aaAnchorId)) {\n process.stderr.write(`INVARIANT VIOLATION: ${aaAnchorId} already in ledger\n`);\n process.exit(1);\n}\nif (aaObs.anchor_id) {\n process.stderr.write(`obs ${assignObsId} already anchored as ${aaObs.anchor_id}\n`);\n process.exit(1);\n}\n

Reviewer: Reliability | Confidence: 82% | Category: Blocking

---

PR #241 Review — Reliability

[dean0x]

Code Review Summary: Lower-Confidence & Non-Anchorable Findings

Medium-Confidence Issues (60–80%)

1. Unbounded Archive (scripts/hooks/json-helper.cjs:272–277, Performance 88%)

   • rotate-observations reads+rewrites entire archive every rotation with no cap. Archive grows monotonically forever.
   • Fix: Make true append (write only new stale, skip read+concat) or add size/row cap.

2. Performance Tests Can Pass Vacuously (tests/decisions/ledger-ops.test.ts:798, render-decisions.test.ts:578, Testing 88%)

   • Ratio tests return without assertion when SMALL < 0.01ms (common on fast runners).
   • Fix: Raise input sizes or add structural assertion (expect.assertions(1)).

3. Under-Modeled LedgerRow in Migration (src/cli/utils/decisions-ledger-migration.ts:80–98, TypeScript 88%)

   • Uses type?: string, decisions_status?: string, [key: string]: unknown instead of canonical LearningObservation.
   • Allows bad status values and field typos to compile.
   • Fix: Reuse LearningObservation or Partial<LearningObservation>; drop index signature.

4. Inline Node.js Heredoc (scripts/hooks/dream-commit:94–100, Complexity 82%)

   • Embeds JS program as shell string with $DREAM_CONFIG interpolated into JS source.
   • Hard to read/lint; fragile if paths contain quotes.
   • Fix: Use existing json-helper.cjs get-field-file helper.

---

Documentation Drift (Not in Changed Lines)

CRITICAL: The PR ships a major subsystem refactor (decisions-append → assign-anchor, append-only → deterministic render) but leaves reference docs dangerously stale:

Project CLAUDE.md (lines 43, 49, 160, 162) — Still references removed decisions-append op:

• Line 43: "id-keyed JSONL records, decisions-append numbering"
• Line 49: "materializes entries via decisions-append"
• Line 160: ".decisions.lock — Lock directory for decisions-append writers"
• Line 162: "written by Dream agent via decisions-append"

This is the highest-traffic onboarding doc; it actively misdirects new contributors. Requires explicit approval before editing.

Feature KB Drift (.devflow/features/hooks/KNOWLEDGE.md lines 78, 185, 190, 204) — Committed KB still documents decisions-append as a live operation:

• Lines document the op signature, lock semantics, and deadlock guidance for a removed op.
• An agent loading this KB will be instructed to call a non-existent op.
• Can be auto-refreshed via devflow knowledge refresh hooks.

In-Changed-File Documentation Contradiction (shared/skills/docs-framework/SKILL.md:135–136) — You edited this file but left the "Append-only" description stale:

• Lines 135–136 still say decisions.md/pitfalls.md are "Append-only (ADR-NNN sequential IDs)".
• Under the shipped model, files are deterministically rendered and entries can be removed via retire-anchor.
• Line 172 correctly describes the new model — lines 135–136 should match.

---

Summary

Three HIGH blocking issues require fixes before merge:

1. Busy-wait CPU pegging in lock primitive
2. Missing invariant assertions on anchor write
3. Committed ledger leaks observation metadata

Type consolidation strongly recommended (three reviewers converge on DecisionsEntryStatus).

Documentation drift should be corrected in-PR (CLAUDE.md requires approval; feature KBs can auto-refresh).

Overall Status: CHANGES_REQUESTED

---

Claude Code Review — PR #241