diff --git a/AGENTS.md b/AGENTS.md index f4010185..872f07ee 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,7 +33,7 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime. - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level. - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Strict insert and upsert both route through the sealed exact-`id`, filter-bearing adapter; bare Lance Append is test-only. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. Mutation/Load keeps one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before recovery arm; mutation update scans stream into the remaining table budget after pending-key shadowing, with blob sizes checked before payload reads. It then arms an identity-bearing recovery-v9 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, and publishes under the same token. Unrelated retryable pre-effect authority movement may fully reprepare without changing logical mode. A proven effect-free strict conflict returns `KeyConflict` only after a fresh exact-ID probe; no exact match triggers bounded full strict-mode reprepare, while an effect-free upsert conflict also fully reprepares. Any earlier effect or ambiguity returns `RecoveryRequired` with the sidecar retained. Strict read-set conflicts return `ReadSetChanged`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary. -- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB), watcher-only durable acknowledgement, conservative replay/fold-only recovery, and one recovery-v11 strict exact-`id` fold whose sole `__manifest` CAS advances the table pointer, lifecycle witness/epoch floor, and graph lineage together. Cheap raw caller row/byte bounds reject obviously over-cap input before recovery I/O; raw-fit input then receives exact post-tombstone validation at that same pre-recovery boundary. After any recovery/authority prelude, every put follows exact charge → shared admission → same-key input queue → worker mode; the permit transfers into the resident generation. Cold replay installs exact accounting plus a fold-only marker before releasing the opener queue; previously charged callers may create a recorded transient overlap above the nominal 32-MiB root budget while they drain, but new charge is refused. A cold fold reserves the full generation/resident/pending envelope before its owned opener and retains it with exclusive authority across the original seal deadline. Shared admission is retained through put/watcher; exclusive admission spans seal, drain proof, table effect, and manifest publication. Post-invocation ambiguity is typed `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The core remains crate-private; its only callable integration seam is a feature-gated, doc-hidden test method. No schema, SDK, CLI, HTTP, Cedar, or OpenAPI streaming contract ships until B2 closes attribution, reclamation, correction, retry sequencing, and persistent operator-control gates. +- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB), watcher-only durable acknowledgement, conservative replay/fold-only recovery, and one recovery-v11 strict exact-`id` fold whose sole `__manifest` CAS advances the table pointer, lifecycle witness/epoch floor, and graph lineage together. Cheap raw caller row/byte bounds reject obviously over-cap input before recovery I/O; raw-fit input then receives exact post-tombstone validation at that same pre-recovery boundary. After any recovery/authority prelude, every put follows exact charge → shared admission → same-key input queue → worker mode; the permit transfers into the resident generation. Cold replay installs exact accounting plus a fold-only marker before releasing the opener queue; previously charged callers may create a recorded transient overlap above the nominal 32-MiB root budget while they drain, but new charge is refused. A cold fold reserves the full generation/resident/pending envelope before its owned opener and retains it with exclusive authority across the original seal deadline. Shared admission is retained through put/watcher; exclusive admission spans seal, drain proof, table effect, and manifest publication. Post-invocation ambiguity is typed `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The core remains crate-private; its only callable integration seam is a feature-gated, doc-hidden test method. B2-0 specifies explicit enrollment, compare-and-chain tokens, trusted attribution, a manifest-selected graph-global current-token participant, monotonic lifecycle revisions with bounded management receipts, strict correction, Lance-owned reclamation with an enforced retained-storage admission watermark, and a manifest-authoritative graph-global `GraphHistoryBudget` checked by every manifest publisher. That budget reserves each publication's source-bounded physical-growth envelope and preserves dynamic per-stream correction/quiesce/`SEALED` closure reserves; none of B2 is implemented. Two RC.1 guards prove generic cleanup does not reclaim `_mem_wal` and raw deletion of the successor's empty WAL fence sentinel is unsafe, so OmniGraph never deletes those paths. Public B2 additionally waits for durable reclaim attempt/receipt recovery, bounded history checkpointing, strong inventory or Lance-owned accounting, and source-derived enforced physical-growth reservations. No schema-v9, SDK, CLI, HTTP, Cedar, or OpenAPI streaming contract ships until the Lance patch and all B2 evidence gates pass. - **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster `, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart. - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table). @@ -280,7 +280,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift | | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: they publish only logical data/schema effects, leaving physical intent pending. `ensure_indices` first runs the roll-forward-only recovery barrier before base capture or planning, then materializes every declared-but-missing artifact for one table through one staged mixed CreateIndex transaction under its identity-bearing recovery-v9 authority/lineage/delta envelope; it continues to report untrainable Vector columns as pending. `Armed` is rollback-only, `EffectsConfirmed` rolls forward only while captured authority holds, and first-touch refs carry exact identity. `optimize` separately restores fragment coverage through its bounded identity-bearing v9 maintenance envelope. | | Strict insert / upsert ingestion | ✅ transaction conflict filters + uncommitted fragment staging | Internal schema v6 introduced the explicit logical mode and v8 preserves it. General strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint the same optional certificate only from its completed effect statistics. Mutation/Load remains one transaction per table, capped before arm at 8,192 rows / 32 MiB. BranchMerge's proven all-new route accepts only a complete certificate chain plus final source/target native-incarnation checks; it stages bounded fragments with `InsertBuilder`, commits them as exact-`id` filtered `Update` transactions, and performs zero target preflights, target merge joins, or committed Appends. Missing or malformed proof falls back to the general ordered diff. Raw Lance writers are outside the supported graph-writer topology, and the certificate is an internal, non-cryptographic capability. The final five-pair production gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes). | -| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v8 carries exact identity-keyed physical binding, current-HEAD witness, `OPEN`/`DRAINING`/`SEALED` state, per-shard epoch floor, and stream-config v2. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. Private B1 adds one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB; watcher-only durable ack; detached ownership across cancellation/timeouts; fold-only replay using the pinned BatchStore watermark bridge; explicit seal/drain proof; and recovery-v11 exact-transaction fold followed by one graph-visibility CAS. Root-shared admission and lifecycle guards exclude competing table effects; a claimed/warm ambiguity retires and aborts before reuse. B1 performs no generation GC and exposes no fresh reads. There is still no public stream schema, enrollment, put/fold/status API, Cedar action, CLI, HTTP route, or OpenAPI contract; those remain B2. | +| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v8 carries exact identity-keyed physical binding, current-HEAD witness, `OPEN`/`DRAINING`/`SEALED` state, per-shard epoch floor, and stream-config v2. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. Private B1 adds one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB; watcher-only durable ack; detached ownership across cancellation/timeouts; fold-only replay using the pinned BatchStore watermark bridge; explicit seal/drain proof; and recovery-v11 exact-transaction fold followed by one graph-visibility CAS. Root-shared admission and lifecycle guards exclude competing table effects; a claimed/warm ambiguity retires and aborts before reuse. B1 performs no generation GC and exposes no fresh reads. B2-0 specifies the inactive schema-v9/config-v3/state-v2/recovery-v12 explicit-enrollment, token, attribution, lifecycle-revision/management-receipt, correction, Lance-owned reclamation, per-binding admission-watermark, and graph-global `GraphHistoryBudget` contracts. The latter is checked by every manifest writer and keeps independent per-stream closure reserves. There is still no public stream schema, enrollment, put/fold/status API, Cedar action, CLI, HTTP route, or OpenAPI contract; those remain B2. | | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema | | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops | | Hybrid ranking | — | `rrf(...)` Reciprocal Rank Fusion in one runtime | diff --git a/WAL-thinking.md b/WAL-thinking.md index ff85733e..12e26ffa 100644 --- a/WAL-thinking.md +++ b/WAL-thinking.md @@ -23,8 +23,11 @@ or one flushed-unmerged generation fold-only, and publish one exact fold at the successor generation can put. This is implementation/evidence machinery, not a product surface. There is still no `@stream`, public enrollment or row admission, SDK/HTTP/CLI/OpenAPI route, operator drain/resume workflow, or fresh -read. RFC-026 remains Draft; Phase B2 is the later public strict activation, -and v7/config-v1 is never adopted in place. +read. RFC-026 remains Draft. Phase B2-0 now specifies the compare-and-chain +token, trusted attribution, persistent lifecycle/correction, and Lance-owned +reclamation contracts, but activates none of them. Phase B2 is the later +implementation and public strict activation, and v7/config-v1 is never adopted +in place. --- @@ -126,10 +129,15 @@ but the watcher outcome is lost, OmniGraph returns typed `AckUnknown`; replay preserves possible durable residue but cannot attribute it to that caller attempt. The attempt remains ambiguous and OmniGraph must never claim “not durable.” Cancellation cannot abandon the worker once invocation starts. -Retrying the same key/payload preserves one-row cardinality, not general -ordering: ambiguous `X(id)`, then durable `Y(id)`, then retry `X(id)` can make -the stale `X` newest again. Public B2 therefore needs an explicit per-key -sequencing or idempotency contract; replay/LWW alone is insufficient. +The current private B1 seam has only cardinality-level same-key retry behavior: +ambiguous `X(id)`, then durable `Y(id)`, then retry `X(id)` can make stale `X` +newest again. B2-0 closes that public-contract hole with an explicit +compare-and-chain token. A caller-stable `write_id` plus the opaque +`predecessor_token` returned by the server derives one stable successor token; +the predecessor must equal the complete current token before Lance is called. +An exact current retry is `already_durable`; a retry behind newer `Y` is +`StreamSequenceConflict`. Replay/LWW realizes the already-admitted chain but is +not itself the sequencing authority. The warm, uncontended WAL persistence path is normally one conditional object store PUT per submitted Lance batch. It is not an unconditional one-call @@ -146,7 +154,8 @@ horizontal writer scaling comes later from deterministic key-based sharding. The strict one-generation fold is implemented privately in B1. Phase A established the binding, recovery, and exclusion preconditions it consumes; -Phase B2 still owns the public fold and operator lifecycle. +B2-0 specifies the token participant and persistent operator lifecycle that +Phase B2 must implement before exposing the public fold. B1 deliberately chooses a smaller contract: one writer owns exactly one generation whose total admitted input is at most 8,192 rows and 32 MiB. It @@ -193,6 +202,16 @@ graph-visible, its recovery sidecar is resolved, maintained indexes have caught up, retained versions no longer need the data, no fresh-read guard pins it, and writer fencing remains safe. +Stock RC.1 cannot perform that reclamation safely. Generic table-version +cleanup leaves `_mem_wal` untouched, and deleting the successor's empty WAL +fence sentinel can let a stale writer report a later WAL PUT as durable because +RC.1 lacks a post-success epoch check. OmniGraph therefore never deletes raw +MemWAL paths. The next substrate slice is Lance-owned durable +inspect/plan/execute with attempt/receipt recovery, bounded history +checkpointing, an enforceable physical-growth reservation, and the +post-success fence check, authored by us and pinned to an exact reviewed fork +commit without waiting for an upstream release. + ### Side by side | | Interactive request | Stream row + later fold | @@ -201,7 +220,7 @@ writer fencing remains safe. | Graph visibility | Before success is returned | After a successful fold; lag depends on triggers, validation, bounds, and fold health | | Pre-ack checks | Complete request validation | Checks that need no graph/base-table read | | Deferred outcome | None after success | Strict fold may block; configured dead-letter may record a reject | -| Same-key behavior | Serialize, retry, or fail loudly | Last-write-wins only when every occurrence of a key maps to the same shard | +| Same-key behavior | Serialize, retry, or fail loudly | Public B2 compare-and-chain admission; Lance LWW only realizes the accepted chain on one shard | | Versions | One graph commit per successful request, which may contain many rows | One graph commit per successful fold, plus one Lance version per affected participant | | Crash story | RFC-022 recovery resolves any table-effect/publication gap | WAL replay preserves durable input; RFC-022 recovery still resolves fold effects/publication | | Best fit | Edits requiring immediate graph visibility and synchronous rejection | High-rate feeds that accept delayed visibility and deferred state-dependent disposition | @@ -363,7 +382,7 @@ These are real substrate capabilities, not a turnkey graph-streaming product: | **`merged_generations`** | Merge progress updated atomically with a base-table merge. It supplies the marker needed for idempotent per-table folding; RFC-022 still owns graph publication and partial-effect recovery. | | **WAL replay** | Durable entries are replayed to reconstruct the MemTable under the next valid claimant. The guarantee depends on respecting fencing, lifecycle, compatibility, and safe GC. | | **LSM merging reads** | A scanner over the base table, selected flushed generations not yet safely replaceable by the base read plan, and optionally same-process active/frozen MemTables. OmniGraph must build and retain a coherent graph-level fresh-read cut. It does not query raw WAL files as a normal read source. | -| **GC eligibility rules** | Upstream specifies when generations may be obsolete and warns that deleting WAL files can weaken fencing. RC.1 does not provide OmniGraph's graph-aware MemWAL garbage collector. | +| **GC eligibility rules** | Upstream specifies when generations may be obsolete and warns that deleting WAL files can weaken fencing. Stock RC.1 does not provide a safe graph-aware MemWAL collector: generic cleanup ignores `_mem_wal`, and there is no post-success WAL epoch check. B2 consumes a Lance-owned opaque reclamation primitive; OmniGraph never deletes raw paths. | | **Staged merge-insert** | `execute_uncommitted` plus atomic `merged_generations` fits the RFC-022 staged shape. There is no one-call MemWAL fold API. | | **Shared sessions/store parameters in RC.1** | Writer-created generations and base-backed scanner paths reuse the base dataset's access context; fresh-only construction still receives that context from its caller. OmniGraph owns long-lived writer/session lifecycle and scanner integration. | | **Key-based sharding** *(later)* | Horizontal scale when every occurrence of one key deterministically maps to one shard. | @@ -374,9 +393,10 @@ adapter, and validation components. “Reusable” does not mean “unchanged” Phase A added stream lifecycle authority and enrollment recovery. Private B1 now adds one bounded admission worker, watcher-only acknowledgement, replay and flushed-generation classification, and one recovery-owned fold. Later phases -still need the public caller, reject/audit participants, durable authenticated -contributor attribution, persistent operator lifecycle, fresh cuts, and cleanup -coordination. +still need implementation of B2-0's token/attribution, persistent operator +lifecycle/correction, Lance-owned reclamation and enforced admission watermark; +then the public +caller, reject participants, fresh cuts, and later cleanup coordination. ### What Phase A added @@ -408,19 +428,20 @@ workflow is implemented. | Responsibility | Required contract | |---|---| -| **Format capability and refusal** | **Phase A historical foundation:** v7 stamp, strict v6↔v7 refusal, export/init/load rebuild, exact lifecycle validation, and uncovered-partial-format refusal. **Private B1 current format:** schema v8, stream-config v2, and recovery-v11 `StreamFold`; genuine v7↔v8 old/new-binary refusal and rebuild evidence remains an acceptance gate. | -| **`@stream` intent and enrollment** | **Foundation only:** Phase A binds stable table/incarnation identity, location/main ref, never-reused enrollment ID, one empty shard, fixed configuration, and the mutable current-HEAD witness under recovery. B1 is implemented but remains private. B2 adds parser/SchemaApply intent and production first use; rebind remains later. | -| **Public surface** | **Phase B2:** `POST /graphs/{graph_id}/streams/{type_name}/ingest`, minimum status/fold/quiesce/resume/abort-drain/rebuild controls, `omnigraph stream …` commands, Cedar, and OpenAPI parity. Existing `/ingest` remains the deprecated load alias. | +| **Format capability and refusal** | **Phase A historical foundation:** v7 stamp, strict v6↔v7 refusal, export/init/load rebuild, exact lifecycle validation, and uncovered-partial-format refusal. **Private B1 current format:** schema v8, stream-config v2, and recovery-v11 `StreamFold`; genuine v7↔v8 old/new-binary refusal and rebuild evidence is green. **Specified B2 format:** schema v9, config-v3, state-v2, and recovery-v12 add hidden token/attribution state plus the manifest-authoritative graph-global `GraphHistoryBudget`; they remain inactive until genuine v8↔v9 refusal/rebuild and budget-init/refusal evidence pass. | +| **`@stream` intent and enrollment** | **Foundation only:** Phase A binds stable table/incarnation identity, location/main ref, never-reused enrollment ID, one empty shard, fixed configuration, and the mutable current-HEAD witness under recovery. B1 is implemented but remains private. B2 makes `@stream` declaration leave the type `UNENROLLED`; an explicit request-idempotent enroll operation creates the logical stream incarnation and physical binding. Rebind remains later. | +| **Public surface** | **Phase B2, after private evidence:** explicit enroll, `POST /graphs/{graph_id}/streams/{type_name}/ingest`, minimum status/block-inspection/fold/quiesce/resume/abort-drain/correct/rebuild-preflight controls, `omnigraph stream …` commands, Cedar, and OpenAPI parity. Every mutating management call after enrollment compares a lifecycle revision and durably returns its bounded terminal receipt. Existing `/ingest` remains the deprecated load alias. | | **Writer registry and routing** | **Phase B1 implemented privately:** one root-scoped, cross-handle registry owns the full binding and reuses the common table-identity plus resolved-physical-ref admission key; one serialized owner serves the initial `(table, main)` profile. One no-rollover generation is capped in total at 8,192 rows / 32 MiB. Puts use exact charge → shared admission → same-key input queue → worker-mode inspection. Claim/replay starts under the shared admission lease. Empty reopen may admit; non-empty replay and one flushed-unmerged generation are fold-only, with exact accounting and the refusal marker installed before the opener releases its queue. Already-charged callers can overlap recovered replay transiently; the ledger records that over-cap overlap while refusing new charge. The exclusive fold validates replayed rows and uses the pinned public BatchStore watermark bridge before reseal. A cold fold reserves the full generation/resident/pending envelope before its owned opener and retains it with exclusive authority across the original seal deadline. Retirement stops puts before public `abort`; `close` is not durability evidence. The private evidence limits are fixed at one resident writer per root/table, a nominal 32-MiB root Arrow admission budget, 32 in-flight calls, four pending generations, 30-second put, 60-second seal/abort, and 60-second idle eviction. They are not public defaults: B2 must requalify public/configurable or multi-resident limits with matched RSS/latency evidence. | | **Durability batching** | **Phase B1 implemented privately:** one admitted call is one non-empty, already-normalized physical `RecordBatch` and one Lance put. The worker owns the final check, invocation, and watcher; only watcher success acknowledges. Anything ambiguous after invocation is typed `AckUnknown`; replay preserves possible residue but never resolves that attempt. Automatic rollover is disabled and the writer retires before a successor-generation put. There is no hidden group-commit policy until an instrument justifies one. | | **Pre-ack validation** | **Phase B1 implemented privately:** apply every rule that needs no graph/base-table read before WAL persistence. The private seam consumes physical vectors supplied in the normalized batch; it neither calls an external embedding provider nor invents unspecified fold-derived fields. | | **Fold adapter** | **Phase B1 implemented privately:** capture one exact stream binding and post-drain shard snapshot, independently prove empty frozen refs plus the exact authoritative generation/cursor (RC.1's drain waiter alone is insufficient), run base-dependent validation, require the LWW output to fit 8,192 rows / 32 MiB, stage one merge-insert with `merged_generations`, and publish the exact table/lifecycle/lineage outcome through recovery schema v11 plus one `__manifest` CAS. Already-normalized physical vectors pass through unchanged. Seal/drain/abort stay background-owned across caller deadlines; recognized unreferenced generation subtrees are retained orphans until proved GC. | -| **Strict and dead-letter disposition** | **B1/B2:** strict only; a permanent deferred-validation failure leaves durable unmerged input and blocks progress loudly. B1 has no correction lane. **Phase C:** dead letters only after restart-stable reject identity and retention are defined. | -| **Policy, lineage, and audit** | B1 records only fixed mechanism lineage and has no public caller. B2 must durably preserve authenticated contributor attribution before public admission. Phase C consumes that evidence for rejects. | -| **Quiescence and rebind** | B2 must provide persistent minimum status/fold/quiesce/resume/abort-drain/rebuild plus a bounded correction path before public activation. Resume rechecks the bounded no-named-branch topology; an incompatible branch operation leaves the stream `SEALED`. `SEALED` permits export/rebuild, not in-place maintenance. Phase D integrates automatic operation drain and physical rebind. | +| **Strict and dead-letter disposition** | **B1/B2:** strict only; a permanent deferred-validation failure leaves durable unmerged input and blocks progress loudly. B1 has no correction lane. B2-0 specifies bounded `REPLACE`/`WITHDRAW` correction over the immutable blocked cut, without a second generation or silent drop. **Phase C:** dead letters only after restart-stable reject identity and retention are defined. | +| **Policy, lineage, and audit** | B1 records only fixed mechanism lineage and has no public caller. B2-0 stores trusted contributor/write metadata with the row, publishes current token state with the base version, and commits a fixed winner summary; it promises durable attribution for visible winners/current withdrawals, not unbounded audit retention for every superseded acknowledgement. Phase C consumes that evidence for rejects. | +| **Quiescence and rebind** | B2-0 specifies durable `OPEN -> DRAINING -> SEALED -> OPEN`, strictly monotonic lifecycle revisions, bounded complete terminal management receipts, authoritative status, roll-forward-only resume/abort-drain recovery, rebuild preflight, and bounded correction. B2 must implement and prove them before public activation. Resume rechecks the bounded no-named-branch topology; an incompatible branch operation leaves the stream `SEALED`. `SEALED` permits export/rebuild, not in-place maintenance. Phase D integrates automatic operation drain and physical rebind. | | **Fresh reads** *(later)* | Explicit committed/fresh IR mode, exact base-plus-MemWAL cut, merged-generation exclusion, retention guards, and documented lack of cross-table atomicity. | -| **Cleanup** | B1 performs no WAL/generation GC. B2 production activation requires bounded reclamation plus a hard retained-storage stop. Later graph-aware GC remains integrated with visibility, recovery, time travel, indexes, fresh-read guards, and fencing safety. | -| **Evidence and operations** | **B1 green privately:** surface guards, genuine v7↔v8 refusal/rebuild, all 24 cells in the graph-level B1 behavior/crash/race suite, qualified local cost/PK-index/RSS evidence, and configured-RustFS cost evidence pass. The instrument keeps warm ack, retained-WAL replay, selected-generation data scan, accumulated retained shard-generation metadata, whole-process RSS, and the non-flat graph-manifest publication-history term separate. B2 still adds public metrics, status, API tests, CLI parity, and operator evidence. | +| **Cleanup** | B1 performs no WAL/generation GC. B2-0 proves stock RC.1 generic cleanup is not the answer and specifies Lance-owned durable inspect/plan/execute, post-success fencing, whole-cut eligibility, strong inventory or durable accounting, bounded history checkpointing, and an enforceable retained-object/byte admission watermark. B2 production activation waits for that patch and local/RustFS crash/bound evidence. Later graph-aware GC remains integrated with visibility, recovery, time travel, indexes, fresh-read guards, and fencing safety. | +| **Graph-manifest lifetime** | The per-binding MemWAL watermark cannot bound shared `__manifest` history. B2-0 therefore specifies one graph-global `GraphHistoryBudget` initialized with schema v9 and checked by every manifest writer. Each publication uses a reserve-first, source-bounded physical-growth envelope; the initial profile holds that global gate from sidecar arm through effect/CAS/finalization, and pending recovery charges settle exactly once. Dynamic per-stream closure reserves cannot be borrowed by ordinary work or another stream, and `GraphRebuildRequired` stops ordinary work at the shared floor while correction/quiesce/`SEALED` rebuild remains funded. The canonical order is relevant stream admission → graph history → schema → main → token → tables. This contract is specified and inactive. | +| **Evidence and operations** | **B1 green privately:** surface guards, genuine v7↔v8 refusal/rebuild, all 24 cells in the graph-level B1 behavior/crash/race suite, qualified local cost/PK-index/RSS evidence, and configured-RustFS cost evidence pass. The instrument keeps warm ack, retained-WAL replay, selected-generation data scan, accumulated retained shard-generation metadata, whole-process RSS, and the non-flat graph-manifest publication-history term separate. **B2-0:** two checked-in guards prove generic cleanup non-ownership and the deleted-successor-sentinel stale-writer hazard. B2 still needs the positive Lance patch matrix, source-derived/enforced growth reservation plus validating instrument, public metrics/status, API tests, CLI parity, and operator evidence. | The table intentionally omits “small/medium” estimates. Atomic rejection, quiescence, fresh-read retention, and GC are correctness protocols; size them @@ -457,6 +478,51 @@ upstream shape still gates overlapping-process topology. Bounded Phase A and the private, evidence-green B1 core are now implemented; B2's product and operational gates remain before public row activation. +### The RC.1 reclamation gap, stated precisely + +RC.1 exposes evidence-level raw listing plus manifest reads, but not a complete +Lance-owned classified inventory. Its generic +`cleanup_old_versions` walks ordinary Lance versions and leaves `_mem_wal` +objects untouched. Its public MemWAL surface has no safe delete/GC operation. +Raw object deletion is not an acceptable substitute: a checked-in adversarial +guard decodes and removes the successor's empty epoch-2 WAL fence sentinel and +demonstrates that the stale writer can still complete its WAL PUT and receive +watcher success even though an explicit check sees `PeerClaimedEpoch`. Deleting +the sentinel weakened the fence. + +The no-wait path is to author the missing capability in Lance ourselves: an +opaque inspect/plan/execute contract with exact shard/base/history witnesses, +durable attempt/receipt recovery, whole-cut cursor proof, manifest-version and +writer-epoch advancement before deletion, bounded history checkpointing, +conservative orphan classification, an enforceable source-derived growth +reservation, and a post-success WAL epoch check. Exact inventory requires +strong HEAD/GET/LIST visibility after PUT/DELETE plus incomplete-multipart +accounting/abort, or Lance-owned durable accounting. Every epoch claim is +sentinel-first and manifest-named; ordinary open/quiesce/resume/checkpoint +claims preserve the prior replay cursor, while only a proved no-data-tail +whole-cut reclaim advances it. Enrollment creates a genesis bootstrap before a +new MemWAL details kind that stock RC.1 rejects. Checkpointing bounds both +ordinary-claim and reclaim receipts, and admission reserves durable +materialization attempts plus control headroom. One bootstrap-selected reserve- +first ledger serializes the physical budget per binding before WAL/upload +effects and settles only from exact inventory. Versioned/soft-delete/Object- +Lock stores are refused unless every retained byte is countable and eligible +versions are permanently removable. We open that patch upstream but pin +OmniGraph to the exact reviewed fork commit immediately. This respects the +substrate boundary without making an upstream merge or release a calendar +dependency. + +That is a MemWAL bound, not a whole-root retention claim. Base-table, token- +table, and shared manifest histories still grow while in-place maintenance is +refused. B2-0 therefore adds a separate manifest-authoritative, graph-global +`GraphHistoryBudget`. Every manifest writer reserves its exact publication and +source-bounded physical-growth envelope before effect. Dynamic per-stream +closure reserves cannot be consumed by ordinary work or by another stream, so +the shared floor refuses new ordinary work with `GraphRebuildRequired` while +preserving enough operations to fold, correct, quiesce, and rebuild each stream +from `SEALED`. Truly indefinite in-place streaming remains a later maintenance +design. + ### Gate E0: a no-wait decision, not activation RFC-026 now has a production-neutral Gate E0. The question is narrower than @@ -553,8 +619,10 @@ fixed binding publication, admission exclusion, restart, and foreign-state refusal. Private B1 now implements first put, durability/lost-response semantics, replay/fold-only reopening, and strict folding. Its genuine cross-version/rebuild, the 24-cell graph-level B1 suite, local cost/RSS, and -configured-RustFS cost gates are green. B2's separate product and operational gates must close before -it may expose a single row acknowledgement. +configured-RustFS cost gates are green. B2-0 now specifies the separate token, +attribution, lifecycle/correction, and reclamation contracts. Their +implementation and evidence gates must close before B2 may expose a single row +acknowledgement. ### Bottom line @@ -580,18 +648,30 @@ The plan is: manifest state, and fold through one recovery-v11 keyed transaction and one exact manifest publication; keep its now-green cross-version, crash, and cost/RSS instruments as regression gates; -4. activate Phase B2 only after durable contributor attribution, bounded - reclamation, strict correction, same-key retry sequencing, and persistent - status/quiesce/resume/abort-drain/rebuild are accepted and implemented; -5. design and measure any later group-commit policy before claiming a +4. retain B2-0's specified compare-and-chain token, trusted attribution, + manifest-selected token participant, protocol-v2 lifecycle, and bounded + `REPLACE`/`WITHDRAW` correction as the implementation contract; +5. author and review the Lance-owned reclamation primitive and post-success + fence check, durable attempt/receipt and bounded-history machinery, pin the + exact fork commit without waiting for release, and pass its local/RustFS + crash matrix plus source-derived/enforced growth reservation and validating + physical retained-storage instrument; +6. implement schema v9/config-v3/state-v2/recovery-v12 privately and keep every + product surface absent until the token, lifecycle, correction, + enforced-watermark, cross-version, and crash matrices are green; +7. add schema/SDK/HTTP/CLI/Cedar/OpenAPI surfaces only after that private core + passes; then activate Phase B2; +8. design and measure any later group-commit policy before claiming a performance multiplier; -6. pursue upstream replay-watermark and drain-error fixes plus the exact +9. pursue upstream replay-watermark and drain-error fixes plus the exact receipt/seal API in parallel; remove the RC.1 bridge when the first pair lands, and use the latter to simplify enrollment and broaden topology. We tested the narrower support contract instead of waiting on the upstream calendar. Phase A's v7 foundation and private B1's current v8/config-v2/v11 -worker/fold core are implemented and evidence-green. RFC-026 remains Draft: private tests can put, -receive watcher-only durability acknowledgement, replay/fold-only, and publish -one fold, but no production enrollment, acknowledgement, fold, operator, or -public stream path exists. +worker/fold core are implemented and evidence-green. RFC-026 remains Draft: +private tests can put, receive watcher-only durability acknowledgement, +replay/fold-only, and publish one fold, and the B2-0 design plus two negative +RC.1 reclamation guards are checked in. No schema-v9 token/lifecycle/ +reclamation implementation, production enrollment, acknowledgement, fold, +operator, or public stream path exists. diff --git a/crates/omnigraph/tests/lance_surface_guards.rs b/crates/omnigraph/tests/lance_surface_guards.rs index 1fbda6d0..ed53f9a3 100644 --- a/crates/omnigraph/tests/lance_surface_guards.rs +++ b/crates/omnigraph/tests/lance_surface_guards.rs @@ -25,22 +25,23 @@ use std::collections::HashMap; use std::fmt; -use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; -use async_trait::async_trait; use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, StringArray}; use arrow_schema::{DataType, Field, Schema}; +use async_trait::async_trait; +use futures::TryStreamExt; use futures::stream::BoxStream; use lance::Dataset; use lance::dataset::builder::DatasetBuilder; use lance::dataset::cleanup::{CleanupPolicy, cleanup_old_versions}; -use lance::dataset::mem_wal::{ - BatchDurableWatcher, DatasetMemWalExt, InitializeMemWalBuilder, ShardManifestStore, - LsmScanner, ShardSnapshot, ShardWriter, ShardWriterConfig, WriteResult, -}; use lance::dataset::mem_wal::scanner::InMemoryMemTables; use lance::dataset::mem_wal::write::{BatchStore, StoredBatch}; +use lance::dataset::mem_wal::{ + BatchDurableWatcher, DatasetMemWalExt, InitializeMemWalBuilder, LsmScanner, ShardManifestStore, + ShardSnapshot, ShardWriter, ShardWriterConfig, WalTailer, WriteResult, +}; use lance::dataset::optimize::{CompactionOptions, compact_files}; use lance::dataset::refs::BranchIdentifier; use lance::dataset::transaction::{Operation, Transaction}; @@ -54,6 +55,7 @@ use lance::dataset::{ use lance::datatypes::LANCE_UNENFORCED_PRIMARY_KEY; use lance::index::DatasetIndexExt; use lance::session::Session; +use lance_core::error::FenceReason; use lance_core::{ ROW_ADDR, ROW_CREATED_AT_VERSION, ROW_ID, ROW_LAST_UPDATED_AT_VERSION, ROW_OFFSET, is_system_column, @@ -72,7 +74,8 @@ use lance_table::system_index::mem_wal::{ use object_store::path::Path; use object_store::{ CopyOptions, GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore, - PutMultipartOptions, PutOptions, PutPayload, PutResult, Result as ObjectStoreResult, + ObjectStoreExt, PutMultipartOptions, PutOptions, PutPayload, PutResult, + Result as ObjectStoreResult, }; use omnigraph_compiler::schema::parser::parse_schema; @@ -289,6 +292,26 @@ fn pk_full_row(dataset: &Dataset, id: &str, value: i32) -> RecordBatch { .unwrap() } +/// Append one row using `fresh_pk_dataset`'s three-column PK schema. Its nullable +/// `note` column distinguishes this from the two-column `append_guard_row`, while +/// V2_2 and stable row IDs preserve the production-table write shape. +async fn append_pk_guard_row(dataset: &mut Dataset, id: &str, value: i32) { + let batch = pk_full_row(dataset, id, value); + let schema = batch.schema(); + dataset + .append( + RecordBatchIterator::new(vec![Ok(batch)], schema), + Some(WriteParams { + mode: WriteMode::Append, + enable_stable_row_ids: true, + data_storage_version: Some(LanceFileVersion::V2_2), + ..Default::default() + }), + ) + .await + .unwrap(); +} + async fn stage_pk_merge( dataset: Arc, batch: RecordBatch, @@ -1283,6 +1306,205 @@ async fn mem_wal_watcher_watermark_is_not_generation_scoped_on_pinned_lance() { writer.abort().await.unwrap(); } +/// RC.1's ordinary dataset cleanup does not own `_mem_wal`. Newer base-table +/// versions therefore are not, by themselves, a reclamation primitive for the +/// WAL, flushed generation, and shard-manifest state in this fixture. B2 counts +/// the complete namespace—including separately classified orphans—toward its +/// enforced retained-storage admission watermark and waits for a Lance-owned +/// inspect/plan/execute API instead of routing these objects through generic +/// version cleanup. +#[tokio::test] +async fn cleanup_old_versions_does_not_reclaim_mem_wal_objects() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().join("memwal-generic-cleanup.lance"); + let mut dataset = fresh_pk_dataset(uri.to_str().unwrap()).await; + dataset + .initialize_mem_wal() + .unsharded() + .execute() + .await + .unwrap(); + + let shard_id = ShardId::new_v4(); + let writer = dataset + .mem_wal_writer( + shard_id, + ShardWriterConfig::new(shard_id) + .with_shard_spec_id(1) + .with_durable_write(true), + ) + .await + .unwrap(); + let (_, watcher) = writer + .put_no_wait(vec![pk_full_row(&dataset, "memwal-retained", 7)]) + .await + .unwrap(); + watcher + .expect("durable MemWAL writer must return a watcher") + .wait() + .await + .unwrap(); + writer.force_seal_active().await.unwrap(); + writer.wait_for_flush_drain().await.unwrap(); + writer.abort().await.unwrap(); + + let object_store = dataset.object_store(None).await.unwrap(); + let mem_wal_root = dataset.branch_location().path.join("_mem_wal"); + // Generic cleanup owns none of this state. Compare bytes as well as paths so + // an unexpected rewrite fails just as loudly as object creation or deletion. + let inventory = || async { + let objects = object_store + .inner + .list(Some(&mem_wal_root)) + .try_collect::>() + .await + .unwrap(); + let mut objects_with_bytes = Vec::with_capacity(objects.len()); + for object in objects { + let bytes = object_store + .inner + .get(&object.location) + .await + .unwrap() + .bytes() + .await + .unwrap(); + objects_with_bytes.push((object.location.to_string(), bytes.to_vec())); + } + let mut objects = objects_with_bytes; + objects.sort(); + objects + }; + let before = inventory().await; + assert!( + before.iter().any(|(path, _)| path.contains("/wal/")), + "fixture must contain durable WAL state" + ); + assert!( + before.iter().any(|(path, _)| path.contains("_gen_")), + "fixture must contain a flushed generation" + ); + assert!( + before.iter().any(|(path, _)| path.contains("/manifest/")), + "fixture must contain shard-manifest state" + ); + + append_pk_guard_row(&mut dataset, "cleanup-v3", 30).await; + append_pk_guard_row(&mut dataset, "cleanup-v4", 40).await; + let current = dataset.version().version; + let removed = cleanup_old_versions( + &dataset, + CleanupPolicy { + before_version: Some(current), + ..Default::default() + }, + ) + .await + .unwrap(); + assert!( + removed.old_versions > 0, + "control must actually remove ordinary Lance versions" + ); + assert_eq!( + inventory().await, + before, + "generic cleanup created, deleted, or rewrote MemWAL state; re-audit RFC-026 reclamation" + ); +} + +/// Lance's MemWAL specification warns that deleting WAL files can weaken +/// epoch fencing. Pinned RC.1 checks the shard epoch after a PUT collision or +/// error, but not after a successful PUT. If GC removes the successor's empty +/// WAL fence sentinel, the stale predecessor can therefore recreate that slot +/// and receive a successful durability watcher even though a fresh fence check +/// rejects it. +/// +/// This is deliberately a production-neutral negative guard: only the test +/// deletes a WAL object. When the Lance-owned post-success fence check lands, +/// invert this assertion and make the patched failure the positive guard. +#[tokio::test] +async fn mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().join("memwal-deleted-fence-slot.lance"); + let mut dataset = fresh_pk_dataset(uri.to_str().unwrap()).await; + dataset + .initialize_mem_wal() + .unsharded() + .execute() + .await + .unwrap(); + + let shard_id = ShardId::new_v4(); + let config = || { + ShardWriterConfig::new(shard_id) + .with_shard_spec_id(1) + .with_durable_write(true) + }; + let stale = dataset.mem_wal_writer(shard_id, config()).await.unwrap(); + assert_eq!(stale.epoch(), 1); + let successor = dataset.mem_wal_writer(shard_id, config()).await.unwrap(); + assert_eq!(successor.epoch(), 2); + + let object_store = dataset.object_store(None).await.unwrap(); + let wal_root = dataset + .branch_location() + .path + .join("_mem_wal") + .join(shard_id.as_hyphenated().to_string()) + .join("wal"); + let sentinel_objects = object_store + .inner + .list(Some(&wal_root)) + .try_collect::>() + .await + .unwrap(); + assert_eq!( + sentinel_objects.len(), + 1, + "epoch-2 claim must leave exactly one data-less fence sentinel" + ); + let tailer = WalTailer::new( + object_store.clone(), + dataset.branch_location().path.clone(), + shard_id, + ); + let sentinel_position = tailer.first_position().await.unwrap(); + let sentinel = tailer + .read_entry(sentinel_position) + .await + .unwrap() + .expect("the listed successor fence sentinel must decode"); + assert_eq!(sentinel.writer_epoch, successor.epoch()); + assert!( + sentinel.batches.is_empty(), + "the deleted WAL object must be the successor's empty fence sentinel" + ); + object_store + .inner + .delete(&sentinel_objects[0].location) + .await + .unwrap(); + + let (_, watcher) = stale + .put_no_wait(vec![pk_full_row(&dataset, "stale-after-gc", 9)]) + .await + .unwrap(); + watcher + .expect("durable MemWAL writer must return a watcher") + .wait() + .await + .expect("pinned RC.1 demonstrates the unsafe post-delete success gap"); + let error = stale.check_fenced().await.unwrap_err(); + assert_eq!( + error.fence_reason(), + Some(FenceReason::PeerClaimedEpoch), + "the stale predecessor must be rejected specifically because its peer claimed epoch 2" + ); + + stale.abort().await.unwrap(); + successor.abort().await.unwrap(); +} + // --- Guard 3: checkout_version + restore async chain ----------------------- // // `db/manifest/recovery.rs:505-522` chains `Dataset::open(...).await? diff --git a/docs/dev/canon.md b/docs/dev/canon.md index 7dc45b17..b138eaf2 100644 --- a/docs/dev/canon.md +++ b/docs/dev/canon.md @@ -3,7 +3,7 @@ **Audience:** maintainers, contributors, and coding agents — internal **Type:** narrative reference ("the book"), read top-to-bottom **Status:** living document -**Surveyed:** OmniGraph 0.8.1 development (`main`), with the RFC-026 Phase A foundation and private Phase B1 core present; Lance 9.0.0-rc.1 (git rev `cec0b7df`); internal manifest schema v8 +**Surveyed:** OmniGraph 0.8.1 development (`main`), with the RFC-026 Phase A foundation and private Phase B1 core present and Phase B2-0 specified; Lance 9.0.0-rc.1 (git rev `cec0b7df`); internal manifest schema v8 --- @@ -513,12 +513,17 @@ witness-update adapter existed. Native branch create/delete was the narrow exception: it did not advance a table HEAD, so it could proceed at `SEALED`; it still refused `OPEN` or `DRAINING`. Schema v8 preserves those exclusions and adds only private B1's exact one-generation fold as a guarded HEAD/witness -transition. RFC-026 B2 must add minimum explicit persistent -status/quiesce/resume/abort-drain/rebuild, durable contributor attribution, -bounded reclamation, correction, and same-key `AckUnknown` sequencing before -any public row surface; Phase D owns future automatic operation-scoped drain -and atomic witness advancement/rebind. Resume rechecks the bounded profile's -no-named-branch topology; an incompatible branch operation stays `SEALED`. +transition. RFC-026 B2-0 now specifies the next contract: compare-and-chain +write/predecessor tokens, trusted hidden row attribution, a manifest-selected +current-token participant, durable `OPEN -> DRAINING -> SEALED -> OPEN`, +bounded `REPLACE`/`WITHDRAW` correction, and Lance-owned reclamation with an +enforced storage admission watermark. None is implemented. Schema +v9/config-v3/state-v2/recovery-v12 +and every public row surface stay inactive until their crash, storage, and +cross-version evidence passes. Phase D owns future automatic operation-scoped +drain and atomic witness advancement/rebind. Resume rechecks the bounded +profile's no-named-branch topology; an incompatible branch operation stays +`SEALED`. --- @@ -730,6 +735,12 @@ Phase-A foundation and adds config-v2 private one-generation admission plus recovery-v11 fold. V6→v7 and v7→v8 each require export/init/load rebuilds; both genuine old/new-binary refusal/rebuild strands are green. +B2-0 assigns the next strict strand—schema v9, stream-config v3, state protocol +v2, and recovery-v12—but does not activate it. V9 is where trusted stream-row +metadata and the manifest-selected current-token dataset would enter the +format. It must refuse/rebuild against v8 in both directions; no serde default +or in-place adoption may manufacture token or contributor evidence. + --- ## The query engine @@ -924,7 +935,9 @@ on the proposer.) parallel log of data. RFC-026 Phase A already consumes Lance's MemWAL initializer for recoverable empty enrollment; private Phase B1 now consumes its bounded data/ack/replay/fold path rather than building one. Public B2 - remains inactive — **(draft RFC-026)**. + remains inactive. B2-0 requires the missing reclamation operation to live in + Lance and forbids OmniGraph from deleting raw `_mem_wal` paths — **(draft + RFC-026)**. - **Mixed constructive/destructive single mutations (D₂).** Keeps in-query read-your-writes unambiguous and each table at one commit per query; the alternative buys an in-query delete-view machine in the hot path. @@ -989,15 +1002,48 @@ physical rows/vectors without external embedding or unspecified derivation. RFC-026 remains Draft: there is no production enrollment, row put/ack/fold, drain/resume, or fresh-read surface. +**Specified, inactive contract:** RFC-026 Phase B2-0 defines explicit +enrollment, compare-and-chain tokens, trusted visible-winner/current-withdrawal +attribution, persistent lifecycle/correction with monotonic revisions and +bounded terminal management receipts, and an enforced retained-storage +admission watermark backed by Lance-owned reclamation. Two checked-in RC.1 +guards prove generic cleanup does not reclaim +`_mem_wal` and deleting the successor's empty WAL fence sentinel can let a +stale writer report WAL success. Those are no-go evidence, not a collector. +The next dependency is Lance-owned durable inspect/plan/execute plus +attempt/receipt recovery, bounded history checkpointing, an enforced physical +growth reservation, bounded materialization attempts/control headroom, and the +post-success epoch patch. Its claims are sentinel-first and manifest-named; +ordinary claims preserve the replay cursor, while only proved whole-cut reclaim +advances it. Enrollment creates the genesis bootstrap before a new MemWAL +details kind that stock RC.1 rejects. A per-binding reserve-first ledger owns +the physical budget; versioned/soft-delete/Object-Lock stores are refused unless +all retained bytes are exactly accountable and eligible versions removable. +OmniGraph will pin the patch at an exact reviewed fork commit without waiting +for release. + +That ledger bounds `_mem_wal`, not base/token or shared manifest history. B2-0 +therefore specifies a separate manifest-authoritative, graph-global +`GraphHistoryBudget`. Every manifest writer participates in its reserve-first +publication and source-bounded physical-growth envelope; pending recovery +charges remain durable. Initial B2 globally serializes this gate from final +reserve/sidecar arm through physical effects, manifest CAS, and charge +finalization. Dynamic per-stream closure reserves cannot be spent +by ordinary work or by another stream. Reaching the shared floor returns +`GraphRebuildRequired` before effect while correction, quiesce, and `SEALED` +export/rebuild remain funded. This keeps B2 a finite-lifetime root profile; +indefinite in-place maintenance remains later work. + **Roadmap / research:** durable table heads / heads format **(RFC-024, research-blocked after Gate A rejected the first physical access shape)**; checkpoint-pinned retention **(RFC-025, research-blocked after Gate 0 rejected the current compacted registry-access shape)**; public MemWAL row admission and later lifecycle/read phases **(RFC-026, draft; -private one-generation Phase B1 implemented; public strict Phase B2 only after -attribution, reclamation, correction, same-key retry sequencing, and persistent -operational escape)**; +private one-generation Phase B1 implemented; B2-0 contracts specified; public +strict Phase B2 only after the Lance patch plus token/attribution, +reclamation/enforced-watermark, correction, lifecycle, cross-version, and product-parity +evidence)**; lineage-based merge deltas **(RFC-027, research-blocked)**; background reconciler; planner statistics/cost model; policy pushdown; ingest-time embeddings; per-query @@ -1015,7 +1061,7 @@ resource budgets. | **R4: Manifest authority access grows with commit count.** Current-state resolution folds history; a selective index does not by itself bound the complete physical read. | Medium | `optimize` compacts internal tables (keeps periodically-optimized shipped paths flat where separately cost-gated). RFC-024 Gate A rejected durable heads because representative RustFS latest-manifest reads/bytes grow despite flat exact-BTREE row/range work. RFC-025 Gate 0 independently rejected checkpoint-registry activation: at local 10→1,000 on RC.1, uncompacted reconciled work and the eight-fragment tail stay flat, but compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; exact-show bytes and operation counts also grow. Both RFCs are research-blocked; v8 retains the journal fold and internal-table *cleanup* remains deferred behind the resurrection watermark. | | **R5: Schema identity corruption or alias/identity drift.** Internal schema v5 introduced stable IDs/incarnation as durable authority; v6, v7, and v8 preserve them. | Medium | Open/init validate the SchemaIR domain and exact bidirectional IR↔manifest identity/path/alias contract; every active recovery envelope carries the identity pair; zero, duplicate, missing, or mismatched identity fails closed. | | **R6: Merge cost at divergence** — full-width classification and history-growing manifest folds. | Medium | Coherent coordinator scans plus retained probe handles reduced the pre-slice measured depth-5/depth-80 baseline from 59/651 manifest reads to 40/410 and cap the common fast-forward route at three internal opens and three scans, but the uncompacted-history slope remains. `merge_cost.rs` keeps both facts visible; O(delta) merge is blocked on a real deletion-delta source **(RFC-027)**; fragment adoption is **(draft RFC-0001)**. | -| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. Phase A makes its opaque initializer + separate shard effect recoverable for one main-only/unsharded/single-live-writer-process empty enrollment, durably records lifecycle authority, and excludes competing local effects. Private B1 now implements and evidence-qualifies one root-scoped, no-rollover generation capped at 8,192 rows/32 MiB: it holds admission from before epoch claim, routes replay/unmerged residue fold-only through the pinned public watermark bridge, proves the drained generation independently, then seals/retires, performs a recovery-v11 strict fold, and reopens at a higher epoch. Every post-invocation ambiguity remains `AckUnknown`; exact already-normalized physical rows/vectors are folded without external embedding or unspecified derivation. The local cost matrix, configured-RustFS warm-ack cell, genuine v7↔v8 run, and 24/24 graph-level suite are green, but the RFC is still Draft and the seam is private. Public B2 additionally requires attribution, bounded reclamation/orphan cleanup, correction, same-key retry sequencing, and persistent status/quiesce/resume/abort-drain/rebuild. A public exact enrollment receipt plus reversible admission seal gates only broader overlapping-process topology. | +| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. Phase A makes its opaque initializer + separate shard effect recoverable for one main-only/unsharded/single-live-writer-process empty enrollment, durably records lifecycle authority, and excludes competing local effects. Private B1 implements and evidence-qualifies one root-scoped, no-rollover generation capped at 8,192 rows/32 MiB, watcher-only acknowledgement, conservative replay, and one recovery-v11 fold; its local/RustFS, cross-version, and 24/24 graph-level gates are green. B2-0 now specifies attribution, compare-and-chain sequencing, current-token authority, correction, persistent lifecycle, and reclamation/enforced watermark. Stock RC.1 generic cleanup and raw successor-fence-sentinel deletion are proven no-gos, so the first implementation dependency is the reviewed Lance durable inspect/plan/execute, reclaim recovery/history checkpoint, enforced growth reservation, and post-success-fence patch, pinned without waiting for release. Public B2 remains closed until that and the private v9 crash/storage/cross-version matrices pass. A public exact enrollment receipt plus reversible admission seal gates only broader overlapping-process topology. | | **R8: Some operations lack enforced memory/time budgets.** | Medium | Known gap, narrowed and accepted for RFC-023. Its direct-substrate instrument rejected the first whole-delta fenced adopt (~447 MB peak at 100K × 256 versus ~74 MB Append), and the first corrected production 10K series failed at 30.0× / 108,625,920 bytes overhead; both negative results remain evidence. Mutation/Load now refuses a keyed table above 8,192 rows / 32 MiB before arm, while BranchMerge uses a recovery-enrolled chain with the same per-chunk bounds and a 1,024-transaction ceiling. The inductive certificate route removes the general diff, temporary delta, target preflight, and target join without weakening that chain. Final five-pair production medians passed at 31/8 ms (3.875×) for 10K and 136/35 ms (~3.886×) for 100K; maximum signed paired RSS overheads were 24,297,472 and 32,604,160 bytes. Inclusive row/transaction ceilings, byte refusal (including materialized blobs), operation-wide validation retention, exact source/target incarnation revalidation, second-generation certificate composition, and both between-chunk recovery directions are pinned; other operations still need explicit bounds. | | **R9: Local-FS conditional-write emulation** (`write_text_if_match` check-then-act gap). | Low | All current callers sit behind the cluster lock protocol; S3 uses true conditional puts; close before admitting any lock-free caller. | | **R10: Doc/spec drift as the system grows** — this document included. | Low | Maintenance contract (same-PR doc updates, `check-agents-md.sh` link CI, "don't lie" stale markers); this canon defers to area docs by construction. | @@ -1046,7 +1092,9 @@ Live design questions, each owned by an RFC or a known gap — not a wishlist: 4. **Which capability owns the next rebuild boundary after v8?** RFC-028 activated stable identity in v5; RFC-023 assigned exact-`id` fencing to v6; RFC-026 Phase A assigned lifecycle/enrollment authority to v7, and private - Phase B1 assigned the data-bearing config-v2/recovery-v11 core to v8. + Phase B1 assigned the data-bearing config-v2/recovery-v11 core to v8. B2-0 + assigns token/lifecycle/correction to schema v9/config-v3/state-v2/ + recovery-v12, but that strand is not active. RFC-024's heads, RFC-025's retention, and later RFC-026 phases remain independently reviewable. Any later format activation requires its own export/init/load rebuild unless capabilities deliberately co-release after @@ -1079,7 +1127,7 @@ The plan of record is the RFC-022…028 family (all under | [0023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | Substrate-native keyed-write fencing via Lance's unenforced-PK filter; fleet/format activation barrier | **Implemented** (2026-07-15) | | [0024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | Materialized head-row research; the first exact-BTREE candidate bounded scan work but failed the full latest-manifest/object-byte cost gate | **Research blocked** | | [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags; current in-manifest registry lookup rejected by Gate 0 | **Research-blocked** | -| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: one-generation private watcher acknowledgement, then graph-atomic fold; public activation has separate attribution/reclamation/correction/sequencing/operator gates | **Draft; Phase A foundation and private Phase B1 core implemented/evidence-green; public B2 inactive** | +| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: one-generation private watcher acknowledgement, then graph-atomic fold; B2-0 specifies compare-and-chain tokens, attribution, lifecycle/correction, and Lance-owned reclamation/enforced admission watermark | **Draft; Phase A/B1 implemented and evidence-green; B2-0 specified; public B2 inactive** | | [0027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | O(delta) merge classification from row-version lineage | Research-blocked | Deliberately split, not one mega-format: identity, key fencing, head rows, diff --git a/docs/dev/index.md b/docs/dev/index.md index 345e090e..04a08be9 100644 --- a/docs/dev/index.md +++ b/docs/dev/index.md @@ -90,7 +90,7 @@ as durable disposition history after closure, so RFC backlinks stay valid. | Area | Read | |---|---| -| RFC-022–028 split architecture review — RFC-022/023/028 implemented; RFC-024/025/027 research-blocked; RFC-026 Phase A and private B1 implemented/evidence-green while public B2 remains inactive; dependency corrections and acceptance evidence remain tracked | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | +| RFC-022–028 split architecture review — RFC-022/023/028 implemented; RFC-024/025/027 research-blocked; RFC-026 Phase A and private B1 implemented/evidence-green, B2-0 specified, and public B2 inactive; dependency corrections and acceptance evidence remain tracked | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | ## Active Implementation Plans diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index 6d87e9b0..2cc06c4a 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -184,7 +184,7 @@ converge the physical state. | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | | Keyed writes | Every current v8 node/edge table declares exact non-null physical `id` as Lance's unenforced PK from creation, using the v6-introduced, v7/v8-preserved exact-`id` fence. General production strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint it only when completed effect statistics prove one attempt inserted every source row with zero updates, deletes, or skipped duplicates. Certificate admission is optional: BranchMerge uses the shortcut only when every transaction in the complete contiguous source interval carries v1 and its persisted operation is the full pure-insert `Update` shape (exact parent, no removed or updated fragments, nonempty new fragments with `physical_rows`, no field or generation rewrites, `RewriteRows`, exact-`id` filter, full nested schema preorder, and matching physical-row total). It then rechecks both source and target native ref incarnations and passes owned batches through an opaque capability whose one production mint site is structurally guarded. The proven publisher stages immutable fragments with `InsertBuilder`, replaces the uncommitted Append operation with that filter-bearing `Update`, and performs zero target preflights, target merge joins, or committed Appends. Missing, cleaned, unknown, or malformed proof uses the general ordered diff. Mutation/Load remains one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before arm; BranchMerge keeps its bounded v4 chain and exact-recovery limits. Raw Lance graph writers are unsupported, and the certificate is an internal non-cryptographic capability rather than an authenticity mechanism. The final production cost gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes) | [RFC-023](../rfcs/0023-key-conflict-fencing.md), [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | -| Streaming ingest | RFC-026 adopts Lance MemWAL as the strategic substrate and remains Draft; public row streaming is inactive. Historical Phase A schema v7 implemented exact bounded enrollment recovery, identity-keyed lifecycle authority, process-local admission/exclusion, current-HEAD witnesses, compatible-open validation, partial-format refusal, and strict v6↔v7 rebuild for the main-only, one-unsharded-shard, one-live-writer-process profile. Current schema v8/config-v2/recovery-v11 adds the private B1 core behind one feature-gated, doc-hidden engine seam: one already-normalized physical batch enters one hard-capped 8,192-row/32-MiB generation; only its successful Lance durability watcher acknowledges; automatic rollover is prevented and the writer retires before a successor-generation put; every post-invocation ambiguity is typed `AckUnknown`; replayed or flushed-unmerged residue is fold-only through the guarded BatchStore watermark bridge; and one exact strict fold publishes the achieved table pointer, lifecycle witness, and lineage only at the `__manifest` CAS. Physical vector columns are supplied before acknowledgement and pass through fold unchanged; B1 neither calls an external embedding provider nor invents unspecified fold-derived fields. B1 is evidence-green at this private boundary: genuine v7↔v8 refusal/rebuild, all 24 cells in the graph-level B1 behavior/crash/race suite, qualified local cost/PK-index/RSS, and configured-RustFS cost gates pass, with the uncompacted-manifest term explicitly non-flat. There is no production first-use caller, schema/SDK/HTTP/CLI/OpenAPI surface, operator drain/resume, correction, GC, or fresh read. B2 remains gated on durable contributor attribution, bounded reclamation/orphan cleanup and retained-storage stop, strict correction, same-key retry sequencing, and persistent operator escape | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | +| Streaming ingest | RFC-026 adopts Lance MemWAL as the strategic substrate and remains Draft; public row streaming is inactive. Historical Phase A schema v7 implemented exact bounded enrollment recovery, identity-keyed lifecycle authority, process-local admission/exclusion, current-HEAD witnesses, compatible-open validation, partial-format refusal, and strict v6↔v7 rebuild for the main-only, one-unsharded-shard, one-live-writer-process profile. Current schema v8/config-v2/recovery-v11 adds the private B1 core behind one feature-gated, doc-hidden engine seam: one already-normalized physical batch enters one hard-capped 8,192-row/32-MiB generation; only its successful Lance durability watcher acknowledges; automatic rollover is prevented and the writer retires before a successor-generation put; every post-invocation ambiguity is typed `AckUnknown`; replayed or flushed-unmerged residue is fold-only through the guarded BatchStore watermark bridge; and one exact strict fold publishes the achieved table pointer, lifecycle witness, and lineage only at the `__manifest` CAS. Physical vector columns are supplied before acknowledgement and pass through fold unchanged; B1 neither calls an external embedding provider nor invents unspecified fold-derived fields. B1 is evidence-green at this private boundary: genuine v7↔v8 refusal/rebuild, all 24 cells in the graph-level B1 behavior/crash/race suite, qualified local cost/PK-index/RSS, and configured-RustFS cost gates pass, with the uncompacted-manifest term explicitly non-flat. There is no production first-use caller, schema/SDK/HTTP/CLI/OpenAPI surface, operator drain/resume, correction, GC, or fresh read. B2-0 now specifies—but does not implement—explicit enrollment, durable contributor attribution, compare-and-chain retry safety, a graph-global current-token participant, monotonic lifecycle revisions with bounded terminal management receipts, strict correction, Lance-owned bounded reclamation, and a graph-global `GraphHistoryBudget` checked by every manifest publisher. Public activation remains gated on implementing those contracts and passing their crash, storage, cross-version, authorization, and product-parity evidence | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | | Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Under schema/source-target/all-table gates, each control uses one operation-local post-gate manifest/namespace capture rather than refreshing the handle-local coordinator around table-gate acquisition; successful ref movement explicitly invalidates derived read caches. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. A target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one identity-bearing v9 recovery envelope; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. Optimize's payload remains a bounded maintenance adapter rather than an exact caller-minted Lance transaction proof, so it is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/0022-unified-write-path.md) | @@ -254,9 +254,24 @@ them explicit. CAS. The private seam consumes already-normalized physical rows and vectors; it performs no external embedding call or unspecified fold-derived-field materialization. It is not a product surface and performs no GC or correction. - B2 is the later public strict activation and remains gated on durable - attribution, bounded reclamation/orphan cleanup and a retained-storage stop, - correction, same-key retry sequencing, and persistent operational escape. + B2 is the later public strict activation. B2-0 now specifies, but does not + implement, compare-and-chain tokens, trusted attribution, a graph-global + manifest-selected current-token participant, persistent lifecycle/correction + with monotonic revisions and bounded terminal management receipts, and Lance- + owned reclamation. The reclamation contract uses sentinel-first + manifest-named claims, a fail-closed bootstrap/checkpoint format, one per- + binding reserve-first physical ledger, exact multipart/versioned-object + accounting or refusal, and hard admission/control bounds; OmniGraph never + deletes raw `_mem_wal` paths. The MemWAL bound does not cover base/token/shared + manifest history. B2-0 therefore adds one manifest-authoritative, graph-global + `GraphHistoryBudget`: every manifest publisher reserves its exact publication + and source-bounded physical-growth envelope before effect and holds the gate + through effect, manifest CAS, and charge finalization, while dynamic per- + stream closure reserves remain independently spendable through correction, + quiesce, and `SEALED` rebuild. Public B2 is finite-lifetime per root until a + later accepted retention strand. Schema v9/config-v3/state-v2/recovery-v12 and + every product surface remain inactive until those contracts and their crash, + storage, cross-version, authorization, and parity evidence are green. B1's genuine v7↔v8 cross-version/rebuild, complete 24-cell graph-level B1 suite, qualified local cost/PK-index/RSS, and configured-RustFS cost gates pass. The non-flat uncompacted-manifest term remains explicit and no product latency or diff --git a/docs/dev/lance.md b/docs/dev/lance.md index 499e7086..ec6b8e10 100644 --- a/docs/dev/lance.md +++ b/docs/dev/lance.md @@ -249,10 +249,12 @@ which caller attempt produced it. Configuration identity binds only correctness/topology/no-rollover fields; explicit runtime policy and injected Session/store capabilities remain separate. Private B1 activates graph schema v8 and stream-config v2 rather than adopting v7/config-v1 in place. Phase B2 is -the later public strict activation and additionally waits for durable -contributor attribution, bounded reclamation, strict correction, a same-key -`AckUnknown` sequencing/idempotency contract, and persistent -quiesce/resume/abort-drain/rebuild. B1's genuine v7↔v8 cross-version/rebuild, +the later public strict activation. B2-0 now specifies explicit enrollment, +durable trusted contributor attribution, compare-and-chain write/predecessor +tokens, a manifest-selected current-token participant, bounded correction, and +persistent revisioned quiesce/resume/abort-drain/rebuild with bounded terminal +management receipts. B1's genuine v7↔v8 +cross-version/rebuild, complete 24-cell graph-level B1 suite, qualified local cost/PK-index/RSS, and configured RustFS cost gates now pass; the measured uncompacted-manifest term remains explicitly non-flat. That makes the private B1 boundary green, not public. The @@ -260,6 +262,59 @@ missing combined enrollment receipt and cross-process admission seal still gate broader topology, not B1 inside the existing single-live-writer-process boundary. +The 2026-07-19 B2-0 reclamation audit adds a separate stock-RC.1 no-go. Generic +`cleanup_old_versions` can reclaim ordinary dataset versions but does not walk +or delete `_mem_wal`; the runtime guard +`cleanup_old_versions_does_not_reclaim_mem_wal_objects` pins that ownership +boundary. RC.1 also has no post-success epoch recheck after a WAL atomic PUT. +`mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance` +decodes and deletes the successor's empty epoch-2 WAL fence sentinel and proves +the stale writer can still complete the put and receive watcher success even +though an explicit check returns `PeerClaimedEpoch`. These guards forbid raw +MemWAL deletion in OmniGraph; they do not implement reclamation. + +RFC-026 §4.5 therefore requires a Lance-owned opaque inspect/plan/execute +primitive with exact base/shard/history and whole-cut/cursor witnesses, durable +attempt/receipt lost-result recovery, manifest-version plus epoch advancement +before deletion, conservative orphan/unknown classification, bounded +shard/reclaim-history checkpointing, and the post-success writer-epoch check. +Every patched epoch claim first persists its attempt, writes an empty successor +sentinel, and only then CASes the manifest that names it. Ordinary open, +quiesce, resume, and checkpoint claims preserve the prior replay cursor and +classify its complete tail; only a proved no-data-tail whole-cut reclaim may +advance that cursor to its new sentinel. Pending attempts block another claim. +Exact inventory also requires strong HEAD/GET/LIST visibility after PUT/DELETE +plus incomplete-multipart accounting/abort, or Lance-owned durable complete +accounting. Versioned/soft-delete/Object-Lock namespaces are refused unless +every retained version/delete marker/locked byte is counted and eligible +versions can be permanently removed. Its bounded bootstrap/checkpoint format +creates a genesis body and pointer before a new details type URL or system-index +kind that stock RC.1 must reject, not an ignored protobuf field or latest- +version hint. One bootstrap-selected reserve-first ledger serializes generation +and control reservations per physical binding before any WAL/upload effect, +reconstructs cold, and settles only from exact inventory; cached checks cannot +double-reserve across shards. Lance must source and enforce the maximum +physical-growth reservation, durable materialization-attempt limit, bounded +claim/reclaim/checkpoint history, and emergency control headroom before +admission; local/RustFS measurement validates those bounds but does not create +them. OmniGraph will author that patch, open it upstream, and pin its exact +reviewed fork commit without waiting for a release. Public B2 remains inactive +until the positive local/RustFS crash and bound-validation matrix passes. WAL- +prefix reclamation is limited to a quiescent whole cut because RC.1 does not +persist a per-generation WAL range. + +That Lance-owned ledger bounds only one physical `_mem_wal` binding. It does not +bound the graph's base/token datasets or shared `__manifest` history. B2-0 +therefore separately specifies a manifest-authoritative graph-global +`GraphHistoryBudget`, checked by every manifest publisher. Each operation +reserves the exact publication and source-bounded physical-growth envelope +before effect, holds the global gate through effect/CAS/finalization, persists +pending recovery charges, and settles them exactly once; +dynamic per-stream closure reserves cannot be borrowed by ordinary work or by +another stream. This OmniGraph authority is specified but inactive together +with schema v9/config-v3/state-v2/recovery-v12 and requires its own bootstrap, +crash, refusal, every-writer, and local/RustFS physical-bound evidence. + The no-roll profile uses fixed portable capacities (`8,193` rows/batches and 1-GiB byte/unflushed thresholds), not architecture-dependent `usize::MAX` metadata. After reopen, the 32-MiB contract is validated from public diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md index 2fec885a..ec36cf41 100644 --- a/docs/dev/rfc-022-027-architecture-review.md +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -2,7 +2,7 @@ **Status:** RFC-022, RFC-023, and RFC-028 implemented; RFC-024, RFC-025, and RFC-027 research-blocked; RFC-026 Phase A and private B1 implemented/evidence- -green, with public B2 inactive +green, Phase B2-0 specified, and public B2 inactive **Date:** 2026-07-11 **Audience:** RFC authors, engine/storage maintainers, and release reviewers **Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at @@ -32,7 +32,14 @@ watcher-only acknowledgement, conservative replay/fold-only reopen, and a recovery-v11 exact graph-atomic fold. Genuine v7↔v8 rebuild/refusal, all 24 graph-level race/crash cells, and the qualified local plus configured-RustFS cost/RSS evidence pass. No production or public caller reaches the row seam; -B2 remains inactive. +B2 remains inactive. B2-0 now specifies explicit enrollment, compare-and-chain +tokens, trusted row attribution, manifest-selected current-token state, +protocol-v2 lifecycle revisions plus bounded management receipts, bounded +correction, Lance-owned reclamation with an enforced storage admission +watermark, and a graph-global `GraphHistoryBudget` checked by every manifest +publisher. Two RC.1 surface guards prove generic cleanup +non-ownership and the stale-writer hazard created by deleting the successor's +empty WAL fence sentinel; neither is an implementation of reclamation. The local RFC-024/025 decision instruments were rerun: both remain research-blocked, with the current RC-specific counters recorded in the Lance audit and RFC-025. @@ -50,7 +57,8 @@ the full MemWAL table and system-index specifications, including the durability/fencing behavior carried forward from beta.17, are reflected in the RFC's survey and acceptance guards. Gate E0 passed at the explicit bounded profile and Phase A consumed it as described above; RFC-026 remains draft and -public row streaming is inactive. +public row streaming is inactive. The B2-0 contract closure and stock-RC.1 +reclamation no-go are recorded in RFC-026 §4.1–§4.6 and BLOCKER-14 below. **RFC-024/025/027 substrate claims revalidated against:** the current RC.1 revision and the full matching table/index/branch/tag/cleanup, transaction, and row-lineage specification sections; their survey headers now record RC.1. @@ -453,11 +461,12 @@ integration. [§9](../rfcs/0026-memwal-streaming-ingest.md#9-fresh-read-cuts), and [§11–13](../rfcs/0026-memwal-streaming-ingest.md#11-format-activation-and-rebuild) -**Status:** Closed through private Phase B1 implementation and evidence on -2026-07-19; public B2 activation remains open. RC.1 lacks the ideal caller-owned enrollment -receipt and cross-process shard-admission seal, but upstream release timing is -not the bounded-profile calendar. Gate E0's exact-classification and ABA cells -passed using known-version probes and strict object-store classification. +**Status:** Closed through private Phase B1 implementation/evidence and Phase +B2-0 format-contract specification on 2026-07-19; public B2 activation remains +open. RC.1 lacks the ideal caller-owned enrollment receipt and cross-process +shard-admission seal, but upstream release timing is not the bounded-profile +calendar. Gate E0's exact-classification and ABA cells passed using +known-version probes and strict object-store classification. Phase A then activated a main-only, unsharded, single-live-writer-process empty enrollment foundation in internal schema v7. Private B1 activated the bounded data-bearing schema-v8/config-v2 core. RFC-026 remains draft and public row @@ -487,6 +496,17 @@ empty-enrollment contract: implemented v8 is the first data-bearing stream format, uses stream-config v2 and recovery schema v11, refuses v7/config-v1, and carries genuine v7↔v8 rebuild/refusal evidence. +B2 is a third strict strand, not an in-place reinterpretation of v8. B2-0 +assigns internal schema v9, stream-config v3, stream-state protocol v2, and +recovery-v12 to trusted hidden row metadata, the manifest-selected current-token +participant, persistent revisioned lifecycle/correction, bounded terminal +management receipts, and their multi-participant recovery. V9 also owns the +manifest-authoritative graph-global `GraphHistoryBudget`; every manifest writer +must reserve and settle its publication plus source-bounded physical-growth +envelope while preserving dynamic per-stream closure reserves. Those versions +are specified but inactive; genuine v8↔v9 refusal/rebuild and budget +initialization/refusal remain implementation gates. + RFC-026 §12 records those tests and §13 places the completed Phase A before the public stream path. RFC-026 §3/§8 also make a rematerialized table a separate physical enrollment boundary: SchemaApply leaves the preserved logical identity @@ -524,12 +544,15 @@ drain proof from empty frozen refs plus the authoritative generation/cursor, immediate writer retirement, one recovery-v11 keyed fold, and higher-epoch reopen only after publication. `AckUnknown` preserves possible residue through replay but remains permanently caller-ambiguous. B2 later owns -public strict activation and cannot start merely because B1 is green: it also -needs durable authenticated-contributor attribution, bounded reclamation with a -retained-storage stop and derived-orphan cleanup, strict correction/disposition, -same-key retry sequencing/idempotency, and persistent -status/quiesce/resume/abort-drain/rebuild. A `SEALED` B2 stream permits -export/rebuild; in-place maintenance still waits for Phase D. +public strict activation and cannot start merely because B1 is green. B2-0 now +specifies durable authenticated-contributor attribution, compare-and-chain +tokens, a manifest-selected current-token participant, bounded +`REPLACE`/`WITHDRAW` correction, persistent revisioned +status/quiesce/resume/abort-drain/rebuild with bounded management receipts, and +the reclamation/enforced-watermark +contract. Their implementation and crash/cross-version/storage evidence remain +gates. A `SEALED` B2 stream permits export/rebuild; in-place maintenance still +waits for Phase D. The bounded candidate instead makes the support restriction load-bearing. It grants an `OPEN` stream exclusive authority to advance that base-table HEAD; @@ -893,6 +916,85 @@ Do not weaken the physical-I/O gate, infer success from flat row/range counters, or add a second authority dataset whose rows cannot share the main-manifest CAS. +### BLOCKER-14 — stock RC.1 cannot safely reclaim MemWAL state + +**Affected:** [RFC-026 §4.5](../rfcs/0026-memwal-streaming-ingest.md#45-reclamation-decision-own-the-missing-lance-primitive), +[§6](../rfcs/0026-memwal-streaming-ingest.md#6-fold-protocol), and +[§12.4](../rfcs/0026-memwal-streaming-ingest.md#124-phase-b2--public-strict-activation-gates) + +**Status:** Design-dispositioned on 2026-07-19; implementation-blocking for +public B2 activation. This blocker does not reopen the MemWAL substrate choice +or require waiting for an upstream release. + +Two production-neutral guards pin the current RC.1 boundary: + +- `cleanup_old_versions_does_not_reclaim_mem_wal_objects` proves Lance's + generic version cleanup can remove ordinary table history while leaving the + present `_mem_wal` fixture unchanged; it does not classify orphans; and +- `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance` + proves that decoding and deleting the successor's empty epoch-2 WAL fence + sentinel can let the stale writer complete a later WAL PUT and receive + watcher success even though an explicit epoch check returns + `PeerClaimedEpoch`. RC.1 does not perform that check after success. + +Those are negative ownership findings. They forbid an OmniGraph raw-path +collector; they do not prove a safe cleanup implementation. WAL positions also +have no per-generation durable range in RC.1, so prefix deletion is limited to +a quiescent whole cut rather than inferred from mutable cursor hints. + +**Required disposition:** author the capability inside Lance as an opaque +inspect/plan/execute operation with durable attempt/receipt recovery. Its plan +binds exact shard-manifest history, whole-cut cursor predicates, base +merge/index state, writer epoch, graph-approved generation, and object/byte +budgets. Execution revalidates those witnesses, advances both shard-manifest +version and epoch before deleting only approved objects, retains unknown/gapped +state, and returns an idempotently classifiable receipt. The patch also needs a +bounded authoritative checkpoint for shard, ordinary-claim, and reclaim +history. All epoch claims must be attempt-owned, successor-sentinel-first, and +manifest-named; ordinary open/quiesce/resume/checkpoint claims preserve the +replay cursor, while only a proved whole-cut reclaim may advance it. Enrollment +creates the genesis bootstrap before a new MemWAL details kind that stock RC.1 +rejects. Exact accounting requires strong HEAD/GET/LIST visibility after +PUT/DELETE plus multipart accounting/abort or durable complete accounting, a +source-derived enforced physical-growth reservation with bounded +materialization attempts and control headroom, and an epoch recheck after every +successful WAL atomic PUT. One bootstrap-selected reserve-first ledger owns +that budget per physical binding and settles only from exact inventory; +versioned/soft-delete/Object-Lock stores are refused unless every retained byte +is accounted and eligible versions are permanently removed. OmniGraph opens the +upstream PR but pins the exact reviewed fork commit immediately, so release +timing is not the calendar. + +Public B2 remains closed until the positive local/RustFS stale-plan, +prune-CAS, partial-delete, lost-result/receipt, +authoritative-checkpoint-plus-successor-chain orphan, +unknown-state, genesis/first-publication, claim/reclaim receipt-expiry, +history-checkpoint, inventory-consistency, and post-success-fence matrices pass. +The source-derived reservation must be enforced before admission; a separate +physical-expansion instrument validates it but cannot create a hard bound. +Runtime never releases budget merely because a fold completed. + +The per-binding ledger bounds `_mem_wal` only. B2-0 separately specifies one +manifest-authoritative, graph-global `GraphHistoryBudget` checked by every +manifest publisher, including ordinary table-disjoint writers. Each operation +reserves its exact logical publication and a source-bounded physical-growth +envelope before effect; durable recovery charges settle exactly once. Initial +B2 holds the gate continuously through physical effects, manifest CAS, and +charge finalization rather than composing speculative pending revisions. Dynamic +per-stream closure reserves remain independently available for block/correction, +quiesce, and `SEALED` rebuild even when ordinary work reaches the shared floor. +The universal order is sorted relevant stream admission → graph history → +schema → main → stream token → sorted tables; late global discovery +releases all of those gates and restarts from the root barrier. + +This closes only `_mem_wal` retention. B2 still refuses in-place maintenance on +an enrolled table, and base/token/shared-manifest histories continue to grow. +The graph-global budget therefore makes B2 a finite-lifetime root profile: +ordinary work returns `GraphRebuildRequired` at the aggregate closure floor, +while each enrolled stream keeps its reserved fold/correction/quiesce path to +`SEALED` export/rebuild. Indefinite in-place service remains a later +maintenance/retention strand. + ### TIGHTENING-01 — symmetric Lance conflicts are not obviously an activation gate **Status:** Closed in implementation and acceptance on 2026-07-15. The beta.21 @@ -1069,13 +1171,18 @@ The review does not require all RFCs to land together. A safe order is: strict-rebuild implementation evidence; 6. retain RFC-026 Gate E0's green exact-version evidence, Phase A's implemented bounded enrollment foundation, and private B1's green schema-v8/config-v2 - one-generation put/watcher/replay/recovery-v11 strict fold evidence; expose - B2 only after accepted designs and evidence for durable - contributor attribution, bounded reclamation, strict correction, same-key - retry sequencing, and - persistent status/quiesce/resume/abort-drain/rebuild. Retain the public - exact enrollment receipt and cross-process admission seal as the gate for - broader topology; + one-generation put/watcher/replay/recovery-v11 strict fold evidence. Retain + B2-0's specified explicit enrollment, compare-and-chain token, attribution, + state-v2 lifecycle revisions/management receipts, bounded correction, + reclamation/enforced-watermark, and graph-global manifest-history budget + contracts. First + author and pin the reviewed Lance durable reclamation, bounded-history, + enforced-growth-reservation, and post-success-fence patch and pass its + local/RustFS evidence; then implement schema-v9/config-v3/recovery-v12 plus + reserve-first graph-history accounting privately. Expose B2 only after every + crash, enforced-bound, cross-version, authorization, and product-parity gate + passes. Retain the public exact enrollment receipt + and cross-process admission seal as the gate for broader topology; 7. keep RFC-027 in research until deletion-delta discovery passes its stated correctness and flat-cost gates. diff --git a/docs/dev/testing.md b/docs/dev/testing.md index faf7972d..8dcba68d 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -25,7 +25,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); RFC-023 pins the inclusive 8,192-row keyed input ceiling, the same exact/+1 boundary on streamed mutation-update matches, no-effect state for both refusals, and oversized stored-Blob rejection before payload read. Crate-internal pending-scan cells pin inclusive/+1 32 MiB accounting plus pending-key shadow-before-charge. The lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | | `src/table_store/staged_tests.rs` | Crate-internal staged primitives. RFC-023 pins one exact target preflight for general StrictInsert, durable v1 mint/commit/reopen/history persistence, exact-`id` filter emission, typed `KeyConflict`, and missing/wrong PK refusal. `all_new_upsert_certifies_insert_absence_and_persists_it_in_history` proves an all-new completed Upsert receives the optional certificate, a mixed/update Upsert does not, unrelated transaction properties survive, and UUID rebinding does not erase it. Proven-insert cells show the opaque path performs zero strict preflights; stages with `InsertBuilder` but commits the full pure-insert `Update` shape (exact parent and `id` filter, `RewriteRows`, no updates/removals, full nested schema preorder, physical rows); persists/re-admits its own output for proof composition; leaves new fragments outside old index coverage; and fails same-key races loudly in proven/proven and proven/general orders. The in-source `exec/merge.rs` certificate unit table rejects missing/unknown properties, wrong parent/filter/full-preorder/mode/offsets, rewrite/removal shapes, missing `physical_rows`, and Append. Source-interval cells pin exact selection, lazy retained-parent splitting, coalescing, and pinned Lance's approximate raw-emission boundary while every normalized/writer chunk remains hard-capped. Generic `stage_append`/`stage_merge_insert` remain primitive tests only. The file also owns index staging and `commit_staged{,_exact}` | | `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. At the RFC-026 Phase-A checkpoint the guard registered only the exact v10 enrollment gateway and feature-gated test seam, counted its sidecar/index/shard durability primitives, and kept every row-put/ack/fold surface absent; the Phase-B1 owner below now exact-counts only the approved crate-private put/fold durable-call sites while retaining the absence of public schema, SDK, HTTP, CLI, and OpenAPI side doors. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. This is intentionally not a Rust macro-expander or general alias analysis; `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | -| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current table version + current `Transaction.uuid` + `ManifestLocation.e_tag` current-HEAD witness; the local/shared-`Session` guard proves unchanged-reopen stability, ordinary-commit movement, and same-version ABA, while RustFS covers object-store ABA. RFC-025 adds exact main/named-branch tag-target, sparse cleanup pin/unpin, and branch-tree-deletion guards. RFC-026 pins doc-hidden `has_successor_version`, initializer/readback/shard-writer/durability/fencing, flush/drain, replay watermark, scanner, and merged-generation shapes; runtime Gate E0 classification belongs to `memwal_enrollment_gate.rs`, while v7 Phase-A and v8 B1 publication/recovery belong to the manifest/failpoint suites. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection. These guards prove substrate shapes/tokens; they do not by themselves prove heads/checkpoint activation or the v8 publisher | +| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current table version + current `Transaction.uuid` + `ManifestLocation.e_tag` current-HEAD witness; the local/shared-`Session` guard proves unchanged-reopen stability, ordinary-commit movement, and same-version ABA, while RustFS covers object-store ABA. RFC-025 adds exact main/named-branch tag-target, sparse cleanup pin/unpin, and branch-tree-deletion guards. RFC-026 pins doc-hidden `has_successor_version`, initializer/readback/shard-writer/durability/fencing, flush/drain, replay watermark, scanner, and merged-generation shapes; runtime Gate E0 classification belongs to `memwal_enrollment_gate.rs`, while v7 Phase-A and v8 B1 publication/recovery belong to the manifest/failpoint suites. B2-0 adds `cleanup_old_versions_does_not_reclaim_mem_wal_objects` and `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance`: the first proves generic cleanup leaves the present MemWAL fixture unchanged and the second proves deleting the successor's empty fence sentinel is unsafe. The pinned source audit, not those two tests alone, establishes that stock RC.1 exposes no owned MemWAL reclamation API. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection. These guards prove substrate shapes/tokens and negative ownership boundaries; they do not by themselves prove heads/checkpoint activation, the v8 publisher, or a safe reclamation implementation | | `memwal_enrollment_gate.rs` | RFC-026's green production-neutral Gate E0 harness, isolated from the production manifest and graph writer. Fourteen substantive local cells plus one explicit unconfigured-S3 skip cover exact no-effect / `N + 1` index / pre-minted empty-shard classification, buried-effect refusal, marker survival, strict inventory/error handling, and the broad fail-closed matrix. The rejected first instrument used `checkout_latest` plus `IOTracker`, which missed local `read_dir`. The accepted exact-version classifier pins doc-hidden `has_successor_version`; its `AttemptTracker` records failed/`NotFound` attempts before forwarding and proves the identical complete six-attempt shape at baseline versions 8/80: four successful manifest HEADs, one `NotFound` manifest HEAD, one successful manifest GET, zero lists. A Unix execute-only `_versions` tripwire proves exact probing works when latest enumeration fails and an unreadable exact HEAD errors. The configured RustFS exact cell passes non-vacuously with the same zero-list shape and owns the positive lost-result/index/empty-shard/reopen sequence plus foreign shard, malformed/loose root, durable WAL, persisted cursor, and corrupt-manifest negatives. S3 ABA remains in `lance_surface_guards.rs`; CI rejects skipped E0/ABA cells. This file never mutates production manifest/schema state or deletes ambiguous artifacts; Phase A consumes its classifier through the private adapter | | `memwal_stream_cost.rs` | Feature-gated RFC-026 B1 decision instrument. It separately measures warm already-claimed durability acknowledgement at compacted graph-history endpoints, cold claim/reopen/replay against retained WAL depth, one selected generation's fold scan, retained already-merged shard metadata, and the shared uncompacted graph-manifest fold/publisher term. It also pins both one-batch and maximally fragmented legal-generation no-auto-roll estimates and records a paired one-batch whole-process peak-RSS delta. The exact local and configured-RustFS evidence is recorded in “RFC-026 Phase B1 coverage ownership” below; debug timings are not a product latency claim. | | `durable_head_lookup_cost.rs` | RFC-024 Gate A decision instrument, isolated from the production manifest schema/publisher. At fixed catalog width 10 it runs the full absent/reconciled/one-uncovered/eight-uncovered/reconciled-after-tail matrix over compacted and uncompacted histories, with cold-open and warm-repeat measurements on local FS and bucket-gated S3/RustFS. Default depths are 20/80; the ignored decision-scale cell runs 10/100/1,000. Correct exact heads, flat indexed `rows_scanned`/range work, an index-absent growing negative control, and observable bounded tails all pass; after the eight-fragment tail, `optimize_indices` returns coverage to zero uncovered and representative `rows_scanned`/range work from 27→10 / 17→10. The test deliberately pins the no-go: uncompacted RustFS cold object reads/bytes and compacted byte terms grow, while RC.1 also crosses a bounded one-operation boundary by 1,000 commits, so RFC-024 remains research-blocked. `rows_scanned` is an RC.1 debug proxy, not a universal decoded-row counter. Object-store wrapper bytes and Lance execution-summary bytes are separate fixture-owned metrics and are not additive | @@ -61,6 +61,11 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `recovery.rs` | Open-time recovery sweep — identity-bearing schema-v9 envelopes for Mutation/Load exact transaction identity, BranchMerge exact chains/ref-only effects + complete delta, SchemaApply exact overwrite/create ownership + schema promotion, EnsureIndices exact mixed-index transactions/authority/lineage/delta/first-touch identity, and Optimize's bounded maintenance payload, plus RFC-026's dedicated schema-v10 roll-forward-only StreamEnrollment envelope and schema-v11 StreamFold envelope binding the exact config-v2 profile, shard cut, merged generation, Lance transaction, and fixed lineage. Explicit refusal replaces alias inference for identity-less input; restartable compensation, fixed logical/rollback IDs, branch-token comparison, fresh under-gate reread/reparse, all-or-nothing roll-forward/rollback/refusal, recovery audit, and read-only guards remain pinned. RFC-023 additionally asserts that restoring a feature-branch sidecar leaves the selected feature ref with exact-`id` PK metadata | | `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). | +RFC-026 reclamation qualification: the two B2-0 runtime guards do not prove +that every possible safe RC.1 API is absent. The source audit establishes that +surface fact; the tests prove the narrower generic-cleanup non-ownership and +deleted-successor-sentinel fencing hazard. + ## Fixtures `crates/omnigraph/tests/fixtures/` holds the canonical schema (`.pg`), seed data (`.jsonl`), and queries (`.gq`) shared across tests. Reuse these before inventing new ones — the helpers harness already knows how to load them. @@ -255,15 +260,90 @@ cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost cargo test -p omnigraph-engine --features failpoints --test failpoints stream_fold ``` -B1 does not add parser/server/CLI tests because it has no public surface. B2 +B1 does not add parser/server/CLI tests because it has no public surface. B2-0 +now specifies the missing contracts without activating them. B2 implementation must extend the existing compiler, server/OpenAPI, CLI parity, Cedar, shutdown, -audit, retention/GC, and genuine rebuild suites together with persistent -status/quiesce/resume/abort-drain, strict correction, and retained-storage-stop -controls; those tests are a B2 gate, not incidental B1 scope. The lifecycle +audit, retention/GC, and genuine rebuild suites together with explicit +enrollment, persistent revisioned status/quiesce/resume/abort-drain, bounded +terminal management receipts, strict correction, the enforced retained-storage +admission watermark, and graph-global manifest-history admission controls; +those tests are a B2 gate, not incidental B1 scope. The lifecycle matrix includes `quiesce -> create named branch -> resume`: bounded resume must recheck branch topology under the closed gates and remain `SEALED`, while a compatible main-only resume advances the epoch and opens. +### RFC-026 Phase B2-0 coverage ownership (specified, inactive) + +B2-0 adds evidence at the boundary where the design depends on Lance, while +leaving all production stream surfaces absent. The two checked-in RC.1 guards +live in `lance_surface_guards.rs` because they characterize substrate behavior: + +- `cleanup_old_versions_does_not_reclaim_mem_wal_objects` creates ordinary + reclaimable table history plus a durable MemWAL entry, runs Lance's generic + cleanup, and proves ordinary versions are removed while every object name and + byte in that present `_mem_wal` fixture is unchanged. It does not contain or + classify an orphan fixture; and +- `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance` + decodes and deletes the successor's exact empty epoch-2 WAL fence sentinel, + then proves the stale writer can still report watcher success even though its + explicit epoch check returns typed `PeerClaimedEpoch`. This is a negative + regression for the required post-success epoch check, not permission for + OmniGraph to delete the sentinel. + +Run them with: + +```bash +cargo test -p omnigraph-engine --test lance_surface_guards cleanup_old_versions_does_not_reclaim_mem_wal_objects -- --exact +cargo test -p omnigraph-engine --test lance_surface_guards mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance -- --exact +``` + +The Lance patch must turn the second guard into a typed fence/unknown outcome +and add local plus object-store inspect/plan/execute coverage for stale plans, +whole-cut/cursor pruning, partial deletion, durable attempt/receipt replay, +lost results, authoritative-checkpoint-plus-successor-chain orphan +classification, unknown retention, strong HEAD/GET/LIST-after-PUT/DELETE and +multipart-accounting refusal or Lance-owned complete accounting, and bounded +history checkpointing. Cold-open/quiesce/resume/checkpoint claims also crash at +attempt/sentinel/manifest boundaries: ordinary sentinel-first claims must +preserve the replay cursor and classify the complete tail, while only a proved +whole-cut reclaim may advance it to the new sentinel. Cross-version tests pin +genesis body/pointer/details publication, a new fail-closed MemWAL details kind, +checkpoint-epoch `ReceiptExpired`/`ClaimReceiptExpired`, and no fallback to a +latest hint. OmniGraph's implementation then extends the existing +`memwal_stream.rs`, recovery, failpoint, forbidden-API, cross-version, +server/OpenAPI, CLI-parity, and Cedar owners; it does not create a parallel +streaming test silo. Lance must expose and enforce a source-derived maximum +physical object/byte growth reservation (or equivalent quota), bounded durable +materialization attempts, exact multipart abort/accounting, and reserved +control headroom before the hard admission watermark is real. The future matrix +also pins one reserve-first ledger per physical binding across concurrent +shards, reserve/effect/settlement/reclaim crashes, cold reconstruction, exact +`observed + unmaterialized remainder` arithmetic, bounded terminal/control/body- +orphan history, and emergency reclaim/quiesce/checkpoint progress at the full +row-admission watermark. Versioned, soft-delete, and Object-Lock storage is +refused unless every retained version/delete marker/locked byte is exactly +accounted and eligible versions are permanently removed. A separate +local/RustFS matrix validates that bound across schemas, fragmentation, crashes, +and retries; measurement alone does not establish it. Until all of that is +green, schema v9/config-v3/state-v2/recovery-v12 and every public B2 route +remain inactive. A separate finite-lifetime matrix initializes and validates the +manifest-authoritative graph-global `GraphHistoryBudget`, then charges every +manifest-writer class and its pending recovery sidecars through reserve, effect, +lost publication acknowledgement, exact settlement, and effect-free release. +It covers source-bounded physical-growth accounting, cap-too-small/bootstrap and +missing/mismatched-authority refusal, ordinary `GraphRebuildRequired` wire/error +mapping, and two simultaneously blocked streams whose dynamic closure reserves +cannot be spent by ordinary work or by each other. At the aggregate floor, each +stream can still consume its own worst-case block/correction/abort-drain/ +requiesce path and reach `SEALED` rebuild. Lifecycle tests also pin monotonic +revision CAS, complete terminal management-receipt replay after later movement, +same-ID/different-digest conflict, stale-revision refusal, and receipt count/byte +closure reserves. Concurrency coverage pins sorted relevant stream admission → +graph history → schema → main → stream token → tables, the history gate held +from sidecar arm through effect/CAS/finalization, a two-publisher fresh-revision +restart, plus release-all/root-restart after late global discovery. The `_mem_wal` watermark is never +asserted to bound whole-root history. + RFC-023's Mutation/Load effect classifier is pinned here, not by ordinary unit tests: `rfc023_effect_free_conflict_is_typed_or_fully_reprepared` proves that a strict same-key conflict is terminal `KeyConflict` while an upsert stages a fresh, diff --git a/docs/dev/writes.md b/docs/dev/writes.md index 7019eb49..c82e483a 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -635,11 +635,63 @@ marked, or authority-mismatched effect fails closed. B1 performs no fresh-tier reads and no generation GC. Acknowledged rows become query-visible only after fold. The support boundary remains main-only, unsharded, one resident stream worker, and one live writer process; the -admission lease is not a distributed fence. Phase B2 must still design and -prove authenticated contributor attribution, bounded WAL/generation -reclamation plus a retained-storage stop, correction/disposition, same-key -retry sequencing after `AckUnknown`, persistent -status/quiesce/resume/abort-drain/rebuild, and every public policy/wire surface. +admission lease is not a distributed fence. + +RFC-026 Phase B2-0 now specifies—but does not implement—the next contract. +`@stream` declares eligibility but explicit, request-idempotent enrollment +creates the stream incarnation and physical binding. A caller-stable `write_id` +and exact predecessor then form an +opaque compare-and-chain token; trusted contributor/digest/tagged-origin and +same-generation chain-certificate metadata travels with the row. A +manifest-selected graph-global token dataset is the sole post-fold per-key +sequencing authority and publishes atomically with the base pointer under the +shared stream-token gate/recovery envelope. Because that participant is +graph-global, every token-moving sidecar blocks every manifest/main writer, +including table-disjoint writers. A final-barrier late discovery releases all +admission, graph-history, schema, branch, token, and table gates and restarts from the root +barrier; it never recovers while retaining a leaf gate. Durable +`OPEN -> DRAINING -> SEALED -> OPEN` state carries a monotonic lifecycle +revision. Every external mutating management call compares that revision and +records a bounded, complete terminal receipt; roll-forward-only resume/abort and +bounded `REPLACE`/`WITHDRAW` correction use recovery-v12. Public activation +also requires Lance-owned durable reclamation with whole-cut proof, +attempt/receipt recovery, bounded history checkpointing, strong PUT/DELETE +inventory plus multipart accounting/abort or durable accounting, a post-success +epoch check, and a source-derived enforced +retained-object/byte admission watermark. Patched epoch claims are +sentinel-first and manifest-named: ordinary open/quiesce/resume/checkpoint +claims preserve the prior replay cursor and classify its tail; only a proved +whole-cut reclaim may advance to its new sentinel. The patch also needs a +genesis bootstrap plus a new stock-RC.1-rejected MemWAL details kind, bounded +ordinary-claim/reclaim receipt history, durable materialization-attempt limits, +exact multipart accounting/abort, and reserved control headroom. A single +bootstrap-selected reserve-first ledger serializes all shards per physical +binding before any WAL/upload effect and reconstructs/settles from exact +inventory; materialized bytes remain `observed` while only possible remainder +stays reserved. Versioned/soft-delete/Object-Lock stores are refused unless all +retained physical bytes are countable and eligible versions permanently +deletable. Measurements validate those bounds; they do not create them. Generic +Lance cleanup does not reclaim `_mem_wal`, and OmniGraph must never delete its +raw paths. Internal +schema v9/config-v3/state-v2/recovery-v12 and every public policy/wire surface +remain inactive until their crash, storage, cross-version, authorization, and +parity evidence passes. The physical watermark is per binding and does not +bound base/token or shared manifest history. B2-0 therefore specifies a separate +manifest-authoritative, graph-global `GraphHistoryBudget`. Every manifest +publisher—stream and ordinary—takes the graph-history gate, reserves the exact +logical publication plus a source-bounded physical-growth envelope before +effect, and settles pending recovery charges exactly once. Initial B2 holds that +gate continuously from final reserve/sidecar arm through physical effects, +manifest CAS, and charge finalization, so only one expected budget revision is +live. Dynamic per-stream +closure reserves cannot be consumed by ordinary commits or by another stream; +at the shared floor ordinary work returns `GraphRebuildRequired`, while the +named stream retains correction, quiesce, and `SEALED` export/rebuild headroom. +The universal B2 gate order is sorted relevant stream admission → graph history +→ schema → main branch → stream token → sorted graph tables; late global +authority discovery releases the complete suffix and restarts at the root +barrier. A writer with no relevant stream still begins at graph history. +Indefinite in-place maintenance remains a later accepted strand. ### Open-time recovery sweep diff --git a/docs/dev/writing-path-state-of-affairs.md b/docs/dev/writing-path-state-of-affairs.md index 92aa398f..124331b3 100644 --- a/docs/dev/writing-path-state-of-affairs.md +++ b/docs/dev/writing-path-state-of-affairs.md @@ -9,21 +9,29 @@ adjacent control and maintenance operations, known blockers, and the next decision points **Change-set boundary:** this page describes current main through RFC-026's -Phase A foundation and private Phase B1 core. RFC-024, RFC-025, and RFC-027 -remain research-blocked at their recorded evidence gates. Gate E0 first proved -the bounded public Lance state classifier; Phase A activated internal schema -v7 with recoverable empty enrollment, durable identity-keyed lifecycle -authority, process-local admission/exclusion, and strict format -refusal/rebuild. Internal schema v8 now preserves that foundation and adds -stream-config v2, a root-scoped one-generation worker, watcher-backed durable -acknowledgement, exact replay/seal/retirement, and recovery-v11 `StreamFold`. -The private B1 evidence is green: the local cost matrix, configured-RustFS -warm-ack cell, genuine v7↔v8 binary refusal/rebuild run, and 24/24 graph-level -suite all pass. The -RFC remains Draft and the implementation remains reachable only through a -feature-gated private engine seam: there is still no `@stream` syntax, -production/public enrollment, put/ack/fold API, operator drain/resume surface, -or fresh-read mode. +Phase A foundation, private Phase B1 core, and Phase B2-0 design closure. +RFC-024, RFC-025, and RFC-027 remain research-blocked at their recorded +evidence gates. Gate E0 first proved the bounded public Lance state classifier; +Phase A activated internal schema v7 with recoverable empty enrollment, durable +identity-keyed lifecycle authority, process-local admission/exclusion, and +strict format refusal/rebuild. Internal schema v8 now preserves that foundation +and adds stream-config v2, a root-scoped one-generation worker, watcher-backed +durable acknowledgement, exact replay/seal/retirement, and recovery-v11 +`StreamFold`. The private B1 evidence is green: the local cost matrix, +configured-RustFS warm-ack cell, genuine v7↔v8 binary refusal/rebuild run, and +24/24 graph-level suite all pass. B2-0 specifies explicit enrollment, +compare-and-chain tokens, trusted row attribution, manifest-selected current- +token state, persistent lifecycle revisions with bounded management receipts, +strict correction, Lance-owned reclamation with an enforced storage admission +watermark, and a graph-global `GraphHistoryBudget` checked by every manifest +publisher. +Together with the RC.1 source audit, two checked-in guards show why stock RC.1 +does not yet provide safe reclamation: the tests themselves prove generic +cleanup non-ownership and the deleted-successor-sentinel fencing hazard. +The RFC remains Draft and the implementation remains reachable only through a +feature-gated private engine seam: there is still no schema v9/config-v3/ +recovery-v12 activation, `@stream` syntax, production/public enrollment, +put/ack/fold API, operator drain/resume surface, or fresh-read mode. This page answers four practical questions: @@ -73,10 +81,12 @@ already-normalized vector values—and neither calls an external embedding provider nor invents unspecified derived fields. Native branch controls alone may proceed at `SEALED`, because they do not move table HEAD. This remains a private format/correctness core, not a streaming product: no production caller -can enroll, put, acknowledge, or fold a row. Phase B2 is the later public -activation and remains gated on attribution, reclamation/orphan cleanup, -correction, same-key retry sequencing, and persistent API/operator escape -work. +can enroll, put, acknowledge, or fold a row. Phase B2-0 has now closed the +explicit-enrollment, attribution, reclamation/orphan cleanup, correction, +same-key retry sequencing, persistent lifecycle/management-receipt, and graph- +global manifest-history designs. Phase B2 is the later implementation and +public activation; its first dependency is the reviewed Lance reclamation and +post-success-fence patch, not product endpoints. The largest remaining correctness boundary is topology: destructive recovery is serialized across every handle in one process, but not against a live writer @@ -196,7 +206,7 @@ retry rules, recovery classification, and the full owned limits. | [023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | **Implemented** | Internal schema v6 introduced exact-`id` PK metadata, closed keyed routing, typed conflicts, bounded replay, rebuild/refusal, and accepted performance evidence; v8 preserves that contract. | | [024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | **Research-blocked** | The first in-manifest BTREE candidate has a specified logical contract and flat indexed row/range work, but fails the complete physical-I/O gate. No head rows or heads format are active. | | [025 — Checkpoint retention](../rfcs/0025-checkpoint-retention.md) | **Research-blocked** | Lance tag/pin semantics pass, but the proposed in-manifest registry access shape is not history-flat after compaction. No checkpoint rows, `ogcp_` production tags, API, or cleanup integration are active. | -| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; Phase A foundation and private Phase B1 core implemented/evidence-green; public inactive** | Schema v8 preserves v7/recovery-v10 enrollment and adds config-v2 one-generation admission plus recovery-v11 strict fold. The local cost matrix, configured-RustFS warm-ack cell, genuine v7↔v8 refusal/rebuild, and 24/24 graph-level suite are green. MemWAL remains the strategic substrate, but only a feature-gated private seam can exercise B1: there is no production enrollment, `@stream`, put/ack/fold, drain/resume, or fresh-read surface. B2 remains separately gated. | +| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; Phase A/B1 implemented and evidence-green; B2-0 specified; public inactive** | Schema v8 preserves v7/recovery-v10 enrollment and adds config-v2 one-generation admission plus recovery-v11 strict fold. The local cost matrix, configured-RustFS warm-ack cell, genuine v7↔v8 refusal/rebuild, and 24/24 graph-level suite are green. B2-0 specifies explicit enrollment, v9 token/attribution, state-v2 lifecycle revisions/management receipts and correction, Lance-owned reclamation, and a graph-global reserve-first manifest-history budget; its two stock-RC.1 guards are checked in. None of that activates schema v9 or a production enrollment, `@stream`, put/ack/fold, drain/resume, or fresh-read surface. | | [027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | **Research-blocked** | The desired O(delta) classifier and fallback contract are specified. Selective live-row and deletion-delta discovery are not yet bounded, so `OrderedTableCursor` remains the correctness path. | | [028 — Stable schema identity](../rfcs/0028-stable-schema-identity.md) | **Implemented** | Rename-stable IDs, table incarnation, identity-derived paths, schema/recovery integration, and strict rebuild activation were introduced in v5 and remain active in v8. | @@ -207,7 +217,8 @@ retry rules, recovery classification, and the full owned limits. | Distributed recovery fence | Process-local queues cannot stop a live foreign process; Lance restore may orphan its commits and native refs lack conditional compare-delete. Supported destructive recovery remains one writer process per graph. | A separately designed and adversarially tested distributed fence before multi-process writers, background compensation, or cross-process exact maintenance recovery. | | History-flat authority | RFC-024/025 show flat BTREE rows/ranges/pages can coexist with history-growing manifest discovery or compacted bytes. No heads/checkpoint format is active; mutable tip caches and a second authority remain rejected. | A new Lance-native access shape—or revised measured operational contract—passes the original cold/warm, compacted/uncompacted, local/object-store gate. | | Internal history GC | Safe live-writer cleanup needs a durable resurrection/retention boundary; otherwise a stalled writer can recreate a collected version. | An evidence-backed cleanup watermark/fence before automated `__manifest` version GC. | -| MemWAL delivery | RC.1 initializes the system index and claims shards as separate effects without a caller-minted combined receipt or cross-process seal. Phase A recovers that gap exactly for main-only, one-shard, one-live-writer-process empty enrollment. At the row boundary, the durability watermark is writer-wide while batch positions reset after MemTable rollover, `put_no_wait` may mutate before returning `Err`, replay leaves its fresh BatchStore WAL watermark unset, and `wait_for_flush_drain` can lose a completed failure before a late waiter snapshots it. Neither batch positions nor WAL statistics are durable receipts. | Private B1 now closes exactly that bounded row case with one root-scoped serialized worker and one hard-bounded generation: it prevents rollover; holds admission before claim through watcher/retirement; owns invocation; routes non-empty replay and an already-flushed unmerged generation fold-only; marks the exact replayed BatchStore prefix WAL-durable through the pinned public RC.1 bridge; proves drain from frozen refs plus the authoritative shard manifest; retires; folds once; and reopens at a higher epoch. Any post-invocation ambiguity remains `AckUnknown`. B2 still requires durable attribution, bounded reclamation/orphan cleanup, correction, and same-key retry sequencing. A public receipt/seal or accepted distributed fence remains the exit for overlapping processes and failover. | +| MemWAL delivery | RC.1 initializes the system index and claims shards as separate effects without a caller-minted combined receipt or cross-process seal. Phase A recovers that gap exactly for main-only, one-shard, one-live-writer-process empty enrollment. At the row boundary, the durability watermark is writer-wide while batch positions reset after MemTable rollover, `put_no_wait` may mutate before returning `Err`, replay leaves its fresh BatchStore WAL watermark unset, and `wait_for_flush_drain` can lose a completed failure before a late waiter snapshots it. Neither batch positions nor WAL statistics are durable receipts. | Private B1 closes exactly that bounded row case. B2-0 closes the remaining logical design with compare-and-chain tokens, trusted hidden row metadata, a manifest-selected current-token participant, protocol-v2 lifecycle, and bounded correction. Implement and prove those contracts privately before adding a public caller. A public receipt/seal or accepted distributed fence remains the exit for overlapping processes and failover. | +| MemWAL reclamation | Stock RC.1 exposes evidence-level raw listing and manifest reads, not a complete classified inventory or safe MemWAL delete/GC primitive. Generic `cleanup_old_versions` leaves `_mem_wal` unchanged. Worse, deleting the successor's empty WAL fence sentinel can let a stale writer complete a WAL PUT and report watcher success because RC.1 has no post-success epoch check. Raw path deletion in OmniGraph is therefore forbidden. | Author Lance-owned durable inspect/plan/execute plus attempt/receipt recovery, post-success fence check, and bounded history checkpoint; open it upstream and pin the exact reviewed fork commit without waiting for a release. Then prove whole-cut/cursor eligibility, strong PUT/DELETE inventory plus multipart accounting/abort or durable accounting, stale plans, prune CAS, partial delete/lost result, orphan/unknown retention, and a source-derived enforced physical-growth reservation on local/RustFS before public admission. Separately prove graph-global manifest-history bootstrap/accounting across every writer and independent per-stream closure reserves. | | O(delta) merge | A version-column predicate is still O(rows) without a selective source, and deleted rows have no live version columns. Full ID differencing remains correct. | Bounded live-row and deletion/change discovery, exact shadow agreement, and a table-size-flat one-row-delete gate. | | Optimize provenance | Compaction/reindex has no stable caller-minted transaction covering the complete effect. Optimize therefore uses bounded, not exact, provenance. | Both an upstream maintenance transaction API and distributed recovery fencing. | | Remaining bounds/operations | Some long-running operations lack complete memory/time budgets; rollback waits for quiesced read-write open; recovery audit has no public query. | Incremental, independently owned hardening without widening format or topology. | @@ -242,6 +253,7 @@ write-path redesign. |---|---| | Lance surfaces consumed by RFC-022/023—transactions, branches, key filters, staged indexes, compaction, and shared sessions—remain compatible; separately surveyed tag/cleanup behavior is also unchanged | Keep the current architecture. PR #364 passed 22 surface guards and 129 runnable failpoint tests; the Gate-0 follow-up adds the 23rd guard plus checkpoint cost evidence. No format redesign is needed. | | Derived MemWAL datasets inherit the base store parameters and `Session`; `put_no_wait` returns an optional watcher whose completion is `Result<()>`, not a durable row coordinate. RC.1's watcher watermark spans the writer while active-MemTable batch positions reset on rollover; `put_no_wait` can also mutate before a later scheduling error. Replay leaves the fresh BatchStore watermark unset, and a late `wait_for_flush_drain` can miss a completed failure. | Shared-session propagation removes one integration concern, but the watcher is safe only inside B1's proved single-generation lifecycle. B1 acknowledges only watcher success, treats every post-invocation error as `AckUnknown`, prevents rollover, uses the pinned public replay-watermark bridge before fold-only reseal, proves the generation from refs plus authoritative manifest state, and retires/reopens before another generation. It does not create an exact combined enrollment receipt or cross-process seal. MemWAL is strategic, not experimental. | +| Generic cleanup ignores `_mem_wal`, and RC.1 does not recheck the writer epoch after a successful WAL PUT | Do not write an OmniGraph-side collector. B2-0 requires Lance-owned durable reclamation, post-success fencing, bounded reclaim-history checkpointing, and an enforceable growth reservation. We will author and pin that reviewed patch ourselves rather than make upstream merge/release timing the calendar; public admission remains closed until its local/RustFS crash and bound-validation evidence pass. | | Experimental, opt-in `DataOverlay` was added | Do not adopt it now. OmniGraph does not enable feature flag 64 or emit the operation; unknown foreign overlay effects remain fail-closed. DataOverlay's experimental status says nothing about MemWAL's status. | | RC.1 expands Lance write rejection from the three row-address names to all five surveyed virtual system-column names | OmniGraph now rejects `_rowid`, `_rowaddr`, `_rowoffset`, `_row_created_at_version`, and `_row_last_updated_at_version` during parsing and accepted-IR validation. A beta.21 development graph using a newly reserved row-version name must be exported with beta.21, renamed, and rebuilt. | | A genuine ordinary-schema beta.21 V2_2 graph forward-opened, queried, and merge-wrote under RC.1 | No general storage-format migration. The reserved-name exception is explicit rather than hidden behind a format bump. | @@ -265,23 +277,44 @@ or slightly lower incremental RSS. That is not a roadmap signal. configured-RustFS cost evidence. Treat those results as acceptance of one main-only, one-shard, one-live-writer-process generation—not as permission to expose a product API or claim general streaming throughput. -2. **Design the Phase B2 contracts before exposing rows.** Contributor - attribution, bounded reclamation/orphan cleanup, strict - correction/disposition, same-key `AckUnknown` sequencing/idempotency, - authoritative status, and persistent quiesce/resume/abort-drain/rebuild all - need accepted designs and crash evidence. Only then should schema, SDK, - HTTP, CLI, Cedar, and OpenAPI converge on the private core. -3. **Keep RFC-024/025/027 stopped at their research no-gos.** Their blockers are - independent of the v8 stream core; do not add heads, checkpoints, or - lineage-delta format state as incidental B2 work. -4. **Give a distributed recovery fence its own design and evidence gate.** +2. **Implement the B2-0 substrate dependency first.** Author the Lance-owned + durable opaque inspect/plan/execute reclamation primitive, attempt/receipt + recovery, sentinel-first manifest-named epoch claims, genesis/bootstrap plus + bounded claim/reclaim history checkpoint, and post-success epoch check; open + the upstream PR and pin the exact reviewed fork commit without waiting for + its release. Add complete local/RustFS whole-cut, lost-result, fail-closed + stock-RC.1 format refusal, strong PUT/DELETE inventory plus multipart + accounting, one per-binding reserve-first durable ledger, versioned/soft- + delete/Object-Lock refusal, bounded materialization attempts/control + headroom, and enforced physical object/byte reservation evidence. Never delete + `_mem_wal` paths from OmniGraph. +3. **Implement the B2 contracts privately in dependency order.** Activate + schema v9/config-v3/state-v2/recovery-v12 only with explicit enrollment, + compare-and-chain token authority, trusted hidden attribution, persistent + revisioned quiesce/resume/abort-drain with bounded management receipts, and + bounded `REPLACE`/`WITHDRAW` correction. Keep the public surface absent while + the crash, rebuild, enforced-watermark, and cross-version matrices converge. + Treat the per-binding watermark as `_mem_wal`-only. Separately initialize one + graph-global `GraphHistoryBudget`, make every manifest writer reserve its + publication and source-bounded physical-growth envelope, and retain dynamic + per-stream closure reserves through `SEALED` rebuild. Base/token/shared- + manifest history remains finite-lifetime until a later maintenance strand. +4. **Add product surfaces last.** Only after the private B2 machinery and + evidence are green should schema intent, SDK, HTTP, CLI, Cedar, OpenAPI, and + shutdown ownership converge on the same core. +5. **Keep RFC-024/025/027 stopped at their research no-gos.** Their blockers are + independent of the v8 stream core; do not add RFC-024 heads, RFC-025 graph + checkpoint rows/format, or RFC-027 lineage-delta state as incidental B2 + work. Lance's private reclaim-history bootstrap in step 2 is substrate + recovery metadata, not RFC-025 graph checkpoint authority. +6. **Give a distributed recovery fence its own design and evidence gate.** Define authority, expiry/renewal, fencing tokens, and crash semantics before implementation; require adversarial multi-process tests on local and object storage. -5. **Continue low-risk v8 hardening.** Add missing resource/time budgets, +7. **Continue low-risk v8 hardening.** Add missing resource/time budgets, preserve cost-at-history-depth gates, and reduce constant factors only where the existing authority model remains intact. -6. **Coordinate upstream without making it the calendar.** The useful +8. **Coordinate upstream without making it the calendar.** The additional Lance asks are replay initializing the per-MemTable WAL watermark, drain completion that cannot lose a finished error, recoverable MemWAL enrollment/admission, conditional native ref operations, exact maintenance @@ -291,13 +324,15 @@ or slightly lower incremental RSS. That is not a roadmap signal. - **Public MemWAL row activation:** Phase A and private B1 are green at their bounded gates; the RFC remains Draft and public activation remains off. - Phase B2 requires an accepted durable contributor-attribution design, bounded - reclamation plus retained-bytes stop, strict correction/disposition, - same-key `AckUnknown` sequencing/idempotency, - schema/SDK/API/CLI parity, cancellation ownership, authoritative status, and - persistent quiesce/resume/abort-drain/rebuild. The exact upstream - receipt/seal remains the preferred simplification and broader-topology gate, - not a reason to widen B1 while waiting. + B2-0 now specifies explicit enrollment, durable contributor attribution, + compare-and-chain sequencing, bounded reclamation plus an enforced retained- + storage admission watermark, strict correction/disposition, persistent + revisioned lifecycle with bounded management receipts, and graph-global + manifest-history admission with per-stream closure reserves. Phase B2 activation still + requires their implementation and evidence, schema/SDK/API/CLI parity, + cancellation ownership, and authoritative status. The exact upstream + enrollment receipt/seal remains the preferred simplification and + broader-topology gate, not a reason to widen B1 while waiting. - **Durable heads or checkpoints:** when a new current-authority access shape exists, run the full decision instrument before adding production rows or a format stamp. @@ -322,9 +357,11 @@ Do not: - implement RFC-024, RFC-025, or RFC-027 production paths behind a nominal feature flag before their blocking gate closes; - treat the private schema-v8 B1 core or its green evidence as permission to - expose a put/ack endpoint before B2 owns attribution, reclamation/orphan - cleanup, correction, same-key retry sequencing, and the persistent - operational escape; + expose a put/ack endpoint before the specified B2 attribution, token, + reclamation/enforced-watermark, correction, and persistent lifecycle contracts are + implemented and evidence-green; +- delete or rewrite `_mem_wal` objects from OmniGraph, or interpret generic + Lance version cleanup as MemWAL reclamation; - permit a base-table writer to advance any Phase A lifecycle's HEAD (including `SEALED`) before the Phase D witness-update/rebind adapter exists, or use a long-lived enrollment tag without first measuring retained files and @@ -355,6 +392,7 @@ Do not: | MemWAL bounded-enrollment Gate E0 (green decision harness; no row path) | [`memwal_enrollment_gate.rs`](../../crates/omnigraph/tests/memwal_enrollment_gate.rs), [`lance_surface_guards.rs`](../../crates/omnigraph/tests/lance_surface_guards.rs), [RFC-026 §12.1](../rfcs/0026-memwal-streaming-ingest.md) | | MemWAL Phase A lifecycle/exclusion/recovery | [`failpoints.rs`](../../crates/omnigraph/tests/failpoints.rs), [`forbidden_apis.rs`](../../crates/omnigraph/tests/forbidden_apis.rs), manifest/write-queue unit tests, [RFC-026 §12.2](../rfcs/0026-memwal-streaming-ingest.md) | | MemWAL private Phase B1 admission/fold/crash and cost evidence | [`memwal_stream.rs`](../../crates/omnigraph/tests/memwal_stream.rs), [`memwal_stream_cost.rs`](../../crates/omnigraph/tests/memwal_stream_cost.rs), worker/recovery unit tests, [RFC-026 §12.3](../rfcs/0026-memwal-streaming-ingest.md) | +| MemWAL Phase B2-0 reclamation ownership/no-go guards | [`lance_surface_guards.rs`](../../crates/omnigraph/tests/lance_surface_guards.rs), [RFC-026 §4.5](../rfcs/0026-memwal-streaming-ingest.md) | | Cross-version refusal/rebuild, including v6↔v7 and v7↔v8 | [`crossversion_upgrade.rs`](../../crates/omnigraph-cli/tests/crossversion_upgrade.rs) | Use [testing.md](testing.md) to find the existing owner before adding coverage. @@ -375,7 +413,10 @@ followed from either gate alone. Private B1 passed its separate gate: the 24/24 graph-level admission/fold/crash suite, genuine v7↔v8 old/new-binary refusal/rebuild, local cost matrix, and configured-RustFS warm-ack cell are green. That evidence activates only the schema-v8 private core; -the RFC remains Draft and every public B2 surface remains inactive. +the RFC remains Draft and every public B2 surface remains inactive. B2-0's two +additional guards prove why stock RC.1 generic cleanup and raw successor +fence-sentinel deletion cannot be the reclamation implementation; they are a +no-go boundary, not a shipped collector. ## Updating this page diff --git a/docs/rfcs/0026-memwal-streaming-ingest.md b/docs/rfcs/0026-memwal-streaming-ingest.md index 290720fc..0dfc5a1a 100644 --- a/docs/rfcs/0026-memwal-streaming-ingest.md +++ b/docs/rfcs/0026-memwal-streaming-ingest.md @@ -11,13 +11,17 @@ owner: OmniGraph maintainers # RFC-026 — MemWAL streaming ingest **Status:** Draft / Phase A foundation and Phase B1 private core evidence-green; -Phase B2 public streaming inactive +Phase B2-0 contracts specified below; public streaming inactive **Date:** 2026-07-10 **Gate E0 evaluated:** 2026-07-18 **Phase A foundation completed:** 2026-07-18 **Phase B1 private core green:** 2026-07-19 (internal schema v8, stream-config v2, recovery-v11, one feature-gated engine seam, and all private B1 acceptance gates in §12.3 passed) +**Phase B2-0 contract closure:** 2026-07-19 (§4.1–§4.6 specifies the +admission/token, attribution, revision-fenced lifecycle, correction, +reclamation, and graph-global closure-budget contracts; it activates no schema +or product surface) **Author track:** Maintainer design series **Depends on:** [RFC-022](0022-unified-write-path.md)'s unified write and generic recovery-sidecar protocol, plus @@ -106,7 +110,8 @@ The implementation remains deliberately split. **Phase B1 is now implemented privately** for one bounded generation, from admission through crash replay and one strict RFC-022 fold. It is reachable only through one feature-gated, doc-hidden engine test seam and is not a product surface. **Phase B2 remains -inactive** and owns schema intent, production first use, +inactive.** B2-0 now specifies its missing contracts in §4.1–§4.6, while B2 +implementation still owns schema intent, production first use, SDK/HTTP/CLI/OpenAPI/Cedar parity, minimum durable contributor attribution and reclamation, and the operator escape/correction path plus an explicit same-key `AckUnknown` retry contract. B1's implementation and evidence ledger is green @@ -119,7 +124,7 @@ folding, fold-time integrity, dead-letter atomicity, branch/schema quiescence, fresh-read cuts, resource bounds, observability, testing, and upgrade posture. It does not replace `load` or `mutate`, provide cross-query transactions, store -manifest mutations in MemWAL, create a second metadata authority, or weaken +manifest mutations in MemWAL, create a competing metadata authority, or weaken default snapshot isolation. Stream-mode deletes remain out of the first delivery and require the Lance tombstone surface plus a separate acceptance pass. @@ -142,6 +147,16 @@ It requires the table's immutable unenforced primary key to equal OmniGraph's merge key: `id` for nodes and edges. All occurrences of one key map to one shard and MemWAL applies last-write-wins ordering. +Initial B2 admission is self-contained and provider-free. The caller supplies +every required physical value, including vectors; the engine may perform only +version-pinned deterministic parsing, defaults, and normalization before token +mint. `@stream` is rejected at accepted-schema validation for a type whose +write path requires `@embed`, an external provider, or any other post-request +derived-field materialization. Runtime rechecks the accepted capability before +admission. A client may compute embeddings first and submit the physical vector, +but streaming never acknowledges a provider-dependent promise and retries never +re-call an external model under the same `write_id`. + Every stream contract is bound to RFC-028's `(stable_table_id, incarnation_id)` pair. A rename preserves that pair, so the logical stream contract and ownership of reject history remain continuous. @@ -158,12 +173,19 @@ logical identity; allowing a retry to append the same `id` twice would violate that contract. A future explicitly keyless, non-graph append-only table class may consume MemWAL append semantics under its own schema/API decision. -Stream ordering intentionally differs from the interactive fence: - -- interactive same-key writes outside an `OPEN` bounded stream serialize or - fail/retry loudly; an `OPEN` bounded stream refuses such a write before - effect until it is drained; -- same-key stream entries resolve by MemWAL generation/position order; +Stream ordering intentionally differs from the interactive fence. Lance still +resolves physical rows by generation/position LWW, but B2 does not expose +unconstrained arrival-order LWW as its retry contract. §4.1 admits a physical +row only when its predecessor token matches the current per-key stream token: + +- interactive same-key writes on an unenrolled table serialize or fail/retry + loudly. Once the table is enrolled, ordinary Mutation/Load remains refused + before effect for `OPEN`, `DRAINING`, **and `SEALED`** in B2. Draining enables + only the proved export/rebuild escape; it does not let a direct write bypass + `_stream_tokens`. Phase D must define a token-aware direct-write transition + and witness/rebind update before relaxing this refusal; +- same-key public stream entries form an explicit compare-and-chain sequence; + MemWAL generation/position order realizes only that already-validated chain; - duplicate keys inside one bulk-load input retain the existing load error. The schema and user docs state all three together. @@ -347,6 +369,25 @@ branch-scoping question is proven by a surface guard and end-to-end test. Later overlapping-process enrollment/failover still requires the upstream receipt / admission lifecycle or a separately accepted distributed fence. +Phase B2 does not hide enrollment inside the first ingest line. `@stream` +declares a table eligible but leaves it `UNENROLLED`; read-only status reports +that declaration and no stream incarnation. An authorized explicit enroll +request carries a caller-minted non-nil `enrollment_request_id`. Under the same +recovery adapter, the engine mints the never-reused physical enrollment/shard +IDs **and** logical `stream_incarnation_id`, creates §4.5's genesis retention +body/pointer and fail-closed details kind, and publishes the exact `OPEN` state. +The request ID and canonical caller intent (declared schema/config plus expected +unenrolled witness), together with the engine-minted binding/incarnation result, +are fixed in recovery before the first effect. The published state also retains an +immutable `EnrollmentReceipt` containing the original request ID, canonical +intent digest, stream incarnation, physical binding, and initial lifecycle +revision. Recovery deletion therefore cannot erase enrollment idempotency. +Same ID/intent after a lost result returns that receipt; same ID with another +intent conflicts; a different ID after enrollment returns effect-free +`already_enrolled` with the current incarnation. `stream ingest` requires that +exact incarnation and returns `StreamNotEnrolled` before body admission on +`UNENROLLED`; it never auto-enrolls or invents a first-row exception. + ## 4. Future public activation This section is the Phase-B2 wire contract, not part of the Phase-B1 private @@ -358,28 +399,1219 @@ into a different root. In-place SchemaApply and maintenance on an enrolled table stay refused until Phase D supplies automatic drain, witness update, and rebind. +### 4.1 Durable row identity and same-key retry safety + +B2 deliberately makes the public contract stricter than blind upsert. Every +input row carries a client-owned logical write token: + +```text +StreamWriteEnvelope { + stream_incarnation_id: UUID, // exact logical stream incarnation + write_id: UUID, // non-nil, stable across retries + predecessor_token: StreamToken?, // opaque exact token returned for id +} + +TrustedStreamRowMetadata { + stream_incarnation_id: UUID, + contributor_id: String, // server/engine derived; never client supplied + write_id: UUID, + predecessor_token: StreamToken?, + stream_token: StreamToken, + fold_base_token: StreamToken?, // token before this generation's chain + chain_depth: u32, // 1-based within this generation for id + origin: Admission { admission_attempt_id: UUID, caller_ordinal: u64 } + | Correction { correction_id: UUID, plan_ordinal: u64 }, + payload_digest: sha256, +} +``` + +`write_id` is the caller's idempotency label and remains stable for an exact +retry. The occurrence/idempotency key is `(table incarnation, stream +incarnation, logical id, predecessor_token, write_id)`. Authenticated +contributor and payload digest are immutable attributes bound to that key, not +a way to turn reuse of the same key into another change. A caller that +deliberately reuses the UUID against a newer predecessor is asking for a +distinct occurrence; SDKs still mint a fresh UUID for each new change. +An `Admission` attempt ID names one possibly ambiguous call to the private B1 +worker; it is not a WAL position or receipt. `Correction` is a distinct durable +origin because correction creates no physical admission attempt. Exactly one +origin variant is present. `predecessor_token = null` means the caller expects +no earlier stream token, which covers both an absent row and a row that predates +public streaming. A blind wildcard predecessor is not supported because it +recreates the stale-retry overwrite. + +`stream_incarnation_id` is minted when the logical token authority is created +and returned by status/enrollment. It survives a same-table physical rebind but +changes whenever a strict rebuild/re-enrollment resets token state. Every +request compares it before Lance is called; a mismatch is effect-free +`StreamBindingChanged`. This closes the null-predecessor ABA in which a delayed +first-write retry from an old root/enrollment could otherwise enter a freshly +empty token authority. Table drop/re-add is already fenced by the distinct +stable table incarnation, and the stream incarnation is checked as well. + +`StreamToken` is an opaque 32-byte, versioned, domain-separated SHA-256 value +computed by the trusted engine from the stable table/incarnation, logical key, +stream incarnation, predecessor token, `write_id`, contributor, and payload +digest. It excludes the physical admission-attempt ID, so an exact retry derives +the same token. The caller only stores and echoes the bytes returned by +`durable`, `already_durable`, `withdrawn`, or authorized +`stream_sequence_conflict` results; it does not construct or parse them. B2's +stream status is lifecycle-level and exposes no per-key token lookup. An +unconfirmed candidate from `AckUnknown` is never a valid predecessor; the +caller retries that same occurrence against its original predecessor until it +receives a confirmed current/durable result or a sequence conflict. This hash +chain prevents a repeated UUID from making an old retry look current after +`X -> Y -> X`: those three occurrences have different full tokens because their +predecessors differ. + +The v1 wire form of every 32-byte stream token, block token, and externally +asserted protocol digest is exactly `sha256:` followed by 64 lowercase +hexadecimal characters. No uppercase, padding, whitespace, alternate prefix, +or base64 form is accepted. Parsing verifies prefix, length, alphabet, and +canonical round-trip before any recovery or Lance call. Stream-config v3 pins +this `sha256-lowerhex-v1` representation; changing it requires a new wire/config +version rather than permissive dual decoding. + +The trusted metadata is stored in one reserved nullable physical struct, +provisionally `__omnigraph_stream_v1`, in every stream-capable base-table +schema. Pre-stream/direct rows have a null top-level struct. For a stream row, +the incarnation, contributor, write ID, stream token, chain depth, payload +digest, and origin tag are non-null; predecessor and fold-base tokens are +nullable when the chain begins at null. The tagged origin has variant-specific +nullability: exactly the `Admission` or `Correction` children selected by its +non-null tag are populated. The compiler reserves +the field; logical query results, ordinary export, user schema reflection, and +user-declared indexes never expose it. Lance therefore carries the exact +metadata atomically with the row through WAL, replay, flushed generation, +fold, and base-table publication. This is intentionally a base-schema field: +RC.1's `put_no_wait` accepts the base schema and adds only Lance's own +`_tombstone` field, so a sidecar column or private WAL rewrite would not be a +valid public integration. + +The engine computes `payload_digest` after all pre-ack normalization, +defaulting, and materialization. Its versioned, domain-separated input includes +the stable table/incarnation, accepted schema hash, and deterministic +type-aware bytes of every logical field while excluding the stream metadata +itself. Blob content is hashed from the stored bytes, not from an external URI +descriptor; a payload that still requires post-ack dereferencing is not +self-contained and is refused. Stream-config v3 pins the digest version and +canonical encoding. + +A graph-internal Lance dataset, provisionally `_stream_tokens.lance`, is the +sole **post-fold** per-key sequencing authority. It is initialized by the new +format strand and addressed only through the exact version selected by +`__manifest`, never by its raw Lance HEAD. Its current row is keyed by +`(stable_table_id, incarnation_id, logical_id)` and carries the immutable +`origin_enrollment_id` under which that token became current, stream +incarnation, current stream token, write ID, predecessor token, +`PRESENT | WITHDRAWN` disposition, contributor, payload digest, exact tagged +origin, and terminal correction actor/operation when present. A token row is +current protocol state, not an admission log; there is at most one current row +per logical graph key. Phase-D physical rebind does not rewrite every token row: +responses report the separately revalidated current binding, while +`origin_enrollment_id` remains attribution history. A token-authority reset +instead mints a new stream incarnation. + +This internal table is not a competing authority: + +- before fold, the metadata embedded in the durable MemWAL row plus the + existing B1 worker cut is authoritative; +- after fold, the manifest-selected token-table version is authoritative; +- a normal fold or correction stages the base-table and token-table effects in + one RFC-022 recovery envelope and exposes both in one `__manifest` CAS; +- there is no per-ack token-table commit and no independently writable token + path; and +- for `PRESENT`, the winning base row's hidden metadata must agree with its + token row. A mismatch is uncovered corruption and fails closed, but the base + copy is evidence/attribution rather than a second sequencing source. + +The token table is necessary for explicit `WITHDRAW`: an acknowledged write +can become terminal while the graph key remains absent or its prior visible +value remains unchanged. A hidden field on the visible base row cannot +represent that state. A current `WITHDRAWN` token is never aged out by GC; it +remains current until a later accepted change explicitly names that full token +as predecessor. The successor then preserves it in the hash chain. Thus an old +retry never becomes admissible merely because its terminal disposition aged +out, while the table still keeps only one current row per graph key. + +Admission keeps B1's `charge -> shared admission -> same-key input queue -> +worker mode` order. Under that order, it derives the current token from the +manifest-selected token table and overlays only watcher-confirmed rows already +in the one live generation. That overlay is a warm derived projection: it is +updated only after watcher success, discarded on ambiguity/retirement, and +never treated as restart authority. Reopen with possible residue remains +fold-only, so a new put cannot race an unclassified token. Repeated IDs inside +one caller request are evaluated in caller order, but one physical Lance put +contains at most one fresh occurrence per logical key. A later occurrence waits +for the prior run's watcher and is then reclassified against its confirmed +overlay; an exact duplicate becomes `already_durable`, while a successor must +have obtained and named the confirmed token in a later request. The adapter +never treats an opaque pending candidate as caller authority. + +The overlay also mints a compact fold certificate for each same-key chain. The +first row for a key in one generation stores the manifest-selected current +token as `fold_base_token` and depth 1; every later admitted row inherits that +base and increments the depth. Lance's LSM scanner may collapse `P -> X -> Y` +to winner `Y` before fold, so the winning row must still prove that its admitted +chain started at manifest token `P`. Fold compares the winner's stream +incarnation and fold base to the manifest-selected token row, recomputes its +candidate token, validates the nonzero bounded depth, and fails closed on a +mismatch. It neither infers the chain from LWW order nor permanently retains +superseded `X`. These reserved fields are trusted engine metadata under the same +no-raw-writer support boundary as contributor attribution. + +For each row, the engine first verifies the stream incarnation, normalizes the +payload, derives the candidate `stream_token`, and classifies the occurrence +**before** minting a new `Admission` origin or fold certificate. Candidate +equality compares only token-bound preimage fields; persisted origin and +certificate belong to the already-current row and are returned rather than +recreated. The rules are: + +1. if `(write_id, predecessor_token)` equals that pair on the current token row + but contributor or payload digest differs, the result is terminal + `StreamIdempotencyConflict`; +2. if the candidate token equals the current token, every token-bound preimage + field must match. A hash-equal field mismatch is corruption. For current + `PRESENT`, the persisted base hidden metadata must agree with the token row + and the result is `already_durable` with its persisted origin/certificate. + For current `WITHDRAWN`, the token row must agree with its terminal correction + receipt/origin; the base row is deliberately allowed to be absent or carry + the older visible value and is not compared as a current-token copy. The + result is the terminal `withdrawn` disposition. Neither case mints an origin, + certificate, or `put_no_wait`; +3. otherwise `predecessor_token` must equal the complete current token, + including null-to-null. A mismatch is an effect-free + `StreamSequenceConflict` that returns the current token; and +4. only the remaining case may call Lance; watcher success advances the warm + overlay to the candidate token. + +The accepted chain is therefore `P -> X -> Y`, where each letter denotes the +full opaque token rather than only its caller UUID. If `X` returns `AckUnknown`, +recovery first makes its possible residue fold-only. A later exact retry sees +either `P` and may write `X`, `X` and returns `already_durable`, or a newer `Y` +and returns `StreamSequenceConflict` without calling Lance. The unsafe +`X(unknown) -> Y(durable) -> retry X` overwrite is no longer representable. +Two concurrent `X(P)` and `Y(P)` calls serialize: one wins and the other +conflicts; `Y` is accepted after `X` only when it explicitly names `X`. + +No automatic retry follows invocation. `AckUnknown` carries its +`admission_attempt_id`, caller ordinal range, binding, and logical write IDs but +makes no durability claim. It may include the deterministic candidate stream +token but must label it unconfirmed; generic stream status never resolves that +attempt. +A later explicit retry may report the exact write current or return a +current-token sequence conflict; neither result retroactively claims whether +the earlier physical attempt was durable. + +### 4.2 Attribution and audit boundary + +The authenticated `contributor_id` is resolved at the trusted engine boundary +after Cedar authorization. Only attempts to supply the reserved +`$stream.contributor_id` or `$stream.origin` metadata are rejected; a logical +schema property that happens to be named `contributor_id` remains ordinary user +data. The HTTP bearer principal, embedded SDK actor, and remote CLI actor all +reach the same engine method. Because contributor identity participates in the +token, an exact retry is actor-bound: retrying the same write tuple after an +actor change returns `StreamIdempotencyConflict` rather than impersonating the +original contributor. The contributor is embedded before `put_no_wait`, so a +durable row cannot become an unattributed fold winner after restart. + +The fold's graph commit is authored by the system actor +`omnigraph:stream-fold`. Its fixed recovery/lineage payload carries a sorted +visible-contributor count, visible-write count, and SHA-256 digest over the +sorted winning `(contributor_id, stream_token, write_id, tagged_origin)` +tuples. Recovery publishes that pre-bound summary rather than +recomputing a possibly different one. Exact winning attribution remains on the +base row and in the current token row. A `WITHDRAWN` token additionally records +the authenticated correction actor and operation. + +B2's minimum audit contract is intentionally about graph-visible winners and +current terminal withdrawals. An acknowledged row that a later caller +explicitly supersedes through the predecessor chain is not a permanent graph +commit and is not promised an unbounded per-attempt audit record after WAL +reclamation. If permanent audit of every acknowledged-but-superseded attempt +becomes a product requirement, it needs a separate retention/cost decision and +an internal Lance participant; it must not appear as per-attempt manifest rows, +sidecars, or a custom log in this RFC. + +### 4.3 Persistent lifecycle and operator authority + +B2 retains `OPEN | DRAINING | SEALED` but upgrades the stream-state payload to +protocol v2. `DRAINING` is a durable operation, not an in-memory observation. +Every state-v2 row carries a strictly monotonic `lifecycle_revision`; every +successful publication of that row increments it exactly once, including a +publication that changes only a witness, retention summary, or last-fold +summary. The row also retains the immutable enrollment receipt and a bounded +set of management receipts described below. The complete manifest row +additionally carries: + +```text +DrainDescriptor { + drain_id, + operation_expected_revision, + operation_request_digest, + goal: SEALED | OPEN_AFTER_FOLD, + initiating_actor, + initiated_at, + expected_binding, + expected_current_head_witness, + target_epoch_floor_by_shard, + guarded_operation: null, // B2; Phase D may fill this +} + +StrictBlock { + block_token, + enrollment_id, + shard_id, + generation, + generation_path, + shard_manifest_version, + replay_cursor, + base_current_head_witness, + validation_contract_version, + violation_code, + violation_digest, + correction_view_digest, + offending_key_count, + correction_revision, +} + +SealedProof { + drain_id, + shard_manifest_version, + writer_epoch, + replay_cursor, + current_generation, + base_merged_generation, + base_current_head_witness, + verified_empty_digest, +} + +LastFoldSummary { + operation_id, + graph_commit_id?, + exact_generation_cut, + outcome: PUBLISHED | STRICT_BLOCKED, + input_rows, + input_bytes, + visible_rows, + visible_bytes, + recorded_at, +} +``` + +`OPEN` has no drain, block, or sealed proof. `DRAINING` has exactly one drain +and at most one exact strict block. `SEALED` has one exact empty proof and no +strict block. Every transition compares the complete prior lifecycle row; no +optional serde default reinterprets a v8 row. Lifecycle-only CASes are audited +metadata changes and do not move graph lineage. + +Every externally initiated mutating management request **after enrollment** is +compare-and-set, not "act on whatever is current." It carries a non-nil +operation ID, the expected +`lifecycle_revision`, and, when it addresses a drain or block, the expected +`drain_id` or `block_token`. Quiesce uses its `drain_id` as the operation ID; +explicit fold uses `fold_operation_id`; resume/abort-drain uses `resume_id`; +correction uses `correction_id`. The idempotency occurrence is `(stream +incarnation, operation kind, operation ID)`; the expected revision is part of +its canonical request digest. Before checking the expected revision, the engine checks the bounded persisted +`ManagementReceipt` set: same occurrence and canonical request digest returns +the recorded result, while the same occurrence with another digest is +`StreamIdempotencyConflict`. With no receipt, a revision or addressed-authority +mismatch is effect-free `StreamLifecycleChanged`; it never retargets the call. +Each terminal successful state-changing request records its kind, canonical +digest, from/to revision, actor, complete bounded canonical result payload, and +result digest in the same terminal publication. A multi-publication quiesce is +not complete at `OPEN -> DRAINING`: its `DrainDescriptor` plus recovery sidecars +are in-progress authority, same-ID retry resumes that exact plan, and its +terminal management receipt appears only with `SEALED` (or an explicit durable +terminal failure). Internal continuations use the persisted operation identity, +request digest, and exact current revision rather than minting a new caller +operation. Enrollment is the one no-prior-state exception: its CAS token is the +exact declared-schema identity/configuration witness plus +`enrollment_request_id`, and its separate immutable `EnrollmentReceipt` supplies +the same lost-result guarantee. + +Stream-config v3 hard-bounds management-receipt count and bytes and reserves +enough slots/bytes for the complete worst-case closure path. Receipts remain for +the stream incarnation; they are not silently evicted. Non-closure management +stops before consuming the closure reserve, and a full stream can still reach +`SEALED` for rebuild. Before enrollment publishes `OPEN`, before each generation +is admitted, and before resume republishes `OPEN`, the engine proves and +reserves the full remaining management-receipt count/byte envelope for that +generation's fold/block/correction/abort/requiesce/seal path. A current fold is +part of that reserve, not an optional later allocation. Status reports receipt +used/reserved/limit values. A new root/stream incarnation makes an old request a +binding mismatch. Thus a delayed retry from an earlier drain/resume cycle can +only return its receipt or a stale-revision error; it can never begin a later +cycle. This bounded receipt set and its worst-case serialized growth are charged +by the graph-global history budget in §4.5. + +`SEALED` is immutable with respect to shard retention state. A new ordinary +claim, reclaim, or checkpoint is refused while sealed; the only epoch movement +from that state is the recovery-covered `StreamResume`, which consumes the old +proof and publishes `OPEN`. Rebuild reads the unchanged proof. Retention work +must finish while `OPEN` or `DRAINING`, and quiesce resolves every pending +claim/reclaim/checkpoint plus its lifecycle publication before constructing the +final proof. + +For every `DRAINING` row, +`drain.expected_current_head_witness == current_head_witness` is an invariant. +Every graph-visible drain fold or correction that advances the base table +rewrites both copies atomically; only `drain_id`, goal, initiating actor/time, +expected binding, and guarded operation remain stable. A `SealedProof` carries +the final equal witness. “Preserve the drain” below always means preserve that +stable operation identity and intent, not preserve stale descriptor bytes. + +The public state machine is: + +```text +OPEN --quiesce or blocked fold--> DRAINING +DRAINING --exact empty-cut proof--> SEALED +SEALED --StreamResume recovery--> OPEN +DRAINING --abort-drain recovery--> OPEN // only after the rules below +``` + +Quiesce requires a caller-minted non-nil `drain_id` and expected lifecycle +revision, acquires the common stream-admission lease exclusively, runs the +recovery barrier, waits for every watcher/retirement owner, and CASes +`OPEN -> DRAINING` before a new physical drain effect. The durable descriptor +is the restart plan. Under it the worker claims a higher epoch, classifies and +folds each exact cut through recovery-v12 `StreamFold` drain mode, independently +proves no active/frozen/replay tail plus shard/base merge agreement, then CASes +`DRAINING -> SEALED` with the exact proof. It never holds a table queue while +waiting for a fold that needs that queue. The management-receipt rules above +make the exact operation retry-safe and reject a stale request instead of +retargeting it. Restart continues `DRAINING` and never auto-opens it. + +Every B2 epoch advance—cold open, quiesce, resume/abort, reclaim, and +checkpoint—uses one patched Lance claim envelope. Lance first persists a +claim-attempt record, conditionally writes the decoded-empty successor sentinel, +then CASes a shard manifest that names the sentinel position/digest and advances +the epoch. It durably terminals the attempt with the exact sentinel/manifest +receipt before returning success. Ordinary claims, including checkpoint claims, +bind the complete authenticated WAL/sentinel tail and conditionally create their +sentinel at checked `max_authenticated_position + 1`; unknown tail, collision, +or overflow refuses. They +preserve the prior +`replay_after_wal_entry_position`. They must replay/classify every object after +that cursor and may advance it only through the existing flush/coverage proof; +they never skip data-bearing WAL merely because the later sentinel is empty. +The sole claim-time cursor exception is §4.5's reclamation successor: only after +its quiescent whole-cut proof establishes no data-bearing tail may its manifest +advance the cursor to the new sentinel. Only the exact manifest-named claim may +proceed to replay, and no new put is admitted until that classifier permits it. +A crash before the CAS is same-attempt recovery state; a foreign object or +movement fails closed. This replaces RC.1's manifest-before-sentinel gap for B2 +and gives every claim recoverable sentinel authority. The open/admission barrier +resolves or refuses pending claim attempts before another claim. +Claim IDs, terminal receipts, retry horizons, and checkpoint compaction follow +§4.5; they are counted durable substrate state rather than an unbounded audit +log. + +Drain mode is not the implemented B1 `OPEN` fold with a relaxed check. Its +sidecar binds the complete expected `DRAINING` row and `drain_id`; publication +preserves the drain identity/intent while advancing the base pointer, both +equal `CurrentHeadWitness` copies, epoch floor, token pointer, merged cut, and +durable `LastFoldSummary`. A different drain or `OPEN`/`SEALED` row is a +read-set conflict. A permanent validation failure attaches `StrictBlock` to +that same descriptor without changing its goal and writes +`LastFoldSummary(outcome = STRICT_BLOCKED, graph_commit_id = null)` with the +exact operation, cut, input rows/bytes, zero visible rows/bytes, and time in the +same lifecycle CAS. + +Resume and abort-drain use one dedicated roll-forward-only recovery-v12 +`StreamResume` payload. Its `Armed` form binds the complete expected lifecycle +row and revision, caller `resume_id`, canonical request digest, binding, +configuration, base witness, graph-branch topology, fixed actor/operation, an +`OPEN` template, management receipt, and minimum next epoch floor. The exact +achieved epoch is unknowable before the writer claim. After claiming, the +adapter durably records `EffectsConfirmed` with the exact sentinel/epoch, +sentinel-naming shard manifest/replay cursor, replay disposition, and final +`OPEN` row; only that row may publish. Recovery +never compensates an epoch or fence sentinel: while admission remains closed it +may claim a still-higher epoch, then records a new exact confirmation before +publication. An already-visible byte-identical `OPEN` row finalizes the +sidecar; a divergent `OPEN` row, binding, witness, topology, or uncovered +residue fails closed. + +Plain resume accepts only `SEALED`, revalidates schema/PK/config/format, exact +physical binding, sealed proof, and the bounded no-named-graph-branch topology. +Abort-drain is explicit and accepts only `DRAINING`; both calls require a +caller-minted `resume_id` and expected lifecycle revision. Abort additionally requires +that no guarded operation began, the binding and complete current DRAINING row +(including its equal current witnesses) still match, every background +seal/abort owner settled, and no unmerged or strict-blocked cut remains. It may +follow graph-visible folds already completed by the drain, but never reopens +around their residue. A named branch created after quiesce leaves resume safely +`SEALED`. + +Abort is therefore not a skip-invalid escape. If acknowledged residue or a +strict block remains, the only forward path is retry fold or exact correction; +after residue is graph-visible and the block is cleared, an otherwise eligible +unguarded drain may abort. A quiesce that keeps `goal = SEALED` may instead +continue to the sealed proof. This rule is identical in §8 and the B2 gates. + +Minimum status is one authoritative manifest row plus a bounded cut-consistent +physical observation. In the supported B2 profile it takes stream admission +exclusively, read-only lists/classifies relevant recovery intents, settles every +writer/watcher/flush/retirement/reclaim owner to the status deadline, reads +lifecycle/token authority plus the shard witness, performs exact inventory, +and rereads those authorities before release. It never resolves recovery. +Movement returns typed `StatusChanged`; failure to settle returns typed +`StatusBusy`. It reports lifecycle/binding/configuration, +`lifecycle_revision`, authoritative and observed epoch, drain ID/goal/phase, +pending generation +rows/bytes, replay and merge cut, retained WAL/generation/orphan/unknown bytes, +strict block token/code/revision, correction and management receipt +spare/use/reserve/limits, the persisted `LastFoldSummary`, graph-global manifest +history used/pending/per-stream reserve/aggregate reserve/limits/source-bound +version, relevant recovery operation, and exact rebuild-readiness reasons. +Mutable WAL cursor statistics are labeled hints and +never used as receipts. A listing error, overflow, or unknown classification is +an explicit status error; the read-only status call does not mutate lifecycle. +Every later admission independently requires its own under-gate token-spare, +management-receipt closure, graph-history, and storage-meter/inventory preflight and refuses before +`put_no_wait` when any proof is unavailable. + +Rebuild preflight requires `SEALED` and re-runs under closed admission plus +schema/branch/table gates. The sealed proof must still match the exact +base/shard state; no relevant sidecar, active/frozen/replayable tail, unmerged +generation, unattributed winner, unresolved token, or uncovered inventory may +remain. Export/cutover re-runs the proof. A prior status response is never an +authority token. + +`verified_empty_digest` is domain-separated over the exact binding, +configuration, stream incarnation, base witness, ordered shard-manifest/ +generation inventory, replay/merge cursors, retained/unknown classes, and every +decoded empty WAL fence sentinel. A sentinel may remain beyond the authoritative +replay cursor only when its exact position/digest/epoch is authenticated by the +retention checkpoint plus retained claim/manifest successor chain. Exactly one +sentinel for the achieved current epoch must remain, be decodable, and be named +by the latest shard manifest. Older authenticated sentinels from the same +shard/enrollment and an epoch no greater than that floor are harmless fence +history and remain included in the digest until reclamation removes them. A +data-bearing entry beyond the cursor, malformed or unauthenticated sentinel, +second current sentinel, or future/foreign epoch fails the proof. + +### 4.4 Bounded strict correction and disposition + +A permanent fold validation failure discovered from `OPEN` CASes the lifecycle +to `DRAINING(goal = OPEN_AFTER_FOLD)` with the exact `StrictBlock` while +admission is still exclusively closed. Before validation, the explicit fold +attempt pre-mints a non-nil `failure_drain_id` and fixes the authenticated +`stream_manage` actor, initiation time, binding, and current witness in its +plan. The conditional failure CAS installs that exact engine-minted ID and +actor in the new `DrainDescriptor`; a lost reply or retry reads the persisted +descriptor and never mints a replacement. Caller-minted `drain_id` is required +only for the public quiesce operation. If the fold already belongs to a +durable `DRAINING` operation, including quiesce, the CAS attaches the block to +that complete row without changing its `drain_id` or goal; a quiesce therefore +remains `goal = SEALED`. The same CAS writes the exact +`LastFoldSummary(outcome = STRICT_BLOCKED, graph_commit_id = null)` described +above. A correction is an operator-authorized management operation carrying the +expected lifecycle revision and keyed by `(block_token, correction_id)`; the +token hashes the immutable cut, base witness, violation, and correction +revision. Its management receipt and correction receipt are published together. +Same operation occurrence plus the same plan is idempotent, the same occurrence +with another digest is a conflict, and a stale revision/block token is +`StreamLifecycleChanged`/`ReadSetChanged` before effect. + +The retained immutable generation is also the authority for a bounded +correction-planning view. `StrictBlock` stores the validator-contract version, +offending-key count, and a canonical digest over sorted entries containing +logical key, current blocked-winner stream token, schema-safe violation code/ +field path/group, stable validator-defined violation-instance ID, and allowed +`REPLACE | WITHDRAW` actions. Byte-identical duplicate entries are coalesced; +the remaining entries sort lexicographically by their **complete canonical +entry bytes**, including the action set and instance ID. Each resulting entry +receives a stable zero-based `entry_ordinal`, which is included in the digest. +Multiple distinct violations for one key therefore remain independently +pageable without a tie whose order can flip. It does not store or expose whole +row payloads. Under exclusive admission, the read-only block- +inspection operation lists relevant recovery without resolving it, binds the +complete `DRAINING` row/block/base witness, scans the at-most-8,192-row/32-MiB +cut, reruns that pinned validator, and requires count/digest equality. It returns +pages in that canonical order, capped by both entry count and serialized bytes, +with an opaque cursor bound to `(block_token, correction_view_digest, +lifecycle_revision, last_entry_ordinal)`, then rereads the complete authority +before release. Each page also returns that bound revision. +Movement is `BlockChanged`; missing cut data or digest disagreement is fail- +closed corruption. GC cannot reclaim the generation while the block exists. +This makes the operator's predecessor tokens and action choices recoverable +after a lost fold response or restart without creating another mutable plan +authority. + +The trusted engine—not the caller—computes the versioned, domain-separated +`correction_plan_digest` from the stream incarnation, block token, correction +ID, authenticated actor, accepted schema hash, and canonical ordered action +encoding (including complete normalized replacement bytes). A client may send +an optional expected digest as an equality assertion, but cannot choose the +receipt identity. Replaying an ID as another actor or with any differently +encoded action is therefore a conflict. + +Idempotency is durable, not inferred after the block disappears. The +manifest-selected token participant also stores an immutable +`CorrectionReceipt` keyed by `(stream_incarnation_id, block_token, +correction_id)` with plan digest, actor, fixed graph commit/result, and final +lifecycle/token digest. A retry checks that receipt before declaring the block +stale: exact ID/digest returns the recorded result, while the same ID with a +different digest is `StreamIdempotencyConflict`. Receipts are rare operator +records, not per-ack admission history. Stream-config v3 contains hard +`max_correction_receipts`, `max_correction_receipt_bytes`, and source-derived +`max_single_correction_receipt_bytes` values. Before the first WAL effect of +each one-live-generation admission, the engine checks under stream admission +and the token gate that persisted receipt usage plus one slot and the full +single-receipt byte bound still fit. That spare cannot be consumed by another +operation while the generation is live: correction requires its exact blocked +generation and admission is closed. Cold replay reconstructs the same spare +from the persisted generation plus current receipt usage. If it cannot be +proved, new admission returns typed `CorrectionReceiptCapacityExceeded` before +`put_no_wait`, while fold/quiesce/rebuild remain available. + +Under the token gate, a correction transfers that generation's spare to the +exact canonical receipt before sidecar arm. The source-derived per-receipt +maximum must cover every legal canonical receipt; exceeding it is a fail-closed +format/invariant violation that the activation matrix proves unreachable, never +an ordinary post-admission capacity outcome. A successful fold releases the +unused logical spare before another generation may admit. +The already-reserved blocked generation therefore always has room for one +terminal correction even when the retained-receipt totals then reach their +caps. Receipts are retained for the stream incarnation. Once no generation or +block remains, full totals stop new admission and a quiesce can still reach +`SEALED` for export/rebuild into a new root/new stream incarnation. Status +reports used, reserved, and limit values. Public defaults require storage +evidence before activation; an unspecified, advisory, or admission-only check +without the terminal spare is invalid. + +Correction targets only keys whose LWW winner is in that blocked cut; adding a +new key is out of B2. It supports two explicit actions: + +- `REPLACE(key, write_id, predecessor_token, complete_row)` supplies one + complete normalized row. Its predecessor must equal the blocked LWW winner's + full stream token for that key. The correction actor is trusted attribution, + and the replacement gets the same versioned payload digest, derived successor + token, and hidden metadata as a public row, but with tagged + `Correction { correction_id, plan_ordinal }` origin rather than a fabricated + admission attempt. It inherits the blocked winner's exact + `fold_base_token` and stores checked `chain_depth + 1`; it does not reset the + certificate merely because correction bypasses MemWAL. Public admission caps + one generation at depth 8,192, and the one terminal correction successor caps + the certificate at `MAX_STREAM_CHAIN_DEPTH = 8,193`. Overflow or any larger + value is effect-free refusal. +- `WITHDRAW(key)` terminally dispositions the blocked current write while + leaving the prior manifest-visible graph value unchanged. It is not a graph + delete. The token table records that current write as `WITHDRAWN`, including + the correction actor/operation, so an absent prior graph row cannot make an + old retry admissible. + +Unmentioned keys retain the original generation's LWW winner. The correction +does not append another MemWAL generation and does not create a custom side +log. Under exclusive admission it scans the one immutable cut, applies the +bounded overlay in memory, reruns complete fold validation, and enforces the +8,192-row/32-MiB post-overlay bound. A validation failure creates no sidecar, +base effect, token effect, lifecycle movement, or correction success. + +Once valid, recovery-v12 `StreamCorrection` binds the prior complete +`DRAINING` row/block token, correction ID/digest, exact generation cut, exact +base and token-table transaction chain, fixed lineage/attribution, immutable +correction and management receipts, and intended lifecycle/token outcomes. The base transaction +applies one bounded keyed-upsert batch with at most one image per resulting +`PRESENT` key and marks the original generation merged. The token participant +then publishes at most 8,192 resulting `PRESENT | WITHDRAWN` current-token rows +in deterministic chunks and the one `CorrectionReceipt` row in a final separate +transaction on that same dataset. Each transaction independently obeys the +sealed 8,192-row/32-MiB bound; the receipt transaction also obeys the stricter +source-derived single-receipt byte bound. Stream-config v3 fixes a source- +derived `max_correction_token_transactions`, and admission proves that the +worst-case token projection for the complete legal generation plus the receipt +fits that chain before the first WAL effect. Recovery binds every exact ordered +transaction and the final token-table version, so no intermediate version or +receipt can become manifest authority independently. +Both pointers, the new base witness, merged +progress, correction lineage, and lifecycle outcome become visible in one +manifest CAS. Until then the original generation is retained. A post-sidecar +cancellation returns `RecoveryRequired` plus the operation ID and recovery +completes the exact outcome once. + +The lifecycle outcome remains `DRAINING`: it clears exactly the matching +`StrictBlock`, preserves `drain_id`, goal, initiating actor, and guarded +operation, advances both equal descriptor/top-level current-HEAD witnesses, +preserves the already-achieved epoch floor, and writes the correction +`LastFoldSummary`. That exact row becomes +the baseline for the next empty-proof or `StreamResume`; correction never +publishes a stale witness and never opens admission. + +After correction, a `SEALED` drain goal continues to the empty proof. An +`OPEN_AFTER_FOLD` goal still uses the `StreamResume` recovery kind through the +explicit `stream resume --abort-drain` operation after all residue is visible; +plain `stream resume` continues to accept only `SEALED`. The fold/correction CAS +never opens admission itself. Skip-invalid, implicit drop, whole-generation +discard, base-row delete, and direct `DRAINING -> OPEN` CAS are forbidden. + +### 4.5 Reclamation decision: own the missing Lance primitive + +The RC.1 audit produces an explicit no-go for physical reclamation through the +stock public surface: + +- generic `cleanup_old_versions` ignores `_mem_wal`; the checked-in + `cleanup_old_versions_does_not_reclaim_mem_wal_objects` guard removes + ordinary Lance versions while proving the present WAL/generation/ + shard-manifest fixture's object names and bytes unchanged. It does not claim + orphan classification; +- `DatasetMemWalExt`, `ShardWriter`, and `ShardManifestStore` expose no + MemWAL GC/delete operation or receipt; +- the MemWAL specification warns that deleting WAL files can weaken writer + fencing. RC.1 checks epoch authority after a PUT conflict/error but returns + immediately after a successful atomic PUT. The checked-in + `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance` + negative guard decodes and deletes the successor's empty epoch-2 WAL fence + sentinel and proves the stale predecessor can receive watcher success even + though an explicit fence check returns `PeerClaimedEpoch`; and +- `FlushedGeneration` carries generation/path but no per-generation WAL range. + Only the shard-wide replay cursor exists, so a WAL prefix is reclaimable only + at B2's quiescent whole-cut boundary. `wal_entry_position_last_seen` is never + substituted. + +OmniGraph therefore does not raw-delete `_mem_wal` objects, route them through +`delete_unverified`, compact shard-manifest history, or wait passively for an +upstream release. The next substrate slice authors a narrow Lance patch, +proposes it upstream, and may pin OmniGraph immediately to the exact reviewed +commit while upstream review proceeds. Deletion stays inside Lance; when the +change is released upstream, the pin returns to the canonical revision rather +than preserving a product fork. + +Stock RC.1 exposes only evidence-level raw object listing plus manifest reads, +not the owned classified inventory below. The required public Lance shape is +an opaque inspect/plan/execute protocol with a durable attempt/receipt, for +example: + +```text +inspect_mem_wal_retention(...) +plan_mem_wal_reclaim(reclaim_id, exact_witnesses, graph_approved_cut, budgets) +execute_mem_wal_reclaim(serializable_opaque_plan) -> exact_receipt +classify_mem_wal_reclaim(reclaim_id, plan_digest) + -> pending | aborted_no_effect(receipt) | complete(receipt) +``` + +The caller cannot forge object paths. A serializable opaque plan binds a +caller-minted `reclaim_id`, plan digest, shard ID; exact latest manifest +version/epoch/status/spec/current generation/replay cursor/last-seen +hint/ordered generations; exact base dataset version with merged-generation +and index-catchup state; the graph-approved whole cut and exact authoritative +replay cursor; the maximum position across the complete authenticated retained +WAL/sentinel inventory; and object/byte delete budgets. The captured +`current_generation` is monotonic allocation authority, not reclaimable data: +the base must prove every generation through it merged or otherwise exactly +classified, and reclamation never resets or reuses that number. +It classifies referenced generations from the authoritative checkpoint summary +plus complete retained successor chain, never-referenced generation-output +orphans, previously referenced retired generations, WAL positions, +malformed/unknown objects, and exact byte totals. The checkpoint preserves the +ever-referenced/retired versus never-referenced evidence needed after history +compaction. A gapped/bounded-out chain, incomplete summary, or unknown object is +retained and reported, never guessed from age or path shape. + +Before pruning, Lance conditionally persists an attempt record containing the +`reclaim_id`, plan digest, exact prestate digest, and the opaque plan bytes. The +record is substrate recovery authority, not an OmniGraph data log. Execution +revalidates every witness byte-for-byte. If pre-sentinel revalidation proves the +plan stale or its conditional authority acquisition loses before the plan has +written its sentinel, changed its manifest, or deleted any object, Lance +conditionally persists an +`aborted_no_effect(receipt)` terminal record. Same ID/digest retry returns that +receipt and a fresh plan must use a fresh ID; another digest for the ID +conflicts. Once any planned physical effect exists, abort is forbidden and +same-plan recovery must complete or fail closed with the pending attempt +retained. +In the supported one-live-writer-process profile, OmniGraph already holds +exclusive admission and has retired every writer. Lance conditionally writes a +decoded-empty successor fence sentinel for `writer_epoch + 1` at a position +equal to checked `max_authenticated_wal_entry_position + 1`, not merely after +the approved data cutoff. Overflow, a gap whose contents were not inventoried, +or an existing object at that exact position refuses before effect. Lance then +appends one exact successor shard manifest with both `version` and +`writer_epoch` incremented, the captured `current_generation` preserved exactly, +an empty complete flushed-generation set, the sentinel position/digest, +`replay_after_wal_entry_position = sentinel_position`, and the exact reclaim +ID/digest. It does not transparently retry around a conflict. A crash after the +sentinel but before the manifest leaves an exact attempt-owned effect that only +same-plan recovery may finish; any other object at that position fails closed. +Only after the successor manifest durably names the new sentinel does execution +delete the planned generation/orphan objects, data WAL prefix through the +captured old cursor, and every separately authenticated older empty sentinel +below the new cursor. Exact inventory proved there was no data-bearing tail in +that interval. The new sentinel is never deleted. + +The same ID/digest re-execution recognizes the exact prestate, exact +attempt-owned successor sentinel before CAS, exact successor manifest, or a +durable completed receipt. It may finish the CAS only from the sentinel state +and resumes missing deletes only from the successor-manifest state. A different +digest for the ID, a foreign successor, or an unclassifiable movement fails +closed. Missing planned objects are +idempotently absent; unknown objects remain. Partial deletion leaves +manifest-unreferenced, base-covered residue owned by the persisted attempt. On +completion Lance durably writes the exact receipt before returning; it carries +before/after manifest/epoch, WAL cutoff, new sentinel position/digest, paths, +the inventoried prior maximum position, equal before/after current-generation +authority, deleted/already-absent/residual object and byte counts, and retained/ +unknown totals. A lost result is resolved +from the attempt/successor/receipt plus fresh inventory, never optimistic +process memory. Attempt and receipt records remain counted and retained through +the supported idempotency/recovery horizon. + +Pending reclaim attempts are part of the mandatory B2 open/admission barrier, +not optional maintenance metadata. Before any stream open, status, put, fold, +quiesce, resume, correction, rebuild preflight, or later reclaim may claim a +writer or touch shard state, it classifies the exact attempt set. Mutating +operations resolve the sole recognized same-plan attempt or refuse; status +reports it without mutation. Patched Lance itself refuses a shard-writer claim +while a pending reclaim attempt exists, so a crash cannot let ordinary reopen +adopt the attempt-owned successor sentinel as foreign WAL. Unknown/multiple +attempts fail closed. + +Reclaim and checkpoint may start only from `OPEN` or `DRAINING` under exclusive +admission; `SEALED` refuses them. Recovery-v12 `StreamRetention` binds the +complete expected lifecycle row, exact Lance attempt/plan, and fixed post- +receipt shard epoch/manifest/cursor/inventory outcome before the first physical +effect. After Lance terminals the receipt, the operation records +`EffectsConfirmed` and conditionally publishes the exact epoch floor, shard +witness, retained-byte classes, lifecycle revision, and graph-history charge +while preserving the lifecycle state and any drain identity/goal. A crash in between is roll- +forward-only and blocks another writer or final sealed proof. Quiesce constructs +`SealedProof` only after every such sidecar and substrate attempt is settled, so +the proof cannot be invalidated by later retention work. + +The patch must also prevent shard-manifest, ordinary claim, and reclaim history +from growing forever. Reclaim IDs are `(checkpoint_epoch, caller_uuid)` and +ordinary claim IDs are `(checkpoint_epoch, claim_kind, claim_uuid)`; the epoch +is part of both identities. Every ordinary claim attempt reaches a durable +`aborted_no_effect(receipt) | complete(receipt)` terminal state under the same +rule as reclaim: once its successor sentinel exists, it cannot abort and +same-attempt recovery must finish or retain the pending record. A checkpoint +may begin only after every claim/reclaim attempt in the current epoch is +terminal and its declared retry horizon has elapsed, and after every older data +or control reservation/materialization attempt is terminal and past its retry +horizon, or represented as an exact still-charged entry that must survive. The +checkpoint operation +allocates `next_checkpoint_epoch = current + 1`; its own checkpoint claim ID and +receipt belong to that next epoch, not the history it is about to expire. It +first advances the writer epoch through the same attempt-owned successor- +sentinel claim and durably terminals that claim. It then writes one +deterministically addressed immutable checkpoint body over the exact +**post-claim** state: complete binding/configuration, latest shard +manifest/epoch/monotonic current-generation/replay witness, retained known/unknown object inventory and +digests, orphan-classification summary, reservation-ledger watermark plus every +still-charged reservation, terminal generation/materialization/settlement and +control-reservation summaries, terminal reclaim summary, and terminal ordinary- +claim summary including the checkpoint claim itself. The claim summary +preserves every retained historical sentinel's exact position/digest/epoch +authority. +Only then does the operation conditionally swap the authoritative retention- +bootstrap pointer to that body. A lost checkpoint response is resolved from the +new body's checkpoint receipt during its declared horizon; only a later +checkpoint may expire it. Older reservation-ledger records are deletable only +after their exact charged/settled state is present in the authoritative body. +The new checkpoint epoch expires summarized terminal reservation IDs with typed +`ReservationExpired`; carried still-charged IDs remain live and cannot reserve +again. The same UUID under the new checkpoint epoch is a distinct identity, so +expiry never requires an unbounded tombstone set. + +All patched readers discover latest state from that bootstrap pointer and then +follow a contiguous successor chain; best-effort latest hints are not authority. +A crash before the pointer CAS leaves a classified checkpoint-body orphan. A +crash after it makes the new body authoritative, after which older manifest, +claim/reclaim attempt, receipt, terminal reservation/materialization, +settlement, and obsolete control-ledger objects may be removed idempotently. The +new checkpoint epoch expires every old reclaim and ordinary-claim ID: retry +returns typed `ReceiptExpired` or `ClaimReceiptExpired`, respectively, and the +same UUID in the new epoch is a different identity. + +There is no absent-pointer bootstrap mode. B2 enrollment or sealed rebuild asks +the patched Lance initializer to create an immutable genesis checkpoint body +with empty claim/reclaim summaries and conditionally install its bootstrap +pointer **before** committing the B2 MemWAL details/index kind. The enrollment +recovery envelope binds the genesis body, pointer, details commit, empty shard, +and final lifecycle publication; a crash before graph publication either +completes that exact prefix or refuses foreign/malformed residue. The first B2 +reader therefore always starts from an authoritative body, while a body/pointer +left before the details commit is classified enrollment-owned state rather than +silently adopted. + +This is not encoded as an unknown protobuf field on RC.1's accepted +`MemWalIndexDetails`/index-version 0 shape, because RC.1 may ignore it. +Stream-config v3 uses a new details type URL or system-index kind that the +reviewed patch understands and stock RC.1 is proved to reject before opening +shard state. Checkpoint activation never strands an already-open old reader on +the legacy kind. Genuine patched↔RC.1 and graph-v8↔v9 tests pin genesis +body/pointer/details publication, every later checkpoint crash boundary, ID +expiry, and refusal to fall back to `version_hint`. If the bootstrap, complete +summary, ID expiry, or bounded compaction cannot be proved, public B2 remains +inactive; periodic rebuild is not a substitute for an unbounded or +unrecoverable MemWAL namespace. Even after that namespace is bounded, the +separate whole-root lifetime limit below still applies. + +Before WAL prefix deletion is enabled, Lance must also recheck shard epoch +after every successful WAL atomic PUT and return a typed fence/unknown outcome +when a successor won. The patch carries the adversarial +deleted-successor-sentinel regression on local and object storage. This bounded +primitive is not advertised for overlapping processes: that later topology +requires a durable +`RECLAIMING`/sealed maintenance intent so a new claimant cannot enter between +prune and physical deletion. + +OmniGraph grants eligibility only at one quiescent whole-cut boundary. All of +the following are necessary: + +1. admission is exclusively closed and every writer, watcher, flush handler, + retirement owner, and reclaim owner is settled or durably retired; +2. the live and frozen MemTables and in-flight flush set are empty, and every + flushed generation has an exact graph-visible base/token-table outcome with + recovery resolved; +3. maintained indexes have caught up, no Fresh-read or retained-version guard + needs the cut, and the planned successor shard manifest has an empty complete + flushed-generation set—not merely no unmerged generation through a partial + cut; +4. the WAL cutoff equals the captured authoritative + `replay_after_wal_entry_position`; because RC.1 has no per-generation WAL + ranges, a smaller data prefix is not treated as proved. Exact inventory proves + no data-bearing WAL entry above that old cutoff and binds the maximum position + of every authenticated historical sentinel/object in the full tail. After the + prune CAS, the empty generation set, unchanged monotonic + `current_generation`, base `merged_generations` coverage, and successor cursor + equal to the new sentinel position are re-proved. The mutable + `wal_entry_position_last_seen` hint is never authority; and +5. the successor-epoch empty fence sentinel is durably created at checked + `max_authenticated_position + 1`, named by the successor manifest/cursor, + excluded from deletion, and remains decodable after reclamation. Position + overflow or a conditional-create collision refuses. An older empty sentinel is deletable + only when its position/digest/epoch is authenticated by the checkpoint plus + retained successor chain and it lies in the opaque plan; a data-bearing entry + is never exempted as a sentinel. + +Data HEAD, merge progress, age, or any one cursor alone is insufficient. The +Lance receipt records the predicates and the post-prune proof; OmniGraph does +not reconstruct them from paths. + +Exact orphan/unknown accounting additionally requires a backend capability that +guarantees HEAD/GET and LIST visibility after successful PUT **and** DELETE for +the `_mem_wal` namespace. If that capability is absent, public streaming is +refused unless the Lance patch provides its own durable complete object +accounting that does not depend on a possibly stale listing. That accounting +includes incomplete multipart uploads, which ordinary object LIST does not +return. Current-key visibility is still insufficient on a versioned, +soft-delete, or Object-Lock/retention-enabled namespace: noncurrent versions, +delete markers, retained locked objects, and incomplete uploads may continue to +consume billed storage after the current key disappears. The initial public B2 +profile refuses those configurations unless the patched backend can enumerate +and byte-account every retained version/marker/lock, permanently delete each +eligible version, and keep ineligible retained bytes charged until verified +removal or expiry. A successful best-effort current-object list is evidence for +the negative RC.1 guards, not a production completeness or cost proof. + +Public activation also requires a hard **admission watermark**, not a maximum +observed in a benchmark and relabeled as a guarantee. Exact cold inventory +counts every object/byte under `_mem_wal`: WAL, referenced generations and +PK/Bloom sidecars, recognized orphans, shard-manifest history/checkpoints, +ordinary claim and reclaim attempt/receipt records, and malformed/unknown +residue. Runtime may cache only a conservative +`observed + reserved_inflight` meter; it never decrements at fold, only after an +exact reclaim receipt plus a new complete inventory. + +The configured object/byte domain is one exact physical stream binding: base +dataset URI/main ref, enrollment ID, and every current or future shard under +that binding's `_mem_wal` namespace. It is not a silently shared graph-wide +quota. Different table bindings have separately configured limits; a later +graph/account quota is an additional admission layer, not a guarantee claimed +here. Inside one binding, the bootstrap-selected Lance retention ledger is the +single reservation authority. Every generation, ordinary claim, reclaim, and +checkpoint reservation conditionally advances that ledger, so concurrent +shards or callers cannot each pass a cached capacity check. Cold open rebuilds +the meter from the authoritative bootstrap/contiguous ledger plus exact +inventory before admission. + +Before the first PUT belonging to a legal generation, Lance must return and +enforce a source-derived `max_physical_growth(bytes, objects)` reservation (or +an equivalent storage quota) covering WAL entries, generation data, +PK/Bloom sidecars, manifests/checkpoints, and the configured bounded retry +allowance. Before any WAL PUT, multipart creation, or generation-output write, +Lance conditionally persists an enumerable ledger record with the durable +`generation_reservation_id = (checkpoint_epoch, generation_uuid)`, exact +binding/generation, maximum object/byte charge, hard +`max_generation_materialization_attempts`, attempt number, and status. Control +reservation IDs likewise use `(checkpoint_epoch, control_kind, control_uuid)`. +Only that reserved ID may create the covered effects. A retry must reuse +the same deterministically addressed output, or reclaim the prior never- +referenced partial output inside that reservation before allocating the next +attempt. It may never create randomized orphan output repeatedly outside the +envelope; an exhausted attempt count stops admission and requires reclaim or +sealed rebuild. One logical generation retains that reservation across +crash/retry and cannot reserve the same allowance repeatedly. Admission +succeeds only when +`observed + reserved_inflight + max_physical_growth` fits both configured +limits. Otherwise it returns `RetainedStorageLimitExceeded` before +`put_no_wait`; fold, correction, status, quiesce, and rebuild remain available. +Under the supported single-writer boundary, physical growth is then bounded by +the already-reserved envelopes even while those writes are in flight. Any +unknown object, incomplete inventory, or write outside an enforced envelope +closes admission. + +Reservation settlement is also durable and conditional. A crash after reserve +but before effect releases the charge only after exact no-effect inventory. Any +owned effect or ambiguity keeps the full maximum charged and permits only +same-ID recovery. After materialization, a complete inventory may atomically +associate the reservation with its exact retained objects/bytes and reduce +`reserved_inflight` only to the still-possible unmaterialized remainder (zero +when the envelope is terminal). Those materialized bytes remain in `observed`, +so they are never counted twice and the remaining legal growth is never +undercharged. A reclaim receipt plus a new complete inventory may reduce +`observed` for verified deleted physical bytes; it does not release them from a +second bucket. Fold success alone never settles storage. Cold open enumerates +every nonterminal reservation before admission, and an object/upload without +its prior reservation is unknown state, not retrospectively adopted. + +On object stores, every multipart upload/part opened for the reservation has a +durable upload identity and charged byte/object allowance. Same-plan recovery +must complete or explicitly abort it before another materialization attempt; +the retained-state inspector must list/account for it through the backend's +multipart API even though ordinary LIST cannot. A backend without exact +multipart accounting/abort semantics cannot enable public streaming. The +local/RustFS/S3-compatible matrix repeatedly crashes before multipart complete +and proves retained charged bytes plus attempts remain within the envelope. + +The configured object/byte limit also withholds a source-derived +`control_headroom` that row admission can never consume. Stream-config v3 hard- +caps outstanding control attempts, unexpired terminal claim/reclaim records and +bytes per checkpoint epoch, checkpoint-body bytes, and checkpoint-body orphans. +At most one checkpoint attempt/body exists per binding: its body path is +deterministic from `(checkpoint_epoch, checkpoint_id)`, same-ID recovery reuses +or removes that body, and another ID is refused while it is pending. Stale +effect-free plans consume a bounded terminal slot; they cannot create an +unbounded receipt or orphan loop. + +The headroom is sized for all capped unexpired terminal history, one pending +same-ID claim recovery, one maximum new-or-pending reclaim attempt/receipt plus +same-ID recovery, **two sequential maximum closure claim envelopes and their +recoveries/terminal receipts** (abort-drain after correction, then requiesce/ +seal), one deterministic checkpoint body/sentinel/manifest/pointer swap, and +the maximum recovery record set. Both closure receipts may coexist until their +retry horizons elapse; the design does not assume an intervening checkpoint can +compact the first. Those control effects reserve through the same durable per- +binding ledger but from the non-admission pool. Row admission stops with typed +`ControlHeadroomExceeded` before it consumes the emergency floor; same-ID +recovery, status, fold/correction, quiesce, reclaim, checkpoint, and sealed +rebuild stay available. If Lance cannot prove these history/orphan limits, the +reserve-first ledger, or forward-progress headroom, B2 must stop before public +activation rather than discover at runtime that a full WAL cannot be reclaimed. + +Local and RustFS measurements across row/byte caps, wide schemas, blobs, +PK/Bloom output, fragmented batches, crashes, and retries validate the +estimator/quota and catch regressions; they do not create the bound. Arrow or +RSS bytes are not disk bytes. A patch that can only report measured growth, or +that cannot checkpoint its own history, does not satisfy B2. + +The per-binding watermark bounds only the physical MemWAL namespace. It does +not bound logical base/token data or their Lance version histories. The shared +`__manifest` history is a separate graph-global finite resource: another stream +or an ordinary graph writer can grow it, so a per-stream publication counter +cannot guarantee that every enrolled stream still has an escape. + +Internal schema v9 therefore adds one manifest-authoritative singleton +`GraphHistoryBudget` for the root. It carries the root/history epoch, every +manifest publication charged since rebuild, a conservative source-derived +physical-growth charge, hard publication/byte limits, the sum of each enrolled +stream's remaining closure obligation, the last finalized budget revision, and +the last finalized budget operation/delta digest. +Each state-v2 stream row carries its own remaining closure reservation; the +singleton aggregate and the set of stream rows must agree in the same CAS. +There is no in-memory or side-dataset budget authority. + +Every publisher of `__manifest` participates—not only stream code. Mutation, +Load, SchemaApply, BranchMerge, branch controls, EnsureIndices, Optimize, +Repair, Cleanup, recovery, enrollment, fold/correction/lifecycle/retention, and +future writers all acquire the graph-history gate and charge the exact logical +publication plus its conservative physical-growth maximum. The complete crash- +safe authority is the committed singleton **plus** globally relevant standard +RFC-022 recovery sidecars. Under the history gate, a writer that may need a +later manifest CAS writes its sidecar with the exact pending budget charge +before its first data effect. The charge binds the history epoch, expected +budget revision, estimator/limits version, operation identity, exact singleton +delta, per-stream reserve delta if any, and conservative object/byte envelope; +same operation with another digest conflicts. This does not require or pretend +there was an extra budget-only manifest CAS. Every publisher's root recovery +barrier sums or resolves those pending charges exactly once. Final manifest +publication applies the bound delta and records its operation/digest in the +singleton; a lost CAS result classifies that exact marker and never charges it +twice. Effect-free finalization removes the pending charge. A normal publication +is admitted only when its charge leaves +the aggregate closure floor untouched. A closure publication may consume only +the named stream's reservation, atomically reducing that stream and the +singleton aggregate; it cannot borrow another stream's reserve. + +The initial B2 profile holds the graph-history gate from that final +reserve/sidecar arm through every owned physical effect, the manifest CAS, and +budget-charge finalization. There is therefore at most one live expected budget +revision; another publisher waits and then restarts its final validation from +the new singleton, while crash recovery resolves the sole pending charge before +releasing the gate. This deliberately serializes graph publication. A future +concurrent-reservation protocol needs a separate accepted composition proof; B2 +does not let multiple pending deltas speculate from one revision. + +The byte charge is a physical retained-storage claim, so it includes every +Lance object a `__manifest` attempt can create: data fragments, version/ +transaction manifests, maintained-index objects, multipart uploads, standard +recovery-sidecar overhead, and failed-CAS/materialization residue. The manifest +adapter obtains a source-derived maximum before materialization, fixes the +Lance transaction/output identity and a hard maximum attempt count in the +sidecar, and either reuses deterministic output or retains the full next-attempt +allowance. Even a manifest-only metadata writer must persist this minimal +standard reservation sidecar before Lance can materialize its CAS attempt, +unless Lance itself supplies an equivalent durable attempt/receipt consumed by +the same barrier. Cold open inventories and classifies the complete manifest namespace +plus pending multipart state; unknown or unreserved output closes all writers. +Settlement moves only observed materialized bytes from reserved to retained, +without double counting, and releases bytes only after an exact Lance-owned +cleanup/rebuild receipt plus fresh inventory. A successful-publication counter +without this reserve-first materialization envelope is depth evidence, not byte +headroom, and does not activate B2. + +Enrollment reserves the complete worst-case closure path for the new stream. +Before a stream admits a generation, resumes to `OPEN`, or begins non-emergency +management, it must replenish and prove that stream's full remaining obligation: +fold or strict block, one terminal correction, abort-drain when the goal is +`OPEN_AFTER_FOLD`, a second quiesce/claim, and final `SEALED` publication, +including the required management receipts and pending recovery charges. The +aggregate is dynamic across **all** enrolled streams, so two blocked tables at +the floor each retain their own correction and seal path. Closing one cannot +consume the other's reserve. The worst-case number/bytes are derived from the +fixed state machine and source-level maximum manifest-row/transaction growth; +configuration cannot lower them. + +A successful `SEALED` publication releases that stream's unused closure +reservation in the same CAS. Any transition back to `OPEN` must first reserve +the full applicable OPEN obligation; if it cannot, it remains closed and +returns `GraphRebuildRequired`. A successful fold may leave an idle stream +`OPEN` with only its shorter no-generation closure reserve, but no next +generation is admitted until the full generation envelope is replenished. + +When an ordinary charge would cross that shared floor, new work returns typed +`GraphRebuildRequired` before effect. Status reports graph-global used, +pending, per-stream reserved, aggregate reserved, limit, and source-bound +version. Operators close every enrolled stream and export/rebuild into a fresh +root. In-place SchemaApply, Optimize, and Cleanup on an enrolled table remain +refused until a later accepted drain/maintenance strand. Indefinite in-place +service requires Phase D or another accepted manifest-retention design; B2 does +not market a bounded MemWAL as a whole-graph storage bound. Public B2 remains +closed until this graph-global charge and its physical-growth upper bound are +implemented and evidence-green; an advisory counter or per-stream-only reserve +does not satisfy the escape invariant. + +These contracts close the design question without pretending RC.1 can already +execute it. B2 implementation and every public surface remain closed until the +reviewed Lance primitive, post-success fence fix, enforced admission watermark, +bounded MemWAL history checkpoint, graph-global manifest-history budget, and +crash matrix are pinned and green. + +### 4.6 Public surface after the gates close + The shipped `POST /graphs/{id}/ingest` path remains the deprecated, compatible alias of `/load`. Streaming receives a new, non-conflicting surface: ```text +POST /graphs/{graph_id}/streams/{type_name}/enroll POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main GET /graphs/{graph_id}/streams GET /graphs/{graph_id}/streams/{type_name} +GET /graphs/{graph_id}/streams/{type_name}/blocks/{block_token} POST /graphs/{graph_id}/streams/{type_name}/fold POST /graphs/{graph_id}/streams/{type_name}/quiesce POST /graphs/{graph_id}/streams/{type_name}/resume +POST /graphs/{graph_id}/streams/{type_name}/correct +POST /graphs/{graph_id}/streams/{type_name}/rebuild-preflight ``` The ingest request and response use `Content-Type: application/x-ndjson` and `Accept: application/x-ndjson`. -Each input line is one row payload. Each output line corresponds to the same -input ordinal: +Each input line is one row payload plus the compare-and-chain envelope. The +contributor is never accepted from the body: ```json -{"ordinal":17,"status":"durable","enrollment_id":"...","shard_id":"...","writer_epoch":8} +{"$stream":{"stream_incarnation_id":"d288f7a0-38b4-4e63-a841-60f323df0dd8","write_id":"8a880f0a-3f41-4a42-9b0e-f34af0a9a4df","predecessor_token":null},"id":"n-17","name":"Ada"} ``` +Each output line corresponds to the same input ordinal: + +```json +{"ordinal":17,"status":"durable","stream_incarnation_id":"d288f7a0-38b4-4e63-a841-60f323df0dd8","write_id":"8a880f0a-3f41-4a42-9b0e-f34af0a9a4df","stream_token":"sha256:...","origin":{"kind":"admission","admission_attempt_id":"...","caller_ordinal":17},"enrollment_id":"...","shard_id":"...","writer_epoch":8} +``` + +The response union is tagged rather than pretending every outcome created a +new admission attempt. Exact JSON `status` values are `durable`, +`ack_unknown`, `already_durable`, `withdrawn`, `invalid`, +`stream_input_too_large`, `stream_binding_changed`, +`stream_lifecycle_changed`, `stream_authority_changed`, +`stream_sequence_conflict`, `stream_idempotency_conflict`, `stream_fold_required`, +`retained_storage_limit_exceeded`, `control_headroom_exceeded`, +`correction_receipt_capacity_exceeded`, `graph_rebuild_required`, +`stream_backpressure`, `recovery_required`, and `stream_retry_required`; +CamelCase names below denote the corresponding engine error/disposition types. +An unresolved recovery found by the request-level barrier before any body line +is admitted may return the ordinary HTTP 503 `RecoveryRequired` envelope. Once +the adapter has accepted an ordinal or emitted any line, the same condition is +represented only by per-line `recovery_required` plus the stop-tail rule below; +partial success never changes into an HTTP error. + +For variants that reach the engine's authoritative binding capture, response +`enrollment_id`, `shard_id`, and `writer_epoch` describe that freshly +revalidated current physical binding. They are not copied from a token row's +immutable `origin_enrollment_id`, so a later Phase-D rebind does not make an +`already_durable` response advertise a stale writer. An adapter-level `invalid` +before binding capture omits those fields. A tail `stream_retry_required` +identifies the blocking attempt's captured binding under explicitly named +`blocking_binding`; it does not pretend to have revalidated a new per-line +binding. OpenAPI makes presence variant-specific rather than nullable-by-habit. + +- `durable` returns the confirmed token and its new `Admission` origin; +- `ack_unknown` / `AckUnknown` returns that attempt plus an explicitly + `candidate_stream_token_unconfirmed`; it never labels the token current; +- `already_durable` returns the persisted token and its persisted + `Admission | Correction` origin, with no new attempt; +- `withdrawn` returns the current terminal token and its original persisted + `Admission | Correction` origin plus the separate withdrawal correction + actor, operation, and receipt, with no new attempt; +- `invalid` is the effect-free per-line parse/schema/normalization error and + creates no attempt; +- `stream_input_too_large` means the exact normalized single row cannot fit an + otherwise empty legal generation. It is terminal for that line, creates no + attempt, and does not ask the caller to fold and retry an impossible row; +- `stream_binding_changed`, `stream_lifecycle_changed`, + `stream_authority_changed`, `stream_sequence_conflict`, and + `stream_idempotency_conflict` are effect-free and return only the + authoritative binding/lifecycle/current-token evidence safe for the + corresponding typed error, with no fabricated attempt. Lifecycle change + covers `OPEN -> DRAINING/SEALED` between request runs; authority change covers + an exhausted pre-invocation reprepare after schema/main/token movement; +- `stream_fold_required`, `retained_storage_limit_exceeded`, + `control_headroom_exceeded`, `correction_receipt_capacity_exceeded`, and + `graph_rebuild_required` are effect-free admission refusals. They carry the + exact relevant usage/limit snapshot and no attempt; `stream_fold_required` + means the next legal row/run fits an empty generation but not the bounded + resident generation and must be folded before retry, while + `graph_rebuild_required` means the shared root publication budget has reached + the aggregate reserved closure floor. The management/rebuild escape in + §4.3–§4.5 remains available for every capacity result; +- `stream_backpressure` is an effect-free admission/queue deadline reached + before `put_no_wait`; it carries retry guidance but no durability claim or + attempt; +- `recovery_required` means a pre-invocation recovery/retirement operation + remains authoritative and could not finish within the request deadline. It + carries that recovery operation ID, no new admission attempt, and no token + claim; and +- `stream_retry_required` means the line was not invoked because an earlier + physical run became `AckUnknown`; it carries that blocking attempt ID but no + attempt or token claim for this line. + RC.1 exposes a durability completion, not an exact per-put WAL receipt. `BatchDurableWatcher::wait()` returns only `Result<()>`; `WriteResult.batch_positions` names positions local to the active @@ -400,45 +1632,124 @@ passes a non-contiguous range to B1. Previously acknowledged runs in the same request remain durable; the response is a stream, not an all-request transaction. Ordering, cancellation, and retry rules are explicit: +Before adding a normalized row to a run, the adapter computes its exact stored +Arrow charge after tombstone/hidden-metadata injection. A single row above +32 MiB is +`stream_input_too_large`; the adapter closes the preceding run and may continue +with later lines. Otherwise it forms only runs that fit both the per-call bound +and the resident generation's exact remaining row/byte allowance. If the next +legal row/run fits an empty generation but not the remaining resident allowance, +the result is `stream_fold_required`. This empty-generation test is mandatory, +so fold/retry cannot loop forever on an intrinsically oversized payload. + - acknowledgements are emitted in input order for one HTTP stream; - disconnecting does not cancel entries whose durability waiter resolved; - cancellation or failure after `put_no_wait` but before a successful watcher result is `AckUnknown`, not proof of non-durability; -- a missing response is ambiguous; retrying the same `id` and payload may add - another WAL entry and preserves one-row cardinality, but is semantically safe - only when no newer same-key value can interleave. If `Y(id)` lands between an - ambiguous `X(id)` and retry `X(id)`, physical LWW order makes the retry - overwrite `Y`. B2 therefore requires an accepted per-key sequencing or - idempotency contract before public activation; +- a missing response is ambiguous. Retrying the same stream incarnation, + authenticated actor, `write_id`, predecessor, and payload follows §4.1: + exact current `X` returns `already_durable`, absent `X` may be submitted again + against its still-current predecessor, and a newer `Y` yields + `StreamSequenceConflict` before any Lance call; - server shutdown stops admission, drains durability waiters up to a bound, and reports any unacknowledged tail as unknown to the client. +Token dispositions are physical-run boundaries too. A row that is +`already_durable`, `withdrawn`, binding-conflicted, sequence-conflicted, or +idempotency-conflicted first waits for the preceding submitted run, then is +evaluated against the resulting confirmed overlay without entering that run. +Later rows are re-evaluated in caller order. For one key, reachable same-request +sequences include `new / already / conflict`: the exact retry of the first +occurrence becomes already-current, while another candidate that still names +the old predecessor conflicts. Mixed-key `new / conflict / new` is also +deterministic. Fresh same-key successors may share one +generation after separate durable calls, but never one physical run: tokens are +opaque, so the later caller cannot name an unconfirmed predecessor. A physical +run contains at most one fresh candidate per key. `AckUnknown` retires that +worker and stops further physical admission for the stream request; rows not +yet invoked receive an effect-free `stream_retry_required` result. +`on_reject="strict"` describes fold validation—it does not turn the NDJSON +response stream into an atomic request. + +An `AckUnknown`, effect-free capacity/backpressure refusal, +`stream_lifecycle_changed`, `stream_authority_changed`, or +`recovery_required` stops further physical admission for that request. The +blocking line receives its exact status; each later otherwise-admissible, +uninvoked line receives the appropriate tail status plus `blocking_ordinal` +(`stream_retry_required` for an `AckUnknown`, otherwise the same blocking +status). The adapter nevertheless continues effect-free parsing, schema +validation, normalization, and intrinsic single-row sizing after the blocker. +A later parse/schema/normalization failure remains `invalid`, and a later row +that cannot fit an empty generation remains `stream_input_too_large`; those +adapter-local terminal results take precedence over any inherited tail blocker. +Every other uninvoked line inherits the blocker. No such line gets an +admission-attempt ID. This rule is identical when the blocker appears after +earlier `durable` output, so the handler never converts partial NDJSON success +into an HTTP-level error or confuses capacity/recovery with `AckUnknown`. + CLI commands mirror the new namespace rather than overloading deprecated `omnigraph ingest`: ```text +omnigraph stream enroll --enrollment-request-id ... omnigraph stream ingest --data ... omnigraph stream status [] ... -omnigraph stream fold [] ... -omnigraph stream quiesce [] ... -omnigraph stream resume [] ... -omnigraph stream resume [] --abort-drain ... +omnigraph stream block show --block-token ... +omnigraph stream fold [] --operation-id --expected-lifecycle-revision ... +omnigraph stream quiesce [] --drain-id --expected-lifecycle-revision ... +omnigraph stream resume [] --resume-id --expected-lifecycle-revision ... +omnigraph stream resume [] --abort-drain --resume-id --expected-lifecycle-revision ... +omnigraph stream correct --block-token --correction-id --expected-lifecycle-revision --plan ... +omnigraph stream rebuild-preflight [] ... ``` +HTTP fold, quiesce, resume/abort-drain, and correction bodies likewise require +their operation ID plus expected lifecycle revision; status and block-view +responses expose the revision to use as the compare token. Exact occurrence +plus intent is retry-safe after a lost response, while stale revision refuses +without retargeting. `enroll` requires caller-minted +`enrollment_request_id` and returns tagged `enrolled | already_enrolled` with +the durable stream incarnation/current binding; ingest on a declared but +unenrolled table returns request-level typed `StreamNotEnrolled` before reading +or acknowledging body rows. `correct` requires `block_token`, +caller-minted `correction_id`, and explicit ordered `REPLACE | WITHDRAW` +actions. The engine derives the canonical plan digest from §4.4; an optional +client digest is only an equality assertion. It returns the immutable +correction receipt. `block show`/the block endpoint returns the digest-verified, +paginated planning view needed to construct those actions; it never returns +whole blocked rows. `rebuild-preflight` is a fresh under-gate proof, not a cached +status alias. + Every endpoint has a dedicated OpenAPI operation and handler tests. Ingest passes the engine `stream_ingest` Cedar action and per-actor admission -accounting before acquiring a shard writer; fold/quiesce/resume use a separate +accounting before acquiring a shard writer; fold, quiesce, resume/abort-drain, +enroll, block inspection, correct, and rebuild-preflight use the separate `stream_manage` action. Status is authorized like other graph operational metadata. The same engine gates apply to embedded and remote CLI use. -Phase B2 initially exposes only strict upsert. Public activation requires the +`GraphRebuildRequired` is graph-wide, so it is part of the ordinary writer +contract too. The engine/SDK error carries history epoch and budget revision, +used and pending publication/byte charges, a bounded per-stream reservation +summary for streams relevant to this operation (empty when none, truncated only +with an explicit summary digest/count), aggregate closure reserve, hard limits, +source-bound version, and the explicit +`quiesce_all_and_rebuild` action; it never suggests retrying ordinary work in +the same root. A streaming request that already emitted rows uses the per-line +`graph_rebuild_required` variant above. Any other HTTP write refuses before its +first durable effect with HTTP 409 and structured `graph_rebuild_required`; +CLI JSON prints the same payload and human output explains that reserved +quiesce/correction/`SEALED` operations remain available. SDK, server, CLI, and +OpenAPI parity tests pin both request-level and partial-NDJSON forms. + +Phase B2 initially exposes only strict compare-and-chain upsert. Public activation requires the minimum `status`, explicit `fold`, persistent `quiesce`, `resume`/abort-drain, rebuild preflight, a bounded strict-correction workflow, durable authenticated-contributor attribution, a same-key `AckUnknown` sequencing/idempotency contract, and a proved WAL/generation reclamation rule. Minimum status includes lifecycle and -binding, current epoch, pending generations/bytes, last fold error, and whether -strict fold is blocked. It never claims to resolve one caller's `AckUnknown`. +binding/revision, current epoch, pending generations/bytes, last fold error, +receipt and graph-history budgets, and whether strict fold is blocked. It never +claims to resolve one caller's `AckUnknown`. Dead-letter row identity and operation, richer status, and configurable policy remain Phase C. Automatic operation-scoped drain, SchemaApply/branch integration, upgrade orchestration, and physical rebind remain Phase D. @@ -456,6 +1767,13 @@ acknowledgement. B1 does not call an external embedding provider or synthesize unspecified fold-derived fields. Any future derivation step requires its own contract and evidence and must complete before the one bounded table effect. +B2 adds one mandatory substrate prelude before any writer claim: under stream +admission it classifies the complete Lance reclaim-attempt set from §4.5. +Mutating entry points finish the sole exact pending plan or refuse, while +read-only status reports it; patched Lance independently refuses +`mem_wal_writer` when an attempt remains pending. No token check, same-key queue, +or final pre-put gate can bypass that prelude after restart. + Phase B1 admits exactly one non-empty normalized `RecordBatch` per Lance `put_no_wait`. Each call and the **entire active generation**, including any duplicate batch submitted while that live generation is still active, are @@ -465,7 +1783,10 @@ representation is the exact stored MemWAL batch after normalization/defaulting a the row put and use the sealed keyed writer's numeric limits; they are not Lance's soft thresholds. Cheap raw row/byte bounds reject obviously over-cap input before recovery I/O; a raw-fit batch then receives exact post-tombstone -validation at that same pre-recovery boundary. After any recovery/authority +validation at that same pre-recovery boundary. An exact batch above either +per-call limit is terminal `ResourceLimitExceeded` in private B1 and maps to +per-line `stream_input_too_large` or bounded adapter chunking in B2; it is never +`FoldRequired`. After any recovery/authority prelude, the exact charge is recomputed and reserved against the aggregate budget. Every put then follows one order: exact charge, shared admission, same-key input queue, and finally worker-mode inspection. A queue-first fast @@ -480,9 +1801,9 @@ opener's queue position. New calls then receive `FoldRequired` before adding a charge. Callers charged before the replay became observable drain normally; their buffers can make the honest ledger transiently exceed the nominal root cap, so the worker does not wait for root-wide budget convergence while it -still holds shared admission. A new batch that would otherwise cross either -generation budget returns typed `FoldRequired` without calling `put_no_wait` -or adding a row/WAL batch. A configured +still holds shared admission. A legal batch that fits an empty generation but +would cross either **remaining** generation budget returns typed `FoldRequired` +without calling `put_no_wait` or adding a row/WAL batch. A configured `durable_write=true` writer must return a durability watcher; `None` after invocation is `AckUnknown` plus writer retirement, never acknowledgement. @@ -504,8 +1825,11 @@ post-invocation `AckUnknown` outcome carries enough of the same binding and caller ordinal context to report the ambiguity, but makes no durability claim. RC.1 cannot later attribute replayed residue to that specific attempt, so the attempt remains permanently ambiguous even when replay preserves its possible -effect. A same-payload retry is only cardinality-idempotent; without per-key -serialization it can overwrite a newer interleaving value. +effect. This is the implemented B1 shape. B2 adds the logical `write_id`, +opaque predecessor/result tokens, trusted contributor, payload digest, and server-minted +`admission_attempt_id` inside the stored row as specified by §4.1. That evidence +does not turn RC.1's watcher into a physical receipt; it lets an explicit retry +enforce current/non-current sequencing without blindly appending stale data. One root-scoped registry is singleflight for the exact physical binding across all `Omnigraph` handles in the process; it is never handle-local. Its serialized @@ -803,7 +2127,7 @@ adopted as a generation, never scanned or folded, and never deleted by B1. It does not make the durable WAL residue unrecoverable. Any loose object outside that recognized generation-output subtree shape still fails closed. B2's accepted reclamation protocol must prove when these orphans can be removed and -count them toward its hard retained-storage stop. +count them toward its enforced retained-storage admission watermark. The fold excludes generations already covered by the base table's exact `merged_generations` and requires exactly one remaining B1 generation. It @@ -854,6 +2178,44 @@ correctness dependency. The commit phase then: `__manifest` CAS, including table heads when RFC-024 is active; 7. deletes the sidecar after successful publication. +That seven-step list is the implemented B1 one-table fold. B2 preserves its +ordering but recovery-v12 adds the exact `_stream_tokens.lance` transaction and +fixed attribution summary. The base and token effects are both classified +before the same manifest CAS; neither may publish independently. A correction +uses the same multi-participant envelope described in §4.4. It does not turn a +generation into multiple base-table keyed transactions. + +Because `_stream_tokens.lance` is one graph-global physical participant, B2 +adds one root-shared stream-token gate. The universal order for a manifest +publisher is `sorted relevant stream admission -> graph history -> schema -> +main branch -> stream token -> sorted graph tables`; a writer with no relevant +stream admission starts at graph history, and no caller acquires a newly +discovered stream lease after that gate. Every fold or +correction captures the token dataset's exact manifest-selected version, +transaction UUID, and e_tag in its `ReadSet`; it never opens raw token HEAD as +current authority. The final gate rechecks that witness plus every winner's +stream incarnation/fold-base certificate before sidecar arm. A B2 recovery +sidecar that can move the token participant is graph-global relevant to +**every** operation that may publish `__manifest` or main authority: stream +operations, Mutation/Load, SchemaApply, BranchMerge, branch controls, +EnsureIndices, Optimize, Repair, Cleanup, recovery, and future writers all +resolve or refuse it before base capture even when its graph table is disjoint. +Quiesce and rebuild preflight use the same barrier. If final relisting discovers +a late global sidecar, stream lifecycle, or required admission lease, the caller +releases every admission/history/schema/branch/token/table gate and restarts +from the root barrier; it never recovers a +disjoint global sidecar while retaining one caller table's gate. This prevents +an unmanifested token HEAD from being buried by an ordinary graph commit. + +The shared gate intentionally serializes token-table physical effects. Folds +still amortize many row acknowledgements, but B2 does not claim independent +per-table token commits. A token conflict proven effect-free across **both** +participants may finalize and fully reprepare from fresh manifest authority; +any base or token effect, or ambiguous classification, retains recovery +ownership and returns `RecoveryRequired`. Recovery rolls the exact pair forward +or compensates it under the captured authority; it never adopts a token-table +HEAD merely because its rows look compatible. + The adapter is a distinct writer kind, not a Mutation payload with extra fields. The sidecar is mandatory even though merge-insert is staged. After `commit_staged`, Lance HEAD and `merged_generations` have moved while the graph @@ -868,12 +2230,13 @@ sidecar and returns `RecoveryRequired`. No later fold is prepared until the synchronous recovery barrier resolves that exact attempt; `merged_generations` is never replanned around an unresolved partial fold. -MemWAL generation GC can start only after the exact fold is graph-visible, its -sidecar is resolved, index catchup permits reclamation, and no `FreshReadCut` -retention guard references the generation. Data-HEAD merge progress alone is -never permission to delete the only fresh-tier copy. B1 deliberately performs -no GC; B2 cannot expose production admission until the reclamation proof and a -hard retained-storage stop are implemented. +MemWAL generation GC can start only after the exact fold and token outcome are +graph-visible, its sidecar is resolved, index catchup permits reclamation, and +no `FreshReadCut` or retained-version guard references the generation. +Data-HEAD merge progress alone is never permission to delete the only +fresh-tier copy. B1 deliberately performs no GC; B2 consumes only the +Lance-owned inspect/plan/execute primitive and enforced admission watermark +specified in §4.5. Concurrent folders reload `merged_generations`: a generation already committed is skipped; otherwise the fold is replanned from current state. The explicit @@ -883,12 +2246,11 @@ the old manifest pointer after the table commit and see the rows only after the single manifest CAS. Phase B1 records fixed fold mechanism lineage only because it has no public -caller. Durable authenticated-contributor attribution is a B2 activation gate: -RC.1 exposes no authoritative per-put WAL range and retaining the WAL cannot -recover actor identity that was never stored. Public audit must not infer -provenance from MemTable positions or WAL cursor statistics. The attribution -mechanism requires its own accepted design/evidence before public rows exist; -restart-stable dead-letter row identity may remain Phase C. +caller. B2's accepted design embeds trusted contributor/write metadata in the +stored row and publishes the §4.2 winner summary plus current token evidence. +It never infers provenance from MemTable positions or WAL cursor statistics. +The design is not active until the v9/config-v3/recovery-v12 implementation and +evidence gates pass; restart-stable dead-letter row identity remains Phase C. ## 7. Fold-time rejection is atomic @@ -897,10 +2259,13 @@ row-validation failure returns a typed blocked outcome, publishes no table pointer or lineage, and leaves the acknowledged generation durable and unmerged. B1 admits no next generation while that one remains unmerged, so it intentionally has no -correction lane and makes no claim that a later row can unblock it. B2 cannot -activate until a bounded correction/disposition protocol preserves every ack, -survives crash, and cannot make the fold exceed one keyed transaction. No row -is silently dropped or credited to a dead letter. +correction lane and makes no claim that a later row can unblock it. B2's +bounded replacement/withdrawal protocol is specified in §4.4: it operates over +the immutable cut, creates no second MemWAL generation, preserves one +base-table keyed transaction, advances the manifest-selected token authority, +and survives crash through recovery-v12. No row is silently dropped or +credited to a dead letter. Public activation still waits for that implementation +and its crash matrix. `dead_letter` remains Phase C. The earlier proposed identity `(stable_table_id, incarnation_id, shard_id, generation, wal_position)` is not @@ -948,30 +2313,36 @@ The bounded-profile drain sequence is: fence a stale owner, and reject or backpressure every new append; 4. flush active MemTables and fold every generation to empty; -5. verify shard manifests and base `merged_generations` agree on emptiness; +5. verify shard manifests and base `merged_generations` agree on emptiness and + classify the exact ordered fence-only WAL inventory from §4.3; 6. publish `DRAINING -> SEALED` with the verified generation cut and exact achieved per-shard epoch map; 7. for an operation-scoped drain, perform the guarded operation; persistent public quiesce stops after step 6. `OPEN -> DRAINING` and `DRAINING -> SEALED` are RFC-022 authority-first -metadata writes; each fold is a separate normal RFC-022 graph write. -`DRAINING` fully encodes every target per-shard floor and current-HEAD witness, -so restart reconstructs the admission gate closed and resumes the drain. A +metadata writes; each fold is a separate recovery-v12 RFC-022 graph write in +the exact drain mode specified by §4.3. +`DRAINING` fully encodes every target per-shard floor and two equal copies of +the current-HEAD witness; each graph-visible drain fold/correction advances both +copies. Restart therefore reconstructs the admission gate closed from the +latest row and resumes the drain. A `SEALED -> OPEN` transition is different: an exact `stream_resume` sidecar covers the higher-epoch claim while the gate remains exclusively closed and is resolved before any ack path proceeds. The drain itself is not one giant sidecar spanning multiple commits. -`DRAINING` is not allowed to become an operator trap. B2 must also implement a +`DRAINING` is not allowed to become an operator trap. B2 implements the +protocol-v2 descriptor and recovery-v12 transition specified in §4.3, including a crash-safe abort transition for a quiesce that cannot finish: only when no -guarded schema/maintenance operation has begun and the exact binding/witness is -unchanged, `stream resume --abort-drain` arms a resume sidecar, claims a higher -epoch under the closed gate, and CASes `DRAINING -> OPEN`. If the drain is -strict-blocked, that transition is permitted only as part of B2's accepted -bounded correction protocol; otherwise it remains `DRAINING`. B1 exposes -neither transition and cannot deadlock a public operator because it has no -public caller. +guarded schema/maintenance operation has begun, the exact current DRAINING row +and its equal witnesses still match, every owner has settled, and no unmerged +or strict-blocked cut remains may `stream resume --abort-drain` arm a resume +sidecar, claim a higher epoch under the closed gate, and CAS +`DRAINING -> OPEN`. A blocked/unmerged cut +must first fold or complete the exact bounded correction; correction remains +`DRAINING` and never opens as part of its CAS. B1 exposes neither transition and +cannot deadlock a public operator because it has no public caller. For the general multi-process profile, step 3 must instead include the public cross-process seal required by §3; it atomically refuses later claims and @@ -1107,10 +2478,13 @@ accepted local cost/RSS and configured-RustFS cost evidence in §12.3 supports the bounded private profile; it is not a public latency SLO, group-commit multiplier, or claim that every metadata term is history-flat. -B2's minimum status surface exposes the exact lifecycle/binding, active epoch, -pending generation/row/byte totals, merged progress, last fold error, -strict-blocked state, and quiescence state. Richer index-catchup, age, and -reject detail may follow in Phase C. Lance's +B2's minimum status surface is the authority-plus-observation contract in +§4.3. It includes the exact lifecycle/binding, active epoch, drain operation, +pending generation/row/byte totals, retained/unknown bytes, merged progress, +last fold outcome, strict block, correction/management-receipt spares, +lifecycle revision, graph-global history used/pending/per-stream and aggregate +closure reserves, relevant recovery, and rebuild readiness. +Richer index-catchup, age, and reject detail may follow in Phase C. Lance's `wal_entry_position_last_seen` and next-position statistics are explicitly labeled hints; neither is presented as a durable per-row receipt or a way to resolve one `AckUnknown` attempt. @@ -1118,6 +2492,10 @@ resolve one `AckUnknown` attempt. Before B2 activation, internal metrics must cover ack latency, durability-wait batching, fenced writers, replayed entries, fold rows/bytes/generations, fold retries, lag, reject counts, and sidecar recovery. +They also cover compare-and-chain conflicts/idempotent hits, token-table +effects, correction outcomes, retained object/byte classes, reclaim +attempt/receipt outcomes, reservation/control-ledger state, admission-watermark +refusals, and finite-lifetime rebuild-required refusals. Defaults for every byte/count/time bound must be documented, and configuration changes are observable behavior. @@ -1165,6 +2543,40 @@ refuses it and requires export/init/load into a different v8 root. The genuine v7↔v8 old-binary/new-format and new-binary/old-format refusal/rebuild evidence now passes (§12.3). V8 remains a private-core format with no public B2 caller. +B2 is a third strict strand: internal schema v9, stream-state protocol v2, +stream-config v3, and recovery-v12. V9 adds the reserved nullable stream-row +metadata to stream-capable physical schemas and initializes the +manifest-selected `_stream_tokens.lance` authority. Root initialization also +creates the singleton `GraphHistoryBudget` at history epoch zero with the exact +source-bound version, baseline retained manifest inventory, zero pending +charges, and closure reserves for every enrollment published in that same +initial outcome. The genesis publication is self-accounting: before any Lance +materialization, the initializer writes a deterministic bootstrap attempt with +the precomputed conservative maximum for its own sidecar, fragments, +manifest/version/index objects, multipart state, and bounded retries. Configured +limits must fit that maximum plus the initial closure aggregate or init refuses +without opening the root. The genesis row records publication one and charges +the **full conservative bootstrap maximum** as retained use; it never attempts +to observe and settle its own publication downward inside that same CAS. Only a +later exact cleanup/rebuild receipt may reduce the charge. Crash/lost-result +recovery either proves that exact singleton/transaction or keeps the root +unopened. No other v9 manifest +publication may precede that authority. A +compatible open validates singleton uniqueness, internal-schema/config/source- +bound versions, committed counter/inventory, unresolved sidecar charges, and +the exact sum of all per-stream closure reservations before any writer runs; +missing, duplicate, or mismatched budget state is uncovered format corruption, +not a default. V8 rows, lifecycle payloads, +and private acknowledgements are never reinterpreted or assigned contributor/ +token evidence with serde defaults. The implementation must prove genuine +v8↔v9 refusal plus export/init/load rebuild before any production caller exists. +Graph-history publication/byte limits and their estimator version are root-init/ +export-rebuild choices in B2; they cannot be lowered or reinterpreted in place. +A later dynamic configuration change requires its own accepted recovery-covered +design that re-proves every committed/pending charge and closure reservation. +These version assignments are part of the B2 contract; this design amendment +does not activate them in code or change the currently served v8 format. + Old binaries refuse a stream-capable graph before reading or writing any table. A stream-capable binary refuses an older graph before running recovery. On a compatible stamp it resolves or refuses every stream recovery kind relevant to @@ -1513,55 +2925,209 @@ The evidence does not claim history-flat uncompacted manifest work, a public latency SLO, group commit, Lance-internal failure injection, or the unavailable forged pre-claim/pre-put stream-sidecar cell. -Phase B2 remains inactive despite B1 being green. It additionally -requires accepted attribution, reclamation/retained-storage-stop, correction, -same-key sequencing/idempotency, and persistent operator escape designs and -implementations. +Phase B2 remains inactive despite B1 being green. §4.1–§4.6 now specify its +attribution, token, reclamation/enforced-admission-watermark, correction, and +persistent operator contracts. None is implemented by this amendment. ### 12.4 Phase B2 — public strict activation gates -Phase B2 may begin only after B1 is green **and** accepted designs exist for -durable authenticated-contributor attribution, bounded WAL/generation -reclamation, strict correction/disposition, and same-key sequencing or -idempotency after `AckUnknown`. Public activation also requires a persistent -escape from an `OPEN` stream; a user must never be left with a table that -ordinary writers refuse but cannot be quiesced or rebuilt. +Phase B2 implementation may begin from the contracts in §4.1–§4.6, but public +activation waits until every implementation/evidence item below is green. The +design does not waive the persistent escape requirement: a user must never be +left with a table that ordinary writers refuse but cannot be corrected, +quiesced, or rebuilt. - `@stream(mode="upsert", on_reject="strict")`, production first use, SDK, HTTP, CLI, Cedar, and OpenAPI all route through the same private B1 core. The existing `/ingest` behavior remains compatible, and embedded/remote command - parity is tested. + parity is tested. Accepted-schema and runtime guards refuse `@stream` on a + type requiring `@embed` or any external/provider-derived field; caller- + supplied physical vectors round-trip without provider invocation. +- Explicit enrollment tests cover `UNENROLLED` status, request-ID same/different + intent, every genesis/body/pointer/details/shard/lifecycle crash boundary, + lost result after sidecar deletion through the persisted request-ID/intent + receipt, `already_enrolled`, `stream_manage` authorization with Cedar/ + embedded/remote parity, first returned stream incarnation, and + request-level `StreamNotEnrolled` before ingest body admission. Ingest never + creates physical enrollment implicitly. - The public acknowledgement is status-only and caller-ordered. It maps the private durable batch result back to each caller ordinal and reports the - enrollment/shard/epoch binding, not a WAL position, generation, or Lance - `batch_positions` value. The B2 adapter owns contiguous-run batching around - invalid NDJSON lines and its bounded reorder buffer. + stream incarnation, write ID, and current binding, not a WAL position, + generation, or Lance `batch_positions` value. Only durable/current outcomes + report a confirmed token and persisted tagged origin; `ack_unknown` labels + its candidate unconfirmed, while invalid/conflict/not-invoked outcomes mint + neither. Every exact response variant in §4.6, including `invalid`, + `stream_input_too_large`, lifecycle/authority/recovery blockers, and + `stream_retry_required`, has a schema. The B2 adapter owns contiguous + physical-run boundaries around invalid lines and token dispositions plus its + bounded reorder buffer. Tests cross a partially full generation, retained- + storage/control/correction/management-receipt spare and graph-history limits, + an intrinsically oversized row on an empty generation, authority/lifecycle + movement or `RecoveryRequired` between runs, and a pre-invocation queue + deadline after earlier lines are already durable; the + blocking and remaining otherwise-admissible lines receive their exact effect- + free capacity/backpressure status while later parse/schema/normalization + failures remain `invalid` and intrinsically oversized rows retain their own + status. The same precedence is pinned after `AckUnknown`. No handler converts + partial NDJSON success into an HTTP error. +- Internal schema v9/config-v3/state-v2/recovery-v12 provisions the hidden row + metadata and manifest-selected token dataset. Every base/token effect is one + recovery-covered publication. The graph-global token participant follows + canonical `sorted relevant stream admission -> graph history -> schema -> + main branch -> stream token -> sorted graph tables` gate order; folds capture + its exact manifest-selected + witness in the read set and never use raw token-table HEAD. If either the base + or token participant has an effect or ambiguous outcome, recovery owns both; + transparent reprepare is allowed only after proving both effect-free. + Current-token mismatch, stream-incarnation mismatch, reused write ID with a + different digest/actor, historic UUID reuse against a newer predecessor, and + the exact `X(unknown) -> Y -> retry X` sequence all follow §4.1 before + `put_no_wait`; idempotent-current retry performs zero WAL writes. Fold proves + each LWW winner's `fold_base_token`/`chain_depth` certificate. An unconfirmed + candidate is never accepted as a predecessor, and each physical run contains + at most one fresh occurrence per key. OpenAPI/SDK/CLI round-trips pin the exact + `sha256:` plus 64 lowercase-hex wire form for stream/block tokens and protocol + digests; uppercase, base64, whitespace, bad prefix, and wrong length refuse + pre-effect. Tests also pin + the universal global-token-sidecar barrier across every ordinary manifest + writer and the release-all-gates/restart rule after late discovery. - Minimum operator controls expose authoritative status, explicit fold, plus - persistent quiesce, resume/abort-drain, and rebuild preflight. Status includes exact - lifecycle/binding, epoch, pending generations/bytes, last fold error, and - strict-blocked state. Quiesce has a crash-safe cut, never auto-reopens, and + persistent quiesce, resume/abort-drain, correction, and rebuild preflight. + Status takes an exclusive cut, settles owners without mutating recovery, and + includes exact lifecycle/binding/revision, epoch, pending generations/bytes, + correction and management receipt usage/reserves/limits, graph-history + budget, last fold summary, and strict-blocked state. Every mutating call after + enrollment carries an operation ID and expected revision; same-ID/same-digest + returns the complete bounded terminal receipt, same-ID/different-digest + conflicts, and a stale revision never retargets. Quiesce requires a caller + drain ID, has a crash-safe cut, never records terminal success at the initial + `DRAINING` CAS, never auto-reopens, and proves that every acknowledged row is folded before export/cutover. Resume revalidates the same binding, no-named-branch topology, and epoch authority - before advancing its epoch. A failed drain can return - `DRAINING -> OPEN` only before a guarded operation and through the accepted - correction protocol. + before advancing its epoch. Abort-drain can return `DRAINING -> OPEN` only + when no guarded operation began, every owner settled, and no unmerged residue + or strict block remains. Correction clears a matching block but preserves + `DRAINING` and the original drain goal; it is never itself a reopen. Tests + delay a quiesce/resume retry across a later cycle and prove receipt/stale- + revision handling. They drive the graph-global budget to its closure floor, + prove new ordinary work returns `GraphRebuildRequired`, hold two streams + strict-blocked simultaneously, then close one while preserving the other's + independent block/correction/abort-drain/requiesce/`SEALED` reserve. The + cap-minus-full-closure case also reserves every required management receipt + and both sequential claim envelopes/terminal records. + The WAL watermark is never reported as a whole-graph history bound. +- Graph-history tests force every manifest writer through the universal gate, + verify late sidecar/lifecycle discovery releases all gates and restarts, and + exercise exact used/pending/finalized accounting. The reserve-first physical + envelope crashes before/after fragment, multipart, failed CAS, manifest/index + output, publication, and cleanup receipt; bounded retries cannot create + uncharged orphans, cold inventory reconstructs every charge, unknown output + closes writers, and two racing publishers prove the globally serialized gate + lifetime, fresh-revision restart, and absence of reverse-order deadlock. Two + simultaneous stream closure reservations cannot be spent by ordinary commits + or by each other. - Every public batch durably records its authenticated contributor before rows can become unattributable, and the fold audit consumes that evidence. A - bounded reclamation protocol plus hard retained-bytes stop prevents public - admission from growing the WAL forever. Neither contract is inferred from - MemTable positions or mutable WAL statistics. + current `WITHDRAWN` token remains durable even when the graph row is absent. + The reviewed Lance inspect/plan/execute patch, durable attempt/receipt, + post-success WAL fence check, bounded history checkpoint, and enforced + retained-byte/object admission watermark bound growth. Exact inventory + requires strong HEAD/GET/LIST visibility after PUT/DELETE plus exact multipart + accounting/abort, or Lance-owned durable accounting. Neither contract is + inferred from MemTable positions, mutable WAL statistics, age, raw path + shape, or benchmark maxima. - Admission cancellation and server shutdown cannot abandon an in-flight durability waiter. A background-owned admission task reaches a durable outcome or typed `AckUnknown`. Status/replay preserves possible residue but never claims to reconcile that caller's attempt. - The accepted retry contract prevents an ambiguous `X(id)`, newer durable `Y(id)`, retry `X(id)` sequence from silently restoring the stale `X` value. - Tests pin that exact interleaving; one-row LWW cardinality alone is not - accepted as semantic idempotency. + Tests pin that exact interleaving, stream-incarnation reset, historic UUID + reuse with a newer predecessor, actor change, same-generation chains across + separate durable puts, the one-fresh-occurrence-per-key physical-run rule, + reachable same-key `new/already/conflict`, and mixed-key + `new/conflict/new` within one request, across separate requests, after restart, + and through correction. One-row LWW cardinality + alone is not accepted as semantic idempotency. +- Correction tests cover `REPLACE` and `WITHDRAW` over an exact blocked cut, + lost fold response/restart followed by paginated block inspection, exact + offending-key/token/action view digest, cursor/block movement, permission + refusal, missing-cut corruption, multiple violations for one key across + count/byte-capped pages with stable full-entry ordering, and the 8,192-row + planning bound, + engine-derived canonical plan digest plus actor binding, same/different + correction-ID replay, stale block token, unknown key, exact row/byte and + receipt-count/receipt-byte cap/+1, admission at cap-minus-one with one full + terminal-receipt spare, cold-replay reconstruction of that spare, a blocked + correction consuming the final reserved slot/bytes at the exact cap, and + no-generation-at-full-cap quiesce through `SEALED` rebuild. They also cover + 8,192 distinct corrected current-token rows plus the separately bounded one- + row receipt transaction, depth-8,192 replacement to exact maximum 8,193, + concurrent fold/resume, and + every sidecar/effect/publication crash boundary. No second generation, + skip-invalid path, or graph delete is admitted. +- Quiesce tests cover crashes before/after `OPEN -> DRAINING`, epoch claim, + seal/drain, each fold, equal descriptor/top-level witness advancement, + historical and reclaim-successor fence sentinels, strict-block summary, + engine-minted failure-drain identity/actor before and after its conditional + CAS, empty proof, and `SEALED` publication. Status covers concurrent + put/flush/reclaim exclusion. Resume and abort-drain cover + arm/claim/confirm/CAS/finalize, + caller operation-ID/expected-revision retry, same-ID/different-intent, + delayed retry after a later drain/resume cycle, named-branch refusal, strict- + block refusal, reopen, and lost replies. Cold open, quiesce, resume/abort, and checkpoint also crash at + claim-attempt/sentinel/manifest boundaries and prove that their ordinary + sentinel-first claims preserve the prior replay cursor and classify its full + tail before admitting a put. Only the whole-cut reclaim case may advance that + cursor to its new sentinel. Quiesce settles and publishes every retention + sidecar before the final proof; reclaim/checkpoint refuse while `SEALED`, and + resume consumes rather than mutates the old proof. `SEALED` never auto-opens. +- Reclamation tests cover stock-cleanup non-ownership, the stale-writer deleted + successor-sentinel slot, exact local/RustFS inventory, HEAD/GET/LIST and + multipart capability refusal or durable accounting, and explicit refusal of + versioned/soft-delete/Object-Lock storage unless every retained version, + delete marker, locked byte, and multipart upload is exactly accounted and + eligible versions are permanently removed, + authoritative-checkpoint-plus-successor-chain orphan classes, exact empty + generation/whole replay cut with no data tail, stale plan and terminal + `aborted_no_effect` plus fresh-ID replan, prune CAS, partial delete, lost + receipt, and attempt/receipt replay. Successor planning binds the maximum + authenticated tail position rather than only the data cutoff, refuses + overflow/collision, and proves the successor manifest preserves the exact + monotonic `current_generation` with consistent base merge coverage. They + crash after successor sentinel but before CAS and prove writer-claim refusal + plus open-barrier recovery; exercise + missing/stale hints, genesis body/pointer/details first publication, and every + later bootstrap-pointer crash boundary. Checkpoint tests fold terminal + ordinary-claim plus reclaim receipts, terminal generation/materialization/ + settlement and control-ledger records, every still-charged reservation, and + historical-sentinel authority into the body before deleting old records; old + IDs return `ReceiptExpired`, `ClaimReceiptExpired`, or `ReservationExpired`, + while the same UUID in a new checkpoint epoch is distinct. A source-derived/ + enforced reservation is tested across bounded + materialization attempts and repeated crash-before-multipart-complete. Its + per-binding reserve-first ledger crashes before/after reserve, first WAL or + upload effect, exact settlement, and reclaim release; cold open reconstructs + it, concurrent shards cannot double-reserve, and unowned effects fail closed. + Capped control attempts/receipts/checkpoint-body orphans cannot consume the + emergency pool; control headroom still allows same-ID recovery, + quiesce/seal, reclaim, and checkpoint at the full admission watermark. + Measurements validate but do not create the bound, and budget is never + released optimistically. - `SEALED` enables safe export/rebuild into a different root. In-place - SchemaApply/maintenance remains refused until Phase D. + Mutation/Load, SchemaApply, and maintenance remain refused for every enrolled + lifecycle, including `SEALED`, until Phase D defines a token-aware direct- + write/witness/rebind transition. - Genuine cross-version tests prove old-binary/new-format and - new-binary/old-format refusal. Rebuild tests prove acknowledged rows survive + new-binary/old-format refusal. They prove stock RC.1 rejects B2's new MemWAL + details type URL/system-index kind rather than ignoring a field or falling + back to a latest-version hint. V9 root-init/open tests prove the singleton + graph-history budget is established before any other v9 publication, its + baseline inventory/source-bound version and aggregate stream reserves match, + its bootstrap publication self-charges under a precomputed maximum, and + cap-too-small, every bootstrap crash/lost-result boundary, changed estimator/ + limits, and missing/duplicate/mismatched state refuse rather than defaulting. + Rebuild tests prove acknowledged rows survive only after quiesce/fold, and fail closed when an acknowledged row remains unfolded. @@ -1591,7 +3157,8 @@ ordinary writers refuse but cannot be quiesced or rebuilt. | E0 | production-neutral public-surface enrollment/witness classifier; no schema, API, sidecar, or format activation | **Passed 2026-07-18:** 14 substantive local cells, complete six-attempt zero-list 8/80 cost shape, Unix no-list/error tripwire, and one non-vacuous configured RustFS positive-plus-negative cell (§12.1) | | A | bounded main/unsharded/single-live-writer enrollment adapter, all-lifecycle effect exclusion with only the `SEALED` native-branch exception, lifecycle/admission lease, then graph-format capability/refusal and strict rebuild | **Implemented 2026-07-18 (§12.2):** internal schema v7, recovery-v10 enrollment, durable lifecycle CAS, process-local exclusion, crash/partial-format refusal, and genuine v6↔v7 strand evidence; no public enrollment or row path | | B1 | **Implemented privately 2026-07-19:** internal schema v8/config-v2, root-scoped one-generation admission worker, durability-watcher acknowledgement, conservative active-state reopen/replay, the pinned RC.1 replay-watermark bridge, and one explicit strict RFC-022 fold; no production caller | **Green privately 2026-07-19 (§12.3):** genuine v7↔v8 binary refusal/rebuild, all 24 cells in the graph-level B1 behavior/crash/race suite, qualified local cost/PK-index/RSS evidence, and configured-RustFS cost evidence pass. The non-flat uncompacted-manifest term is retained explicitly; upstream-internal flush/channel failure and forged pre-claim/pre-put stream-sidecar injection remain unclaimed seam limitations, not public guarantees | -| B2 | public strict activation across schema, SDK, HTTP, CLI, Cedar, and OpenAPI, with minimum attribution, reclamation, status/fold, correction, same-key retry safety, and persistent quiesce/resume/abort-drain/rebuild escape | B1 green; accepted attribution/GC/correction/sequencing designs; API compatibility; cancellation/shutdown ownership; quiesce/fold/resume/abort-drain/rebuild survival matrix | +| B2-0 | contract closure for compare-and-chain tokens, trusted row attribution, manifest-selected current-token state, revision-fenced protocol-v2 lifecycle/management receipts, bounded `REPLACE`/`WITHDRAW` correction, Lance-owned reclamation, enforced retained-storage admission, and graph-global manifest-history/closure reservations; no product surface | **Specified 2026-07-19 (§4.1–§4.6):** two stock-RC.1 negative/ownership guards are checked in. Implementation remains inactive; the owned Lance durable inspect/plan/execute, post-success-fence check, bounded MemWAL checkpoint, strong-inventory proof, enforceable MemWAL growth reservation, and reserve-first graph-manifest materialization envelope are next | +| B2 | public strict activation across schema v9/config-v3/recovery-v12, SDK, HTTP, CLI, Cedar, and OpenAPI, with the B2-0 contracts implemented | B1 green; reviewed Lance patch pinned; token/attribution/correction/revision-fenced lifecycle/reclamation and graph-history crash matrices; local and RustFS enforced-bound validation; genuine v8↔v9 refusal/rebuild; API compatibility; cancellation/shutdown ownership; public parity | | C | restart-stable reject-row identity, atomic dead letter, richer status, and evidence-backed configurable bounds | reject crash matrix; reject-retention proof; backpressure and RSS/latency evidence | | D | automatic operation drain, schema/branch/upgrade integration, and rematerialization rebind | two-coordinator race, old/new physical-binding crash matrix, and format-transition suite | | E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion |