Implement RFC-026 private MemWAL B1 core

[aaltshuler]

Summary

• activate internal schema v8, stream-config v2, and recovery-v11 StreamFold
• add a root-scoped private MemWAL worker for one no-roll generation, with watcher-only durable acknowledgement, detached ownership across cancellation, and typed AckUnknown
• classify replayed or already-flushed residue as fold-only, bridge the pinned Lance replay watermark, and publish one exact-ID fold only through the __manifest CAS
• enforce one admission order (charge → shared admission → same-key queue → worker mode), exact replay accounting, and fail-closed ownership for cold open/seal/abort timeouts
• add cross-version, crash, race, API-boundary, cost, and RSS evidence; refresh RFC-026 and the developer architecture/testing docs

Why

Phase A established bounded MemWAL enrollment and lifecycle authority but intentionally shipped no row path. This slice implements and proves the smallest data-bearing B1 profile without waiting for a broader upstream receipt/seal API and without exposing a product contract.

Acknowledged rows remain invisible until the exact fold publishes through the graph manifest. Lance continues to own the WAL, shard fencing, replay, generation storage, and scan primitives; OmniGraph adds only the graph authority, recovery envelope, and visibility boundary around them.

Support boundary

This remains private and deliberately narrow:

• canonical main only
• one unsharded MemWAL shard
• one resident writer / one live writer process
• one generation capped at 8,192 rows and 32 MiB
• no public schema, SDK, CLI, HTTP, Cedar, or OpenAPI streaming surface
• no fresh-tier reads, correction lane, or generation GC

Public activation remains RFC-026 B2 and is still gated on attribution, reclamation, correction, retry sequencing, and persistent operator controls.

Validation

• cargo test --workspace --locked
• cargo test -p omnigraph-engine --features failpoints --test memwal_stream --locked — 24 passed
• worker ownership/accounting unit suite — 12 passed
• cargo test -p omnigraph-engine --features failpoints --test failpoints --locked — 141 passed, 1 helper ignored
• cargo test -p omnigraph-engine --features failpoints --test forbidden_apis --locked — 16 passed
• cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost --locked — 7 passed, 1 helper ignored
• genuine v7 ↔ v8 refusal/export-rebuild compatibility test
• cargo test -p omnigraph-server --features aws
• scripts/check-agents-md.sh
• targeted rustfmt --check and git diff --check

Greptile Summary

This PR adds the private RFC-026 B1 MemWAL core. The main changes are:

• Root-scoped, bounded MemWAL admission and worker ownership.
• Watcher-based durable acknowledgement with typed ambiguity handling.
• Replay classification, exact-generation sealing, and fold-only recovery.
• Recovery-v11 publication through one manifest update.
• Internal schema v8 and stream-config v2 activation.
• Crash, race, compatibility, resource, and cost coverage.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.
• Ambiguous post-invocation outcomes return AckUnknown while ownership remains detached.
• Failed or stalled retirement keeps the worker fenced instead of releasing potentially live authority.
• Fold recovery distinguishes the exact planned effect from no effect before publication.
• The v7-to-v8 format change is protected by the strict version fence.

Important Files Changed

Filename	Overview
crates/omnigraph/src/table_store/mem_wal/worker.rs	Adds worker registration, admission accounting, replay modes, durable acknowledgement, detached timeout ownership, and seal/retirement sequencing.
crates/omnigraph/src/db/omnigraph/stream_ingest.rs	Adds the private put and fold orchestration with authority checks, exact generation scans, and bounded retries.
crates/omnigraph/src/db/manifest/recovery.rs	Adds recovery-v11 StreamFold state, exact effect classification, confirmation, and manifest publication.
crates/omnigraph/src/db/manifest/stream.rs	Activates stream-config v2 and validates identity-bound stream lifecycle state.
crates/omnigraph/src/storage_layer.rs	Adds exact stream-fold staging to the sealed storage interface.
crates/omnigraph/src/error.rs	Adds typed capacity and post-invocation acknowledgement errors.
crates/omnigraph-cli/tests/crossversion_upgrade.rs	Adds bidirectional v7/v8 format refusal and rebuild coverage.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller
    participant Registry as Worker Registry
    participant Writer as Lance MemWAL
    participant Recovery as Recovery v11
    participant Table as Lance Table
    participant Manifest as __manifest

    Caller->>Registry: Reserve charge and queue input
    Registry->>Writer: Claim or reuse writer
    Registry->>Writer: Put one bounded batch
    Writer-->>Registry: Durability watcher result
    Registry-->>Caller: Durable ack or AckUnknown

    Caller->>Registry: Request fold
    Registry->>Registry: Acquire exclusive admission
    Registry->>Writer: Seal and prove generation cut
    Registry->>Writer: Abort and retire writer
    Registry->>Recovery: Arm exact StreamFold sidecar
    Recovery->>Table: Commit exact-ID update
    Recovery->>Manifest: Publish table, lifecycle, epoch, and lineage
    Manifest-->>Caller: Fold visible

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller
    participant Registry as Worker Registry
    participant Writer as Lance MemWAL
    participant Recovery as Recovery v11
    participant Table as Lance Table
    participant Manifest as __manifest

    Caller->>Registry: Reserve charge and queue input
    Registry->>Writer: Claim or reuse writer
    Registry->>Writer: Put one bounded batch
    Writer-->>Registry: Durability watcher result
    Registry-->>Caller: Durable ack or AckUnknown

    Caller->>Registry: Request fold
    Registry->>Registry: Acquire exclusive admission
    Registry->>Writer: Seal and prove generation cut
    Registry->>Writer: Abort and retire writer
    Registry->>Recovery: Arm exact StreamFold sidecar
    Recovery->>Table: Commit exact-ID update
    Recovery->>Manifest: Publish table, lifecycle, epoch, and lineage
    Manifest-->>Caller: Fold visible

Loading

_{Reviews (1): Last reviewed commit: "Implement RFC-026 private MemWAL B1 core" | Re-trigger Greptile}

Context used:

• Context used - AGENTS.md (source)
• Context used - CLAUDE.md (source)