Implement approved multi-process DataStore coordination

[enricopiovesan]

Ticket ID

implement-multiprocess-datastore-coordination

Summary

Implement the approved multi-process DataStore coordination model without weakening committed-record integrity.

Blocked By

An approved successor specification and ADR from the multi-process coordination-model ticket.

Definition of Done

• Approved model governs the implementation
• Cross-process stress tests cover contention, owner crash, recovery, fairness, and deterministic failure behavior
• PR is linked to this Project 1 item

Non-goals

Defining the coordination model, distributed synchronization, or bypassing the single-owner lock.

Validation

Approved-spec alignment and cross-process conformance suite.

[enricopiovesan]

Operational blocker record

Blocked by: absent approved multi-process coordination-model successor spec and ADR.

Blocking party: DataStore architecture approval. Accountable owner: multi-process coordination-model owner. Escalation owner: DataStore architecture maintainer.

Next action: create and approve a model deciding process identity, lock-versus-lease semantics, stale-owner/crash recovery, fairness, timeouts, and per-platform support.

Where: successor to Specs 518/519; future cross-process suite in crates/traverse-runtime/tests/ or scripts/ci/.

Ready when: approved model/ADR names the failure taxonomy and stress matrix for contention, owner crash, recovery, fairness, and deterministic failures.

Follow-on: implement #879 without weakening committed-record integrity.

[enricopiovesan]

Decision record

Decision: Use one OS-backed exclusive owner lock for each DataStore root. A contending process receives stable store_busy and retries explicitly; no lease or heartbeat takeover is used.

Rationale: A single OS lock is the smallest cross-platform correctness boundary and avoids clock-, pause-, and partition-driven split-brain writers. Unsupported locking semantics fail closed.

Options considered: OS-backed exclusive lock; lease/heartbeat takeover; local coordinator daemon.

Consequences and follow-up: The successor spec/ADR must define lock acquisition, stable contention/unsupported errors, crash release assumptions, and cross-process stress cases.

Governing changes required: Include owner-lock behavior and failure taxonomy in the multi-process coordination spec and ADR.

[enricopiovesan]

Decision record

Decision: Record a random per-process instance ID, PID, and lock-acquisition time as diagnostic-only ownership evidence.

Rationale: The instance ID avoids PID-reuse ambiguity across restarts while avoiding a host identity dependency. The operating-system lock, not this metadata, remains authoritative.

Options considered: PID only; generated process-instance ID plus PID/time; host-supplied stable identity.

Consequences and follow-up: The spec must define metadata privacy, lifecycle, and diagnostic output without implying that the advisory identity grants authority.

Governing changes required: Include diagnostic owner metadata in the multi-process coordination spec and ADR.

[enricopiovesan]

Decision record

Decision: Hold the root-wide owner lock for mutations, migration, restore, and maintenance. Allow readers to access only completed committed records without the writer lock.

Rationale: This preserves a single writer and prevents partial reads while avoiding unnecessary read serialization. Readers may observe the prior committed snapshot until a writer commits.

Options considered: root lock for all operations; root lock for mutations with committed-snapshot reads; per-record locks.

Consequences and follow-up: The spec must define commit visibility, reader snapshot semantics, and conformance tests proving that a reader never observes partial data.

Governing changes required: Include root lock scope and committed-snapshot read semantics in the multi-process coordination spec and ADR.

[enricopiovesan]

Decision record

Decision: Support only local OS filesystems with reliable advisory locking. Return stable locking_unsupported for network, virtual, removable, or otherwise unreliable filesystem semantics.

Rationale: The local multi-process model must not claim integrity where advisory locking behavior is unknowable. Shared/network storage belongs to a future distributed coordination design.

Options considered: local reliable filesystems only; attempt any filesystem accepting a lock; selected network-filesystem support profiles.

Consequences and follow-up: The spec must define capability detection, stable unsupported behavior, and a local-filesystem cross-process stress matrix.

Governing changes required: Create the multi-process coordination spec and ADR using all recorded decisions before implementing this issue.