Proposal: split verified search, read-only, and embedding fixes

[saagpatel]

Context

I ran a measure-first review of a local 22-commit performance exploration branch against current main. The branch mixed correctness fixes, deterministic search, model metadata, profiling harnesses, rejected batching experiments, and public drafts, so it should not be merged as one unit.

This issue proposes extracting only the independently reviewable pieces. It also relates to #34, but does not attempt to solve long-query scaling as one large performance PR.

Proposed focused changes

1. Deterministic search ordering plus an index for chunks(file_id) lookups.
2. Read-only serve safety: no file watcher or startup cleanup/reconciliation writes, with a stale-index warning.
3. Embedding startup/indexing improvement: prefer local tokenizer sources and reuse one llama.cpp context sequentially across embed_batch inputs.
4. Persist the actual embedding model URI and dimension for engraph status.
5. Replace obsolete HNSW/ONNX integration suites with current, model-free Store + MockLlm coverage.

I plan to open these as separate draft PRs so each can be accepted, revised, or declined independently.

Evidence reproduced locally

• cargo fmt --check, cargo clippy -- -D warnings, unit tests, and all-target tests pass on the extracted branch.
• Deterministic golden search repeats the same expected rankings ten times.
• Real embedding-model parity is exact for sequential context reuse: max_abs_diff=0, with 0/32 vectors beyond 1e-6.
• A copied 11,251-vector lab showed the shared-context path about 24-25% faster than fresh contexts in the sampled runs.
• Read-only staleness and non-reconciliation behavior have focused tests.

Explicitly excluded

• The in-memory vector cache: it was faster, but a fresh deterministic battery reproduced a top-k set mismatch on 1 of 8 probes.
• True packed multi-sequence batching: it remains rejected because Metal and CPU results failed the existing 1e-6 parity gate.
• An implicit FTS porter-tokenizer rebuild during Store::open: this needs a separate transactional migration design and explicit search-semantics review.
• Research notes, Fable harnesses, handoffs, and public drafts.
• Broad prepared-statement caching for now; its isolated A/B evidence is not strong enough to ride with the correctness changes.

Requested direction

Please use the draft PRs linked below to review the pieces independently. The deterministic-search and read-only correctness groups are the strongest candidates; performance and metadata changes can remain draft until their scope is accepted.

[saagpatel]

The focused draft PRs are now available for independent review:

• perf(llm): reuse embedding context across batch inputs #45 — embedding tokenizer/context reuse with an exact-parity harness
• fix(search): make equal-score ranking deterministic #46 — deterministic search ordering and indexed best-chunk lookup
• test: replace obsolete model integration suites #47 — current model-free integration suites
• feat(status): report indexed embedding metadata #48 — indexed embedding model URI/dimension in status
• fix(serve): disable writer maintenance in read-only mode #49 — read-only watcher/reconciliation safety and stale-index warning

The vector cache, true batching, implicit FTS migration, broad statement-cache sweep, and Fable/public material remain excluded as described above.

[saagpatel]

Cross-platform integration evidence is now available for the exact union of draft PRs #45 through #49.

• Temporary validation PR: ci: validate extracted fable stack saagpatel/engraph#1
• GitHub Actions run: https://github.com/saagpatel/engraph/actions/runs/29167258380
• macOS: cargo fmt --check, cargo clippy -- -D warnings, and cargo test --lib passed (471 tests)
• Ubuntu: the same three gates passed (471 tests)
• Local combined-tree verification also passed cargo test --all-targets

The temporary PR will be closed and its aggregate remote branch deleted now that the evidence is captured. The five draft PRs remain the intended review units.

The corresponding upstream Actions runs remain in action_required because GitHub requires an upstream maintainer to approve workflows from a first-time fork contributor; attempting to approve them as the fork owner correctly returned HTTP 403 (Must have admin rights to Repository).