[Bug]: Cascade rebuild-from-md livelocks after a full index wipe — last_processed frozen at 0, memcells never commit (1.1.2)

[dolphinsue319]

Summary

After upgrading 1.1.0 → 1.1.2, the episode.subject_vector schema change forces an index rebuild (LanceDBSchemaMismatchError). Following the documented recovery (rm -rf ~/.everos/.index/lancedb and restart), the daemon's cascade livelocks: it re-scans and re-processes forever but never advances the checkpoint (cascade status → last_processed: 0), so almost nothing is durably committed. No exceptions, no failed rows, and /health stays 200 the whole time. A single-file everos cascade sync <one.md> commits fine — only the daemon's concurrent bulk rebuild is stuck.

Environment

• EverOS 1.1.2 (PyPI), installed via uv into a venv
• lancedb 0.33.0, pylance 7.0.0, pyarrow 24.0.0
• Python 3.12.13, macOS (arm64)
• Corpus: ~106 markdown files, 1087 memcells (single user, EVEROS_MEMORIZE__MODE=agent)
• Embedding/rerank: DeepInfra Qwen3-Embedding-4B / Qwen3-Reranker-4B; LLM OpenRouter gpt-5-mini (all remote, healthy)

Steps to reproduce

1. Have a populated 1.1.0 index (sqlite + lancedb), then uv pip install everos==1.1.2.
2. Restart → startup aborts: LanceDBSchemaMismatchError: table 'episode' schema drift: missing=['subject_vector']. The index is rebuildable from md — rm -rf ~/.everos/.index/lancedb and restart.
3. rm -rf ~/.everos/.index/lancedb (or wipe both lancedb + sqlite) and restart the daemon so it rebuilds from the md source of truth.

Expected

Cascade drains the queue, re-embeds the memcells, last_processed climbs to max, done → 106, search returns results.

Actual — silent livelock (observed for 45+ min)

• everos cascade status: pending: 106, done: 0, last_processed: 0 (frozen), max climbing (e.g. 8919 → 9449 over a few min), lag == max.
• everos cascade fix → no failed rows.
• sqlite memcell committed only ~5 of 1087; md_change_state = 106 rows, and every 30 s scan logs cascade_scanner_decisions added=0 count=106 modified=106 (all files perpetually "modified").
• lancedb oscillates (e.g. 372 MB → 238 MB) as vectors are written then pruned; atomic_fact fragment count stays low.
• everos.out.log: only cascade_scanner_decisions (every 30 s) + occasional cascade_worker_processed change_type=added kind=atomic_fact — no error/exception/warning, and /health returns 200 throughout.
• system.db-wal grows steadily; the max LSN growth means the queue is effectively self-feeding.

It looks like the per-item processing writes vectors to LanceDB (non-transactional) but the sqlite transaction that would persist the memcell + advance last_processed / md_change_state never commits (or rolls back), so the scanner re-enqueues the same files indefinitely.

Isolating signal

Running the CLI on a single file with the daemon stopped commits durably:

everos cascade sync ~/.everos/evermem/global/users/<user>/episodes/episode-YYYY-MM-DD.md
# → cascade_worker_processed … kind=episode
# → cascade_lancedb_optimized kind=episode pruned=True
# → sync complete — processed 1 row(s)

So the checkpoint/commit path works per-file; the failure is specific to the daemon processing the whole corpus concurrently after a wipe. (One caveat: single-file episode sync populated episode.lance but not atomic_fact.lance, so the derived-kind rebuild may be a separate concern.)

Impact

A schema change on a minor upgrade forces a from-scratch rebuild, and that rebuild cannot complete on a modest corpus — the memory store is left effectively empty (search returns almost nothing) and the WAL/LSN keeps growing. Had to roll back to 1.1.0 + the pre-upgrade index to restore service.

Possibly related

• [Bug]: everOS /health hangs after 2-5 successful KB uploads — single asyncio event loop deadlocks between cascade + extract_atomic_facts #316 (single asyncio event loop deadlocks between cascade + extract_atomic_facts) — same subsystem; here /health does not hang, but the cascade makes no durable progress.
• [Bug]: LanceDB index grows unbounded and fills the disk when cascade compaction/prune silently fails #315 (compaction/prune silently fails) — the optimize failure itself appears fixed on 1.1.2 (optimize succeeds on a fresh index), but the rebuild needed to get to 1.1.2 is what breaks.

Ask

• A cascade bulk-rebuild path that checkpoints incrementally / converges on a normal corpus (or an everos index rebuild that doesn't rely on the live scanner re-enqueue loop).
• Ideally, avoid forcing a full from-scratch rebuild on a minor schema add — migrate/backfill subject_vector from existing memcells instead of requiring rm -rf lancedb.