fix(lancedb): reclaim stale versions via write-locked prune

docs/cascade_runbook.md

@@ -79,8 +79,50 @@ everos cascade sync users/u1/episodes/X.md    # re-enqueue + drain
 ```
 
 The CLI builds the same `CascadeOrchestrator` as the daemon but only
-calls `sync_once` / `drain_once` — no watcher / scanner background
-task. So it's safe to run in parallel with a live `everos server`.
+calls `sync_once` / `drain_once` — no watcher / scanner background task.
+Its drain still runs the same compaction + version-cleanup (`prune`) as
+the daemon, but `prune` uses `delete_unverified=False`, so it never
+deletes a file another process may be mid-commit on. Safe to run in
+parallel with a live `everos server`.
+
+## Rebuild the index: `everos cascade rebuild`
+
+The safe recovery from a drifted or corrupt LanceDB index. It rebuilds
+the whole index from markdown (the source of truth) in one shot:
+
+```bash
+everos cascade rebuild          # prompts for confirmation
+everos cascade rebuild --yes    # non-interactive
+```
+
+> **Stop the `everos server` first.** Unlike `cascade sync`, rebuild
+> **drops and recreates** the LanceDB tables. A running daemon holds
+> cached table handles that would keep pointing at (and writing to) the
+> dropped dataset, corrupting the rebuild. This is the one cascade
+> command that is **not** safe to run alongside a live server.
+
+What it does, in order:
+
+1. **Drops** every business LanceDB table (`drop_business_tables`) and
+   evicts them from the connection cache.
+2. **Recreates** them empty from the current schema + FTS indexes
+   (`ensure_business_indexes`).
+3. **Clears** the cascade queue (`md_change_state.reset_all`) so every
+   md file re-enqueues as `added` on the next scan.
+4. **Re-scans + drains** (`sync_once`): re-embeds and re-inserts every
+   md entry.
+
+It deliberately **skips `verify_business_schemas`** — the drift it
+recovers from would otherwise trip that guard on startup before the
+rebuild could run (chicken-and-egg).
+
+Why not a bare `rm`:
+
+| Recovery | Re-populates `done` entries | Preserves `unprocessed_buffer` |
+|---|---|---|
+| `rm -rf .index/lancedb` | ❌ scanner skips `done` rows → empty index | ✅ |
+| `rm -rf .index` | ✅ | ❌ deletes un-extracted messages |
+| `everos cascade rebuild` | ✅ | ✅ |
 
 ## Recovery paths
 
@@ -91,15 +133,28 @@ an on-disk table has columns the current Pydantic schema does not
 declare (or vice versa), the boot fails with:
 
 ```
-LanceDB table 'episode' schema drift: missing=[...], extra=[...].
-The index is rebuildable from md — recover with
-`rm -rf ~/.everos/.index/lancedb` and restart.
+LanceDB table 'episode' schema drift: missing=[...], extra=[...],
+type_drift=[...]. The index is rebuildable from md — recover with
+`everos cascade rebuild`.
 ```
 
-This is the documented recovery: delete the index, restart the
-server, the scanner will pick up every md file on its first sweep and
-the worker repopulates LanceDB. Markdown is the source of truth, so
-no data is lost.
+`verify_business_schemas` compares both the column **names** and their
+**Arrow types** against the current schema. Catching type drift matters:
+an `episode.subject_vector` column left as `string` (or `null`) by an
+older build, while the schema now declares a 1024-d `fixed_size_list`,
+has the same column *name* — so a name-only check would wave it through
+and it would detonate later inside `merge_insert` as an opaque
+`LanceError(IO): Spill has sent an error` (EverOS #337). The type check
+turns that into this clean startup error.
+
+Recover with **`everos cascade rebuild`** (documented above). Do **not** just
+`rm -rf ~/.everos/.index/lancedb`: that clears the vectors but leaves
+`md_change_state` marked `done`, so the scanner skips every already-
+indexed file and the index comes back **empty**. And do **not**
+`rm -rf ~/.everos/.index`: that also deletes `unprocessed_buffer`
+(messages received but not yet extracted — not rebuildable from md).
+`cascade rebuild` is correct on both counts. Markdown is the source of
+truth, so no memory content is lost.
 
 ### inotify watch-limit exhaustion (Linux)
 
@@ -262,7 +317,9 @@ is a deployment-side change with no schema work.
 
 ## What cascade does NOT do (yet)
 
-- **Schema migration**: LanceDB column changes require `rm -rf`.
+- **Schema migration**: LanceDB has no in-place column migration; a
+  schema change is recovered by rebuilding from md (`everos cascade
+  rebuild`), not an automatic `ALTER`.
 - **Parent-id back-link**: Episode rows currently carry
   `parent_id=None`; the writer doesn't preserve the source memcell id
   in the entry inline. Tracked separately.

docs/cli.md

@@ -43,8 +43,10 @@ Each subcommand lives in its own module under
 registered in `cli/main.py`. The CLI is intentionally small — hot-path
 business (`/add` `/flush` `/search` `/get`) is the **HTTP API**, not the
 CLI; the CLI covers setup (`init`), running the server, and index ops
-(`cascade`). There is no `reindex` command — rebuild by deleting
-`<root>/.index/lancedb` and restarting, or run `everos cascade sync`.
+(`cascade`). There is no `reindex` command — for an incremental
+catch-up run `everos cascade sync`; to rebuild the whole index from
+markdown (recovery from drift / corruption) run `everos cascade
+rebuild`.
 
 ## `everos server start`
 

docs/how-memory-works.md

@@ -90,7 +90,8 @@ visually distinct from a user-named one).
     table `md_change_state`) — there is no `.cascade.log` / `.manifest.json`
     file in the current implementation. The `<app>/<project>` nesting is
     real and always present (`default_app/default_project` for the default
-    scope). There is **no `everos reindex` command** (see
+    scope). There is no command literally named `reindex`, but
+    `everos cascade rebuild` rebuilds the index from markdown (see
     [Operating it](#operating-it)).
 
 The path manager is
@@ -296,12 +297,17 @@ The CLI ([cli.md](cli.md)) is intentionally small:
 | `everos cascade status` | queue / LSN summary |
 | `everos cascade sync` | drain the cascade queue now (force md → LanceDB) |
 | `everos cascade fix` | list failed rows / re-enqueue retryable ones |
+| `everos cascade rebuild` | rebuild the whole index from markdown (drift / corruption recovery) |
 
 !!! warning "There is no `everos reindex` or `everos flush`"
-    - **Reindex** = the index is rebuildable: stop the server,
-      `rm -rf <memory-root>/.index/lancedb`, restart — the cascade
-      rebuilds from markdown. For an incremental catch-up, use
-      `everos cascade sync`.
+    - **Reindex** = the index is rebuildable from markdown. To rebuild
+      the whole index, run `everos cascade rebuild` — it drops the
+      LanceDB tables and re-indexes from md, re-populating even entries
+      the queue already marked `done` and preserving un-extracted
+      buffered messages. (A bare `rm -rf <memory-root>/.index/lancedb`
+      is **not** enough: the cascade queue still shows those files
+      `done`, so the scanner skips them and the index comes back empty.)
+      For an incremental catch-up, use `everos cascade sync`.
     - **Flush** is an HTTP endpoint (`POST /api/v2/memory/flush`), not a
       CLI command — it forces *extraction* of the session buffer, which is
       a different thing from forcing *index sync* (`cascade sync`).

docs/openapi.json

@@ -12,7 +12,7 @@
           "health"
         ],
         "summary": "Health",
-        "description": "Liveness probe with capabilities and disabled features.",
+        "description": "Liveness + capabilities + cascade readiness probe.\n\n``status`` stays ``\"ok\"`` whenever the process is up — the HTTP code\nis a *liveness* signal and a degraded cascade must not trigger a\nrestart (a crash-loop fixes neither a bad md file nor disk bloat).\nThe ``cascade`` block is the *readiness* signal: ``healthy=false``\nwith human-readable ``reasons`` **only** when the projection pipeline\nitself is stuck (drain failing, optimize stuck, version cleanup\nstalled). ``failed_permanent`` — files awaiting ``cascade fix`` — is\na data-quality backlog reported as an informational count that does\nnot flip ``healthy``. Alert on ``cascade.healthy``.",
         "operationId": "health_health_get",
         "responses": {
           "200": {
@@ -1805,6 +1805,63 @@
         ],
         "title": "Body_replace_document_route_api_v2_knowledge_documents__doc_id__put"
       },
+      "CascadeHealthBlock": {
+        "properties": {
+          "healthy": {
+            "type": "boolean",
+            "title": "Healthy"
+          },
+          "reasons": {
+            "items": {
+              "type": "string"
+            },
+            "type": "array",
+            "title": "Reasons"
+          },
+          "pending": {
+            "type": "integer",
+            "title": "Pending"
+          },
+          "failed_permanent": {
+            "type": "integer",
+            "title": "Failed Permanent"
+          },
+          "failed_retryable": {
+            "type": "integer",
+            "title": "Failed Retryable"
+          },
+          "drain_consecutive_failures": {
+            "type": "integer",
+            "title": "Drain Consecutive Failures"
+          },
+          "unrecoverable_total": {
+            "type": "integer",
+            "title": "Unrecoverable Total"
+          },
+          "optimize_failure_streak": {
+            "type": "integer",
+            "title": "Optimize Failure Streak"
+          },
+          "prune_stale_seconds": {
+            "type": "number",
+            "title": "Prune Stale Seconds"
+          }
+        },
+        "type": "object",
+        "required": [
+          "healthy",
+          "reasons",
+          "pending",
+          "failed_permanent",
+          "failed_retryable",
+          "drain_consecutive_failures",
+          "unrecoverable_total",
+          "optimize_failure_streak",
+          "prune_stale_seconds"
+        ],
+        "title": "CascadeHealthBlock",
+        "description": "Readiness of the md → LanceDB projection (cascade) subsystem.\n\n``healthy`` reflects **operational** health only — drain loop alive,\noptimize not stuck, version cleanup (prune) not stalled — and is what\nalerting should watch. ``failed_permanent`` (md files awaiting\n``cascade fix``) is a normal data-quality backlog reported as an\ninformational count; it does **not** flip ``healthy``, otherwise the\nsignal would sit red forever."
+      },
       "CategoryDTO": {
         "properties": {
           "category_id": {
@@ -2768,6 +2825,16 @@
             },
             "type": "array",
             "title": "Disabled Features"
+          },
+          "cascade": {
+            "anyOf": [
+              {
+                "$ref": "#/components/schemas/CascadeHealthBlock"
+              },
+              {
+                "type": "null"
+              }
+            ]
           }
         },
         "type": "object",
@@ -2778,7 +2845,7 @@
           "disabled_features"
         ],
         "title": "HealthResponse",
-        "description": "Response schema for ``GET /health``.\n\nDeclared as a Pydantic model (not ``dict``) so the generated\nOpenAPI schema carries the full field shape — ``capabilities`` and\n``disabled_features`` are typed. A bare ``-> dict`` return type\ndegrades the OpenAPI response to ``additionalProperties: true``,\nwhich robs clients (and codegen) of any structure to lean on."
+        "description": "Response schema for ``GET /health``.\n\nDeclared as a Pydantic model (not ``dict``) so the generated\nOpenAPI schema carries the full field shape — ``capabilities``,\n``disabled_features`` and ``cascade`` are typed. A bare ``-> dict``\nreturn type degrades the OpenAPI response to\n``additionalProperties: true``, which robs clients (and codegen) of\nany structure to lean on."
       },
       "KnowledgeSearchRequest": {
         "properties": {

pyproject.toml

@@ -26,7 +26,12 @@ dependencies = [
     "pydantic-settings>=2.0.0",
 
     # Storage stack (md-first three-piece set)
-    "lancedb>=0.13.0",            # Vector + BM25 + scalar filter (Arrow-based)
+    # Upper bound: 0.35.0 embeds lance-rust v9 (large storage/encoding jump, not
+    # yet stable-released). 0.32-0.34 carry a compaction offset-overflow regression
+    # (lance-format/lance#7653); we run 0.34.0 safely via the with_position=False
+    # FTS workaround. Do not float past 0.34.x until 0.35 is validated by the soak
+    # harness. Never widen this floor back below 0.34 (older lance can't read v8 data).
+    "lancedb>=0.34.0,<0.35.0",    # Vector + BM25 + scalar filter (Arrow-based)
     "aiosqlite>=0.20.0",          # Async SQLite driver (used by SA async engine)
     "sqlmodel>=0.0.22",           # ORM (Pydantic + SQLAlchemy 2.0 async)
     "alembic>=1.13.0",            # SQLite schema migrations