Skip to content

fix(lancedb): detect schema type drift and add cascade rebuild recovery#354

Open
gloryfromca wants to merge 2 commits into
mainfrom
fix/verify-schema-type-drift
Open

fix(lancedb): detect schema type drift and add cascade rebuild recovery#354
gloryfromca wants to merge 2 commits into
mainfrom
fix/verify-schema-type-drift

Conversation

@gloryfromca

@gloryfromca gloryfromca commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Problem — the root cause behind #337

verify_business_schemas (the startup guard) only compared column names against the current schema. A column whose on-disk Arrow type had drifted — name unchanged — sailed through and detonated much later, deep inside merge_insert, as the notoriously opaque:

RuntimeError: lance error: LanceError(IO): Execution error: Spill has sent an error, poll.rs:291:44

This is exactly #337: an episode.subject_vector column left as string (or null) by an older build, while the current schema declares a 1024-d fixed_size_list. Writing a real vector into that column is a pyarrow type error that lance wraps as the "Spill" IO error.

Verified

  • Reproduced byte-identically: a drifted subject_vector (string and null) column + a real 1024-d vector through the real LanceRepoBase.upsert() path yields the exact cascade worker fails to index episodes: LanceDB Spill has sent an error (IO) #337 error; a correctly-typed table accepts the same records.
  • No lancedb version produces the bad column (checked 0.13→0.34: ≤0.19 raise TypeError, 0.20+ render the correct fixed_size_list), and lance never silently adds a mistyped column — so the drift originates from a build that declared the field differently, and the guard is what should have caught it at startup.

Changes

1. Type-aware verify_business_schemas — compare each shared column's Arrow type against schema.to_arrow_schema() (the exact schema get_table builds tables from, so a healthy table never false-positives). On drift the startup error now names the drifted column and both types instead of the runtime Spill.

2. everos cascade rebuild — the safe recovery from a drifted/corrupt index:

  • drops the business LanceDB tables and re-indexes from markdown (the source of truth);
  • skips the verify guard (whose whole point is the drift it would otherwise trip on at startup — chicken-and-egg);
  • unlike rm -rf ~/.everos/.index/lancedb, re-populates already-done entries (reset_all clears the cascade queue so every md file re-enqueues) — a bare rm of the lancedb dir leaves the queue marked done and the index comes back empty;
  • unlike rm -rf ~/.everos/.index, preserves unprocessed_buffer (messages received but not yet extracted — not rebuildable from md).

Tests

  • test_verify_schemas.py — healthy tables pass (false-positive guard); subject_vector as string/null raises with type_drift; missing column raises (unchanged); drop_business_tables drops + recreates the correct vector type. The type-drift tests were confirmed to fail without the fix (old guard DID NOT RAISE).
  • test_md_change_state.pyreset_all clears every row regardless of status; noop on empty.
  • test_cascade_cli_integration.pycascade rebuild runs despite a drift that trips a normal command's startup verify, recreates the table with the correct type, and re-indexes md from scratch.

Full unit suite + make lint (ruff, import-linter, datetime, openapi) green.

Fixes #337.

verify_business_schemas only compared column names, so a column whose
on-disk Arrow type had drifted (name unchanged) slipped through and
detonated later inside merge_insert as an opaque LanceError(IO)
"Spill has sent an error" (#337). Now compare each shared column's Arrow
type against schema.to_arrow_schema() — the exact schema get_table
builds tables from, so a healthy table never false-positives.

Reproduced #337 byte-identically: an episode.subject_vector column left
as string or null by an older build, plus a real 1024-d vector on
upsert, yields the exact crash. No lancedb version (0.13-0.34) renders
Optional[Vector] as a non-vector type, so the startup guard is what
should catch it — not the runtime.

Add `everos cascade rebuild` as the safe recovery: it drops the business
LanceDB tables and re-indexes from markdown, skipping the verify guard
(which the drift would otherwise trip on startup). Unlike removing only
.index/lancedb it re-populates already-done entries (reset_all clears
the cascade queue); unlike removing all of .index it preserves
unprocessed_buffer (messages not yet extracted).

Fixes #337.
@gloryfromca
gloryfromca force-pushed the fix/verify-schema-type-drift branch from 07b4704 to fe65af2 Compare July 24, 2026 05:55
@gloryfromca gloryfromca changed the title 🐛 fix(lancedb): verify_business_schemas compares column types, not just names fix(lancedb): detect schema type drift and add cascade rebuild recovery Jul 24, 2026
@gloryfromca gloryfromca reopened this Jul 24, 2026
Add the `everos cascade rebuild` command to the runbook, CLI, and
how-memory-works docs. Correct the old recovery guidance: a bare
`rm -rf .index/lancedb` leaves md_change_state marked `done`, so the
scanner skips those files and the index comes back empty — the runbook
previously claimed a full repopulation that does not happen. `cascade
rebuild` is the safe path (re-populates done entries, preserves
unprocessed_buffer). Also document that verify now checks column types.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cascade worker fails to index episodes: LanceDB Spill has sent an error (IO)

1 participant