fix(lancedb): detect schema type drift and add cascade rebuild recovery#354
Open
gloryfromca wants to merge 2 commits into
Open
fix(lancedb): detect schema type drift and add cascade rebuild recovery#354gloryfromca wants to merge 2 commits into
gloryfromca wants to merge 2 commits into
Conversation
verify_business_schemas only compared column names, so a column whose on-disk Arrow type had drifted (name unchanged) slipped through and detonated later inside merge_insert as an opaque LanceError(IO) "Spill has sent an error" (#337). Now compare each shared column's Arrow type against schema.to_arrow_schema() — the exact schema get_table builds tables from, so a healthy table never false-positives. Reproduced #337 byte-identically: an episode.subject_vector column left as string or null by an older build, plus a real 1024-d vector on upsert, yields the exact crash. No lancedb version (0.13-0.34) renders Optional[Vector] as a non-vector type, so the startup guard is what should catch it — not the runtime. Add `everos cascade rebuild` as the safe recovery: it drops the business LanceDB tables and re-indexes from markdown, skipping the verify guard (which the drift would otherwise trip on startup). Unlike removing only .index/lancedb it re-populates already-done entries (reset_all clears the cascade queue); unlike removing all of .index it preserves unprocessed_buffer (messages not yet extracted). Fixes #337.
gloryfromca
force-pushed
the
fix/verify-schema-type-drift
branch
from
July 24, 2026 05:55
07b4704 to
fe65af2
Compare
Add the `everos cascade rebuild` command to the runbook, CLI, and how-memory-works docs. Correct the old recovery guidance: a bare `rm -rf .index/lancedb` leaves md_change_state marked `done`, so the scanner skips those files and the index comes back empty — the runbook previously claimed a full repopulation that does not happen. `cascade rebuild` is the safe path (re-populates done entries, preserves unprocessed_buffer). Also document that verify now checks column types.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem — the root cause behind #337
verify_business_schemas(the startup guard) only compared column names against the current schema. A column whose on-disk Arrow type had drifted — name unchanged — sailed through and detonated much later, deep insidemerge_insert, as the notoriously opaque:This is exactly #337: an
episode.subject_vectorcolumn left asstring(ornull) by an older build, while the current schema declares a 1024-dfixed_size_list. Writing a real vector into that column is a pyarrow type error that lance wraps as the "Spill" IO error.Verified
subject_vector(stringandnull) column + a real 1024-d vector through the realLanceRepoBase.upsert()path yields the exact cascade worker fails to index episodes: LanceDBSpill has sent an error(IO) #337 error; a correctly-typed table accepts the same records.TypeError, 0.20+ render the correctfixed_size_list), and lance never silently adds a mistyped column — so the drift originates from a build that declared the field differently, and the guard is what should have caught it at startup.Changes
1. Type-aware
verify_business_schemas— compare each shared column's Arrow type againstschema.to_arrow_schema()(the exact schemaget_tablebuilds tables from, so a healthy table never false-positives). On drift the startup error now names the drifted column and both types instead of the runtime Spill.2.
everos cascade rebuild— the safe recovery from a drifted/corrupt index:rm -rf ~/.everos/.index/lancedb, re-populates already-doneentries (reset_allclears the cascade queue so every md file re-enqueues) — a barermof the lancedb dir leaves the queue marked done and the index comes back empty;rm -rf ~/.everos/.index, preservesunprocessed_buffer(messages received but not yet extracted — not rebuildable from md).Tests
test_verify_schemas.py— healthy tables pass (false-positive guard);subject_vectorasstring/nullraises withtype_drift; missing column raises (unchanged);drop_business_tablesdrops + recreates the correct vector type. The type-drift tests were confirmed to fail without the fix (old guardDID NOT RAISE).test_md_change_state.py—reset_allclears every row regardless of status; noop on empty.test_cascade_cli_integration.py—cascade rebuildruns despite a drift that trips a normal command's startup verify, recreates the table with the correct type, and re-indexes md from scratch.Full unit suite +
make lint(ruff, import-linter, datetime, openapi) green.Fixes #337.