feat(v2): Phase 3 — LanceDB index#228
Merged
Merged
Conversation
…erge-insert upsert
…n-scalar guard; strengthen evolution test
…; remove dead identifier-char set
Single list_tables() call only returns first page; iterate until page_token is None to avoid missing tables beyond the first page in ensure_table, drop_table and search existence checks.
- table_columns: add ORDER BY version ASC so earliest-version dtype wins deterministically (prevents dtype divergence between successive table_columns calls on the same schema). - rebuild_schema_index: add V2Record.id as tiebreaker to the OFFSET pagination sort key so rows with equal inserted_at cannot be skipped across page boundaries.
…ild; add response_model - RecordFilter: pydantic model_validator rejects op='in' with a scalar value (str/non-iterable) raising ValueError → 422 instead of an unmapped TypeError → 500 - LanceIndexEngine.upsert: add optimize=False kwarg; new optimize_table() method - rebuild_schema_index: upsert all batches with optimize=False, call optimize_table() once at the end — avoids O(batches) FTS rebuilds during a full reindex - rebuild_schema_index endpoint: add response_model=dict[str,int] and timeout advisory docstring - Test: assert op='in' + scalar → 422
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 of 6 — LanceDB index engine (v2)
Adds a LanceDB-backed index derived from Postgres, exposing full-text (BM25) + scalar-filter search over v2 records via
POST /api/v2/schemas/{id}/records:search, kept in sync best-effort on write and rebuildable. Additive only — the v1search_engine/(ES/OpenSearch) package is untouched (branch diff is empty over that path); v1 retires in Phase 6.Spec:
docs/superpowers/specs/2026-06-27-schema-centric-data-model-design.md§15 (design) + new §16 (implementation resolutions).Plan:
docs/superpowers/plans/2026-07-07-v2-lancedb-index.md.What's included
index/package — pure Arrow/rowmapping.py(no LanceDB/DB import); v2-shapedIndexEngineABC + result/filter models (base.py);LanceIndexEngineon the async LanceDB API (lancedb_engine.py);get_index_engineDI provider mirroringget_search_engine.contexts/v2/index_sync.py) — ensure/evolve on schema publish, upsert after bulk-upsert, delete on delete. Any engine error is logged at WARNING and swallowed; the API request still succeeds. Only:rebuild-index,rebuild_schema_index, and the reindex CLI surface errors.POST /schemas/{id}/records:search— Lance supplies matchingrecord_ids + scores; payloads are hydrated from Postgres (source of truth), returned in Lance hit order, tolerant of stale-index rows.POST /schemas/{id}:rebuild-indexrecovery endpoint +extralit_server index reindex|listCLI.EXTRALIT_LANCEDB_URIsetting (defaults to{home_path}/lance).Design guarantees
schema_{id.hex}): system columns + typed columns (superset across versions viaadd_columns) + a derivedtextcolumn carrying the FTS index.similarity_search, no litellm this phase.Testing / review
search_engine+ v2indexcoexist on import;create_server_app()boots;ruff check/formatclean.ge/le+None→ always-false>= NULL;lancedb_uriannotation).🤖 Generated with Claude Code