Feat/persistent storage durability#96
Merged
luis-gasparschroeder merged 6 commits intoJul 18, 2026
Merged
Conversation
Adds two opt-in, drop-in strategy implementations so the cache can survive process restarts: SQLiteEmbeddingMetadataStorage persists metadata to a SQLite file, and PersistentHNSWLibVectorDB persists its hnswlib index plus a small bookkeeping sidecar to disk. Existing defaults (in-memory storage, non-persistent HNSWLib) are unchanged.
…rites Addresses maintainer feedback on the persistent-storage PR: the previous implementation called index.save_index() plus a separate meta.json write on every add/remove, which rewrote the whole index each time and could leave the index and metadata desynchronized if a crash landed between the two writes. Replaces this with a checkpoint + write-ahead-log scheme: mutations are appended to a WAL (cheap, O(1) per call) and only folded into a full index snapshot periodically. Checkpoints publish a new generation file and manifest via temp-file-plus-os.replace, so the previous generation stays valid and visible until the new one is fully committed, and a crash can never surface a partially-written index or an index/metadata mismatch. On load, the last checkpoint is restored and any WAL entries written after it are replayed.
Contributor
Author
Safaael25
force-pushed
the
feat/persistent-storage-durability
branch
from
July 17, 2026 17:50
16afb2d to
a9412f5
Compare
This reverts commit 80ed092.
Contributor
Author
|
@luis-gasparschroeder |
Collaborator
|
@Safaael25, good to be merged, thanks. |
Co-authored-by: Cursor <cursoragent@cursor.com>
vcache-project
approved these changes
Jul 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make persistent HNSWLib writes atomic and avoid full index rewrites
PersistentHNSWLibVectorDB used to call save_index() (a full index rewrite) plus a separate meta.json write on every add/remove, and neither write was atomic — a crash between the two could desync the index from its metadata.
This replaces that with a checkpoint + write-ahead-log scheme:
Mutations are appended to a WAL (cheap, O(1)) instead of rewriting the full index each time.
The index is only fully serialized on a periodic checkpoint, published atomically via temp-file + os.replace alongside a manifest that pins the current generation and its metadata — so the two can never disagree.
On restart, the last checkpoint is loaded and any WAL entries written after it are replayed.
Single-writer crash safety only; concurrent multi-process writers to the same path are still unsupported (noted in the docstring).