Make SQLite the default local/Docker database#13
Open
mkonopelski-gd wants to merge 8 commits into
Open
Conversation
Replaces the Firestore emulator sidecar with a new SqliteDatabase (IDatabase impl) as the sole local/Docker-dev backend. Firestore now only connects to an already-hosted, GCP-managed instance (DATABASE_TYPE=firestore) or a manually-run emulator process (DATABASE_TYPE=emulator) — SpecFlow never deploys/manages Firestore itself locally anymore. - docker-compose.yml: drop firestore-emulator/firestore-exporter services; backend defaults to DATABASE_TYPE=sqlite and bind-mounts the host's ~/.specflow/ directory (one central db shared across local projects/MCP sessions, matching the old shared-emulator model). - Makefile: init-firestore(-dry) -> init-db(-dry) (aliases kept), isolated test-stack gets its own nested sqlite path so tests never touch the real central db. - init_firestore.py -> init_db.py; emulator-host requirement now conditional on DATABASE_TYPE. - create_generation_session_repos.py: DATABASE_TYPE gate now rejects only memory (was firestore-only). - startup_validation.py: generalized Firestore-only connectivity check to a DB-agnostic one; TOKEN_ENCRYPTION_KEY/GitHub-secrets requirement now covers sqlite too (it persists across restarts, unlike memory). - Extracted shared IDatabase contract tests (db_contract.py) run against both memory and sqlite backends.
The TUI's container-readiness check and specflow-init.sh's onboarding flow still assumed a firestore-emulator container existed. Fixes the resulting breakage from the sqlite-default switch: - local_env.py: containers_running() now checks only the backend container (sqlite has no separate container — it's a bind-mounted file). - specflow-init.sh: seeding step no longer hardcodes FIRESTORE_EMULATOR_HOST unconditionally; --reset-local-db now clears the SQLite file + WAL/SHM sidecars instead of an emulator export directory. - .env.quickstart.example: DATABASE_TYPE default flips to sqlite, documents SQLITE_DB_PATH/SPECFLOW_HOME_MOUNT_PATH, reframes Firestore vars as hosted-connect-only. - mcp_server/tests/e2e/runner.py + cli.py: default to sqlite for API-key fetch and update --reset-local-db help text.
Follow-up to the sqlite migration — these still described the removed Firestore emulator.
mkonopelski-gd
force-pushed
the
feature/use-sqllite-instead-firestore
branch
from
d9a1896 to
6712186
Compare
mkonopelski-gd
commented
Jul 2, 2026
- Nest specflow.db/-wal/-shm under a db/ subdirectory (~/.specflow/db/, container /root/.specflow/db/) instead of directly in ~/.specflow/, so the database files stay separate from other files (e.g. config.json) in that directory. Updated docker-compose.yml, config.py, Makefile, specflow-init.sh, .env.quickstart.example, and the mcp_server e2e runner default accordingly. - docs/ARCHITECTURE.md: answer where ~/.specflow/ and its db/ subdir get created on a fresh install (Docker bind-mount + SqliteDatabase.__init__). - Trim two over-long inline comments (backend/Dockerfile, local_env.py) to one-liners per review feedback.
…tion - init_db.py: default DATABASE_TYPE to sqlite (not memory) when unset, matching the sqlite-is-the-default intent everywhere else. Previously, running the script directly without exporting DATABASE_TYPE silently fell back to the ephemeral in-memory backend. - scripts/README.md: documents the sqlite default; drops the stale "defaults to memory, data won't persist" warning. - docs/backend/ARCHITECTURE.md, docs/backend/DEVELOPMENT.md: backend contributor docs updated for the sqlite migration (missed in the earlier top-level docs/ARCHITECTURE.md and QUICKSTART.md pass), using the db/ subdirectory path from the PR review fixup.
Host-side seeding (uv run scripts/init_db.py) resolved SQLITE_DB_PATH from .env, which is the container-internal path (/root/.specflow/db/specflow.db). On the host that path is unwritable for non-root users (init_db.py crashes on mkdir) and, even when writable, seeds a file the container never reads (it reads the bind-mount source ~/.specflow/db/specflow.db) — so the backend boots with an unseeded DB and LocalAuthMiddleware rejects every request. This broke the from-scratch local quickstart. Derive the host seed path from SPECFLOW_HOME_PATH (the bind-mount source), the single knob (via SPECFLOW_HOME_MOUNT_PATH) shared by host and container. Comment out the container-internal SQLITE_DB_PATH in .env.quickstart.example (already defaulted by docker-compose) and add a regression test. Addresses backend-review Finding #1 (CRITICAL).
Both host-side seeders carried their own copy of the workspace-document builder, id-assignment, and upsert loop. The two create_workspace_document copies had already drifted — the provisioner's was missing the cleaning_started_at field. Introduce app/services/workspace_pool_seeding.py as the single source of truth: WorkspacePoolEntry (typed, validated), parse_pool_entries, assign_pool_entries (ordered + prefix id assignment), build_workspace_document (one schema, incl. cleaning_started_at), and seed_workspace_pool (idempotent upsert with created/ updated/skipped counts). init_db.py and create_generation_session_repos.py now delegate to it. No behavior change; file-based seeding contracts preserved. Adds unit tests for the module. Part of moving local workspace config to SQLite as the single source of truth.
…spaces.json
The workspace pool was duplicated: a transient .specflow-local/workspaces.json flat
file AND the DB workspaces collection. Eliminate the flat file so the SQLite DB is the
single source of truth for the local pool.
- create_generation_session_repos.py now seeds the DB directly for BOTH the {prefix}{num}
and --provide-own-repos (arbitrary-name) paths — id assignment handles ordered repos, so
the arbitrary-name limitation that forced the file handoff is gone.
- init_db.py: --workspace-config is now optional. Without it (local quickstart) it seeds only
the bootstrap API key + local-auth identity; the file input is retained for e2e / BYO repos.
- specflow-init.sh: provisioning writes the pool straight into the DB (drops --skip-firestore
and --output-workspace-config, passes the host SQLITE_DB_PATH); init_db.py then seeds API key
+ identity. .specflow-local/workspaces.json is gone entirely.
Tests updated to the new contract; MEMORY.md note refreshed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SqliteDatabase(IDatabaseimpl) as the sole local/Docker-dev backend. Firestore now only connects to an already-hosted, GCP-managed instance (DATABASE_TYPE=firestore) or a manually-run emulator process (DATABASE_TYPE=emulator) — SpecFlow never deploys/manages Firestore itself locally anymore.docker-compose.yml: dropsfirestore-emulator/firestore-exporterservices; backend defaults toDATABASE_TYPE=sqliteand bind-mounts the host's~/.specflow/directory (one central db shared across local projects/MCP sessions, matching the old shared-emulator model).Makefile:init-firestore(-dry)→init-db(-dry)(aliases kept); isolated test-stack gets its own nested sqlite path so tests never touch the real central db.init_firestore.py→init_db.py; emulator-host requirement now conditional onDATABASE_TYPE.IDatabasecontract tests (db_contract.py) run against both memory and sqlite backends.specflow-init.shand.env.quickstart.examplenow describe/seed sqlite (not the removed emulator),mcp_server's CLI help text, container-readiness check, and e2e test-credential fetch follow the new sqlite default.Test plan
make unit-tests— 560 passed./specflow-init.sh --dry-runand a real local quickstart against~/.specflow/specflow.dbCo-Authored-By: Claude Sonnet 5 noreply@anthropic.com