Add demo proof artifacts and self-test tooling

CHANGELOG.md

@@ -1,4 +1,13 @@
 # Changelog
 
+# Changelog
+
+## 0.2.0 - 2024-05-15 "Proof Artifacts"
+- Added offline demo corpus with cross-platform run scripts and documentation.
+- Implemented `raglite self-test` CLI command with backend detection and stats output.
+- Bundled tiny evaluation and benchmarking scripts with accompanying pytest coverage.
+- Clarified vector backend selection with Python fallback and SQLite extension support.
+- Documented architecture, limitations, and quickstart workflow with new README badges.
+
 ## 0.1.0 - 2024-05-01
 - Initial release of raglite with SQLite-backed RAG pipeline, Typer CLI, FastAPI server, and demo corpus.

README.md

@@ -1,49 +1,147 @@
 # raglite
 
-Local-first retrieval augmented generation toolkit built on SQLite. Raglite bundles ingestion, chunking, embeddings, hybrid BM25/vector search, a Typer CLI, and a FastAPI microservice. Everything runs on CPU and stores state in a single SQLite database.
+[![CI](https://github.com/mmprotest/raglite-sqlite/actions/workflows/ci.yml/badge.svg)](https://github.com/mmprotest/raglite-sqlite/actions/workflows/ci.yml)
+![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)
+![Python: 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)
+![PyPI: TODO](https://img.shields.io/badge/PyPI-TODO-lightgrey.svg)
 
-## Features
-
-- SQLite FTS5 BM25 search with optional vector reranking.
-- Pluggable embedding backends with automatic fallback to a deterministic hash model for testing.
-- Python API, Typer CLI, and FastAPI server.
-- Demo corpus and integration examples for LangChain and LlamaIndex.
+Local-first retrieval augmented generation toolkit built on SQLite. Raglite bundles
+ingestion, chunking, hybrid BM25/vector search, Typer CLI workflows, and a FastAPI
+microservice. Everything runs on CPU and stores state in a single SQLite database.
 
-## Quickstart
+## Quickstart in 6 lines
 
 ```bash
-pip install -e .[server,dev]
+python -m venv .venv && source .venv/bin/activate
+pip install -e .[server]
 raglite init-db --db demo.db
-raglite ingest --db demo.db --path src/demo/mini_corpus --embed-model debug
-raglite query --db demo.db --text "quick start guide"
+raglite ingest --db demo.db --path demo/mini_corpus
+raglite query --db demo.db --text "quick start guide" --k 5 --alpha 0.6
+raglite serve --db demo.db --host 127.0.0.1 --port 8080
 ```
 
+_On Windows use `\.venv\Scripts\activate` for step two._
+
+## Features
+
+- Deterministic chunking and debug embeddings for air-gapped demos, with an easy upgrade
+  path to SentenceTransformer models.
+- Hybrid BM25 + cosine search with rerank option and automatic Python fallback when SQLite
+  vector extensions are unavailable.
+- Typer CLI (`raglite`), FastAPI server (`raglite serve`), and Python API for scripted use.
+- Offline demo kit (`demo/mini_corpus`) with one-shot run scripts and proof artifacts.
+- Tiny eval and benchmark scripts that run in seconds and provide reproducible metrics.
+
+## Demo & proof artifacts
+
+Explore the bundled two-minute corpus and scripts inside [`demo/`](demo/README.md):
+
+- `demo/mini_corpus/`: 12 varied documents (how-to, FAQ, articles, product docs).
+- `demo/run_demo.sh` / `demo/run_demo.ps1`: create a virtual environment, install Raglite,
+  ingest the demo corpus, run a sample query, and start the FastAPI server with curl hints.
+- `demo/eval_set.jsonl`: 25 ground-truth query snippets for offline evaluation.
+
+A placeholder animation (`demo/demo.gif`) can be replaced with your own screenshots once
+you run the scripts.
+
+## CLI highlights
+
+- `raglite self-test` builds a temporary database from the demo corpus, runs three canned
+  queries, prints titles/snippets, reports the active vector backend, and dumps stats.
+- `raglite stats` now returns document, chunk, embedding counts plus backend, embedding
+  model/dimensions, FTS status, and alpha.
+- `raglite benchmark` / `raglite eval` invoke the new tiny scripts under `scripts/`.
+
+## Vector backends
+
+Raglite automatically selects the most capable vector backend:
+
+1. **SQLite extension** (`sqlite-vec` or `sqlite-vss`): if loadable, cosine similarity runs
+   directly inside SQLite for best performance. Example load step:
+   ```python
+   import sqlite3
+   conn = sqlite3.connect("raglite.db")
+   conn.enable_load_extension(True)
+   conn.load_extension("sqlite_vec")
+   conn.enable_load_extension(False)
+   ```
+2. **Python fallback (default)**: BM25 prefilters the top 200 rows and cosine similarity is
+   computed with NumPy arrays in Python. This works cross-platform with zero extra
+   dependencies.
+3. **None**: if the embeddings table is absent, vector search is skipped and BM25 answers
+   requests alone.
+
+`raglite self-test`, `raglite stats`, and the benchmark script print which path you are on.
+Expect the Python fallback to be a few milliseconds slower per query but fully portable.
+
 ## Architecture
 
+```mermaid
+erDiagram
+  documents ||--o{ chunks : contains
+  chunks ||--o{ embeddings : has
+  documents {
+    int id
+    text path
+    text title
+    text mime
+    text meta_json
+    datetime created_at
+  }
+  chunks {
+    int id
+    int document_id
+    int chunk_idx
+    text text
+    int tokens
+    text tags_json
+  }
+  embeddings {
+    int id
+    int chunk_id
+    text model
+    int dim
+    blob embedding
+  }
+
+  %% Additional structure
+  %% chunk_fts is an FTS5 virtual table with triggers syncing chunk text changes.
 ```
-+-----------------+
-|   Typer CLI     |
-+-----------------+
-        |
-+-----------------+     +-----------------------+
-|    API Layer    |<--->|    FastAPI Service    |
-+-----------------+     +-----------------------+
-        |
-+-----------------+     +-----------------------+
-|  Search Engine  |<--->| Vector Backends       |
-+-----------------+     +-----------------------+
-        |
-+-----------------------------------------------+
-|                   SQLite DB                    |
-+-----------------------------------------------+
-```
 
-See `src/raglite/schema.sql` for the full schema.
+`chunk_fts` is a virtual FTS5 table kept in sync with the `chunks` table via insert/update
+triggers; see [`src/raglite/schema.sql`](src/raglite/schema.sql) for details.
+
+## Evaluations & benchmarks
+
+- `python scripts/eval_small.py` compares BM25, Hybrid (α=0.6), and optional rerank on the
+  bundled dataset. Hybrid matches BM25 on this micro-set by default and rerank (if
+  installed) can further improve conversational queries.
+- `python scripts/bench_basic.py` duplicates the demo corpus to ~2k chunks, reports indexing
+  throughput, query latency (p50/p95) with and without vector fallback, database size, and
+  backend selection.
+- Both scripts support `--tiny` for <30s smoke runs and fall back to packaged data when
+  installed from wheels.
+
+> **Limitations & Guidance**
+> - Designed for small/medium local corpora, not massive ANN workloads. For millions of
+>   vectors, adopt a dedicated vector database.
+> - Best with a single writer; readers work fine with WAL mode. Remember to back up both
+>   `*.db` and `*.db-wal` files.
+> - Increase α to ≥0.6 when keyword precision matters; lower it or enable rerank (with
+>   `raglite[rerank]`) for conversational questions.
+> - Raglite is local-first—avoid ingesting secrets or PII. Use `.ragliteignore` patterns or
+>   preprocess files to filter sensitive content.
+
+## Tests & quality
 
-## Tests
+Run the full suite that matches CI:
 
 ```bash
+ruff check src tests
+black --check src tests
+mypy src
 pytest -q
+raglite self-test
 ```
 
 ## License

demo/README.md

@@ -0,0 +1,25 @@
+# Raglite demo kit
+
+![Demo animation placeholder](demo.gif)
+
+Run the offline two-minute demo end-to-end:
+
+```bash
+./demo/run_demo.sh
+```
+
+Or on Windows PowerShell:
+
+```powershell
+./demo/run_demo.ps1
+```
+
+The scripts create a virtual environment, install Raglite with the server extra, build a
+database from `demo/mini_corpus`, run a sample query, and start the API server on
+`http://127.0.0.1:8080` with helpful `curl` examples.
+
+You can also reproduce the self-test in isolation:
+
+```bash
+raglite self-test
+```

demo/eval_set.jsonl

@@ -0,0 +1,26 @@
+{"query": "create a status dashboard", "relevant_substring": "select the \"Service Health\" template"}
+{"query": "dashboard live preview toggle", "relevant_substring": "toggle \"Live Preview\""}
+{"query": "schedule raglite backups", "relevant_substring": "Create a cron entry"}
+{"query": "enabling wal prior to exports", "relevant_substring": "Enable WAL mode before exporting"}
+{"query": "tune hybrid alpha", "relevant_substring": "Run `raglite query --alpha 0.6`"}
+{"query": "alpha setting for conversational questions", "relevant_substring": "Decrease to 0.3"}
+{"query": "installation admin rights", "relevant_substring": "No. Raglite ships as pure Python"}
+{"query": "air-gapped embeddings", "relevant_substring": "Use `--embed-model debug`"}
+{"query": "run without gpus", "relevant_substring": "All pipelines run on CPU"}
+{"query": "documents table metadata", "relevant_substring": "Inside the `documents` table"}
+{"query": "chunk tagging metadata field", "relevant_substring": "Each chunk row has a JSON column"}
+{"query": "network drive wal file", "relevant_substring": "copy the `.db` and `-wal` files together"}
+{"query": "search returning empties", "relevant_substring": "Verify the ingest step ran"}
+{"query": "vector cache slow first run", "relevant_substring": "vector cache builds the first time"}
+{"query": "reset index", "relevant_substring": "Delete the SQLite file"}
+{"query": "local-first raglite", "relevant_substring": "Raglite keeps retrieval pipelines"}
+{"query": "hybrid ranking blends", "relevant_substring": "Hybrid ranking blends BM25"}
+{"query": "debug embedding hashes tokens", "relevant_substring": "hashes tokens for speed"}
+{"query": "sentence transformer smoother similarity", "relevant_substring": "plug in a sentence-transformer"}
+{"query": "backup strategy nightly exports", "relevant_substring": "snapshot production data"}
+{"query": "storing db wal partner", "relevant_substring": "Store both the `.db` file and its `.db-wal` partner"}
+{"query": "raglite console query builder", "relevant_substring": "A query builder that exposes alpha"}
+{"query": "offline installers for windows", "relevant_substring": "Offline installers for Windows"}
+{"query": "replicating indexes synchronization service", "relevant_substring": "watches the `.db` and `.db-wal` files"}
+{"query": "newest file prevails", "relevant_substring": "latest file wins"}
+{"query": "self-test command release notes", "relevant_substring": "Added a self-test command"}

demo/mini_corpus/article_backup_strategy.md

@@ -0,0 +1,6 @@
+# Backup strategy for local indexes
+
+Treat the Raglite database like source code. Commit small demo indexes, but snapshot
+production data using nightly exports. Store both the `.db` file and its `.db-wal` partner,
+then verify the archive by opening it with `sqlite3` in read-only mode. Regular integrity
+checks keep retrieval fast and avoid fragmented pages.

demo/mini_corpus/article_embeddings.md

@@ -0,0 +1,6 @@
+# Why embeddings still matter
+
+Even with a small corpus, embeddings capture paraphrased questions. The debug embedding
+model hashes tokens for speed yet still distinguishes "backup schedule" from "restore run".
+For production, plug in a sentence-transformer to get smoother similarity curves and better
+recall for conversations that wander between topics.

demo/mini_corpus/article_local_rag.md

@@ -0,0 +1,7 @@
+# Local-first RAG in practice
+
+Raglite keeps retrieval pipelines on your laptop. The SQLite core fits alongside existing
+notebooks, so teams avoid provisioning a remote vector database. Hybrid ranking blends
+BM25 filtering with cosine similarity to surface precise answers from even tiny datasets.
+Because ingestion produces deterministic chunks, you can commit the resulting database to
+version control and reproduce experiments later without drift.

demo/mini_corpus/faq_installation.md

@@ -0,0 +1,10 @@
+# Installation FAQ
+
+**Q: Do I need admin rights?**
+A: No. Raglite ships as pure Python and SQLite. Install inside a virtual environment.
+
+**Q: What about embeddings?**
+A: Use `--embed-model debug` for air-gapped demos or install `sentence-transformers` for production.
+
+**Q: Can I run without GPUs?**
+A: Yes. All pipelines run on CPU and complete within seconds for the demo corpus.

demo/mini_corpus/faq_storage.txt

@@ -0,0 +1,11 @@
+FAQ: Storage layout
+-------------------
+
+Q: Where does Raglite keep metadata?
+A: Inside the `documents` table. Paths, titles, and MIME types stay normalized.
+
+Q: How do I track chunk tags?
+A: Each chunk row has a JSON column. Use `raglite add-tags` to patch keys atomically.
+
+Q: Can I use network drives?
+A: Yes, but enable WAL mode and copy the `.db` and `-wal` files together when backing up.

demo/mini_corpus/faq_troubleshooting.md

@@ -0,0 +1,10 @@
+# Troubleshooting FAQ
+
+**Q: My search returns empty results.**
+A: Verify the ingest step ran with the same embedding model that your server uses.
+
+**Q: Why is the CLI slow on first run?**
+A: The vector cache builds the first time you query. Subsequent runs are instant.
+
+**Q: How do I reset the index?**
+A: Delete the SQLite file and re-run `raglite init-db` followed by ingestion.

demo/mini_corpus/howto_create_dashboard.md

@@ -0,0 +1,12 @@
+# How to create a status dashboard
+
+1. Open Raglite Studio and sign in with your local profile.
+2. Click **New Dashboard** and select the "Service Health" template.
+3. Add the latency, uptime, and error rate widgets from the component tray.
+4. Connect each widget to an existing monitor or paste static values for demos.
+5. Share the dashboard snapshot with your team through the local share link.
+
+Tip: dashboards refresh every minute; toggle "Live Preview" for second-by-second updates.
+
+Need a quick start guide? Run `raglite self-test` to build the demo database and
+inspect sample answers before customizing your own widgets.

demo/mini_corpus/howto_schedule_backups.txt

@@ -0,0 +1,8 @@
+How to schedule Raglite backups
+-------------------------------
+
+1. Create a cron entry that runs `raglite export --db ~/data/raglite.db --out ~/backups/$(date +%Y%m%d).zip`.
+2. Enable WAL mode before exporting to keep readers online: `raglite pragma --db ~/data/raglite.db --set journal_mode=WAL`.
+3. Copy the generated archive to an offline disk once per week.
+
+Backups remain compact because Raglite stores content inside a single SQLite file.

demo/mini_corpus/howto_tune_alpha.md

@@ -0,0 +1,8 @@
+# How to tune hybrid alpha
+
+- Run `raglite query --alpha 0.6` for balanced lexical/vector ranking.
+- Increase to 0.8 for keyword-heavy compliance documents.
+- Decrease to 0.3 when questions are phrased conversationally.
+- Validate results against the built-in tiny eval dataset before rolling out.
+
+Record the chosen alpha inside `raglite.toml` so the server and CLI stay aligned.

demo/mini_corpus/product_datasheet_console.md

@@ -0,0 +1,7 @@
+# Raglite Console Datasheet
+
+The Raglite Console is a lightweight desktop UI for browsing documents and running saved
+queries. It ships with:
+- Offline installers for Windows, macOS, and Linux.
+- A query builder that exposes alpha blending and rerank toggles.
+- Export tools that package highlighted answers into Markdown briefs.

demo/mini_corpus/product_datasheet_sync.md

@@ -0,0 +1,5 @@
+# Raglite Sync Service Overview
+
+Raglite Sync replicates a SQLite index between laptops. It watches the `.db` and `.db-wal`
+files, compresses new pages, and pushes them over SSH. Conflict resolution is simple: the
+latest file wins, so schedule syncs right after ingestion jobs finish.

demo/mini_corpus/product_release_notes.md

@@ -0,0 +1,6 @@
+# Raglite 0.2 Release Notes
+
+- Added a self-test command that builds a demo database and prints backend stats.
+- Bundled a two-minute corpus for offline demos and quickstart experiments.
+- Improved hybrid scoring so NumPy fallback can surface conversational matches.
+- Fixed minor UI bugs in Raglite Console around query history refresh.

demo/run_demo.ps1

@@ -0,0 +1,45 @@
+Param(
+    [string]$Python = "python"
+)
+
+$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
+$RepoRoot = Resolve-Path (Join-Path $ScriptDir "..")
+$VenvPath = Join-Path $RepoRoot ".venv"
+$DbPath = Join-Path $RepoRoot "demo/demo.db"
+$CorpusDir = Join-Path $RepoRoot "demo/mini_corpus"
+
+if (-not (Test-Path $VenvPath)) {
+    Write-Host "[raglite-demo] Creating virtual environment at $VenvPath"
+    & $Python -m venv $VenvPath
+}
+
+$Activate = Join-Path $VenvPath "Scripts/Activate.ps1"
+. $Activate
+
+Set-Location $RepoRoot
+pip install --upgrade pip
+pip install -e .[server]
+
+if (Test-Path $DbPath) {
+    Remove-Item $DbPath
+}
+
+raglite init-db --db $DbPath
+raglite ingest --db $DbPath --path $CorpusDir --embed-model debug --strategy fixed
+raglite query --db $DbPath --text "quick start guide" --k 5 --alpha 0.6 --embed-model debug
+
+Write-Host ""
+Write-Host "[raglite-demo] Database ready at $DbPath"
+Write-Host "[raglite-demo] Starting server on http://127.0.0.1:8080"
+Write-Host "[raglite-demo] Try:"
+Write-Host "  curl -s http://127.0.0.1:8080/health"
+Write-Host "  curl -s -X POST http://127.0.0.1:8080/query `\
+    -H 'Content-Type: application/json' `\
+    -d '{\"text\": \"backup schedule\", \"k\": 3}'"
+
+try {
+    raglite serve --db $DbPath --host 127.0.0.1 --port 8080 --embed-model debug
+}
+finally {
+    Write-Host "[raglite-demo] Shutting down server"
+}