Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# raglite
# raglite-SQLite

[![CI](https://github.com/mmprotest/raglite-sqlite/actions/workflows/ci.yml/badge.svg)](https://github.com/mmprotest/raglite-sqlite/actions/workflows/ci.yml)
![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)
![Python: 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)
[![PyPI](https://img.shields.io/pypi/v/raglite.svg)](https://pypi.org/project/raglite/)
[![PyPI](https://img.shields.io/pypi/v/raglite-sqlite.svg)](https://pypi.org/project/raglite-SQLite/)

Local-first retrieval augmented generation toolkit built on SQLite. Raglite bundles
ingestion, chunking, hybrid BM25/vector search, Typer CLI workflows, and a FastAPI
Expand All @@ -13,7 +13,7 @@ microservice. Everything runs on CPU and stores state in a single SQLite databas

```bash
python -m venv .venv && source .venv/bin/activate
pip install "raglite[server]"
pip install "raglite-sqlite[server]"
raglite init-db --db demo.db
raglite ingest --db demo.db --path demo/mini_corpus
raglite query --db demo.db --text "quick start guide" --k 5 --alpha 0.6
Expand All @@ -24,24 +24,24 @@ _On Windows use `\.venv\Scripts\activate` for step two._

## Installation

Raglite is published as [`raglite` on PyPI](https://pypi.org/project/raglite/). Install the
Raglite is published as [`raglite-SQLite` on PyPI](https://pypi.org/project/raglite-SQLite/). Install the
core toolkit with:

```bash
pip install raglite
pip install raglite-sqlite
```

To enable the optional FastAPI server, install the `server` extra:

```bash
pip install "raglite[server]"
pip install "raglite-sqlite[server]"
```

Development dependencies (linters, type checkers, packaging utilities) can be installed
with the `dev` extra:

```bash
pip install "raglite[dev]"
pip install "raglite-sqlite[dev]"
```

## Features
Expand Down Expand Up @@ -150,7 +150,7 @@ triggers; see [`src/raglite/schema.sql`](src/raglite/schema.sql) for details.
> - Best with a single writer; readers work fine with WAL mode. Remember to back up both
> `*.db` and `*.db-wal` files.
> - Increase α to ≥0.6 when keyword precision matters; lower it or enable rerank (with
> `raglite[rerank]`) for conversational questions.
> `raglite-sqlite[rerank]`) for conversational questions.
> - Raglite is local-first—avoid ingesting secrets or PII. Use `.ragliteignore` patterns or
> preprocess files to filter sensitive content.

Expand Down
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ known_first_party = ["raglite"]

[tool.ruff]
line-length = 100

[tool.ruff.lint]
select = ["E", "F", "I", "B"]

[tool.mypy]
Expand Down
45 changes: 26 additions & 19 deletions src/raglite/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import sqlite3
from dataclasses import dataclass
from pathlib import Path
from types import ModuleType
from typing import Iterator, List, Sequence, Tuple

from . import chunk as chunk_utils
Expand All @@ -15,26 +16,35 @@
from .embed import get_embedding_store

try:
from readability import Document
import readability as _readability
except Exception: # pragma: no cover - optional dependency
Document = None
readability: ModuleType | None = None
else:
readability = _readability

try:
from bs4 import BeautifulSoup
import bs4 as _bs4
except Exception: # pragma: no cover - optional dependency
BeautifulSoup = None
bs4: ModuleType | None = None
else:
bs4 = _bs4

try:
from pypdf import PdfReader
import pypdf as _pypdf
except Exception: # pragma: no cover - optional dependency
PdfReader = None
pypdf: ModuleType | None = None
else:
pypdf = _pypdf

try: # optional OCR
import pytesseract
from PIL import Image
import pytesseract as _pytesseract
from PIL import Image as _Image
except Exception: # pragma: no cover - optional dependency
pytesseract = None
Image = None
pytesseract: ModuleType | None = None
Image: ModuleType | None = None
else:
pytesseract = _pytesseract
Image = _Image


@dataclass
Expand Down Expand Up @@ -80,26 +90,23 @@ def read_text_file(path: Path) -> str:

def read_html_file(path: Path) -> str:
html = path.read_text(encoding="utf-8", errors="ignore")
if Document is None or BeautifulSoup is None:
if readability is None or bs4 is None:
return html
doc = Document(html)
doc = readability.Document(html)
summary = doc.summary()
soup = BeautifulSoup(summary, "html.parser")
soup = bs4.BeautifulSoup(summary, "html.parser")
return soup.get_text(" ")


def read_pdf_file(path: Path, *, ocr: bool = False) -> str:
if PdfReader is None:
if pypdf is None:
raise UnsupportedDocument("pypdf is required to read PDF files")
reader = PdfReader(str(path))
reader = pypdf.PdfReader(str(path))
texts: List[str] = []
for page in reader.pages:
text = page.extract_text() or ""
if not text.strip() and ocr and pytesseract and Image:
try:
images = page.images
except AttributeError: # pragma: no cover
images = []
images = list(getattr(page, "images", []))
for image in images:
try:
img = Image.open(image.data)
Expand Down