Knowable

Answers point-in-time monetary-policy questions from a frozen subset of Fed/BLS/BEA corpus. An LLM writes a small retrieval program; the server validates it against an AST allowlist and executes it through SQLite primitives that enforce available_at <= as_of in SQL — so no plan, however bad, can cite evidence from after the question's cutoff.

Quickstart

Needs Python 3.12+ only — no dependencies, no install. The corpus (evidence.sqlite) and built UI ship in the repo.

git clone https://github.com/jevers9788/knowable.git && cd knowable
cp .env.example .env          # add PERPLEXITY_API_KEY for real answers
source .env
python3 -m knowable.server    # open http://127.0.0.1:8000

Without a key: plan generation can't run, so requests return a labeled fallback instead of local answers. Node/npm are needed only for UI tests or rebuilding (just test-ui, just build-ui); just is only used for the shortcuts in justfile.

How it works

1. The planner LLM sees the primitive contract and corpus manifest, and emits straight-line calls — add_latest, add_between, add_passages, add_facts, gap, trend — ending in one answer_local("rationale") or route("live"|"clarify", reason).
2. The program is ast-parsed against an allowlist (no variables, loops, attributes, imports, operators, builtins) and dispatched by walking the tree; model text is never executed as Python.
3. Every primitive caps time boundaries at as_of in SQL. A packet-integrity validator checks evidence and references before a local answer is accepted. A rejected plan is regenerated once, then routed to live search — never passed off as a local answer.
4. Synthesis sees only the accepted evidence packet; claims must reference real evidence IDs. Live requests use Perplexity pro-search, stay in a separate live_evidence ledger, and carry live_cutoff_guarantee: false.

All model calls (planning, synthesis, live) go through the Perplexity Agent API with one key; the default model is google/gemini-3-flash-preview on both sides of the baseline comparison.

Tests and evals

just test                                 # python unit tests + eval self-test + ui tests
just check-corpus                         # corpus integrity
just eval-retrieval                       # keyed server required; writes results/*.jsonl
python3 -m knowable.eval score results/<run.jsonl> --cases evals/eval_cases.json

Three question sets: evals/eval_cases.json (12 original v1 regression cases), evals/eval_development.json (30 cases: paraphrases, novel questions, off-ramps, adversarial false-local traps — used freely during development), evals/eval_holdout.json (13 cases written after the code/prompt freeze, run once).

Freeze timeline: code and prompt frozen 13:21 EDT July 20, 2026; holdout written 13:22, run once 13:23; commit 28cec6a records that state. The commit prevents later tuning from hiding, but as a post-holdout commit it is not retroactive proof of the freeze.

Retrieval results (zero temporal leaks on all local evidence)

Set	Runs	Document recall	Fact accuracy	Relationship accuracy	Routing
v1 regression (×3)	33	97.6%	100%	93.3%	100%
Development (×3)	87	96.8%	97.1%	91.7%	98.9%
Sealed holdout (one shot)	12	100%	100%	100%	91.7%

The v1 system (keyword routing, hand-built retrieval) scored 56.5% / 17.4% / 16.7% / 66.7% on the development set — its 100% on its own 12 cases was overfitting, which this redesign replaced. Planner overhead: ~1.8–2.3 s and ~$0.0014 per question, 99–100% valid programs, zero invalid ASTs and integrity failures. Plans are not deterministic run-to-run (stability ≤ 0.03); execution of any accepted plan is.

End to end

Set	Local answers	Documents	Facts	Relationships	Claim refs	Median latency	Median cost
v1 regression	8 of 9	92.3%	100%	100%	100%	7.710 s	$0.00630
Development	18 of 21	94.2%	100%	88.9%	100%	7.101 s	$0.00577
Holdout (round 2)	8 of 9	100%	100%	100%	100%	5.552 s	$0.00368

Versus the saved Perplexity pro-search baseline on the shared regression questions (same synthesis model):

Metric	Perplexity baseline (9)	Knowable (8 of 9 local)
Median / p95 latency	28.4 s / 36.7 s	7.7 s / 9.3 s
Median cost	$0.02825	$0.00630
Required-document recall	32.1%	92.3%
Numeric / relationship accuracy	70% / 80%	100% / 100%
Post-cutoff evidence	34 / 193	0 / 134

A holdout-set baseline run showed the gap widening on unseen questions: 18.2% document recall and 15/163 post-cutoff citations for the baseline (its fact accuracy could not be auto-graded — see future work). This is a narrow fast-path comparison over a frozen corpus, not a claim of general-search superiority.

Technical limits

• Named misses: april-lookahead misrouted to clarify (regression e2e); holdout-inflation-expectations falsely answered locally — caught downstream by claim-reference integrity in round 2.
• Development runs: one plan execution failure (correctly fell back to live), three invalid-JSON synthesis responses, one live result citing a post-cutoff page (the live route has no cutoff guarantee, by design).
• Live routes cost ~20–42 s and ~$0.02 — the fallback is real but not fast.
• Post-freeze changes, disclosed: (1) synthesis gained a salvage parse and one retry for malformed JSON; (2) on July 21 provider-side drift on the preview model broke minimal-effort planning (0/6 correct on a smoke question, with the model ignoring the input); planner effort is now low (6/6, no measurable latency cost). All recorded runs predate both changes; no retrieval, routing, or scoring logic changed.
• Independence caveats: the holdout was authored by the same session that wrote the planner prompt; roughly half its cases re-test development question families in new words (novel families also scored 100%); expected relationship IDs mirror server-side naming; the planner prompt's rules contain development-set vocabulary. Part of the measured agreement is inherited structure, not pure generalization.
• The scorer separates deterministic prechecks from generated plans and omits route mismatches from its failures array; rates above include them explicitly. Claim checks validate references, not semantic entailment.

Not built / future work

• Multiturn. Designed: prior turns feed planner/synthesis as context, follow-ups inherit the thread's as_of, every follow-up still compiles to a fresh cutoff-bound plan. Blocked on multiturn eval cases — shipping it unevaluated would undercut the protocol.
• Perplexity baseline on the development set (generalization comparison on paraphrases).
• Fact-level grading of the baseline holdout run — holdout facts lack answer_pattern regexes, so prose can't be auto-graded; grade manually or amend patterns with disclosure.
• Smaller: token-by-token streaming, semantic search (only if a measured FTS miss appears), corpus auto-refresh, UI persistence and real auth (both stubs), live-source date verification.