From bdfb2a29c30f5c673330f75ea9a64817f80b5084 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Thu, 2 Jul 2026 17:17:51 +1000 Subject: [PATCH 01/38] feat(v2): scaffold the V2 information architecture (CIP-3325) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Foundation for the docs overhaul tracked in CIP-3307: - IA.md: living migration checklist at the repo root (one checkbox per planned page; sections tick off as they land on this branch) - New `v2docs` collection (content/docs) served from the site root via a required catch-all route, alongside the legacy tree (content/stack) at /stack until every section migrates - Frontmatter facet model: Diátaxis `type`, `components`, `audience`, `integration` (category / setup / pairsWith), and review-tracking fields (`verifiedAgainst`, `reviewBy`) - Section scaffold: get-started, integrations (incl. the /integrations/supabase stub the Supabase listing links to), concepts, compare, guides, security, solutions, reference — meta.json + stubs - v2-redirects.mjs: full legacy→v2 map (85/85 pages covered), gated behind ENABLE_V2_REDIRECTS=1 so the preview serves both trees during migration; /quickstart vanity redirect ships ungated - scripts/validate-v2-redirects.ts wired into prebuild: CI fails if a legacy page has no v2 mapping - llms.txt, llms-full.txt, sitemap, and the .mdx raw-markdown mirror now cover both trees Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- IA.md | 173 ++++++++ content/docs/compare/index.mdx | 9 + content/docs/compare/meta.json | 5 + content/docs/concepts/index.mdx | 9 + content/docs/concepts/meta.json | 5 + content/docs/get-started/index.mdx | 9 + content/docs/get-started/meta.json | 5 + content/docs/guides/deployment/index.mdx | 8 + content/docs/guides/deployment/meta.json | 4 + content/docs/guides/development/index.mdx | 8 + content/docs/guides/development/meta.json | 4 + content/docs/guides/index.mdx | 9 + content/docs/guides/meta.json | 5 + content/docs/guides/migration/index.mdx | 8 + content/docs/guides/migration/meta.json | 4 + content/docs/guides/troubleshooting/index.mdx | 8 + content/docs/guides/troubleshooting/meta.json | 4 + content/docs/integrations/index.mdx | 9 + content/docs/integrations/meta.json | 5 + content/docs/integrations/supabase/index.mdx | 20 + content/docs/integrations/supabase/meta.json | 5 + content/docs/meta.json | 12 + content/docs/reference/auth/index.mdx | 8 + content/docs/reference/auth/meta.json | 4 + content/docs/reference/cli/index.mdx | 8 + content/docs/reference/cli/meta.json | 4 + content/docs/reference/eql/index.mdx | 8 + content/docs/reference/eql/meta.json | 4 + content/docs/reference/index.mdx | 9 + content/docs/reference/meta.json | 5 + content/docs/reference/proxy/index.mdx | 8 + content/docs/reference/proxy/meta.json | 4 + content/docs/reference/stack/index.mdx | 8 + content/docs/reference/stack/meta.json | 4 + content/docs/reference/workspace/index.mdx | 8 + content/docs/reference/workspace/meta.json | 4 + content/docs/security/compliance/index.mdx | 8 + content/docs/security/compliance/meta.json | 4 + content/docs/security/index.mdx | 9 + content/docs/security/meta.json | 5 + content/docs/solutions/index.mdx | 9 + content/docs/solutions/meta.json | 5 + next.config.mjs | 23 ++ package.json | 5 +- scripts/validate-v2-redirects.ts | 61 +++ source.config.ts | 48 +++ src/app/(home)/page.tsx | 2 +- src/app/[...slug]/layout.tsx | 15 + src/app/[...slug]/page.tsx | 76 ++++ src/app/api/search/route.ts | 2 +- src/app/layout.tsx | 2 +- src/app/llms-full.txt/route.ts | 4 +- src/app/llms.mdx/v2/[...slug]/route.ts | 44 ++ src/app/llms.txt/route.ts | 5 +- src/app/og/docs/[...slug]/route.tsx | 4 +- src/app/sitemap.ts | 4 +- src/app/stack/[[...slug]]/page.tsx | 8 +- src/app/stack/layout.tsx | 2 +- src/components/icons/supabase.tsx | 2 +- src/lib/posthog/provider.tsx | 6 +- src/lib/source.ts | 17 +- src/proxy.ts | 2 +- v2-redirects.mjs | 375 ++++++++++++++++++ 63 files changed, 1135 insertions(+), 26 deletions(-) create mode 100644 IA.md create mode 100644 content/docs/compare/index.mdx create mode 100644 content/docs/compare/meta.json create mode 100644 content/docs/concepts/index.mdx create mode 100644 content/docs/concepts/meta.json create mode 100644 content/docs/get-started/index.mdx create mode 100644 content/docs/get-started/meta.json create mode 100644 content/docs/guides/deployment/index.mdx create mode 100644 content/docs/guides/deployment/meta.json create mode 100644 content/docs/guides/development/index.mdx create mode 100644 content/docs/guides/development/meta.json create mode 100644 content/docs/guides/index.mdx create mode 100644 content/docs/guides/meta.json create mode 100644 content/docs/guides/migration/index.mdx create mode 100644 content/docs/guides/migration/meta.json create mode 100644 content/docs/guides/troubleshooting/index.mdx create mode 100644 content/docs/guides/troubleshooting/meta.json create mode 100644 content/docs/integrations/index.mdx create mode 100644 content/docs/integrations/meta.json create mode 100644 content/docs/integrations/supabase/index.mdx create mode 100644 content/docs/integrations/supabase/meta.json create mode 100644 content/docs/meta.json create mode 100644 content/docs/reference/auth/index.mdx create mode 100644 content/docs/reference/auth/meta.json create mode 100644 content/docs/reference/cli/index.mdx create mode 100644 content/docs/reference/cli/meta.json create mode 100644 content/docs/reference/eql/index.mdx create mode 100644 content/docs/reference/eql/meta.json create mode 100644 content/docs/reference/index.mdx create mode 100644 content/docs/reference/meta.json create mode 100644 content/docs/reference/proxy/index.mdx create mode 100644 content/docs/reference/proxy/meta.json create mode 100644 content/docs/reference/stack/index.mdx create mode 100644 content/docs/reference/stack/meta.json create mode 100644 content/docs/reference/workspace/index.mdx create mode 100644 content/docs/reference/workspace/meta.json create mode 100644 content/docs/security/compliance/index.mdx create mode 100644 content/docs/security/compliance/meta.json create mode 100644 content/docs/security/index.mdx create mode 100644 content/docs/security/meta.json create mode 100644 content/docs/solutions/index.mdx create mode 100644 content/docs/solutions/meta.json create mode 100644 scripts/validate-v2-redirects.ts create mode 100644 src/app/[...slug]/layout.tsx create mode 100644 src/app/[...slug]/page.tsx create mode 100644 src/app/llms.mdx/v2/[...slug]/route.ts create mode 100644 v2-redirects.mjs diff --git a/IA.md b/IA.md new file mode 100644 index 0000000..4c94f3d --- /dev/null +++ b/IA.md @@ -0,0 +1,173 @@ +# Docs V2 — Information Architecture & migration checklist + +Living checklist for the docs overhaul. Tracked in Linear under +[CIP-3307](https://linear.app/cipherstash/issue/CIP-3307); the full IA rationale +(design principles, audience doors, correctness strategy) lives in +`CipherStash docs IA v1.md` in the content repo. **Tick items here as they land +on the `v2` branch.** Legend: `[ ]` todo · `[x]` done · 🚧 stub exists · ⛔ blocked +on a product decision (see CIP-3307 checklist). + +## How this branch works + +- New IA lives in `content/docs`, served from the site root (`/docs/
/…`). +- The legacy tree (`content/stack`) is served alongside it at `/docs/stack/…` + until every section migrates, then deleted (CIP-3335). +- The full legacy→v2 redirect map is `v2-redirects.mjs`, gated behind + `ENABLE_V2_REDIRECTS=1` (flipped on at merge). `bun run validate-redirects` + enforces that every legacy page has a mapping. +- Frontmatter facets (`type`, `components`, `audience`, `integration`, + `verifiedAgainst`, `reviewBy`) are defined in `source.config.ts` (`v2docs`). +- **Moving a page = ** move the file into `content/docs`, update its facets, + fix inbound links, confirm its `v2-redirects.mjs` entry, tick it here. + +## URL conventions + +Lowercase, hyphens, no trailing slashes, no version numbers in paths. +Integrations are **flat** (no category segment). Error pages (future, miette) +live at `/docs/errors/` — permanent, never restructured (CIP-3338). + +--- + +## Get started — CIP-3327 + +- [x] Section scaffold 🚧 +- [ ] `/get-started/what-is-cipherstash` — mental model, components map, audience router +- [ ] `/get-started/quickstart` — rewritten on EQL v3 (fixes `cs_match_v1`, broken scaffold imports) +- [ ] `/get-started/choose-your-stack` — static matrix v1 (platform × ORM × auth) +- [ ] `/get-started/examples` — runnable example apps index +- [ ] `/docs` landing page (replaces/updates `(home)` route: what-is + audience router) + +## Integrations — CIP-3328 (Supabase), CIP-3330 (auth), CIP-3336 (rest) + +- [x] Section scaffold 🚧 (index + supabase stub with facet exemplar) +- [ ] `/integrations` index — category grid w/ setup badges +- [ ] `/integrations/supabase` — flagship tutorial (CIP-3328) +- [ ] `/integrations/supabase/database` +- [ ] `/integrations/supabase/auth` +- [ ] `/integrations/supabase/dashboard-experience` — Table Editor, expose eql schema +- [ ] ⛔ `/integrations/supabase/edge-functions` — pending Deno/FFI answer +- [ ] ⛔ `/integrations/supabase/realtime` — pending product verification +- [ ] `/integrations/drizzle` — merge the two divergent Drizzle pages +- [ ] `/integrations/prisma-next` +- [ ] `/integrations/aws/rds-aurora` — Proxy path +- [ ] `/integrations/aws/dynamodb` +- [ ] `/integrations/clerk` +- [ ] `/integrations/auth0` — end-to-end example (Clerk parity) +- [ ] `/integrations/okta` — end-to-end example (Clerk parity) +- [ ] `/integrations/nextjs` +- [ ] `/integrations/typescript` — thin router to Stack SDK reference +- [ ] `/integrations/serverless` — Vercel/Lambda, bundling, CS_CONFIG_PATH +- [ ] `/integrations/docker` +- [ ] ⛔ `/integrations/edge-workers` — pending Deno/workerd answer + +## Concepts — CIP-3333 (searchable-encryption), others per section tickets + +- [x] Section scaffold 🚧 +- [ ] `/concepts/privacy-first-design` +- [ ] `/concepts/application-level-encryption` — vs TDE/pgcrypto/RLS +- [ ] `/concepts/searchable-encryption` — REWRITE with honest leakage model (canonical leakage page) +- [ ] `/concepts/eql` — the typed-column model (declare capability in the schema) +- [ ] `/concepts/key-management` — per-value keys, rotation, crypto-shredding +- [ ] `/concepts/identity-aware-encryption` — lock contexts, CTS (CIP-3330) +- [ ] `/concepts/threat-modelling` + +## Comparisons — CIP-3333 + +- [x] Section scaffold 🚧 +- [ ] `/compare/aws-kms` (port) +- [ ] `/compare/fhe` (port) +- [ ] `/compare/rls-and-tde` (new — expand the Supabase-listing RLS contrast) +- [ ] `/compare/hashicorp-vault` (in flight on `docs/vault-comparison` branch — land there or here, then port) + +## Guides + +- [x] Section scaffold 🚧 (development, migration, deployment, troubleshooting) +- [ ] `/guides/development/local-setup` — profiles, device auth, workspaces, keys +- [ ] `/guides/development/schema-design` — which encrypted type/variant per column (CIP-3327) +- [ ] `/guides/development/testing-and-ci` (port deploy/testing) +- [ ] `/guides/development/team-onboarding` (port) +- [ ] `/guides/migration/encrypt-existing-data` — the backfill guide, runnable (CIP-3329) +- [ ] ⛔ `/guides/migration/upgrading-from-eql-v2` — REQUIRED; mechanics pending product answer (CIP-3329) +- [ ] `/guides/migration/adopting-incrementally` (CIP-3329) +- [ ] `/guides/migration/key-rotation-operations` +- [ ] `/guides/deployment/going-to-production` (port) +- [ ] `/guides/deployment/serverless-and-bundling` (merge bundling + sst) +- [ ] `/guides/deployment/proxy-deployment` (merge proxy Docker + aws-ecs) +- [ ] `/guides/troubleshooting` index — symptom-based router +- [ ] `/guides/troubleshooting/query-performance` — seq-scan diagnosis, typed-operand gotcha +- [ ] `/guides/troubleshooting/runtime-errors` +- [ ] `/guides/troubleshooting/cli` (port) +- [ ] `/guides/troubleshooting/proxy` (port) + +## Architecture & security — CIP-3331, CIP-3332 (compliance) + +- [x] Section scaffold 🚧 +- [ ] `/security/architecture` — ONE reconciled ZeroKMS mechanism story (kills the 3 conflicting accounts) +- [ ] `/security/zerokms` +- [ ] `/security/cts` — auth layer architecture (CIP-3330) +- [ ] `/security/stack-sdk` +- [ ] `/security/proxy` +- [ ] `/security/threat-scenarios` +- [ ] ⛔ `/security/availability-and-continuity` — DR (port) + SLA + exit story; pending SLA answer +- [ ] ⛔ `/security/audit-logging` — pending retention answer +- [ ] ⛔ `/security/key-ownership` — BYOK/self-hosted; pending product answer +- [ ] `/security/compliance` index — framework mapping (port, good) +- [ ] `/security/compliance/hipaa` — BAA scope, §164.312 mapping (CIP-3332) +- [ ] `/security/compliance/soc2` — verify Type II report exists +- [ ] `/security/compliance/gdpr` + +## Solutions + +- [x] Section scaffold 🚧 +- [ ] `/solutions/protecting-pii` (new) +- [ ] `/solutions/healthcare-hipaa` (new; pairs with compliance/hipaa) +- [ ] `/solutions/ai-and-rag` (port use-cases/ai-rag) +- [ ] `/solutions/data-residency` (port) +- [ ] `/solutions/provable-access` (port) + +## Reference + +- [x] Section scaffold 🚧 (eql, stack, auth, cli, proxy, workspace) +- **EQL (v3 rewrite — CIP-3326):** +- [ ] `/reference/eql` — overview + install (single SQL file, permissions split, dbdev, Docker) +- [ ] `/reference/eql/types` — 10 scalar families × variants + `eql_v3.json` +- [ ] `/reference/eql/operators` — per-variant matrix incl. what RAISES; typed-operand rule +- [ ] `/reference/eql/indexes` — functional indexes on extractors; Supabase-compatible +- [ ] `/reference/eql/json` — ste_vec, path queries +- [ ] `/reference/eql/functions` — incl. aggregates (min/max only) +- [ ] `/reference/eql/payload-format` — v/i/c envelope, hm/ob/bf (absorbs cipher-cell) +- **Stack SDK:** +- [ ] `/reference/stack` — client + configuration (port encryption/* pages) +- [ ] `/reference/stack/schema` +- [ ] `/reference/stack/encrypt-decrypt` (+ bulk, models) +- [ ] `/reference/stack/supabase` — THE canonical `encryptedSupabase` page, ONE signature (CIP-3328) +- [ ] `/reference/stack/drizzle-operators` +- [ ] `/reference/stack/errors` — port error-handling; miette catalog later (CIP-3338) +- [ ] `/reference/stack/upgrading-from-protect` (retitled package-rename guide) +- **Auth (CIP-3330):** +- [ ] `/reference/auth/lock-contexts` +- [ ] `/reference/auth/cts-tokens` +- [ ] `/reference/auth/oidc-configuration` +- [ ] `/reference/auth/access-keys` (+ clients) +- **CLI / Proxy / Workspace:** +- [ ] `/reference/cli/*` (port 9 pages) +- [ ] `/reference/proxy/*` (configuration, message-flow, multitenant, errors) +- [ ] `/reference/workspace/billing` + `/members` + `/configuration` +- **Cross-cutting:** +- [ ] `/reference/benchmarks` — listing numbers + methodology (CIP-3334) +- [ ] `/reference/agent-skills` (port; expand per CIP-3339) +- [ ] `/reference/glossary` (port) +- [ ] Repoint `scripts/generate-docs.ts` TypeDoc output → `content/docs/reference/stack` + +## Infrastructure / final pass + +- [x] `v2` branch + this checklist +- [x] `v2docs` collection + facet schema (`source.config.ts`) +- [x] Root catch-all routes (`src/app/[...slug]`), llms.mdx mirror, sitemap/llms.txt include v2 +- [x] `v2-redirects.mjs` (flag-gated) + `validate-redirects` gate in prebuild +- [x] `/quickstart` vanity redirect +- [ ] OG images for v2 pages (route only covers legacy tree) +- [ ] Correctness CI: snippet type-checking, SQL-vs-EQL-Docker, terminology lint (CIP-3337) +- [ ] llms.txt curation + Cloudflare AI crawl policy + md-degradation check (CIP-3339) +- [ ] Flip `ENABLE_V2_REDIRECTS=1`, delete `content/stack` + `/stack` routes + legacy loader (CIP-3335) +- [ ] Consistency sweep + Supabase listing v3 revision (CIP-3335) diff --git a/content/docs/compare/index.mdx b/content/docs/compare/index.mdx new file mode 100644 index 0000000..0f07e95 --- /dev/null +++ b/content/docs/compare/index.mdx @@ -0,0 +1,9 @@ +--- +title: Comparisons +description: How CipherStash compares to other approaches to protecting data. +type: concept +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/compare/meta.json b/content/docs/compare/meta.json new file mode 100644 index 0000000..3e374e5 --- /dev/null +++ b/content/docs/compare/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Comparisons", + "icon": "Scale", + "pages": ["index", "..."] +} diff --git a/content/docs/concepts/index.mdx b/content/docs/concepts/index.mdx new file mode 100644 index 0000000..282c15b --- /dev/null +++ b/content/docs/concepts/index.mdx @@ -0,0 +1,9 @@ +--- +title: Concepts +description: How CipherStash works and how to think about searchable encryption, keys, and identity. +type: concept +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/concepts/meta.json b/content/docs/concepts/meta.json new file mode 100644 index 0000000..ca4b08b --- /dev/null +++ b/content/docs/concepts/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Concepts", + "icon": "Lightbulb", + "pages": ["index", "..."] +} diff --git a/content/docs/get-started/index.mdx b/content/docs/get-started/index.mdx new file mode 100644 index 0000000..af235c6 --- /dev/null +++ b/content/docs/get-started/index.mdx @@ -0,0 +1,9 @@ +--- +title: Get started +description: What CipherStash is, a 10-minute quickstart, and how to choose your integration path. +type: tutorial +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/get-started/meta.json b/content/docs/get-started/meta.json new file mode 100644 index 0000000..2d9cfb8 --- /dev/null +++ b/content/docs/get-started/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Get started", + "icon": "Rocket", + "pages": ["index", "..."] +} diff --git a/content/docs/guides/deployment/index.mdx b/content/docs/guides/deployment/index.mdx new file mode 100644 index 0000000..771a593 --- /dev/null +++ b/content/docs/guides/deployment/index.mdx @@ -0,0 +1,8 @@ +--- +title: Deployment +description: Deployment documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/guides/deployment/meta.json b/content/docs/guides/deployment/meta.json new file mode 100644 index 0000000..0bbd244 --- /dev/null +++ b/content/docs/guides/deployment/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Deployment", + "pages": ["index", "..."] +} diff --git a/content/docs/guides/development/index.mdx b/content/docs/guides/development/index.mdx new file mode 100644 index 0000000..c0d4ab1 --- /dev/null +++ b/content/docs/guides/development/index.mdx @@ -0,0 +1,8 @@ +--- +title: Development +description: Development documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/guides/development/meta.json b/content/docs/guides/development/meta.json new file mode 100644 index 0000000..24f8878 --- /dev/null +++ b/content/docs/guides/development/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Development", + "pages": ["index", "..."] +} diff --git a/content/docs/guides/index.mdx b/content/docs/guides/index.mdx new file mode 100644 index 0000000..f034020 --- /dev/null +++ b/content/docs/guides/index.mdx @@ -0,0 +1,9 @@ +--- +title: Guides +description: Task-oriented guides: development workflow, data migration, deployment, and troubleshooting. +type: guide +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/guides/meta.json b/content/docs/guides/meta.json new file mode 100644 index 0000000..810cf95 --- /dev/null +++ b/content/docs/guides/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Guides", + "icon": "Wrench", + "pages": ["index", "..."] +} diff --git a/content/docs/guides/migration/index.mdx b/content/docs/guides/migration/index.mdx new file mode 100644 index 0000000..690a307 --- /dev/null +++ b/content/docs/guides/migration/index.mdx @@ -0,0 +1,8 @@ +--- +title: Data migration +description: Data migration documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/guides/migration/meta.json b/content/docs/guides/migration/meta.json new file mode 100644 index 0000000..db4e68b --- /dev/null +++ b/content/docs/guides/migration/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Data migration", + "pages": ["index", "..."] +} diff --git a/content/docs/guides/troubleshooting/index.mdx b/content/docs/guides/troubleshooting/index.mdx new file mode 100644 index 0000000..f1e926e --- /dev/null +++ b/content/docs/guides/troubleshooting/index.mdx @@ -0,0 +1,8 @@ +--- +title: Troubleshooting +description: Troubleshooting documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/guides/troubleshooting/meta.json b/content/docs/guides/troubleshooting/meta.json new file mode 100644 index 0000000..e5715ad --- /dev/null +++ b/content/docs/guides/troubleshooting/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Troubleshooting", + "pages": ["index", "..."] +} diff --git a/content/docs/integrations/index.mdx b/content/docs/integrations/index.mdx new file mode 100644 index 0000000..dfcf78e --- /dev/null +++ b/content/docs/integrations/index.mdx @@ -0,0 +1,9 @@ +--- +title: Integrations +description: Set up CipherStash with your platform, ORM, framework, auth provider, and runtime. +type: tutorial +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/integrations/meta.json b/content/docs/integrations/meta.json new file mode 100644 index 0000000..62604e2 --- /dev/null +++ b/content/docs/integrations/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Integrations", + "icon": "Blocks", + "pages": ["index", "..."] +} diff --git a/content/docs/integrations/supabase/index.mdx b/content/docs/integrations/supabase/index.mdx new file mode 100644 index 0000000..6066937 --- /dev/null +++ b/content/docs/integrations/supabase/index.mdx @@ -0,0 +1,20 @@ +--- +title: Supabase +description: Searchable, application-level encryption for your Supabase project — encrypt in your app, query in Postgres. +type: tutorial +components: [encryption, eql, auth] +audience: [developer] +integration: + category: platform + setup: dashboard-required + pairsWith: [drizzle, prisma-next, clerk, nextjs] +--- + +CipherStash adds application-level encryption to your Supabase project: +sensitive fields are encrypted in your application before they reach Postgres, +and stay queryable with the same Supabase.js calls you already use. + +This page is being rebuilt as part of the docs V2 overhaul +([CIP-3328](https://linear.app/cipherstash/issue/CIP-3328)). Until it lands, +the current Supabase integration guide lives at +[CipherStash + Supabase](/docs/stack/cipherstash/supabase). diff --git a/content/docs/integrations/supabase/meta.json b/content/docs/integrations/supabase/meta.json new file mode 100644 index 0000000..a61c668 --- /dev/null +++ b/content/docs/integrations/supabase/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Supabase", + "icon": "Supabase", + "pages": ["index", "..."] +} diff --git a/content/docs/meta.json b/content/docs/meta.json new file mode 100644 index 0000000..8935d68 --- /dev/null +++ b/content/docs/meta.json @@ -0,0 +1,12 @@ +{ + "pages": [ + "get-started", + "integrations", + "concepts", + "compare", + "guides", + "security", + "solutions", + "reference" + ] +} diff --git a/content/docs/reference/auth/index.mdx b/content/docs/reference/auth/index.mdx new file mode 100644 index 0000000..2dea44b --- /dev/null +++ b/content/docs/reference/auth/index.mdx @@ -0,0 +1,8 @@ +--- +title: Auth +description: Auth documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/reference/auth/meta.json b/content/docs/reference/auth/meta.json new file mode 100644 index 0000000..943c560 --- /dev/null +++ b/content/docs/reference/auth/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Auth", + "pages": ["index", "..."] +} diff --git a/content/docs/reference/cli/index.mdx b/content/docs/reference/cli/index.mdx new file mode 100644 index 0000000..90d8cb3 --- /dev/null +++ b/content/docs/reference/cli/index.mdx @@ -0,0 +1,8 @@ +--- +title: CLI +description: CLI documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/reference/cli/meta.json b/content/docs/reference/cli/meta.json new file mode 100644 index 0000000..075b6da --- /dev/null +++ b/content/docs/reference/cli/meta.json @@ -0,0 +1,4 @@ +{ + "title": "CLI", + "pages": ["index", "..."] +} diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx new file mode 100644 index 0000000..e312de2 --- /dev/null +++ b/content/docs/reference/eql/index.mdx @@ -0,0 +1,8 @@ +--- +title: EQL +description: EQL documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/reference/eql/meta.json b/content/docs/reference/eql/meta.json new file mode 100644 index 0000000..be23fb4 --- /dev/null +++ b/content/docs/reference/eql/meta.json @@ -0,0 +1,4 @@ +{ + "title": "EQL", + "pages": ["index", "..."] +} diff --git a/content/docs/reference/index.mdx b/content/docs/reference/index.mdx new file mode 100644 index 0000000..fc2c7e7 --- /dev/null +++ b/content/docs/reference/index.mdx @@ -0,0 +1,9 @@ +--- +title: Reference +description: Precise API documentation for EQL, the Stack SDK, Auth, the CLI, and Proxy. +type: reference +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/reference/meta.json b/content/docs/reference/meta.json new file mode 100644 index 0000000..38c38b5 --- /dev/null +++ b/content/docs/reference/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Reference", + "icon": "Library", + "pages": ["index", "..."] +} diff --git a/content/docs/reference/proxy/index.mdx b/content/docs/reference/proxy/index.mdx new file mode 100644 index 0000000..961c67b --- /dev/null +++ b/content/docs/reference/proxy/index.mdx @@ -0,0 +1,8 @@ +--- +title: Proxy +description: Proxy documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/reference/proxy/meta.json b/content/docs/reference/proxy/meta.json new file mode 100644 index 0000000..a64115c --- /dev/null +++ b/content/docs/reference/proxy/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Proxy", + "pages": ["index", "..."] +} diff --git a/content/docs/reference/stack/index.mdx b/content/docs/reference/stack/index.mdx new file mode 100644 index 0000000..993c909 --- /dev/null +++ b/content/docs/reference/stack/index.mdx @@ -0,0 +1,8 @@ +--- +title: Stack SDK +description: Stack SDK documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/reference/stack/meta.json b/content/docs/reference/stack/meta.json new file mode 100644 index 0000000..3d44235 --- /dev/null +++ b/content/docs/reference/stack/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Stack SDK", + "pages": ["index", "..."] +} diff --git a/content/docs/reference/workspace/index.mdx b/content/docs/reference/workspace/index.mdx new file mode 100644 index 0000000..9bebe76 --- /dev/null +++ b/content/docs/reference/workspace/index.mdx @@ -0,0 +1,8 @@ +--- +title: Workspace & account +description: Workspace & account documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/reference/workspace/meta.json b/content/docs/reference/workspace/meta.json new file mode 100644 index 0000000..2c03700 --- /dev/null +++ b/content/docs/reference/workspace/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Workspace & account", + "pages": ["index", "..."] +} diff --git a/content/docs/security/compliance/index.mdx b/content/docs/security/compliance/index.mdx new file mode 100644 index 0000000..4a25ab6 --- /dev/null +++ b/content/docs/security/compliance/index.mdx @@ -0,0 +1,8 @@ +--- +title: Compliance +description: Compliance documentation — being built as part of the docs V2 overhaul. +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/security/compliance/meta.json b/content/docs/security/compliance/meta.json new file mode 100644 index 0000000..346ed94 --- /dev/null +++ b/content/docs/security/compliance/meta.json @@ -0,0 +1,4 @@ +{ + "title": "Compliance", + "pages": ["index", "..."] +} diff --git a/content/docs/security/index.mdx b/content/docs/security/index.mdx new file mode 100644 index 0000000..6bb327d --- /dev/null +++ b/content/docs/security/index.mdx @@ -0,0 +1,9 @@ +--- +title: Architecture & security +description: Trust model, components, availability, audit, and compliance — self-contained for security review. +type: concept +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/security/meta.json b/content/docs/security/meta.json new file mode 100644 index 0000000..7a3eb64 --- /dev/null +++ b/content/docs/security/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Architecture & security", + "icon": "Shield", + "pages": ["index", "..."] +} diff --git a/content/docs/solutions/index.mdx b/content/docs/solutions/index.mdx new file mode 100644 index 0000000..5189ccf --- /dev/null +++ b/content/docs/solutions/index.mdx @@ -0,0 +1,9 @@ +--- +title: Solutions +description: What CipherStash solves: PII protection, HIPAA, AI/RAG, data residency, provable access. +type: concept +--- + +This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, current documentation lives in the [existing docs](/docs/stack). diff --git a/content/docs/solutions/meta.json b/content/docs/solutions/meta.json new file mode 100644 index 0000000..6b5983d --- /dev/null +++ b/content/docs/solutions/meta.json @@ -0,0 +1,5 @@ +{ + "title": "Solutions", + "icon": "Target", + "pages": ["index", "..."] +} diff --git a/next.config.mjs b/next.config.mjs index 5825239..c6a0f69 100644 --- a/next.config.mjs +++ b/next.config.mjs @@ -1,13 +1,29 @@ import { createMDX } from "fumadocs-mdx/next"; +import { v2Redirects } from "./v2-redirects.mjs"; const withMDX = createMDX(); +// V2 IA migration (CIP-3325): the full legacy→v2 redirect map is gated so the +// preview site serves BOTH trees while sections migrate (legacy at /stack, v2 +// at the root). Flip on at merge; once content/stack is deleted the map +// becomes unconditional (CIP-3335). Coverage is enforced by +// `bun run validate-redirects` regardless of the flag. +const enableV2Redirects = process.env.ENABLE_V2_REDIRECTS === "1"; + /** @type {import('next').NextConfig} */ const config = { basePath: "/docs", reactStrictMode: true, async redirects() { return [ + // Vanity URL for the new IA (safe to ship ungated: the path has no + // legacy traffic). Temporary until the v2 quickstart is canonical. + { + source: "/quickstart", + destination: "/get-started/quickstart", + permanent: false, + }, + ...(enableV2Redirects ? v2Redirects : []), // === 4-section consolidation: product sections under /cipherstash/ === { source: "/stack/encryption/:path*", @@ -317,6 +333,13 @@ const config = { source: "/stack/:path*.mdx", destination: "/llms.mdx/stack/:path*", }, + // Raw-markdown mirror for the v2 tree (Cloudflare/agents fetch + // .mdx). Listed after the /stack rule so legacy paths keep + // resolving to the legacy collection. + { + source: "/:path*.mdx", + destination: "/llms.mdx/v2/:path*", + }, ], afterFiles: [ { diff --git a/package.json b/package.json index 7cc2dfd..3b4681a 100644 --- a/package.json +++ b/package.json @@ -3,7 +3,7 @@ "version": "0.0.0", "private": true, "scripts": { - "prebuild": "bun run generate-docs && bun run generate-docs:eql && bun run validate-links", + "prebuild": "bun run generate-docs && bun run generate-docs:eql && bun run validate-links && bun run validate-redirects", "build": "next build", "dev": "next dev -p 3001", "start": "next start", @@ -13,7 +13,8 @@ "format": "biome format --write", "generate-docs": "tsx scripts/generate-docs.ts", "generate-docs:eql": "tsx scripts/generate-eql-docs.ts", - "validate-links": "tsx scripts/validate-links.ts" + "validate-links": "tsx scripts/validate-links.ts", + "validate-redirects": "tsx scripts/validate-v2-redirects.ts" }, "dependencies": { "fumadocs-core": "16.6.0", diff --git a/scripts/validate-v2-redirects.ts b/scripts/validate-v2-redirects.ts new file mode 100644 index 0000000..97c90d0 --- /dev/null +++ b/scripts/validate-v2-redirects.ts @@ -0,0 +1,61 @@ +#!/usr/bin/env tsx +/** + * V2 redirect gate (CIP-3325 / CIP-3337 item 7). + * + * Every page in the legacy tree (content/stack) must be covered by an entry + * in v2-redirects.mjs — exact match or `:path*` wildcard — so that no URL is + * orphaned when the v2 IA ships. Run via `bun run validate-redirects`; wired + * into prebuild so a page added to content/stack without a mapping fails CI. + * + * This checks map *coverage*, not destination existence — destinations are + * stubs until each section's migration ticket lands. CIP-3335 verifies + * destinations resolve before merge. + */ +import fs from "node:fs"; +import path from "node:path"; +// eslint-disable-next-line -- .mjs import is intentional; the map is shared with next.config.mjs +import { v2Redirects } from "../v2-redirects.mjs"; + +const LEGACY_DIR = path.join(process.cwd(), "content/stack"); + +function collectSlugs(dir: string, prefix: string[] = []): string[] { + const slugs: string[] = []; + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + if (entry.isDirectory()) { + slugs.push( + ...collectSlugs(path.join(dir, entry.name), [...prefix, entry.name]), + ); + } else if (entry.name.endsWith(".mdx") || entry.name.endsWith(".md")) { + const base = entry.name.replace(/\.mdx?$/, ""); + const parts = base === "index" ? prefix : [...prefix, base]; + slugs.push(`/stack${parts.length ? `/${parts.join("/")}` : ""}`); + } + } + return slugs; +} + +function matches(url: string, source: string): boolean { + if (source.endsWith("/:path*")) { + const base = source.slice(0, -"/:path*".length); + return url === base || url.startsWith(`${base}/`); + } + return url === source; +} + +const urls = collectSlugs(LEGACY_DIR); +const unmatched = urls.filter( + (url) => !v2Redirects.some((r: { source: string }) => matches(url, r.source)), +); + +if (unmatched.length > 0) { + console.error( + `✗ ${unmatched.length} legacy page(s) have no v2 redirect mapping:\n`, + ); + for (const url of unmatched.sort()) { + console.error(` ${url}`); + } + console.error("\nAdd entries to v2-redirects.mjs (see IA.md migration map)."); + process.exit(1); +} + +console.log(`✓ all ${urls.length} legacy pages covered by v2-redirects.mjs`); diff --git a/source.config.ts b/source.config.ts index 7769cf3..18ad3f5 100644 --- a/source.config.ts +++ b/source.config.ts @@ -23,6 +23,54 @@ export const docs = defineDocs({ }, }); +// V2 information architecture (CIP-3325). New content lives in content/docs +// and is served from the site root (e.g. /docs/get-started/...). The legacy +// `docs` collection above (content/stack) is served alongside it during the +// migration and is deleted once the last section moves. See IA.md. +export const v2docs = defineDocs({ + dir: "content/docs", + docs: { + schema: pageSchema.extend({ + seoTitle: z.string().optional(), + // Diátaxis page type. Every page should declare one; enforced by the + // docs lint (CIP-3337) rather than the schema so stubs can land first. + type: z.enum(["tutorial", "guide", "concept", "reference"]).optional(), + // Facets powering index pages, filtered views, and the future + // tailored-quickstart picker (CIP-3339). Nav position never depends on + // these — the sidebar tree comes from meta.json alone. + components: z + .array(z.enum(["encryption", "auth", "zerokms", "eql", "proxy", "cli"])) + .optional(), + audience: z.array(z.enum(["developer", "cto", "ciso"])).optional(), + integration: z + .object({ + category: z.enum([ + "platform", + "orm", + "framework", + "auth-provider", + "language", + "runtime", + ]), + setup: z.enum(["code-only", "dashboard-required"]), + pairsWith: z.array(z.string()).optional(), + }) + .optional(), + // Review tracking (CIP-3337): API pages pin the releases they were + // verified against (e.g. { stack: "1.2.0", eql: "3.0.0" }); claims pages + // (compliance, pricing, comparisons) carry a review-by date instead. + verifiedAgainst: z.record(z.string(), z.string()).optional(), + reviewBy: z.string().optional(), + }), + postprocess: { + includeProcessedMarkdown: true, + }, + }, + meta: { + schema: metaSchema, + }, +}); + // Parse the leftover code-fence meta string (what remains after Fumadocs // extracts `title`, `tab`, and line-number directives) for the analytics // attributes documented for authors: `example-id`, `cta`, and `cta-type`. diff --git a/src/app/(home)/page.tsx b/src/app/(home)/page.tsx index 5311eab..7cad1aa 100644 --- a/src/app/(home)/page.tsx +++ b/src/app/(home)/page.tsx @@ -11,6 +11,7 @@ import { ShieldCheck, Zap, } from "lucide-react"; +import type { Metadata } from "next"; import Link from "next/link"; import type { ComponentType } from "react"; import { @@ -19,7 +20,6 @@ import { PrismaLogo, SupabaseLogo, } from "@/components/integration-logos"; -import type { Metadata } from "next"; // The /docs landing page had no metadata (no ). `absolute` bypasses the // root layout's "%s | CipherStash Docs" template so the title isn't doubled. diff --git a/src/app/[...slug]/layout.tsx b/src/app/[...slug]/layout.tsx new file mode 100644 index 0000000..2045bab --- /dev/null +++ b/src/app/[...slug]/layout.tsx @@ -0,0 +1,15 @@ +import { DocsLayout } from "fumadocs-ui/layouts/docs"; +import { baseOptions } from "@/lib/layout.shared"; +import { v2source } from "@/lib/source"; + +// Layout for the V2 IA tree (content/docs), served from the site root. +// A *required* catch-all (`[...slug]`, not `[[...slug]]`) so it never +// competes with the (home) route for "/". Static routes (/stack, /api, +// /og, …) take precedence over this segment as usual. +export default function Layout({ children }: LayoutProps<"/[...slug]">) { + return ( + <DocsLayout tree={v2source.getPageTree()} {...baseOptions()}> + {children} + </DocsLayout> + ); +} diff --git a/src/app/[...slug]/page.tsx b/src/app/[...slug]/page.tsx new file mode 100644 index 0000000..bc6b521 --- /dev/null +++ b/src/app/[...slug]/page.tsx @@ -0,0 +1,76 @@ +import { + DocsBody, + DocsDescription, + DocsPage, + DocsTitle, +} from "fumadocs-ui/layouts/docs/page"; +import { createRelativeLink } from "fumadocs-ui/mdx"; +import type { Metadata } from "next"; +import { notFound } from "next/navigation"; +import { LLMCopyButton, ViewOptions } from "@/components/ai/page-actions"; +import { gitConfig } from "@/lib/layout.shared"; +import { v2source } from "@/lib/source"; +import { getMDXComponents } from "@/mdx-components"; + +// Page route for the V2 IA tree (content/docs). Mirrors the legacy +// /stack/[[...slug]] route; the legacy route is deleted when the migration +// completes (see IA.md). +export default async function Page(props: PageProps<"/[...slug]">) { + const params = await props.params; + const page = v2source.getPage(params.slug); + if (!page) notFound(); + + const MDX = page.data.body; + + return ( + <DocsPage toc={page.data.toc} full={page.data.full}> + <DocsTitle>{page.data.title}</DocsTitle> + <DocsDescription className="mb-0"> + {page.data.description} + </DocsDescription> + <div className="flex flex-row gap-2 items-center border-b pb-6"> + <LLMCopyButton markdownUrl={`/docs${page.url}.mdx`} /> + <ViewOptions + markdownUrl={`/docs${page.url}.mdx`} + githubUrl={`https://github.com/${gitConfig.user}/${gitConfig.repo}/blob/${gitConfig.branch}/content/docs/${page.path}`} + /> + </div> + <DocsBody> + <MDX + components={getMDXComponents({ + a: createRelativeLink(v2source, page), + })} + /> + </DocsBody> + </DocsPage> + ); +} + +export async function generateStaticParams() { + return v2source.generateParams(); +} + +export async function generateMetadata( + props: PageProps<"/[...slug]">, +): Promise<Metadata> { + const params = await props.params; + const page = v2source.getPage(params.slug); + if (!page) notFound(); + + const title = page.data.seoTitle ?? page.data.title; + const url = `https://cipherstash.com/docs${page.url}`; + + return { + title, + description: page.data.description, + alternates: { canonical: url }, + openGraph: { + type: "article", + url, + title, + description: page.data.description, + // TODO(v2): OG images — the /og route only covers the legacy tree. + // Add a v2 OG route when the first real (non-stub) pages land. + }, + }; +} diff --git a/src/app/api/search/route.ts b/src/app/api/search/route.ts index aa9d5cd..ec6bc8d 100644 --- a/src/app/api/search/route.ts +++ b/src/app/api/search/route.ts @@ -1,5 +1,5 @@ -import { source } from "@/lib/source"; import { createFromSource } from "fumadocs-core/search/server"; +import { source } from "@/lib/source"; export const { GET } = createFromSource(source, { // https://docs.orama.com/docs/orama-js/supported-languages diff --git a/src/app/layout.tsx b/src/app/layout.tsx index e6ff9a2..33912be 100644 --- a/src/app/layout.tsx +++ b/src/app/layout.tsx @@ -2,7 +2,7 @@ import { RootProvider } from "fumadocs-ui/provider/next"; import { PostHogProvider } from "@/lib/posthog/provider"; import "./global.css"; import type { Metadata } from "next"; -import { Inter, Fira_Code } from "next/font/google"; +import { Fira_Code, Inter } from "next/font/google"; // Site-wide title template so every page gets a descriptive, branded // <title>. Per-page metadata returns a bare title (e.g. "Keysets") which diff --git a/src/app/llms-full.txt/route.ts b/src/app/llms-full.txt/route.ts index 8e2efe8..9cd76be 100644 --- a/src/app/llms-full.txt/route.ts +++ b/src/app/llms-full.txt/route.ts @@ -1,5 +1,5 @@ import { getPostHogClient } from "@/lib/posthog/server"; -import { getLLMText, source } from "@/lib/source"; +import { getLLMText, source, v2source } from "@/lib/source"; export const revalidate = false; @@ -18,7 +18,7 @@ export async function GET(request: Request) { await posthog.flush(); } - const scan = source.getPages().map(getLLMText); + const scan = [...v2source.getPages(), ...source.getPages()].map(getLLMText); const scanned = await Promise.all(scan); return new Response(scanned.join("\n\n")); diff --git a/src/app/llms.mdx/v2/[...slug]/route.ts b/src/app/llms.mdx/v2/[...slug]/route.ts new file mode 100644 index 0000000..77dfd1e --- /dev/null +++ b/src/app/llms.mdx/v2/[...slug]/route.ts @@ -0,0 +1,44 @@ +import { notFound } from "next/navigation"; +import { getPostHogClient } from "@/lib/posthog/server"; +import { getLLMText, v2source } from "@/lib/source"; + +// Raw-markdown mirror for the V2 IA tree, reached via the +// `/:path*.mdx` rewrite in next.config.mjs (same pattern as the legacy +// /llms.mdx/stack route). +export const revalidate = false; + +export async function GET( + req: Request, + { params }: RouteContext<"/llms.mdx/v2/[...slug]">, +) { + const { slug } = await params; + const page = v2source.getPage(slug); + if (!page) notFound(); + + const posthog = getPostHogClient(); + if (posthog) { + posthog.capture({ + distinctId: "llm-agent", + event: "llms_mdx_page_fetched", + properties: { + $current_url: req.url, + page_slug: slug?.join("/") ?? "", + page_title: page.data.title, + referer: req.headers.get("referer") ?? "", + user_agent: req.headers.get("user-agent") ?? "", + }, + }); + await posthog.flush(); + } + + return new Response(await getLLMText(page), { + headers: { + "Content-Type": "text/markdown", + "Access-Control-Allow-Origin": "*", + }, + }); +} + +export function generateStaticParams() { + return v2source.generateParams(); +} diff --git a/src/app/llms.txt/route.ts b/src/app/llms.txt/route.ts index 5d6bcbb..2c6696e 100644 --- a/src/app/llms.txt/route.ts +++ b/src/app/llms.txt/route.ts @@ -1,5 +1,5 @@ import { getPostHogClient } from "@/lib/posthog/server"; -import { source } from "@/lib/source"; +import { source, v2source } from "@/lib/source"; export const revalidate = false; @@ -21,7 +21,8 @@ export async function GET(request: Request) { const lines: string[] = []; lines.push("# Documentation"); lines.push(""); - for (const page of source.getPages()) { + // V2 tree first: it's the canonical IA once the migration completes. + for (const page of [...v2source.getPages(), ...source.getPages()]) { lines.push(`- [${page.data.title}](${page.url}): ${page.data.description}`); } return new Response(lines.join("\n")); diff --git a/src/app/og/docs/[...slug]/route.tsx b/src/app/og/docs/[...slug]/route.tsx index 208fdfd..801a890 100644 --- a/src/app/og/docs/[...slug]/route.tsx +++ b/src/app/og/docs/[...slug]/route.tsx @@ -1,7 +1,7 @@ -import { getPageImage, source } from "@/lib/source"; +import { generate as DefaultImage } from "fumadocs-ui/og"; import { notFound } from "next/navigation"; import { ImageResponse } from "next/og"; -import { generate as DefaultImage } from "fumadocs-ui/og"; +import { getPageImage, source } from "@/lib/source"; export const revalidate = false; diff --git a/src/app/sitemap.ts b/src/app/sitemap.ts index 515283d..77fd867 100644 --- a/src/app/sitemap.ts +++ b/src/app/sitemap.ts @@ -1,10 +1,10 @@ import type { MetadataRoute } from "next"; -import { source } from "@/lib/source"; +import { source, v2source } from "@/lib/source"; const BASE_URL = "https://cipherstash.com/docs"; export default function sitemap(): MetadataRoute.Sitemap { - return source.getPages().map((page) => ({ + return [...v2source.getPages(), ...source.getPages()].map((page) => ({ url: `${BASE_URL}${page.url}`, lastModified: new Date(), changeFrequency: "weekly", diff --git a/src/app/stack/[[...slug]]/page.tsx b/src/app/stack/[[...slug]]/page.tsx index 6f5e79a..ef050ee 100644 --- a/src/app/stack/[[...slug]]/page.tsx +++ b/src/app/stack/[[...slug]]/page.tsx @@ -1,16 +1,16 @@ -import { getPageImage, source } from "@/lib/source"; import { DocsBody, DocsDescription, DocsPage, DocsTitle, } from "fumadocs-ui/layouts/docs/page"; -import { notFound } from "next/navigation"; -import { getMDXComponents } from "@/mdx-components"; -import type { Metadata } from "next"; import { createRelativeLink } from "fumadocs-ui/mdx"; +import type { Metadata } from "next"; +import { notFound } from "next/navigation"; import { LLMCopyButton, ViewOptions } from "@/components/ai/page-actions"; import { gitConfig } from "@/lib/layout.shared"; +import { getPageImage, source } from "@/lib/source"; +import { getMDXComponents } from "@/mdx-components"; export default async function Page(props: PageProps<"/stack/[[...slug]]">) { const params = await props.params; diff --git a/src/app/stack/layout.tsx b/src/app/stack/layout.tsx index d5b93ec..78d2389 100644 --- a/src/app/stack/layout.tsx +++ b/src/app/stack/layout.tsx @@ -1,6 +1,6 @@ -import { source } from "@/lib/source"; import { DocsLayout } from "fumadocs-ui/layouts/docs"; import { baseOptions } from "@/lib/layout.shared"; +import { source } from "@/lib/source"; export default function Layout({ children }: LayoutProps<"/stack">) { return ( diff --git a/src/components/icons/supabase.tsx b/src/components/icons/supabase.tsx index c6336f9..492ac61 100644 --- a/src/components/icons/supabase.tsx +++ b/src/components/icons/supabase.tsx @@ -44,5 +44,5 @@ export function SupabaseIcon(props: React.SVGProps<SVGSVGElement>) { </linearGradient> </defs> </svg> - ) + ); } diff --git a/src/lib/posthog/provider.tsx b/src/lib/posthog/provider.tsx index 711982a..9d7a5cb 100644 --- a/src/lib/posthog/provider.tsx +++ b/src/lib/posthog/provider.tsx @@ -1,13 +1,13 @@ "use client"; -import posthog from "posthog-js"; import { usePathname, useSearchParams } from "next/navigation"; +import posthog from "posthog-js"; import { - Suspense, createContext, + type ReactNode, + Suspense, useContext, useEffect, - type ReactNode, } from "react"; const PostHogContext = createContext<typeof posthog | null>(null); diff --git a/src/lib/source.ts b/src/lib/source.ts index 97ae84f..d0bcb1c 100644 --- a/src/lib/source.ts +++ b/src/lib/source.ts @@ -1,7 +1,7 @@ -import { docs } from "fumadocs-mdx:collections/server"; +import { docs, v2docs } from "fumadocs-mdx:collections/server"; import { type InferPageType, loader } from "fumadocs-core/source"; -import { createElement } from "react"; import { icons } from "lucide-react"; +import { createElement } from "react"; import { SupabaseIcon } from "@/components/icons/supabase"; const customIcons: Record<string, () => React.ReactElement> = { @@ -23,6 +23,15 @@ export const source = loader({ icon: resolveIcon, }); +// V2 IA tree (CIP-3325): content/docs served from the site root, e.g. +// /docs/get-started/quickstart. Lives alongside the legacy `source` during +// the migration; the legacy loader and /stack routes are deleted at the end. +export const v2source = loader({ + baseUrl: "/", + source: v2docs.toFumadocsSource(), + icon: resolveIcon, +}); + export function getPageImage(page: InferPageType<typeof source>) { const segments = [...page.slugs, "image.png"]; @@ -32,7 +41,9 @@ export function getPageImage(page: InferPageType<typeof source>) { }; } -export async function getLLMText(page: InferPageType<typeof source>) { +export async function getLLMText( + page: InferPageType<typeof source> | InferPageType<typeof v2source>, +) { const processed = await page.data.getText("processed"); return `# ${page.data.title} diff --git a/src/proxy.ts b/src/proxy.ts index 028dd8f..5d45773 100644 --- a/src/proxy.ts +++ b/src/proxy.ts @@ -1,5 +1,5 @@ -import { NextResponse } from "next/server"; import type { NextFetchEvent, NextRequest } from "next/server"; +import { NextResponse } from "next/server"; import { getPostHogClient } from "@/lib/posthog/server"; const SKIP_PATHS = ["/api", "/_next/static", "/_next/image", "/ingest"]; diff --git a/v2-redirects.mjs b/v2-redirects.mjs new file mode 100644 index 0000000..1f068b1 --- /dev/null +++ b/v2-redirects.mjs @@ -0,0 +1,375 @@ +// V2 IA redirect map (CIP-3325): every legacy /stack/* URL → its new home. +// Derived from the migration map in IA.md; completeness is enforced by +// `scripts/validate-v2-redirects.ts` (every content/stack page must match an +// entry here, exact or wildcard). +// +// Gated behind ENABLE_V2_REDIRECTS=1 in next.config.mjs: during the migration +// the preview site serves BOTH trees (legacy at /stack, v2 at the root), so +// unmigrated content stays reachable. The flag flips on at merge; once +// content/stack is deleted these entries become unconditional (CIP-3335). +// +// Conventions (matching next.config.mjs): sources/destinations omit the +// "/docs" basePath. Order matters — specific entries before wildcards. +export const v2Redirects = [ + // === Roots === + { source: "/stack", destination: "/", permanent: true }, + { + source: "/stack/quickstart", + destination: "/get-started/quickstart", + permanent: true, + }, + { source: "/stack/cipherstash", destination: "/", permanent: true }, + { + source: "/stack/cipherstash/postgres", + destination: "/reference/eql", + permanent: true, + }, + { + source: "/stack/cipherstash/supabase", + destination: "/integrations/supabase", + permanent: true, + }, + + // === Encryption SDK section → Reference/stack + new homes === + { + source: "/stack/cipherstash/encryption", + destination: "/reference/stack", + permanent: true, + }, + { + source: "/stack/cipherstash/encryption/searchable-encryption", + destination: "/concepts/searchable-encryption", + permanent: true, + }, + { + source: "/stack/cipherstash/encryption/identity", + destination: "/concepts/identity-aware-encryption", + permanent: true, + }, + { + source: "/stack/cipherstash/encryption/drizzle", + destination: "/integrations/drizzle", + permanent: true, + }, + { + source: "/stack/cipherstash/encryption/prisma-next", + destination: "/integrations/prisma-next", + permanent: true, + }, + { + source: "/stack/cipherstash/encryption/dynamodb", + destination: "/integrations/aws/dynamodb", + permanent: true, + }, + { + source: "/stack/cipherstash/encryption/supabase", + destination: "/reference/stack/supabase", + permanent: true, + }, + { + source: "/stack/cipherstash/encryption/indexes", + destination: "/reference/eql/indexes", + permanent: true, + }, + { + source: "/stack/cipherstash/encryption/queries", + destination: "/reference/eql/operators", + permanent: true, + }, + // configuration, encrypt-decrypt, bulk-operations, models, schema, storing-data + { + source: "/stack/cipherstash/encryption/:path*", + destination: "/reference/stack/:path*", + permanent: true, + }, + + // === KMS section → Security + Reference/auth + Concepts === + { + source: "/stack/cipherstash/kms", + destination: "/security/zerokms", + permanent: true, + }, + { + source: "/stack/cipherstash/kms/cts", + destination: "/security/cts", + permanent: true, + }, + { + source: "/stack/cipherstash/kms/oidc", + destination: "/reference/auth/oidc-configuration", + permanent: true, + }, + { + source: "/stack/cipherstash/kms/access-keys", + destination: "/reference/auth/access-keys", + permanent: true, + }, + { + source: "/stack/cipherstash/kms/clients", + destination: "/reference/auth/clients", + permanent: true, + }, + { + source: "/stack/cipherstash/kms/disaster-recovery", + destination: "/security/availability-and-continuity", + permanent: true, + }, + { + source: "/stack/cipherstash/kms/keysets", + destination: "/concepts/key-management", + permanent: true, + }, + { + source: "/stack/cipherstash/kms/regions", + destination: "/security/zerokms", + permanent: true, + }, + { + source: "/stack/cipherstash/kms/configuration", + destination: "/reference/workspace/configuration", + permanent: true, + }, + + // === Proxy section → Reference/proxy + new homes === + { + source: "/stack/cipherstash/proxy", + destination: "/reference/proxy", + permanent: true, + }, + { + source: "/stack/cipherstash/proxy/audit", + destination: "/security/audit-logging", + permanent: true, + }, + { + source: "/stack/cipherstash/proxy/getting-started", + destination: "/integrations/aws/rds-aurora", + permanent: true, + }, + { + source: "/stack/cipherstash/proxy/encrypt-tool", + destination: "/guides/migration/encrypt-existing-data", + permanent: true, + }, + { + source: "/stack/cipherstash/proxy/searchable-json", + destination: "/reference/eql/json", + permanent: true, + }, + { + source: "/stack/cipherstash/proxy/troubleshooting", + destination: "/guides/troubleshooting/proxy", + permanent: true, + }, + // configuration, message-flow, multitenant + { + source: "/stack/cipherstash/proxy/:path*", + destination: "/reference/proxy/:path*", + permanent: true, + }, + + // === CLI section → Reference/cli === + { + source: "/stack/cipherstash/cli", + destination: "/reference/cli", + permanent: true, + }, + { + source: "/stack/cipherstash/cli/troubleshooting", + destination: "/guides/troubleshooting/cli", + permanent: true, + }, + { + source: "/stack/cipherstash/cli/:path*", + destination: "/reference/cli/:path*", + permanent: true, + }, + + // === Deploy section → Guides === + { + source: "/stack/deploy", + destination: "/guides/deployment", + permanent: true, + }, + { + source: "/stack/deploy/going-to-production", + destination: "/guides/deployment/going-to-production", + permanent: true, + }, + { + source: "/stack/deploy/aws-ecs", + destination: "/guides/deployment/proxy-deployment", + permanent: true, + }, + { + source: "/stack/deploy/bundling", + destination: "/guides/deployment/serverless-and-bundling", + permanent: true, + }, + { + source: "/stack/deploy/sst", + destination: "/guides/deployment/serverless-and-bundling", + permanent: true, + }, + { + source: "/stack/deploy/testing", + destination: "/guides/development/testing-and-ci", + permanent: true, + }, + { + source: "/stack/deploy/team-onboarding", + destination: "/guides/development/team-onboarding", + permanent: true, + }, + { + source: "/stack/deploy/troubleshooting", + destination: "/guides/troubleshooting", + permanent: true, + }, + + // === Reference section === + { source: "/stack/reference", destination: "/reference", permanent: true }, + { + source: "/stack/reference/what-is-cipherstash", + destination: "/get-started/what-is-cipherstash", + permanent: true, + }, + { + source: "/stack/reference/security-architecture", + destination: "/security/architecture", + permanent: true, + }, + { + source: "/stack/reference/compliance", + destination: "/security/compliance", + permanent: true, + }, + { + source: "/stack/reference/comparisons", + destination: "/compare", + permanent: true, + }, + { + source: "/stack/reference/comparisons/:path*", + destination: "/compare/:path*", + permanent: true, + }, + { + source: "/stack/reference/use-cases", + destination: "/solutions", + permanent: true, + }, + { + source: "/stack/reference/use-cases/ai-rag", + destination: "/solutions/ai-and-rag", + permanent: true, + }, + { + source: "/stack/reference/use-cases/compliance", + destination: "/security/compliance", + permanent: true, + }, + { + source: "/stack/reference/use-cases/:path*", + destination: "/solutions/:path*", + permanent: true, + }, + { + source: "/stack/reference/billing", + destination: "/reference/workspace/billing", + permanent: true, + }, + { + source: "/stack/reference/members", + destination: "/reference/workspace/members", + permanent: true, + }, + { + source: "/stack/reference/cipher-cell", + destination: "/reference/eql/payload-format", + permanent: true, + }, + { + source: "/stack/reference/eql-guide", + destination: "/reference/eql", + permanent: true, + }, + { + source: "/stack/reference/eql", + destination: "/reference/eql", + permanent: true, + }, + { + source: "/stack/reference/eql/:path*", + destination: "/reference/eql/:path*", + permanent: true, + }, + { + source: "/stack/reference/encryption-sdk", + destination: "/reference/stack", + permanent: true, + }, + { + source: "/stack/reference/error-handling", + destination: "/reference/stack/errors", + permanent: true, + }, + // NOTE: legacy "migration" page is the @cipherstash/protect→stack package + // rename guide, NOT data migration (see IA.md). + { + source: "/stack/reference/migration", + destination: "/reference/stack/upgrading-from-protect", + permanent: true, + }, + { + source: "/stack/reference/proxy-errors", + destination: "/reference/proxy/errors", + permanent: true, + }, + { + source: "/stack/reference/proxy-reference", + destination: "/reference/proxy/configuration", + permanent: true, + }, + { + source: "/stack/reference/drizzle", + destination: "/integrations/drizzle", + permanent: true, + }, + { + source: "/stack/reference/dashboard-supabase-integration", + destination: "/integrations/supabase", + permanent: true, + }, + { + source: "/stack/reference/discovery-session", + destination: "/get-started/choose-your-stack", + permanent: true, + }, + { + source: "/stack/reference/planning-guide", + destination: "/get-started/choose-your-stack", + permanent: true, + }, + { + source: "/stack/reference/supported-solutions", + destination: "/integrations", + permanent: true, + }, + { + source: "/stack/reference/agent-skills", + destination: "/reference/agent-skills", + permanent: true, + }, + { + source: "/stack/reference/glossary", + destination: "/reference/glossary", + permanent: true, + }, + // Generated TypeDoc API reference (scripts/generate-docs.ts output) + { + source: "/stack/reference/stack/:path*", + destination: "/reference/stack/:path*", + permanent: true, + }, +]; From af92d2c5cf3cda8eb11f3cbe03dc096351ae3097 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 17:20:00 +1000 Subject: [PATCH 02/38] fix(v2): quote colon-bearing frontmatter descriptions; unshadow /reference/eql MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two issues caught by smoke-testing the scaffold: - Stub descriptions containing ":" broke YAML frontmatter parsing (500s across the v2 tree) — descriptions are now quoted. - The AI-citation redirect "/reference/eql" → "/stack/reference/eql" shadowed the v2 /reference/eql page (redirects run before the filesystem); removed since the v2 page now serves that path. Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- content/docs/compare/index.mdx | 2 +- content/docs/concepts/index.mdx | 2 +- content/docs/get-started/index.mdx | 2 +- content/docs/guides/deployment/index.mdx | 2 +- content/docs/guides/development/index.mdx | 2 +- content/docs/guides/index.mdx | 2 +- content/docs/guides/migration/index.mdx | 2 +- content/docs/guides/troubleshooting/index.mdx | 2 +- content/docs/integrations/index.mdx | 2 +- content/docs/integrations/supabase/index.mdx | 2 +- content/docs/reference/auth/index.mdx | 2 +- content/docs/reference/cli/index.mdx | 2 +- content/docs/reference/eql/index.mdx | 2 +- content/docs/reference/index.mdx | 2 +- content/docs/reference/proxy/index.mdx | 2 +- content/docs/reference/stack/index.mdx | 2 +- content/docs/reference/workspace/index.mdx | 2 +- content/docs/security/compliance/index.mdx | 2 +- content/docs/security/index.mdx | 2 +- content/docs/solutions/index.mdx | 2 +- next.config.mjs | 9 ++++----- 21 files changed, 24 insertions(+), 25 deletions(-) diff --git a/content/docs/compare/index.mdx b/content/docs/compare/index.mdx index 0f07e95..8d19d4b 100644 --- a/content/docs/compare/index.mdx +++ b/content/docs/compare/index.mdx @@ -1,6 +1,6 @@ --- title: Comparisons -description: How CipherStash compares to other approaches to protecting data. +description: "How CipherStash compares to other approaches to protecting data." type: concept --- diff --git a/content/docs/concepts/index.mdx b/content/docs/concepts/index.mdx index 282c15b..26a565e 100644 --- a/content/docs/concepts/index.mdx +++ b/content/docs/concepts/index.mdx @@ -1,6 +1,6 @@ --- title: Concepts -description: How CipherStash works and how to think about searchable encryption, keys, and identity. +description: "How CipherStash works and how to think about searchable encryption, keys, and identity." type: concept --- diff --git a/content/docs/get-started/index.mdx b/content/docs/get-started/index.mdx index af235c6..0e08a05 100644 --- a/content/docs/get-started/index.mdx +++ b/content/docs/get-started/index.mdx @@ -1,6 +1,6 @@ --- title: Get started -description: What CipherStash is, a 10-minute quickstart, and how to choose your integration path. +description: "What CipherStash is, a 10-minute quickstart, and how to choose your integration path." type: tutorial --- diff --git a/content/docs/guides/deployment/index.mdx b/content/docs/guides/deployment/index.mdx index 771a593..8fb92ca 100644 --- a/content/docs/guides/deployment/index.mdx +++ b/content/docs/guides/deployment/index.mdx @@ -1,6 +1,6 @@ --- title: Deployment -description: Deployment documentation — being built as part of the docs V2 overhaul. +description: "Deployment documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/guides/development/index.mdx b/content/docs/guides/development/index.mdx index c0d4ab1..a7a049f 100644 --- a/content/docs/guides/development/index.mdx +++ b/content/docs/guides/development/index.mdx @@ -1,6 +1,6 @@ --- title: Development -description: Development documentation — being built as part of the docs V2 overhaul. +description: "Development documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/guides/index.mdx b/content/docs/guides/index.mdx index f034020..44308b2 100644 --- a/content/docs/guides/index.mdx +++ b/content/docs/guides/index.mdx @@ -1,6 +1,6 @@ --- title: Guides -description: Task-oriented guides: development workflow, data migration, deployment, and troubleshooting. +description: "Task-oriented guides: development workflow, data migration, deployment, and troubleshooting." type: guide --- diff --git a/content/docs/guides/migration/index.mdx b/content/docs/guides/migration/index.mdx index 690a307..97ac0dc 100644 --- a/content/docs/guides/migration/index.mdx +++ b/content/docs/guides/migration/index.mdx @@ -1,6 +1,6 @@ --- title: Data migration -description: Data migration documentation — being built as part of the docs V2 overhaul. +description: "Data migration documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/guides/troubleshooting/index.mdx b/content/docs/guides/troubleshooting/index.mdx index f1e926e..7354565 100644 --- a/content/docs/guides/troubleshooting/index.mdx +++ b/content/docs/guides/troubleshooting/index.mdx @@ -1,6 +1,6 @@ --- title: Troubleshooting -description: Troubleshooting documentation — being built as part of the docs V2 overhaul. +description: "Troubleshooting documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/integrations/index.mdx b/content/docs/integrations/index.mdx index dfcf78e..6d6e1bb 100644 --- a/content/docs/integrations/index.mdx +++ b/content/docs/integrations/index.mdx @@ -1,6 +1,6 @@ --- title: Integrations -description: Set up CipherStash with your platform, ORM, framework, auth provider, and runtime. +description: "Set up CipherStash with your platform, ORM, framework, auth provider, and runtime." type: tutorial --- diff --git a/content/docs/integrations/supabase/index.mdx b/content/docs/integrations/supabase/index.mdx index 6066937..6330cf9 100644 --- a/content/docs/integrations/supabase/index.mdx +++ b/content/docs/integrations/supabase/index.mdx @@ -1,6 +1,6 @@ --- title: Supabase -description: Searchable, application-level encryption for your Supabase project — encrypt in your app, query in Postgres. +description: "Searchable, application-level encryption for your Supabase project — encrypt in your app, query in Postgres." type: tutorial components: [encryption, eql, auth] audience: [developer] diff --git a/content/docs/reference/auth/index.mdx b/content/docs/reference/auth/index.mdx index 2dea44b..1c05374 100644 --- a/content/docs/reference/auth/index.mdx +++ b/content/docs/reference/auth/index.mdx @@ -1,6 +1,6 @@ --- title: Auth -description: Auth documentation — being built as part of the docs V2 overhaul. +description: "Auth documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/reference/cli/index.mdx b/content/docs/reference/cli/index.mdx index 90d8cb3..2897d5c 100644 --- a/content/docs/reference/cli/index.mdx +++ b/content/docs/reference/cli/index.mdx @@ -1,6 +1,6 @@ --- title: CLI -description: CLI documentation — being built as part of the docs V2 overhaul. +description: "CLI documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index e312de2..276450a 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -1,6 +1,6 @@ --- title: EQL -description: EQL documentation — being built as part of the docs V2 overhaul. +description: "EQL documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/reference/index.mdx b/content/docs/reference/index.mdx index fc2c7e7..287825a 100644 --- a/content/docs/reference/index.mdx +++ b/content/docs/reference/index.mdx @@ -1,6 +1,6 @@ --- title: Reference -description: Precise API documentation for EQL, the Stack SDK, Auth, the CLI, and Proxy. +description: "Precise API documentation for EQL, the Stack SDK, Auth, the CLI, and Proxy." type: reference --- diff --git a/content/docs/reference/proxy/index.mdx b/content/docs/reference/proxy/index.mdx index 961c67b..d3d231e 100644 --- a/content/docs/reference/proxy/index.mdx +++ b/content/docs/reference/proxy/index.mdx @@ -1,6 +1,6 @@ --- title: Proxy -description: Proxy documentation — being built as part of the docs V2 overhaul. +description: "Proxy documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/reference/stack/index.mdx b/content/docs/reference/stack/index.mdx index 993c909..6d79f4c 100644 --- a/content/docs/reference/stack/index.mdx +++ b/content/docs/reference/stack/index.mdx @@ -1,6 +1,6 @@ --- title: Stack SDK -description: Stack SDK documentation — being built as part of the docs V2 overhaul. +description: "Stack SDK documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/reference/workspace/index.mdx b/content/docs/reference/workspace/index.mdx index 9bebe76..e97a9a6 100644 --- a/content/docs/reference/workspace/index.mdx +++ b/content/docs/reference/workspace/index.mdx @@ -1,6 +1,6 @@ --- title: Workspace & account -description: Workspace & account documentation — being built as part of the docs V2 overhaul. +description: "Workspace & account documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/security/compliance/index.mdx b/content/docs/security/compliance/index.mdx index 4a25ab6..70c433a 100644 --- a/content/docs/security/compliance/index.mdx +++ b/content/docs/security/compliance/index.mdx @@ -1,6 +1,6 @@ --- title: Compliance -description: Compliance documentation — being built as part of the docs V2 overhaul. +description: "Compliance documentation — being built as part of the docs V2 overhaul." --- This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). diff --git a/content/docs/security/index.mdx b/content/docs/security/index.mdx index 6bb327d..da86585 100644 --- a/content/docs/security/index.mdx +++ b/content/docs/security/index.mdx @@ -1,6 +1,6 @@ --- title: Architecture & security -description: Trust model, components, availability, audit, and compliance — self-contained for security review. +description: "Trust model, components, availability, audit, and compliance — self-contained for security review." type: concept --- diff --git a/content/docs/solutions/index.mdx b/content/docs/solutions/index.mdx index 5189ccf..350020d 100644 --- a/content/docs/solutions/index.mdx +++ b/content/docs/solutions/index.mdx @@ -1,6 +1,6 @@ --- title: Solutions -description: What CipherStash solves: PII protection, HIPAA, AI/RAG, data residency, provable access. +description: "What CipherStash solves: PII protection, HIPAA, AI/RAG, data residency, provable access." type: concept --- diff --git a/next.config.mjs b/next.config.mjs index c6a0f69..5b12a91 100644 --- a/next.config.mjs +++ b/next.config.mjs @@ -303,11 +303,10 @@ const config = { destination: "/stack/deploy/aws-ecs", permanent: true, }, - { - source: "/reference/eql", - destination: "/stack/reference/eql", - permanent: false, - }, + // NOTE(v2): the AI-citation redirect "/reference/eql" → + // "/stack/reference/eql" was removed here — its source collides with + // the v2 IA's /reference/eql page, which now serves that traffic + // directly (CIP-3325). { source: "/platform/workspaces/key-sets", destination: "/stack/cipherstash/kms/keysets", From a6a4274828e98c56dae3148428822fcfeb1bd19c Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 17:43:40 +1000 Subject: [PATCH 03/38] feat(v2): serve the /docs landing inside the docs navigation; redirect bare root MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fixes from preview review: - The bare domain root (Vercel preview URLs) 404'd because the app lives under the /docs basePath — added a basePath:false redirect / → /docs. In production only /docs/* reaches this app, so previews-only. - The /docs landing was a standalone (home) page disconnected from the v2 nav, with every link pointing at legacy /stack URLs. It's now content/docs/index.mdx rendered inside DocsLayout (sidebar + search), linking the v2 sections. The catch-all became optional ([[...slug]]) and the (home) route group is deleted (recoverable from history; CIP-3327 refines the landing content). - The landing's raw-markdown mirror serves at /docs/index.mdx (its URL is "/", which can't carry the .mdx suffix). Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- content/docs/index.mdx | 38 ++ content/docs/meta.json | 1 + next.config.mjs | 10 + src/app/(home)/layout.tsx | 6 - src/app/(home)/page.tsx | 346 ------------------ src/app/{[...slug] => [[...slug]]}/layout.tsx | 10 +- src/app/{[...slug] => [[...slug]]}/page.tsx | 24 +- .../v2/{[...slug] => [[...slug]]}/route.ts | 10 +- 8 files changed, 77 insertions(+), 368 deletions(-) create mode 100644 content/docs/index.mdx delete mode 100644 src/app/(home)/layout.tsx delete mode 100644 src/app/(home)/page.tsx rename src/app/{[...slug] => [[...slug]]}/layout.tsx (51%) rename src/app/{[...slug] => [[...slug]]}/page.tsx (70%) rename src/app/llms.mdx/v2/{[...slug] => [[...slug]]}/route.ts (73%) diff --git a/content/docs/index.mdx b/content/docs/index.mdx new file mode 100644 index 0000000..e46f40e --- /dev/null +++ b/content/docs/index.mdx @@ -0,0 +1,38 @@ +--- +title: CipherStash Docs +seoTitle: CipherStash Docs — Searchable encryption for Postgres +description: "Searchable field-level encryption, identity-bound keys, and cryptographic audit trails — built into your existing Postgres stack." +type: concept +audience: [developer, cto, ciso] +--- + +CipherStash encrypts your data at the field level. Every value gets its own +key, bound to an identity — and the ciphertext stays queryable in Postgres. +A breach, a compromised agent, a curious insider: they all see ciphertext +with no key. + +## Start here + +<Cards> + <Card title="Get started" href="/docs/get-started" description="What CipherStash is and your first encrypted fields in 10 minutes." /> + <Card title="Quickstart" href="/docs/stack/quickstart" description="Encrypt, store, query, and decrypt your first fields in any Postgres." /> + <Card title="Supabase" href="/docs/integrations/supabase" description="Searchable, application-level encryption for your Supabase project." /> + <Card title="Agent skills" href="/docs/stack/reference/agent-skills" description="CipherStash knowledge for Cursor, Copilot, and Claude Code." /> +</Cards> + +## Browse the docs + +<Cards> + <Card title="Integrations" href="/docs/integrations" description="Platforms, ORMs, frameworks, auth providers, and runtimes." /> + <Card title="Concepts" href="/docs/concepts" description="How searchable encryption, key management, and identity-aware encryption work." /> + <Card title="Guides" href="/docs/guides" description="Development workflow, data migration, deployment, and troubleshooting." /> + <Card title="Architecture & security" href="/docs/security" description="Trust model, components, availability, audit, and compliance — for security review." /> + <Card title="Solutions" href="/docs/solutions" description="PII protection, HIPAA, AI/RAG, data residency, and provable access." /> + <Card title="Reference" href="/docs/reference" description="EQL, the Stack SDK, Auth, the CLI, and Proxy — precise API documentation." /> +</Cards> + +## AI-ready documentation + +Every page is available as clean markdown: append `.mdx` to any page URL, or +fetch the whole corpus via [llms.txt](/docs/llms.txt) and +[llms-full.txt](/docs/llms-full.txt). diff --git a/content/docs/meta.json b/content/docs/meta.json index 8935d68..74f486e 100644 --- a/content/docs/meta.json +++ b/content/docs/meta.json @@ -1,5 +1,6 @@ { "pages": [ + "index", "get-started", "integrations", "concepts", diff --git a/next.config.mjs b/next.config.mjs index 5b12a91..990a7b3 100644 --- a/next.config.mjs +++ b/next.config.mjs @@ -16,6 +16,16 @@ const config = { reactStrictMode: true, async redirects() { return [ + // The app lives under the /docs basePath, so the bare domain root + // (e.g. on Vercel preview URLs) would otherwise 404. In production + // "/" never reaches this app — cipherstash.com routes only /docs/* + // here — so this only affects previews. + { + source: "/", + destination: "/docs", + basePath: false, + permanent: false, + }, // Vanity URL for the new IA (safe to ship ungated: the path has no // legacy traffic). Temporary until the v2 quickstart is canonical. { diff --git a/src/app/(home)/layout.tsx b/src/app/(home)/layout.tsx deleted file mode 100644 index c16b056..0000000 --- a/src/app/(home)/layout.tsx +++ /dev/null @@ -1,6 +0,0 @@ -import { HomeLayout } from "fumadocs-ui/layouts/home"; -import { baseOptions } from "@/lib/layout.shared"; - -export default function Layout({ children }: LayoutProps<"/">) { - return <HomeLayout {...baseOptions()}>{children}</HomeLayout>; -} diff --git a/src/app/(home)/page.tsx b/src/app/(home)/page.tsx deleted file mode 100644 index 7cad1aa..0000000 --- a/src/app/(home)/page.tsx +++ /dev/null @@ -1,346 +0,0 @@ -import { - ArrowRight, - BookOpen, - Code, - Database, - ExternalLinkIcon, - FileText, - KeyRound, - Lock, - Search, - ShieldCheck, - Zap, -} from "lucide-react"; -import type { Metadata } from "next"; -import Link from "next/link"; -import type { ComponentType } from "react"; -import { - DrizzleLogo, - DynamoDBLogo, - PrismaLogo, - SupabaseLogo, -} from "@/components/integration-logos"; - -// The /docs landing page had no metadata (no <title>). `absolute` bypasses the -// root layout's "%s | CipherStash Docs" template so the title isn't doubled. -export const metadata: Metadata = { - title: { - absolute: "CipherStash Docs — Searchable encryption for Postgres", - }, - description: - "Data Level Access Control for Postgres. Searchable field-level encryption, identity-bound keys, and cryptographic audit trails.", - alternates: { canonical: "https://cipherstash.com/docs" }, -}; - -const monoClass = "font-[family-name:var(--font-fira-code)] tracking-[-0.02em]"; -const eyebrowClass = - "font-[family-name:var(--font-fira-code)] text-[10px] font-medium tracking-[0.16em] uppercase text-fd-primary"; - -const products = [ - { - title: "Encryption", - description: - "Searchable field-level encryption. Range queries, exact match, and free-text search over ciphertext. Sub-millisecond overhead.", - href: "/stack/cipherstash/encryption", - icon: Lock, - }, - { - title: "ZeroKMS", - description: - "The key management layer. Unique key per value, derived on demand, never stored. 100x faster than AWS KMS.", - href: "/stack/cipherstash/kms", - icon: KeyRound, - }, - { - title: "Proxy", - description: - "Transparent searchable encryption for existing PostgreSQL databases. Zero application code changes.", - href: "/stack/cipherstash/proxy", - icon: Database, - }, -]; - -const integrations: { - title: string; - description: string; - href: string; - logo: ComponentType<{ className?: string }>; -}[] = [ - { - title: "Supabase", - description: "Field-level encryption for your Supabase project.", - href: "/stack/cipherstash/supabase", - logo: SupabaseLogo, - }, - { - title: "Drizzle ORM", - description: "Encrypted column types and query operators for Drizzle.", - href: "/stack/cipherstash/encryption/drizzle", - logo: DrizzleLogo, - }, - { - title: "Prisma Next", - description: - "Searchable field-level encryption for Postgres with Prisma Next.", - href: "/stack/cipherstash/encryption/prisma-next", - logo: PrismaLogo, - }, - { - title: "DynamoDB", - description: - "Encrypted DynamoDB attributes with searchable equality lookups.", - href: "/stack/cipherstash/encryption/dynamodb", - logo: DynamoDBLogo, - }, -]; - -const resources = [ - { - title: "What is CipherStash?", - description: "DLAC, threat model, how it works", - href: "/stack/reference/what-is-cipherstash", - icon: ShieldCheck, - }, - { - title: "API Reference", - description: "SDK and API reference docs", - href: "/stack/reference", - icon: Code, - }, - { - title: "Agent Skills", - description: "CipherStash knowledge for your AI coding agent", - href: "/stack/reference/agent-skills", - icon: Zap, - }, - { - title: "Use Cases", - description: "AI/RAG, compliance, data residency", - href: "/stack/reference/use-cases", - icon: BookOpen, - }, -]; - -export default function HomePage() { - return ( - <main className="flex flex-col"> - {/* Hero */} - <section className="border-b border-fd-border"> - <div className="mx-auto w-full max-w-[1200px] px-6 pt-24 pb-16 md:px-12 md:pt-32 md:pb-20"> - <p className={eyebrowClass}>DLAC / DATA LEVEL ACCESS CONTROL</p> - <h1 - className={`mt-4 text-3xl font-medium text-fd-foreground md:text-5xl ${monoClass}`} - > - CipherStash Docs - </h1> - <p className="mt-4 max-w-2xl text-[17px] leading-relaxed text-fd-muted-foreground"> - Searchable field-level encryption. Identity-bound keys. - Cryptographic audit trails. Built into your existing Postgres stack. - </p> - - {/* Getting started cards */} - <div className="mt-10 grid gap-px bg-fd-border sm:grid-cols-2 border border-fd-border rounded-[2px] overflow-hidden"> - {[ - { - href: "/stack/quickstart", - icon: Zap, - title: "Quickstart", - desc: "Encrypt your first fields in 15 minutes.", - }, - { - href: "/stack/cipherstash/supabase", - icon: Database, - title: "Supabase", - desc: "Field-level encryption for Supabase.", - }, - { - href: "/stack/cipherstash/encryption/searchable-encryption", - icon: Search, - title: "Searchable encryption", - desc: "Equality, free text, range, ordering, and JSON queries over ciphertext.", - }, - { - href: "/stack/reference/agent-skills", - icon: Zap, - title: "Agent Skills", - desc: "CipherStash knowledge for Cursor, Copilot, Claude Code.", - }, - ].map((card) => ( - <Link - key={card.href} - href={card.href} - className="group flex items-center gap-4 bg-fd-background p-5 transition-colors hover:bg-fd-accent/50" - > - <div className="flex size-10 shrink-0 items-center justify-center rounded-[2px] bg-fd-primary/10 text-fd-primary"> - <card.icon className="size-5" /> - </div> - <div className="min-w-0"> - <p - className={`font-medium text-fd-foreground text-[15px] ${monoClass}`} - > - {card.title} - </p> - <p className="text-sm text-fd-muted-foreground"> - {card.desc} - </p> - </div> - <ArrowRight className="ml-auto size-4 shrink-0 text-fd-muted-foreground transition-colors group-hover:text-fd-primary" /> - </Link> - ))} - </div> - </div> - </section> - - {/* Products */} - <section className="mx-auto w-full max-w-[1200px] px-6 py-16 md:px-12 md:py-24"> - <p className={eyebrowClass}>§ 01 / THE STACK</p> - <h2 - className={`mt-3 text-xl font-medium text-fd-foreground md:text-2xl ${monoClass}`} - > - The Stack - </h2> - <p className="mt-2 text-fd-muted-foreground"> - Encryption, key management, and proxy. - </p> - - <div className="mt-8 grid gap-px bg-fd-border sm:grid-cols-3 border border-fd-border rounded-[2px] overflow-hidden"> - {products.map((product) => ( - <Link - key={product.title} - href={product.href} - className="group relative flex flex-col overflow-hidden bg-fd-background transition-colors hover:bg-fd-accent/50" - > - <div className="flex h-32 items-center justify-center border-b border-fd-border bg-fd-muted/20"> - <product.icon className="size-10 text-fd-muted-foreground/30" /> - </div> - <div className="flex flex-1 flex-col p-5"> - <div className="flex items-center gap-2"> - <product.icon className="size-4 text-fd-primary" /> - <h3 className={`font-medium text-fd-foreground ${monoClass}`}> - {product.title} - </h3> - </div> - <p className="mt-2 flex-1 text-sm leading-relaxed text-fd-muted-foreground"> - {product.description} - </p> - </div> - </Link> - ))} - </div> - </section> - - {/* Integrations */} - <section className="border-t border-fd-border"> - <div className="mx-auto w-full max-w-[1200px] px-6 py-16 md:px-12 md:py-24"> - <p className={eyebrowClass}>§ 02 / INTEGRATIONS</p> - <h2 - className={`mt-3 text-xl font-medium text-fd-foreground md:text-2xl ${monoClass}`} - > - Integrations - </h2> - <p className="mt-2 text-fd-muted-foreground"> - Drop-in encryption for the databases and ORMs you already use. - </p> - - <div className="mt-8 grid gap-px bg-fd-border sm:grid-cols-2 lg:grid-cols-4 border border-fd-border rounded-[2px] overflow-hidden"> - {integrations.map((integration) => ( - <Link - key={integration.title} - href={integration.href} - className="group flex flex-col items-center bg-fd-background p-6 text-center transition-colors hover:bg-fd-accent/50" - > - <div className="flex size-24 items-center justify-center"> - <integration.logo className="h-12 w-auto" /> - </div> - <h3 - className={`mt-4 font-medium text-fd-foreground ${monoClass}`} - > - {integration.title} - </h3> - <p className="mt-1 text-sm text-fd-muted-foreground"> - {integration.description} - </p> - </Link> - ))} - </div> - </div> - </section> - - {/* Resources */} - <section className="border-t border-fd-border"> - <div className="mx-auto w-full max-w-[1200px] px-6 py-16 md:px-12 md:py-24"> - <p className={eyebrowClass}>§ 03 / RESOURCES</p> - <h2 - className={`mt-3 text-xl font-medium text-fd-foreground md:text-2xl ${monoClass}`} - > - Resources - </h2> - - <div className="mt-8 grid gap-px bg-fd-border sm:grid-cols-2 lg:grid-cols-4 border border-fd-border rounded-[2px] overflow-hidden"> - {resources.map((resource) => ( - <Link - key={resource.title} - href={resource.href} - className="group flex items-start gap-3 bg-fd-background p-4 transition-colors hover:bg-fd-accent/50" - > - <resource.icon className="mt-0.5 size-5 shrink-0 text-fd-muted-foreground group-hover:text-fd-primary" /> - <div> - <p - className={`font-medium text-fd-foreground text-[14px] ${monoClass}`} - > - {resource.title} - </p> - <p className="mt-0.5 text-sm text-fd-muted-foreground"> - {resource.description} - </p> - </div> - </Link> - ))} - </div> - </div> - </section> - - {/* AI/LLM + CTA footer */} - <section className="border-t border-fd-border bg-fd-card/50"> - <div className="mx-auto flex w-full max-w-[1200px] flex-col items-center px-6 py-16 text-center md:px-12 md:py-20"> - <div className="flex size-10 items-center justify-center rounded-[2px] bg-fd-primary/10 text-fd-primary"> - <FileText className="size-5" /> - </div> - <h2 - className={`mt-4 text-xl font-medium text-fd-foreground md:text-2xl ${monoClass}`} - > - AI-ready documentation - </h2> - <p className="mx-auto mt-2 max-w-lg text-sm text-fd-muted-foreground"> - Every page is clean markdown. Feed it to your LLM. - </p> - <div className="mt-6 flex flex-wrap justify-center gap-3"> - <Link - href="/llms.txt" - className="inline-flex items-center gap-2 rounded-[2px] border border-fd-border px-4 py-2 text-sm font-medium text-fd-foreground transition-colors hover:border-fd-primary/40 hover:bg-fd-accent/50" - > - <FileText className="size-4" /> - llms.txt - </Link> - <Link - href="/llms-full.txt" - className="inline-flex items-center gap-2 rounded-[2px] border border-fd-border px-4 py-2 text-sm font-medium text-fd-foreground transition-colors hover:border-fd-primary/40 hover:bg-fd-accent/50" - > - <FileText className="size-4" /> - llms-full.txt - </Link> - <a - href="https://github.com/cipherstash/stack" - target="_blank" - rel="noopener noreferrer" - className="inline-flex items-center gap-2 rounded-[2px] border border-fd-border px-4 py-2 text-sm font-medium text-fd-foreground transition-colors hover:border-fd-primary/40 hover:bg-fd-accent/50" - > - <ExternalLinkIcon className="size-4" /> - GitHub - </a> - </div> - </div> - </section> - </main> - ); -} diff --git a/src/app/[...slug]/layout.tsx b/src/app/[[...slug]]/layout.tsx similarity index 51% rename from src/app/[...slug]/layout.tsx rename to src/app/[[...slug]]/layout.tsx index 2045bab..719ab35 100644 --- a/src/app/[...slug]/layout.tsx +++ b/src/app/[[...slug]]/layout.tsx @@ -2,11 +2,11 @@ import { DocsLayout } from "fumadocs-ui/layouts/docs"; import { baseOptions } from "@/lib/layout.shared"; import { v2source } from "@/lib/source"; -// Layout for the V2 IA tree (content/docs), served from the site root. -// A *required* catch-all (`[...slug]`, not `[[...slug]]`) so it never -// competes with the (home) route for "/". Static routes (/stack, /api, -// /og, …) take precedence over this segment as usual. -export default function Layout({ children }: LayoutProps<"/[...slug]">) { +// Layout for the V2 IA tree (content/docs), served from the site root — +// including the /docs landing page (content/docs/index.mdx), which renders +// inside the same navigation shell as every other page. Static routes +// (/stack, /api, /og, …) take precedence over this segment as usual. +export default function Layout({ children }: LayoutProps<"/[[...slug]]">) { return ( <DocsLayout tree={v2source.getPageTree()} {...baseOptions()}> {children} diff --git a/src/app/[...slug]/page.tsx b/src/app/[[...slug]]/page.tsx similarity index 70% rename from src/app/[...slug]/page.tsx rename to src/app/[[...slug]]/page.tsx index bc6b521..5efba49 100644 --- a/src/app/[...slug]/page.tsx +++ b/src/app/[[...slug]]/page.tsx @@ -12,10 +12,18 @@ import { gitConfig } from "@/lib/layout.shared"; import { v2source } from "@/lib/source"; import { getMDXComponents } from "@/mdx-components"; -// Page route for the V2 IA tree (content/docs). Mirrors the legacy -// /stack/[[...slug]] route; the legacy route is deleted when the migration -// completes (see IA.md). -export default async function Page(props: PageProps<"/[...slug]">) { +// Page route for the V2 IA tree (content/docs), including the /docs landing +// page. Mirrors the legacy /stack/[[...slug]] route; the legacy route is +// deleted when the migration completes (see IA.md). + +// The landing page's URL is "/", which would produce "/docs/.mdx" — serve its +// raw-markdown mirror at /docs/index.mdx instead (normalized back to the root +// slug in the llms.mdx/v2 route). +function markdownUrl(pageUrl: string): string { + return `/docs${pageUrl === "/" ? "/index" : pageUrl}.mdx`; +} + +export default async function Page(props: PageProps<"/[[...slug]]">) { const params = await props.params; const page = v2source.getPage(params.slug); if (!page) notFound(); @@ -29,9 +37,9 @@ export default async function Page(props: PageProps<"/[...slug]">) { {page.data.description} </DocsDescription> <div className="flex flex-row gap-2 items-center border-b pb-6"> - <LLMCopyButton markdownUrl={`/docs${page.url}.mdx`} /> + <LLMCopyButton markdownUrl={markdownUrl(page.url)} /> <ViewOptions - markdownUrl={`/docs${page.url}.mdx`} + markdownUrl={markdownUrl(page.url)} githubUrl={`https://github.com/${gitConfig.user}/${gitConfig.repo}/blob/${gitConfig.branch}/content/docs/${page.path}`} /> </div> @@ -51,14 +59,14 @@ export async function generateStaticParams() { } export async function generateMetadata( - props: PageProps<"/[...slug]">, + props: PageProps<"/[[...slug]]">, ): Promise<Metadata> { const params = await props.params; const page = v2source.getPage(params.slug); if (!page) notFound(); const title = page.data.seoTitle ?? page.data.title; - const url = `https://cipherstash.com/docs${page.url}`; + const url = `https://cipherstash.com/docs${page.url === "/" ? "" : page.url}`; return { title, diff --git a/src/app/llms.mdx/v2/[...slug]/route.ts b/src/app/llms.mdx/v2/[[...slug]]/route.ts similarity index 73% rename from src/app/llms.mdx/v2/[...slug]/route.ts rename to src/app/llms.mdx/v2/[[...slug]]/route.ts index 77dfd1e..fc9f251 100644 --- a/src/app/llms.mdx/v2/[...slug]/route.ts +++ b/src/app/llms.mdx/v2/[[...slug]]/route.ts @@ -9,10 +9,14 @@ export const revalidate = false; export async function GET( req: Request, - { params }: RouteContext<"/llms.mdx/v2/[...slug]">, + { params }: RouteContext<"/llms.mdx/v2/[[...slug]]">, ) { const { slug } = await params; - const page = v2source.getPage(slug); + // The landing page's markdown mirror is served at /docs/index.mdx (its URL + // is "/", which can't carry an .mdx suffix) — normalize back to the root. + const normalized = + !slug || (slug.length === 1 && slug[0] === "index") ? [] : slug; + const page = v2source.getPage(normalized); if (!page) notFound(); const posthog = getPostHogClient(); @@ -22,7 +26,7 @@ export async function GET( event: "llms_mdx_page_fetched", properties: { $current_url: req.url, - page_slug: slug?.join("/") ?? "", + page_slug: normalized.join("/"), page_title: page.data.title, referer: req.headers.get("referer") ?? "", user_agent: req.headers.get("user-agent") ?? "", From 5c34cefc6f16721076c71b4f254c0d5537e792bc Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 17:45:47 +1000 Subject: [PATCH 04/38] docs(v2): note landing-page state in IA.md checklist Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- IA.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/IA.md b/IA.md index 4c94f3d..f9c6f9e 100644 --- a/IA.md +++ b/IA.md @@ -35,7 +35,9 @@ live at `/docs/errors/<code>` — permanent, never restructured (CIP-3338). - [ ] `/get-started/quickstart` — rewritten on EQL v3 (fixes `cs_match_v1`, broken scaffold imports) - [ ] `/get-started/choose-your-stack` — static matrix v1 (platform × ORM × auth) - [ ] `/get-started/examples` — runnable example apps index -- [ ] `/docs` landing page (replaces/updates `(home)` route: what-is + audience router) +- [ ] `/docs` landing page 🚧 — now `content/docs/index.mdx` rendered inside the docs + nav (the old standalone `(home)` route is deleted; recoverable from git history). + CIP-3327 refines the content (what-is + audience router) ## Integrations — CIP-3328 (Supabase), CIP-3330 (auth), CIP-3336 (rest) From b604d4ff128b992849692afe6f673ae4992eb715 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 17:54:54 +1000 Subject: [PATCH 05/38] fix(v2): merge folder index pages into their nav rows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Listing "index" explicitly in meta.json pages forced each section's index out as a separate child item with the same title as its folder (clicking "Get started" opened a sub-nav containing another "Get started"). With index unlisted, Fumadocs merges it into the folder row: the folder itself links to the page, and children are only real sub-pages (Integrations → Supabase, not Integrations → Integrations). The root meta.json keeps "index" — at tree root there is no folder row to merge into, so the landing needs its own sidebar item. Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- content/docs/compare/meta.json | 2 +- content/docs/concepts/meta.json | 2 +- content/docs/get-started/meta.json | 2 +- content/docs/guides/deployment/meta.json | 2 +- content/docs/guides/development/meta.json | 2 +- content/docs/guides/meta.json | 2 +- content/docs/guides/migration/meta.json | 2 +- content/docs/guides/troubleshooting/meta.json | 2 +- content/docs/integrations/meta.json | 2 +- content/docs/integrations/supabase/meta.json | 2 +- content/docs/reference/auth/meta.json | 2 +- content/docs/reference/cli/meta.json | 2 +- content/docs/reference/eql/meta.json | 2 +- content/docs/reference/meta.json | 2 +- content/docs/reference/proxy/meta.json | 2 +- content/docs/reference/stack/meta.json | 2 +- content/docs/reference/workspace/meta.json | 2 +- content/docs/security/compliance/meta.json | 2 +- content/docs/security/meta.json | 2 +- content/docs/solutions/meta.json | 2 +- 20 files changed, 20 insertions(+), 20 deletions(-) diff --git a/content/docs/compare/meta.json b/content/docs/compare/meta.json index 3e374e5..76e9696 100644 --- a/content/docs/compare/meta.json +++ b/content/docs/compare/meta.json @@ -1,5 +1,5 @@ { "title": "Comparisons", "icon": "Scale", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/concepts/meta.json b/content/docs/concepts/meta.json index ca4b08b..521f756 100644 --- a/content/docs/concepts/meta.json +++ b/content/docs/concepts/meta.json @@ -1,5 +1,5 @@ { "title": "Concepts", "icon": "Lightbulb", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/get-started/meta.json b/content/docs/get-started/meta.json index 2d9cfb8..3f92dab 100644 --- a/content/docs/get-started/meta.json +++ b/content/docs/get-started/meta.json @@ -1,5 +1,5 @@ { "title": "Get started", "icon": "Rocket", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/guides/deployment/meta.json b/content/docs/guides/deployment/meta.json index 0bbd244..e2ffe90 100644 --- a/content/docs/guides/deployment/meta.json +++ b/content/docs/guides/deployment/meta.json @@ -1,4 +1,4 @@ { "title": "Deployment", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/guides/development/meta.json b/content/docs/guides/development/meta.json index 24f8878..203f9c9 100644 --- a/content/docs/guides/development/meta.json +++ b/content/docs/guides/development/meta.json @@ -1,4 +1,4 @@ { "title": "Development", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/guides/meta.json b/content/docs/guides/meta.json index 810cf95..498dd61 100644 --- a/content/docs/guides/meta.json +++ b/content/docs/guides/meta.json @@ -1,5 +1,5 @@ { "title": "Guides", "icon": "Wrench", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/guides/migration/meta.json b/content/docs/guides/migration/meta.json index db4e68b..941c504 100644 --- a/content/docs/guides/migration/meta.json +++ b/content/docs/guides/migration/meta.json @@ -1,4 +1,4 @@ { "title": "Data migration", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/guides/troubleshooting/meta.json b/content/docs/guides/troubleshooting/meta.json index e5715ad..82c3c83 100644 --- a/content/docs/guides/troubleshooting/meta.json +++ b/content/docs/guides/troubleshooting/meta.json @@ -1,4 +1,4 @@ { "title": "Troubleshooting", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/integrations/meta.json b/content/docs/integrations/meta.json index 62604e2..13995d5 100644 --- a/content/docs/integrations/meta.json +++ b/content/docs/integrations/meta.json @@ -1,5 +1,5 @@ { "title": "Integrations", "icon": "Blocks", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/integrations/supabase/meta.json b/content/docs/integrations/supabase/meta.json index a61c668..b4690bf 100644 --- a/content/docs/integrations/supabase/meta.json +++ b/content/docs/integrations/supabase/meta.json @@ -1,5 +1,5 @@ { "title": "Supabase", "icon": "Supabase", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/reference/auth/meta.json b/content/docs/reference/auth/meta.json index 943c560..d801d12 100644 --- a/content/docs/reference/auth/meta.json +++ b/content/docs/reference/auth/meta.json @@ -1,4 +1,4 @@ { "title": "Auth", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/reference/cli/meta.json b/content/docs/reference/cli/meta.json index 075b6da..0a67892 100644 --- a/content/docs/reference/cli/meta.json +++ b/content/docs/reference/cli/meta.json @@ -1,4 +1,4 @@ { "title": "CLI", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/reference/eql/meta.json b/content/docs/reference/eql/meta.json index be23fb4..d58bc8f 100644 --- a/content/docs/reference/eql/meta.json +++ b/content/docs/reference/eql/meta.json @@ -1,4 +1,4 @@ { "title": "EQL", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/reference/meta.json b/content/docs/reference/meta.json index 38c38b5..b74408a 100644 --- a/content/docs/reference/meta.json +++ b/content/docs/reference/meta.json @@ -1,5 +1,5 @@ { "title": "Reference", "icon": "Library", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/reference/proxy/meta.json b/content/docs/reference/proxy/meta.json index a64115c..85de4fd 100644 --- a/content/docs/reference/proxy/meta.json +++ b/content/docs/reference/proxy/meta.json @@ -1,4 +1,4 @@ { "title": "Proxy", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/reference/stack/meta.json b/content/docs/reference/stack/meta.json index 3d44235..d0f86af 100644 --- a/content/docs/reference/stack/meta.json +++ b/content/docs/reference/stack/meta.json @@ -1,4 +1,4 @@ { "title": "Stack SDK", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/reference/workspace/meta.json b/content/docs/reference/workspace/meta.json index 2c03700..1dc0214 100644 --- a/content/docs/reference/workspace/meta.json +++ b/content/docs/reference/workspace/meta.json @@ -1,4 +1,4 @@ { "title": "Workspace & account", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/security/compliance/meta.json b/content/docs/security/compliance/meta.json index 346ed94..e7c6fa5 100644 --- a/content/docs/security/compliance/meta.json +++ b/content/docs/security/compliance/meta.json @@ -1,4 +1,4 @@ { "title": "Compliance", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/security/meta.json b/content/docs/security/meta.json index 7a3eb64..5aa7273 100644 --- a/content/docs/security/meta.json +++ b/content/docs/security/meta.json @@ -1,5 +1,5 @@ { "title": "Architecture & security", "icon": "Shield", - "pages": ["index", "..."] + "pages": ["..."] } diff --git a/content/docs/solutions/meta.json b/content/docs/solutions/meta.json index 6b5983d..ac4f22b 100644 --- a/content/docs/solutions/meta.json +++ b/content/docs/solutions/meta.json @@ -1,5 +1,5 @@ { "title": "Solutions", "icon": "Target", - "pages": ["index", "..."] + "pages": ["..."] } From f5c9e81192d2aaed038128a3a7da7a7ce0686a2f Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 18:33:33 +1000 Subject: [PATCH 06/38] fix(v2): no chevron on nav items without sub-pages Folders whose only page is their index still rendered as collapsible sidebar folders with a chevron pointing at nothing. getV2PageTree() now collapses such folders into plain page items (recursively, so guides/* and reference/* leaves flatten too); a section becomes a folder again automatically when its first real sub-page lands. Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- src/app/[[...slug]]/layout.tsx | 4 ++-- src/lib/source.ts | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/src/app/[[...slug]]/layout.tsx b/src/app/[[...slug]]/layout.tsx index 719ab35..d121273 100644 --- a/src/app/[[...slug]]/layout.tsx +++ b/src/app/[[...slug]]/layout.tsx @@ -1,6 +1,6 @@ import { DocsLayout } from "fumadocs-ui/layouts/docs"; import { baseOptions } from "@/lib/layout.shared"; -import { v2source } from "@/lib/source"; +import { getV2PageTree } from "@/lib/source"; // Layout for the V2 IA tree (content/docs), served from the site root — // including the /docs landing page (content/docs/index.mdx), which renders @@ -8,7 +8,7 @@ import { v2source } from "@/lib/source"; // (/stack, /api, /og, …) take precedence over this segment as usual. export default function Layout({ children }: LayoutProps<"/[[...slug]]">) { return ( - <DocsLayout tree={v2source.getPageTree()} {...baseOptions()}> + <DocsLayout tree={getV2PageTree()} {...baseOptions()}> {children} </DocsLayout> ); diff --git a/src/lib/source.ts b/src/lib/source.ts index d0bcb1c..c056250 100644 --- a/src/lib/source.ts +++ b/src/lib/source.ts @@ -1,4 +1,5 @@ import { docs, v2docs } from "fumadocs-mdx:collections/server"; +import type * as PageTree from "fumadocs-core/page-tree"; import { type InferPageType, loader } from "fumadocs-core/source"; import { icons } from "lucide-react"; import { createElement } from "react"; @@ -32,6 +33,25 @@ export const v2source = loader({ icon: resolveIcon, }); +// Sidebar folders whose only page is their index render with a collapse +// chevron pointing at nothing. Collapse such folders into plain page items; +// they become folders again automatically once real sub-pages land. +function flattenEmptyFolders(nodes: PageTree.Node[]): PageTree.Node[] { + return nodes.map((node) => { + if (node.type !== "folder") return node; + const children = flattenEmptyFolders(node.children); + if (children.length === 0 && node.index) { + return { ...node.index, icon: node.index.icon ?? node.icon }; + } + return { ...node, children }; + }); +} + +export function getV2PageTree(): PageTree.Root { + const tree = v2source.getPageTree(); + return { ...tree, children: flattenEmptyFolders(tree.children) }; +} + export function getPageImage(page: InferPageType<typeof source>) { const segments = [...page.slugs, "image.png"]; From 85149324539c82644c0c6eaf9842afb1ad417257 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 18:40:54 +1000 Subject: [PATCH 07/38] fix(v2): make all MDX links basePath-relative MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MDX links (markdown and Card hrefs) render through the Link component, which prefixes the /docs basePath — hardcoded /docs/... links rendered as /docs/docs/... and 404'd. Convention (enforce via CIP-3349 lint): internal links in content are always basePath-relative. Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- content/docs/compare/index.mdx | 2 +- content/docs/concepts/index.mdx | 2 +- content/docs/get-started/index.mdx | 2 +- content/docs/guides/deployment/index.mdx | 2 +- content/docs/guides/development/index.mdx | 2 +- content/docs/guides/index.mdx | 2 +- content/docs/guides/migration/index.mdx | 2 +- content/docs/guides/troubleshooting/index.mdx | 2 +- content/docs/index.mdx | 24 +++++++++---------- content/docs/integrations/index.mdx | 2 +- content/docs/integrations/supabase/index.mdx | 2 +- content/docs/reference/auth/index.mdx | 2 +- content/docs/reference/cli/index.mdx | 2 +- content/docs/reference/eql/index.mdx | 2 +- content/docs/reference/index.mdx | 2 +- content/docs/reference/proxy/index.mdx | 2 +- content/docs/reference/stack/index.mdx | 2 +- content/docs/reference/workspace/index.mdx | 2 +- content/docs/security/compliance/index.mdx | 2 +- content/docs/security/index.mdx | 2 +- content/docs/solutions/index.mdx | 2 +- 21 files changed, 32 insertions(+), 32 deletions(-) diff --git a/content/docs/compare/index.mdx b/content/docs/compare/index.mdx index 8d19d4b..5c34ec1 100644 --- a/content/docs/compare/index.mdx +++ b/content/docs/compare/index.mdx @@ -6,4 +6,4 @@ type: concept This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/concepts/index.mdx b/content/docs/concepts/index.mdx index 26a565e..3d36567 100644 --- a/content/docs/concepts/index.mdx +++ b/content/docs/concepts/index.mdx @@ -6,4 +6,4 @@ type: concept This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/get-started/index.mdx b/content/docs/get-started/index.mdx index 0e08a05..a92e606 100644 --- a/content/docs/get-started/index.mdx +++ b/content/docs/get-started/index.mdx @@ -6,4 +6,4 @@ type: tutorial This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/guides/deployment/index.mdx b/content/docs/guides/deployment/index.mdx index 8fb92ca..f269e42 100644 --- a/content/docs/guides/deployment/index.mdx +++ b/content/docs/guides/deployment/index.mdx @@ -5,4 +5,4 @@ description: "Deployment documentation — being built as part of the docs V2 ov This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/guides/development/index.mdx b/content/docs/guides/development/index.mdx index a7a049f..3ac286f 100644 --- a/content/docs/guides/development/index.mdx +++ b/content/docs/guides/development/index.mdx @@ -5,4 +5,4 @@ description: "Development documentation — being built as part of the docs V2 o This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/guides/index.mdx b/content/docs/guides/index.mdx index 44308b2..8d6647e 100644 --- a/content/docs/guides/index.mdx +++ b/content/docs/guides/index.mdx @@ -6,4 +6,4 @@ type: guide This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/guides/migration/index.mdx b/content/docs/guides/migration/index.mdx index 97ac0dc..cd728e5 100644 --- a/content/docs/guides/migration/index.mdx +++ b/content/docs/guides/migration/index.mdx @@ -5,4 +5,4 @@ description: "Data migration documentation — being built as part of the docs V This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/guides/troubleshooting/index.mdx b/content/docs/guides/troubleshooting/index.mdx index 7354565..d049ef4 100644 --- a/content/docs/guides/troubleshooting/index.mdx +++ b/content/docs/guides/troubleshooting/index.mdx @@ -5,4 +5,4 @@ description: "Troubleshooting documentation — being built as part of the docs This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/index.mdx b/content/docs/index.mdx index e46f40e..a2f2d22 100644 --- a/content/docs/index.mdx +++ b/content/docs/index.mdx @@ -14,25 +14,25 @@ with no key. ## Start here <Cards> - <Card title="Get started" href="/docs/get-started" description="What CipherStash is and your first encrypted fields in 10 minutes." /> - <Card title="Quickstart" href="/docs/stack/quickstart" description="Encrypt, store, query, and decrypt your first fields in any Postgres." /> - <Card title="Supabase" href="/docs/integrations/supabase" description="Searchable, application-level encryption for your Supabase project." /> - <Card title="Agent skills" href="/docs/stack/reference/agent-skills" description="CipherStash knowledge for Cursor, Copilot, and Claude Code." /> + <Card title="Get started" href="/get-started" description="What CipherStash is and your first encrypted fields in 10 minutes." /> + <Card title="Quickstart" href="/stack/quickstart" description="Encrypt, store, query, and decrypt your first fields in any Postgres." /> + <Card title="Supabase" href="/integrations/supabase" description="Searchable, application-level encryption for your Supabase project." /> + <Card title="Agent skills" href="/stack/reference/agent-skills" description="CipherStash knowledge for Cursor, Copilot, and Claude Code." /> </Cards> ## Browse the docs <Cards> - <Card title="Integrations" href="/docs/integrations" description="Platforms, ORMs, frameworks, auth providers, and runtimes." /> - <Card title="Concepts" href="/docs/concepts" description="How searchable encryption, key management, and identity-aware encryption work." /> - <Card title="Guides" href="/docs/guides" description="Development workflow, data migration, deployment, and troubleshooting." /> - <Card title="Architecture & security" href="/docs/security" description="Trust model, components, availability, audit, and compliance — for security review." /> - <Card title="Solutions" href="/docs/solutions" description="PII protection, HIPAA, AI/RAG, data residency, and provable access." /> - <Card title="Reference" href="/docs/reference" description="EQL, the Stack SDK, Auth, the CLI, and Proxy — precise API documentation." /> + <Card title="Integrations" href="/integrations" description="Platforms, ORMs, frameworks, auth providers, and runtimes." /> + <Card title="Concepts" href="/concepts" description="How searchable encryption, key management, and identity-aware encryption work." /> + <Card title="Guides" href="/guides" description="Development workflow, data migration, deployment, and troubleshooting." /> + <Card title="Architecture & security" href="/security" description="Trust model, components, availability, audit, and compliance — for security review." /> + <Card title="Solutions" href="/solutions" description="PII protection, HIPAA, AI/RAG, data residency, and provable access." /> + <Card title="Reference" href="/reference" description="EQL, the Stack SDK, Auth, the CLI, and Proxy — precise API documentation." /> </Cards> ## AI-ready documentation Every page is available as clean markdown: append `.mdx` to any page URL, or -fetch the whole corpus via [llms.txt](/docs/llms.txt) and -[llms-full.txt](/docs/llms-full.txt). +fetch the whole corpus via [llms.txt](/llms.txt) and +[llms-full.txt](/llms-full.txt). diff --git a/content/docs/integrations/index.mdx b/content/docs/integrations/index.mdx index 6d6e1bb..6148388 100644 --- a/content/docs/integrations/index.mdx +++ b/content/docs/integrations/index.mdx @@ -6,4 +6,4 @@ type: tutorial This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/integrations/supabase/index.mdx b/content/docs/integrations/supabase/index.mdx index 6330cf9..86e2ef6 100644 --- a/content/docs/integrations/supabase/index.mdx +++ b/content/docs/integrations/supabase/index.mdx @@ -17,4 +17,4 @@ and stay queryable with the same Supabase.js calls you already use. This page is being rebuilt as part of the docs V2 overhaul ([CIP-3328](https://linear.app/cipherstash/issue/CIP-3328)). Until it lands, the current Supabase integration guide lives at -[CipherStash + Supabase](/docs/stack/cipherstash/supabase). +[CipherStash + Supabase](/stack/cipherstash/supabase). diff --git a/content/docs/reference/auth/index.mdx b/content/docs/reference/auth/index.mdx index 1c05374..9bea074 100644 --- a/content/docs/reference/auth/index.mdx +++ b/content/docs/reference/auth/index.mdx @@ -5,4 +5,4 @@ description: "Auth documentation — being built as part of the docs V2 overhaul This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/reference/cli/index.mdx b/content/docs/reference/cli/index.mdx index 2897d5c..8eeffd3 100644 --- a/content/docs/reference/cli/index.mdx +++ b/content/docs/reference/cli/index.mdx @@ -5,4 +5,4 @@ description: "CLI documentation — being built as part of the docs V2 overhaul. This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index 276450a..551c51e 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -5,4 +5,4 @@ description: "EQL documentation — being built as part of the docs V2 overhaul. This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/reference/index.mdx b/content/docs/reference/index.mdx index 287825a..997dff4 100644 --- a/content/docs/reference/index.mdx +++ b/content/docs/reference/index.mdx @@ -6,4 +6,4 @@ type: reference This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/reference/proxy/index.mdx b/content/docs/reference/proxy/index.mdx index d3d231e..8e59184 100644 --- a/content/docs/reference/proxy/index.mdx +++ b/content/docs/reference/proxy/index.mdx @@ -5,4 +5,4 @@ description: "Proxy documentation — being built as part of the docs V2 overhau This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/reference/stack/index.mdx b/content/docs/reference/stack/index.mdx index 6d79f4c..edac1c3 100644 --- a/content/docs/reference/stack/index.mdx +++ b/content/docs/reference/stack/index.mdx @@ -5,4 +5,4 @@ description: "Stack SDK documentation — being built as part of the docs V2 ove This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/reference/workspace/index.mdx b/content/docs/reference/workspace/index.mdx index e97a9a6..177bb82 100644 --- a/content/docs/reference/workspace/index.mdx +++ b/content/docs/reference/workspace/index.mdx @@ -5,4 +5,4 @@ description: "Workspace & account documentation — being built as part of the d This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/security/compliance/index.mdx b/content/docs/security/compliance/index.mdx index 70c433a..9af190d 100644 --- a/content/docs/security/compliance/index.mdx +++ b/content/docs/security/compliance/index.mdx @@ -5,4 +5,4 @@ description: "Compliance documentation — being built as part of the docs V2 ov This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/security/index.mdx b/content/docs/security/index.mdx index da86585..33b00fe 100644 --- a/content/docs/security/index.mdx +++ b/content/docs/security/index.mdx @@ -6,4 +6,4 @@ type: concept This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). diff --git a/content/docs/solutions/index.mdx b/content/docs/solutions/index.mdx index 350020d..36278b5 100644 --- a/content/docs/solutions/index.mdx +++ b/content/docs/solutions/index.mdx @@ -6,4 +6,4 @@ type: concept This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). -Until it lands, current documentation lives in the [existing docs](/docs/stack). +Until it lands, current documentation lives in the [existing docs](/stack). From d415a48032f46fcb002cbbb52ed788a198b1de46 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 18:55:57 +1000 Subject: [PATCH 08/38] feat(v2): EQL v3 reference section (CIP-3326) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven pages replacing the v2-era EQL reference, written against the eql_v3 branch of cipherstash/encrypt-query-language (3.0.0): - index: what EQL is, the v3 domain-variant model, install (single SQL script, idempotent), dbdev, Docker, migration/runtime permission split, managed-Postgres rationale - types: 10 scalar families × variants matrix; bool storage-only; _ord/_ord_ore twins; index terms per variant - operators: per-variant support matrix, typed-operand rule, no-LIKE, fail-loud blockers, query shapes, function-form equivalents - indexes: functional indexes on term extractors, engagement requirements, sort-key form for index-streamed ORDER BY, EXPLAIN checklist, large-table build guidance - json: ste_vec model, per-node-type terms (hm XOR oc), containment + GIN, field access, path queries, blocked native jsonb operators - functions: comparisons, extractors, min/max only (no SUM/AVG), version() - payload-format: v/i/c envelope (wire version still v:2), hm/ob/bf term keys, sv document shape, annotated examples (absorbs the legacy CipherCell page) Cross-page consistency verified against the shipped SQL: equality on _ord variants compares ORE terms (no hm in _ord payloads), and bare ORDER BY is correct but extractor-form sort keys stream from the index. Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- IA.md | 14 +- content/docs/reference/eql/functions.mdx | 112 +++++++++ content/docs/reference/eql/index.mdx | 135 ++++++++++- content/docs/reference/eql/indexes.mdx | 182 ++++++++++++++ content/docs/reference/eql/json.mdx | 228 ++++++++++++++++++ content/docs/reference/eql/meta.json | 9 +- content/docs/reference/eql/operators.mdx | 153 ++++++++++++ content/docs/reference/eql/payload-format.mdx | 123 ++++++++++ content/docs/reference/eql/types.mdx | 95 ++++++++ 9 files changed, 1040 insertions(+), 11 deletions(-) create mode 100644 content/docs/reference/eql/functions.mdx create mode 100644 content/docs/reference/eql/indexes.mdx create mode 100644 content/docs/reference/eql/json.mdx create mode 100644 content/docs/reference/eql/operators.mdx create mode 100644 content/docs/reference/eql/payload-format.mdx create mode 100644 content/docs/reference/eql/types.mdx diff --git a/IA.md b/IA.md index f9c6f9e..b51956c 100644 --- a/IA.md +++ b/IA.md @@ -131,13 +131,13 @@ live at `/docs/errors/<code>` — permanent, never restructured (CIP-3338). - [x] Section scaffold 🚧 (eql, stack, auth, cli, proxy, workspace) - **EQL (v3 rewrite — CIP-3326):** -- [ ] `/reference/eql` — overview + install (single SQL file, permissions split, dbdev, Docker) -- [ ] `/reference/eql/types` — 10 scalar families × variants + `eql_v3.json` -- [ ] `/reference/eql/operators` — per-variant matrix incl. what RAISES; typed-operand rule -- [ ] `/reference/eql/indexes` — functional indexes on extractors; Supabase-compatible -- [ ] `/reference/eql/json` — ste_vec, path queries -- [ ] `/reference/eql/functions` — incl. aggregates (min/max only) -- [ ] `/reference/eql/payload-format` — v/i/c envelope, hm/ob/bf (absorbs cipher-cell) +- [x] `/reference/eql` — overview + install (single SQL file, permissions split, dbdev, Docker) +- [x] `/reference/eql/types` — 10 scalar families × variants + `eql_v3.json` +- [x] `/reference/eql/operators` — per-variant matrix incl. what RAISES; typed-operand rule +- [x] `/reference/eql/indexes` — functional indexes on extractors; Supabase-compatible +- [x] `/reference/eql/json` — ste_vec, path queries +- [x] `/reference/eql/functions` — incl. aggregates (min/max only) +- [x] `/reference/eql/payload-format` — v/i/c envelope, hm/ob/bf (absorbs cipher-cell) - **Stack SDK:** - [ ] `/reference/stack` — client + configuration (port encryption/* pages) - [ ] `/reference/stack/schema` diff --git a/content/docs/reference/eql/functions.mdx b/content/docs/reference/eql/functions.mdx new file mode 100644 index 0000000..210ca31 --- /dev/null +++ b/content/docs/reference/eql/functions.mdx @@ -0,0 +1,112 @@ +--- +title: Functions +description: "The eql_v3 function surface: comparison functions, index-term extractors, MIN/MAX aggregates, JSON functions, and version reporting." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Everything EQL exposes lives in the `eql_v3` schema. Most functions are generated per [domain variant](/reference/eql/types), so PostgreSQL's overload resolution picks the right implementation from the argument type. As with operators, arguments must be typed — see [the typed-operand rule](/reference/eql/operators). + +## Comparison functions + +Function forms of the comparison operators, for platforms that disallow custom operators. Each is generated per capable domain variant, with overloads accepting the domain on either side and `jsonb` on the other: + +```sql +eql_v3.eq(a, b) RETURNS boolean -- = on _eq / _ord / _ord_ore / text_search +eql_v3.neq(a, b) RETURNS boolean -- <> +eql_v3.lt(a, b) RETURNS boolean -- < on _ord / _ord_ore / text_search +eql_v3.lte(a, b) RETURNS boolean -- <= +eql_v3.gt(a, b) RETURNS boolean -- > +eql_v3.gte(a, b) RETURNS boolean -- >= +eql_v3.contains(a, b) RETURNS boolean -- @> on text_match / text_search / eql_v3.json +eql_v3.contained_by(a, b) RETURNS boolean -- <@ +``` + +```sql +SELECT * FROM users WHERE eql_v3.eq(email, $1::eql_v3.text_eq); +SELECT * FROM users WHERE eql_v3.lt(created_at, $1::eql_v3.timestamp_ord); +``` + +Calling a comparison function a variant doesn't support resolves to a blocker that raises `operator … is not supported` — the same [fail-loud behavior](/reference/eql/operators) as the operators. There are no `like` / `ilike` functions: text matching is `eql_v3.contains` on a `text_match` value. + +## Index-term extractors + +These extract the encrypted index term from a domain value. They're generated per eq-, ord-, and match-capable variant of every scalar type, and they return the self-contained `eql_v3` index-term types: + +```sql +-- Equality term (hm) +eql_v3.eq_term(a eql_v3.<T>_eq) RETURNS eql_v3.hmac_256 + +-- Ordering term (ob) +eql_v3.ord_term(a eql_v3.<T>_ord) RETURNS eql_v3.ore_block_256 +eql_v3.ord_term(a eql_v3.<T>_ord_ore) RETURNS eql_v3.ore_block_256 + +-- Text-match term (bf) +eql_v3.match_term(a eql_v3.text_match) RETURNS eql_v3.bloom_filter +``` + +`eql_v3.text_search` carries all three terms, so all three extractors work on it. + +The extractors exist for **indexing**: EQL indexes through a functional index on the extractor, never an operator class on the column. The extractors are inlinable, so bare-form predicates (`WHERE email = $1`) engage the index without rewriting. Sort keys are the exception — see [Range and ORDER BY](/reference/eql/indexes#range-and-order-by): + +```sql +CREATE INDEX users_email_eq ON users USING hash (eql_v3.eq_term(email)); +CREATE INDEX users_salary_ord ON users USING btree (eql_v3.ord_term(salary)); +CREATE INDEX users_name_match ON users USING gin (eql_v3.match_term(name)); +``` + +See [Indexes](/reference/eql/indexes) for the full recipes and performance guidance. + +## Aggregates: `eql_v3.min` and `eql_v3.max` + +`MIN` / `MAX` over encrypted values, defined per ord-capable variant of every scalar type. The input type selects the aggregate; the return type matches the input: + +```sql +eql_v3.min(eql_v3.<T>_ord) RETURNS eql_v3.<T>_ord +eql_v3.max(eql_v3.<T>_ord) RETURNS eql_v3.<T>_ord +eql_v3.min(eql_v3.<T>_ord_ore) RETURNS eql_v3.<T>_ord_ore +eql_v3.max(eql_v3.<T>_ord_ore) RETURNS eql_v3.<T>_ord_ore +``` + +Comparison routes through the variant's `<` / `>` operator, which uses the ORE block term — no decryption happens in the database. `NULL` inputs are skipped, and an all-`NULL` input set returns `NULL`. + +```sql +SELECT eql_v3.min(salary) FROM users; +SELECT eql_v3.max(salary) FROM users WHERE department = 'engineering'; + +-- On a generic jsonb column, cast to the right domain at the call site +SELECT eql_v3.min(salary_jsonb::eql_v3.int8_ord) FROM users; +``` + +<Callout type="warn"> +**`SUM`, `AVG`, and other arithmetic aggregates are not supported** on encrypted columns — they would require homomorphic encryption. `MIN` / `MAX` work because they only need comparison. For sums and averages, decrypt at the application boundary and aggregate client-side. +</Callout> + +## JSON functions + +The encrypted-JSON document type `eql_v3.json` has its own function surface: + +- `eql_v3.jsonb_path_query(doc, selector)` — set-returning path query yielding encrypted entries; also `jsonb_path_query_first` and `jsonb_path_exists` +- `eql_v3.jsonb_array_length` / `jsonb_array_elements` / `jsonb_array_elements_text` — array helpers +- `eql_v3.to_ste_vec_query(doc)` — builds the GIN-indexable containment query form +- Entry-level term extractors: `eql_v3.eq_term(eql_v3.ste_vec_entry)` and `eql_v3.ore_cllw(eql_v3.ste_vec_entry)` + +These are documented with worked examples in [JSON support](/reference/eql/json). + +## `eql_v3.version()` + +Returns the installed EQL version string, baked in at build time: + +```sql +SELECT eql_v3.version(); +-- '3.0.0' +``` + +The same version string is mirrored as a comment on the `eql_v3` schema, so you can read it without calling a function: + +```sql +SELECT obj_description('eql_v3'::regnamespace); +-- '3.0.0' +``` diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index 551c51e..68b95c2 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -1,8 +1,137 @@ --- title: EQL -description: "EQL documentation — being built as part of the docs V2 overhaul." +description: "Encrypt Query Language (EQL) installs encrypted column types and operators into Postgres as plain SQL — encryption itself happens in your client." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" --- -This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). +Encrypt Query Language (EQL) is a set of types, operators, and functions for storing and querying encrypted data in PostgreSQL. It installs as a single plain-SQL script — no extension packaging, no superuser, no operator classes — so it runs on Supabase, RDS, Cloud SQL, and self-hosted Postgres alike. -Until it lands, current documentation lives in the [existing docs](/stack). +EQL itself never encrypts anything. Encryption and decryption happen in the client, using the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). EQL provides the database-side surface those clients query against: encrypted column types, the operators that compare them, and the term-extractor functions that make indexes work. + +## The v3 model + +Every encrypted column is a `jsonb`-backed **domain type** in the `eql_v3` schema. The domain variant you choose declares the column's searchable capability: `eql_v3.text_eq` supports equality (`=` / `<>`), `eql_v3.text_match` supports encrypted text containment (`@>` / `<@`), `eql_v3.int4_ord` adds range comparisons, `ORDER BY`, and `MIN` / `MAX`. Each domain carries a `CHECK` constraint that validates the encrypted payload on insert, so a malformed or wrong-version value is rejected at write time rather than surfacing at query time. + +There is no database-side configuration table. Earlier EQL versions tracked encryption config in the database (`config_add_table`, `config_add_column`, and friends) — those are gone in v3. The searchable surface of a column is fixed by the domain variant you type it as, and which index terms travel in a value's payload is decided by the encryption client. Operators that a variant doesn't support raise an "operator not supported" error rather than silently falling through to native `jsonb` semantics — and `LIKE` / `ILIKE` are blocked on every encrypted column. + +```sql +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email eql_v3.text_eq, -- equality only + salary eql_v3.int4_ord, -- equality + range + ORDER BY + created_at eql_v3.timestamp_ord +); +``` + +## Install + +<Steps> +<Step> + +### Download the install script + +Each [GitHub release](https://github.com/cipherstash/encrypt-query-language/releases) publishes a versioned `cipherstash-encrypt.sql`: + +```sh +curl -sLo cipherstash-encrypt.sql https://github.com/cipherstash/encrypt-query-language/releases/latest/download/cipherstash-encrypt.sql +``` + +</Step> +<Step> + +### Run it against each database + +```sh +psql -f cipherstash-encrypt.sql +``` + +The script installs the `eql_v3` schema with all domain types, operators, functions, and aggregates. It is idempotent: re-running it upgrades the `eql_v3` surface in place and won't remove anything you've built on top of it. To upgrade, download the latest script and run it again. + +</Step> +<Step> + +### Verify + +```sql +SELECT eql_v3.version(); +-- '3.0.0' +``` + +</Step> +</Steps> + +<Callout type="warn"> +`DROP SCHEMA eql_v3 CASCADE` drops every column typed as an `eql_v3` domain. The domain types live in the schema, and your columns depend on them. +</Callout> + +### dbdev + +EQL is also published to [dbdev](https://database.dev/cipherstash/eql). The dbdev release can lag behind GitHub releases, so prefer the install script when you need the latest version. + +### Docker for local development + +Run a Postgres image with EQL pre-installed: + +```sh +docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=postgres \ + ghcr.io/cipherstash/postgres-eql:17 +``` + +EQL installs automatically on first boot. Images are available for PostgreSQL 14–17 (`:14` through `:17`), and you can pin a specific EQL version with a suffixed tag (for example `:17-3.0.0`). + +## Permissions + +Installing EQL and running queries against it need different privileges. A common production pattern splits them across two users. + +**Migration user** — installs EQL and adds encrypted columns during migrations: + +```sql +GRANT CREATE ON DATABASE your_database TO your_migration_user; +GRANT CREATE ON SCHEMA public TO your_migration_user; +GRANT ALTER ON ALL TABLES IN SCHEMA public TO your_migration_user; +``` + +`CREATE ON DATABASE` creates the `eql_v3` schema and its types; `CREATE ON SCHEMA` and `ALTER` are needed to add encrypted columns (typed as `eql_v3` domains, with their `CHECK` constraints) to your tables. + +**Runtime user** — the application's day-to-day access: + +```sql +-- EQL schema usage (resolves the encrypted operators / extractors) +GRANT USAGE ON SCHEMA eql_v3 TO your_app_user; +GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA eql_v3 TO your_app_user; + +-- User table access (normal application permissions) +GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE your_tables TO your_app_user; +``` + +Schema changes — adding or removing encrypted columns — always go through the migration user. + +## Managed Postgres and Supabase + +EQL v3 is designed to install without superuser. There are no custom operator classes (which managed platforms typically block), no `postgresql.conf` changes, and no separate Supabase build — the single install script is the same artefact everywhere. Indexing works through ordinary functional indexes over EQL's term-extractor functions, which any user who can `CREATE INDEX` can build. See the [Supabase integration](/integrations/supabase) for platform-specific setup. + +## In this section + +<Cards> + <Card title="Types" href="/reference/eql/types"> + The encrypted domain type families and the capability each variant carries. + </Card> + <Card title="Operators" href="/reference/eql/operators"> + Which SQL operators resolve on which variant, and what raises. + </Card> + <Card title="Indexes" href="/reference/eql/indexes"> + Functional-index recipes for equality, range, and text match. + </Card> + <Card title="JSON" href="/reference/eql/json"> + Encrypted JSON documents: containment, field access, and GIN indexing. + </Card> + <Card title="Functions" href="/reference/eql/functions"> + The function equivalents of every operator, extractors, and aggregates. + </Card> + <Card title="Payload format" href="/reference/eql/payload-format"> + The encrypted payload envelope and index terms. + </Card> +</Cards> diff --git a/content/docs/reference/eql/indexes.mdx b/content/docs/reference/eql/indexes.mdx new file mode 100644 index 0000000..83f354e --- /dev/null +++ b/content/docs/reference/eql/indexes.mdx @@ -0,0 +1,182 @@ +--- +title: Indexes +description: "Create Postgres indexes on encrypted columns using functional indexes over EQL's term-extractor functions." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +EQL indexes are ordinary PostgreSQL functional indexes over **term-extractor functions** — never an index or operator class on the column itself. Each extractor returns a small per-row index term whose return type already carries a default operator class: + +| Extractor | Index method | Term | Capability | +| --- | --- | --- | --- | +| `eql_v3.eq_term(col)` | `hash` (or `btree`) | `hm` (HMAC-256) | equality | +| `eql_v3.ord_term(col)` | `btree` | `ob` (ORE block) | range, `ORDER BY`, `MIN` / `MAX` | +| `eql_v3.match_term(col)` | `gin` | `bf` (bloom filter) | text containment | + +The extractors are inlinable SQL functions, so the planner rewrites a bare-form predicate into the same expression the index was built on. You don't rewrite queries to use the index: + +```sql +SELECT * FROM users WHERE email = $1::eql_v3.text_eq; +-- planner inlines `=` to: eql_v3.eq_term(email) = eql_v3.eq_term($1) +-- Index Cond on USING hash (eql_v3.eq_term(email)) +``` + +<Callout type="warn"> +EQL v3 deliberately ships no operator class for encrypted columns. Operators resolve against the domain's `jsonb` base type, so an opclass on the column would bypass the encrypted surface. Always index through the extractor. +</Callout> + +## Index recipes + +Type the column as the domain variant that carries the term ([Types](/reference/eql/types)), then index the matching extractor: + +```sql +-- Equality: hash index on eq_term +-- (columns typed eql_v3.<T>_eq or text_search; equality on _ord columns +-- compares ORE terms, so the btree on ord_term below serves it) +CREATE INDEX users_email_eq + ON users USING hash (eql_v3.eq_term(email)); + +-- Range / ordering: btree index on ord_term +-- (columns typed eql_v3.<T>_ord or _ord_ore) +CREATE INDEX users_created_at_ord + ON users USING btree (eql_v3.ord_term(created_at)); + +-- Text match: GIN index on match_term +-- (columns typed eql_v3.text_match or text_search) +CREATE INDEX users_name_match + ON users USING gin (eql_v3.match_term(name)); + +ANALYZE users; +``` + +Run `ANALYZE` after every index build. `CREATE INDEX` on an expression gathers no statistics for that expression — without `ANALYZE`, the planner has no histogram for `eql_v3.eq_term(email)` and can misjudge the index it just built. + +Create indexes when the table has a significant number of rows (typically more than 1,000) and you query the column with the matching operator. Drop indexes for capabilities you no longer query — duplicate indexes compete for cache and slow writes. + +## Requirements for an index to engage + +All three must hold: + +1. **The value carries the required term.** Equality needs `hm`, range needs `ob`, containment needs `bf`. Which terms travel in a value's payload is decided by the encryption client — a value with only a bloom term will not drive an equality index. +2. **The index was built after the data carried the term.** If you change which terms a column's values carry, recreate the index. +3. **The query operand is typed.** A typed parameter (`$1`, which CipherStash Proxy supplies) or an explicit cast resolves the encrypted operator; a bare `jsonb` literal falls through to native `jsonb` semantics and skips the index entirely: + +```sql +-- ✓ resolves the encrypted operator → uses the index +WHERE email = $1; +WHERE email = $1::eql_v3.text_eq; + +-- ✗ falls through to native jsonb semantics +WHERE email = '{"hm":"abc"}'::jsonb; +``` + +## Query shapes + +### Equality + +```sql +SELECT * FROM users WHERE email = $1; +-- Index Scan using users_email_eq +-- Index Cond: (eql_v3.eq_term(email) = eql_v3.eq_term($1)) +``` + +### Range and ORDER BY + +The `<`, `<=`, `>`, `>=` operators inline to comparisons on `eql_v3.ord_term`, so natural-form range predicates match the btree: + +```sql +SELECT * FROM users WHERE created_at < $1; +``` + +`ORDER BY` needs care. The planner inlines operators in *predicates* but does not rewrite *sort keys*: `ORDER BY created_at` uses the index for the `WHERE` clause but still adds a `Sort` node, which scales linearly with the rows passing the filter. To stream rows out of the btree already ordered, write the sort key in extractor form: + +```sql +SELECT * FROM users + WHERE created_at < $1 + ORDER BY eql_v3.ord_term(created_at) DESC + LIMIT 10; +``` + +ORE terms are order-preserving, so this sorts identically to the natural form — it just lets the index do the ordering. At large row counts this is the difference between seconds and milliseconds. + +<Callout type="warn"> +If you `SELECT col::jsonb ... ORDER BY col`, Postgres folds the cast into the scan and uses `(col)::jsonb` as the sort key — which matches no index. Project the column raw, or write the sort key as `eql_v3.ord_term(col)`, which sidesteps this entirely. +</Callout> + +### GROUP BY and DISTINCT + +Group on the extractor, not the raw column: + +```sql +SELECT eql_v3.eq_term(email), count(*) + FROM users + GROUP BY eql_v3.eq_term(email); +``` + +`GROUP BY email` uses the entire encrypted payload (1–2 KB per row) as the hash key; Postgres estimates a hash table far larger than the default `work_mem` and falls back to a disk-spilling `GroupAggregate`. The extractor key is a small deterministic term, so the hash table fits in `work_mem` and the planner picks `HashAggregate` reliably. If an ORM forces the raw-column form, raising `work_mem` is the rescue knob — but the extractor form is the design. + +## Encrypted JSON + +Containment (`@>` / `<@`) on `eql_v3.json` document columns uses a GIN index over `eql_v3.to_ste_vec_query(col)::jsonb`, and field-level equality and ordering have their own extractor recipes. See [JSON](/reference/eql/json). + +## Verify with EXPLAIN + +The first move on a slow query is `EXPLAIN (COSTS OFF)`: + +- **`Index Scan using <your-index>`** — the functional index is engaged. +- **`Index Cond:` referencing the extractor** (`eql_v3.eq_term(...)`, `eql_v3.ord_term(...)`) — the inlined predicate matched the index. +- **`Seq Scan`** — no index used. Check the three requirements above. +- **`Filter:` showing the raw operator** — inlining did not happen. Usual causes: a pinned `search_path` on a customised function, or the planner judging another plan cheaper. +- **`Sort` node above an Index Scan** — natural-form `ORDER BY`. Switch the sort key to `eql_v3.ord_term(col)` to eliminate it. + +Once the plan looks right, repeat with `EXPLAIN ANALYZE` to measure actual timings. For a full diagnosis walkthrough, see [query performance troubleshooting](/guides/troubleshooting/query-performance). + +## Building indexes on large tables + +Index *build* time is a separate axis from query time — a functional index that queries in a millisecond can take hours to `CREATE` on a large table. + +**Raise `maintenance_work_mem`.** `CREATE INDEX` draws on `maintenance_work_mem` (default 64 MB — far too small for a multi-million-row build). It's the single highest-leverage knob: + +```sql +SET maintenance_work_mem = '2GB'; +CREATE INDEX users_email_eq ON users USING btree (eql_v3.eq_term(email)); +``` + +**Prefer `btree` over `hash` for equality on large tables.** A btree build sorts then bulk-loads with sequential writes and can parallelise; a hash build scatters rows to random buckets and degrades to random I/O once the index outgrows cache — it cannot parallelise. A btree on `eql_v3.eq_term(col)` serves `=` exactly as well as a hash index, with no query-side cost. Hash is fine up to mid-six-figure row counts. + +**Expect a de-TOAST floor.** A functional index over a large encrypted column de-TOASTs the whole stored value once per row to evaluate the extractor. This cost is identical across access methods and sets the build's floor rate. Index builds are also I/O-heavy in a way queries are not — containerised Postgres on a virtualised filesystem (Docker Desktop on macOS, notably) pays a steep penalty, so run large builds on native storage. + +**Watch the build.** From a second session while `CREATE INDEX` runs: + +```sql +SELECT phase, tuples_done, tuples_total, + round(100.0 * tuples_done / nullif(tuples_total, 0), 1) AS pct +FROM pg_stat_progress_create_index; +``` + +A steady `tuples_done` rate is healthy. A rate that decays over time is the cache/memory wall — raise `maintenance_work_mem`, and if it's a hash index, rebuild it as a btree. + +## Why this works on managed Postgres + +Everything above is a functional index over an `IMMUTABLE` SQL function — no operator class on a column, no superuser, no `postgresql.conf` changes. Managed platforms that block custom operator classes (Supabase among them) run these recipes unchanged, so the indexing model is identical on Supabase, RDS, Cloud SQL, and self-hosted Postgres. See the [Supabase integration](/integrations/supabase). + +## Troubleshooting + +**Index not being used:** + +1. Verify the value carries the term: + + ```sql + SELECT email::jsonb ? 'hm' AS has_hmac, + email::jsonb ? 'ob' AS has_ore_block, + email::jsonb ? 'bf' AS has_bloom + FROM users LIMIT 1; + ``` + +2. Verify the operand is typed (`$1::eql_v3.text_eq`, not `$1::jsonb`). +3. Recreate the index if the column's terms changed after it was built. +4. Run `ANALYZE`. Very small tables may still choose a sequential scan — that's correct. + +**`=` returns zero rows on a populated column:** equality requires the term its variant compares — `hm` on `_eq` / `text_search`, `ob` on `_ord` variants. Type the column as an equality-capable variant and confirm the encryption client is emitting that term. diff --git a/content/docs/reference/eql/json.mdx b/content/docs/reference/eql/json.mdx new file mode 100644 index 0000000..6ef2e60 --- /dev/null +++ b/content/docs/reference/eql/json.mdx @@ -0,0 +1,228 @@ +--- +title: Encrypted JSON +description: "Store and query encrypted JSON documents with eql_v3.json — containment, field access, and path queries over ciphertext, with the native jsonb operators that don't apply blocked outright." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +`eql_v3.json` is EQL's encrypted JSON document type, built on structured encryption (**ste_vec**). The document is encrypted as a vector of encrypted entries — one entry per path inside the document — and every path is queryable without decryption: containment, field and array access, and equality or range comparisons on extracted leaves. + +Like every EQL type, `eql_v3.json` holds ciphertext the database can't read. Encryption, decryption, and selector generation happen in the client — the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). See [Searchable encryption](/concepts/searchable-encryption) for how querying ciphertext works at all. + +## The types + +Three `jsonb`-backed domains make up the encrypted JSON surface: + +| Type | What it is | +| --- | --- | +| `eql_v3.json` | The column type. An encrypted document envelope carrying an `sv` array — one encrypted entry per path in the document. | +| `eql_v3.ste_vec_entry` | A single entry from the vector: a selector, a ciphertext, and exactly one index term. This is what `->` returns. | +| `eql_v3.ste_vec_query` | A containment needle: entries with selectors and index terms but **no ciphertext**. This is what you cast a `@>` operand to. | + +The full wire shape of each is documented in [Payload format](/reference/eql/payload-format). + +## Storing encrypted JSON + +Type the column as `eql_v3.json`: + +```sql +CREATE TABLE orders ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + metadata eql_v3.json +); +``` + +There is no database-side configuration step. Which index terms a document carries is decided by the encryption client; typing the column as `eql_v3.json` is what makes the encrypted operators and functions resolve. The domain's `CHECK` constraint validates the payload shape on insert, so malformed values are rejected at write time. + +Insert and read through the Stack SDK or Proxy, which encrypt the document into the ste_vec payload on write and decrypt it on read. + +## What each node type supports + +During encryption, the client flattens the document: each unique path gets a deterministic **selector** hash, and each node gets an entry in the `sv` vector carrying index terms for its JSON type: + +| JSON node type | Index term | Equality (`=`, `<>`, `GROUP BY`) | Ordering (`<` … `>=`, `MIN`/`MAX`) | +| --- | --- | --- | --- | +| Object | `hm` (HMAC-256) | Yes | No | +| Array | `hm` (HMAC-256) | Yes | No | +| Boolean / JSON `null` | `hm` (HMAC-256) | Yes | No | +| String | `oc` (CLLW ORE, string domain) | Yes | Yes | +| Number | `oc` (CLLW ORE, numeric domain) | Yes | Yes | + +Each entry carries exactly one of `hm` or `oc` — the domain `CHECK` enforces the exclusivity. `hm` is a deterministic hash, so it supports equality only. `oc` is a CLLW ORE term that reveals ordering and, being deterministic, collapses to equality on matching selectors — `eql_v3.eq_term` reads whichever term an entry carries, so equality works uniformly across all node types. Earlier payload versions split the ORE term into `ocf` (fixed-width, numeric) and `ocv` (variable-width, string); current payloads emit a single `oc` whose leading domain-tag byte carries the numeric/string distinction. + +JSON `null` here means a `null` literal *inside* the document. A SQL `NULL` column value is not encrypted at all. + +## Blocked native jsonb operators + +These native PostgreSQL `jsonb` operators are **blocked** on `eql_v3.json`. They raise an error rather than silently running plaintext-jsonb semantics against the encrypted payload: + +- Key/path existence: `?`, `?|`, `?&`, `@?`, `@@` +- Path extraction: `#>`, `#>>` +- Mutation: `-`, `#-`, `||` +- Root-document comparison: `=`, `<>`, `<`, `<=`, `>`, `>=` + +Use containment (`@>` / `<@`), field access (`->` / `->>`), or the `eql_v3.jsonb_path_*` functions instead. There is no server-side mutation of an encrypted document — updates re-encrypt in the client. + +<Callout type="warn"> +**Type your operands.** `eql_v3.json` is a domain over `jsonb`, and PostgreSQL resolves `domain OP untyped_literal` to the **native** `jsonb` operator — bypassing both the encrypted operator and the blockers. `WHERE doc -> 'email'` silently runs native `jsonb ->` and returns `NULL`; `WHERE doc -> 'email'::text` resolves the encrypted operator. This is the same rule as the [scalar operators](/reference/eql/operators). Queries through CipherStash Proxy always bind typed parameters, so this only bites hand-written ad-hoc SQL. +</Callout> + +## Containment: `@>` and `<@` + +`@>` tests whether the encrypted document contains a structure; `<@` is the reverse. Build the needle with the client and cast it to `eql_v3.ste_vec_query` (a typed `eql_v3.json` or `eql_v3.ste_vec_entry` operand also works): + +```sql +SELECT * FROM orders +WHERE metadata @> $1::eql_v3.ste_vec_query; +``` + +This is the encrypted equivalent of the plaintext `metadata @> '{"customer": {"tier": "premium"}}'`: containment checks that every encrypted term in the needle exists in the document's `sv` vector. `eql_v3.to_ste_vec_query(doc)` converts a stored document into the needle shape, and `eql_v3.ste_vec_contains(a, b)` is the function form backing `@>`. + +For large tables, back containment with a GIN index. The typed `@>` overload inlines to a native `jsonb @>` over `eql_v3.to_ste_vec_query(col)::jsonb`, so a GIN index on that same expression engages: + +```sql +CREATE INDEX orders_metadata_gin + ON orders USING gin (eql_v3.to_ste_vec_query(metadata)::jsonb jsonb_path_ops); +ANALYZE orders; +``` + +See [Indexes](/reference/eql/indexes) for the full recipes. + +## Field access: `->` and `->>` + +Fields are addressed by **selector hash** — the deterministic identifier the client emits for a JSON path during encryption — not a plaintext path string like `$.customer.tier`. + +```sql +-- Field access by selector (returns eql_v3.ste_vec_entry) +SELECT metadata -> 'selector_hash'::text FROM orders; + +-- The entry serialized as text (ciphertext JSON, not decrypted plaintext) +SELECT metadata ->> 'selector_hash'::text FROM orders; + +-- Array element by 0-based index +SELECT metadata -> 0 FROM orders; +``` + +The extracted `eql_v3.ste_vec_entry` is itself comparable: + +- `=` / `<>` resolve via `eql_v3.eq_term` — works on every node type +- `<` / `<=` / `>` / `>=` resolve via `eql_v3.ore_cllw` — String and Number leaves only +- `MIN` / `MAX` over an extracted ordered leaf use the `eql_v3.min` / `eql_v3.max` aggregates + +```sql +-- Equality on an extracted leaf +SELECT * FROM orders +WHERE metadata -> 'email_selector'::text = $1::eql_v3.ste_vec_entry; + +-- Group by an extracted leaf's equality term +SELECT eql_v3.eq_term(metadata -> 'region_selector'::text) AS region, COUNT(*) +FROM orders +GROUP BY eql_v3.eq_term(metadata -> 'region_selector'::text); +``` + +A hash index on `eql_v3.eq_term(col -> '<selector>'::text)` engages the equality lookup; a btree on `eql_v3.ore_cllw(...)` engages range and `ORDER BY`. See [Indexes](/reference/eql/indexes). + +## Path queries and array helpers + +The function forms take the same selector hashes: + +```sql +-- All entries matching a selector +SELECT eql_v3.jsonb_path_query(metadata, 'selector_hash') FROM orders; + +-- First match only +SELECT eql_v3.jsonb_path_query_first(metadata, 'selector_hash') FROM orders; + +-- Does the selector exist in this document? +SELECT eql_v3.jsonb_path_exists(metadata, 'selector_hash') FROM orders; +``` + +For encrypted array nodes: + +```sql +SELECT eql_v3.jsonb_array_length(metadata -> 'items_selector'::text) FROM orders; +SELECT eql_v3.jsonb_array_elements(metadata -> 'items_selector'::text) FROM orders; +SELECT eql_v3.jsonb_array_elements_text(metadata -> 'items_selector'::text) FROM orders; +``` + +`jsonb_array_elements` yields encrypted entries; `jsonb_array_elements_text` yields each element as ciphertext text. + +## Worked example + +An `orders` table with an encrypted `metadata` document. The plaintext your application works with: + +```json +{ + "customer": { + "tier": "premium", + "region": "apac" + }, + "items": ["sku-1042", "sku-2210"] +} +``` + +The client encrypts this into a ste_vec payload with selectors for `$`, `$.customer`, `$.customer.tier`, `$.customer.region`, `$.items`, and each array element — every path becomes queryable. + +<Steps> +<Step> + +### Create the table and insert + +```sql +CREATE TABLE orders ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + metadata eql_v3.json +); + +INSERT INTO orders (metadata) VALUES ($1); +-- $1 is the encrypted ste_vec payload produced by the Stack SDK or Proxy +``` + +</Step> +<Step> + +### Query by containment + +Find premium orders. The client encrypts the needle `{"customer": {"tier": "premium"}}` into a `ste_vec_query`: + +```sql +SELECT id FROM orders +WHERE metadata @> $1::eql_v3.ste_vec_query; +``` + +Add the GIN index from above once the table grows. + +</Step> +<Step> + +### Query by path + +Count orders per region, grouping on the encrypted leaf — the database never sees `"apac"`: + +```sql +SELECT eql_v3.eq_term(metadata -> 'region_selector'::text) AS region, COUNT(*) +FROM orders +WHERE eql_v3.jsonb_path_exists(metadata, 'region_selector') +GROUP BY 1; +``` + +The rows come back as ciphertext; decrypt them in the client. + +</Step> +</Steps> + +## In this section + +<Cards> + <Card title="Payload format" href="/reference/eql/payload-format"> + The wire shape of the ste_vec envelope and its entries. + </Card> + <Card title="Indexes" href="/reference/eql/indexes"> + GIN containment and field-level functional index recipes. + </Card> + <Card title="Operators" href="/reference/eql/operators"> + The full operator surface, including the typed-operand rule. + </Card> +</Cards> diff --git a/content/docs/reference/eql/meta.json b/content/docs/reference/eql/meta.json index d58bc8f..48fe2e7 100644 --- a/content/docs/reference/eql/meta.json +++ b/content/docs/reference/eql/meta.json @@ -1,4 +1,11 @@ { "title": "EQL", - "pages": ["..."] + "pages": [ + "types", + "operators", + "indexes", + "json", + "functions", + "payload-format" + ] } diff --git a/content/docs/reference/eql/operators.mdx b/content/docs/reference/eql/operators.mdx new file mode 100644 index 0000000..60a5fff --- /dev/null +++ b/content/docs/reference/eql/operators.mdx @@ -0,0 +1,153 @@ +--- +title: Operators +description: "Which SQL operators work on each eql_v3 encrypted-domain variant, how unsupported operators fail, and why operands must be typed." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +EQL overloads standard PostgreSQL operators on the [encrypted-domain types](/reference/eql/types). Type the column as the variant that carries the right index term and the operator resolves — and engages a matching [functional index](/reference/eql/indexes). + +<Callout type="warn"> +**Operands must be typed.** The `eql_v3` domains are backed by `jsonb`. When an operand has no known type — a bare string literal, an untyped parameter — PostgreSQL reduces the domain to its `jsonb` base type and resolves the **native `jsonb` operator** instead of the encrypted one. The query doesn't fail; it silently returns native `jsonb` semantics, which are meaningless for encrypted payloads. + +Always type the operand: a typed parameter (`$1::eql_v3.text_eq`) or an explicit cast (`'…'::eql_v3.int4_ord`). The [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) type bound parameters automatically — raw SQL must do it by hand. +</Callout> + +## Operator support by variant + +A ✅ means the operator resolves on a column typed as that variant. A ❌ means it is blocked — it raises, it does not return wrong rows. + +| SQL operator | Meaning | `eql_v3.<T>` | `_eq` | `_ord` / `_ord_ore` | `text_match` | `text_search` | +| --- | --- | :---: | :---: | :---: | :---: | :---: | +| `=` | Equality | ❌ | ✅ | ✅ | ❌ | ✅ | +| `<>` / `!=` | Inequality | ❌ | ✅ | ✅ | ❌ | ✅ | +| `<` `<=` `>` `>=` | Ordered comparison | ❌ | ❌ | ✅ | ❌ | ✅ | +| `@>` / `<@` | Bloom-filter token containment | ❌ | ❌ | ❌ | ✅ | ✅ | +| `LIKE` / `ILIKE` (`~~` / `~~*`) | SQL pattern match | ❌ | ❌ | ❌ | ❌ | ❌ | +| `IS NULL` / `IS NOT NULL` | Null check | ✅ | ✅ | ✅ | ✅ | ✅ | + +A SQL `NULL` column value is not encrypted, so `IS NULL` / `IS NOT NULL` always work regardless of variant. + +## There is no `LIKE` + +`LIKE` and `ILIKE` (`~~` / `~~*`) raise on **every** encrypted-domain variant. SQL pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter token containment — `@>` on a `text_match` or `text_search` column: + +```sql +-- ❌ Raises: operator not supported +SELECT * FROM users WHERE email LIKE '%alice%'; + +-- ✅ Encrypted free-text match +SELECT * FROM users WHERE email @> $1::eql_v3.text_match; +``` + +`@>` / `<@` here is **probabilistic ngram-bloom containment** — it tests whether the encrypted text contains the (encrypted) search terms. It is not JSONB containment and not `LIKE`. The client encrypts the search term into a bloom-filter query value; false positives are possible, false negatives are not. + +## Unsupported operators fail loudly + +Unsupported operators are not silent no-ops. Every operator that a variant doesn't support is still *defined* — it routes to a blocker function that raises an `operator … is not supported` exception. A mis-typed query fails loudly instead of silently returning wrong results: + +```sql +-- salary is eql_v3.int8_eq (equality only) +SELECT * FROM users WHERE salary > $1::eql_v3.int8_eq; +-- ERROR: operator > is not supported for eql_v3.int8_eq +``` + +A `NULL` operand still raises — the blockers are deliberately not `STRICT`, so PostgreSQL can't skip the check. + +## Query shapes + +### Equality: `=` and `<>` + +Works on `_eq`, `_ord` / `_ord_ore`, and `text_search`. On `_eq` and `text_search`, equality compares the HMAC (`hm`) term; on `_ord` variants it compares the ORE (`ob`) term, which collapses to equality — so `_ord` columns get equality without carrying an `hm` term: + +```sql +SELECT * FROM users WHERE email = $1::eql_v3.text_eq; +SELECT * FROM users WHERE email <> $1::eql_v3.text_eq; +``` + +### Comparison, `BETWEEN`, and `ORDER BY` + +Works on `_ord` / `_ord_ore` and `text_search` (variants carrying an `ob` ORE term): + +```sql +SELECT * FROM users WHERE salary >= $1::eql_v3.int8_ord; + +-- BETWEEN desugars to >= and <= +SELECT * FROM users +WHERE created_at BETWEEN $1::eql_v3.timestamp_ord AND $2::eql_v3.timestamp_ord; + +-- ORDER BY is meaningful only with an ORE term +SELECT * FROM users ORDER BY salary DESC; +``` + +`ORDER BY` on a variant without an `ob` term won't produce a meaningful order — type the column as an `_ord` variant when ordering matters. + +Bare `ORDER BY col` sorts correctly, but the planner doesn't rewrite sort keys, so it adds a `Sort` node even when a btree index exists. To stream rows out of the index already ordered, write the sort key in extractor form (`ORDER BY eql_v3.ord_term(col)`) — see [Range and ORDER BY](/reference/eql/indexes#range-and-order-by). + +### Text containment: `@>` and `<@` + +Works on `text_match` and `text_search` only: + +```sql +SELECT * FROM users WHERE email @> $1::eql_v3.text_match; +``` + +### `IN` + +Desugars to `=`, so it needs an equality-capable variant (`_eq`, `_ord`, `text_search`): + +```sql +SELECT * FROM users +WHERE email IN ($1::eql_v3.text_eq, $2::eql_v3.text_eq); +``` + +### `GROUP BY` and `DISTINCT` + +Need an equality term (`_eq`, `_ord`, `text_search`): + +```sql +SELECT email, COUNT(*) FROM logins GROUP BY email; +SELECT DISTINCT email FROM logins; +``` + +Plain `COUNT(col)` needs no term and works on any variant; `COUNT(DISTINCT col)` needs an equality term. + +### Joins + +Equijoins work on equality-capable variants, with one extra constraint: **both sides must have been encrypted with the same keyset and typed as a matching variant** — otherwise the equality terms can never match: + +```sql +SELECT u.*, o.total +FROM users u +JOIN orders o ON u.email = o.customer_email; -- both eql_v3.text_eq, same keyset +``` + +The same rule applies to `IN (subquery)` and set-operation deduplication. + +## Function-form equivalents + +Some managed platforms disallow custom operators. Every operator has a function form, generated per domain variant, taking the same domain types: + +| Function | Operator | Available on | +| --- | --- | --- | +| `eql_v3.eq(a, b)` | `=` | `_eq`, `_ord` / `_ord_ore`, `text_search` | +| `eql_v3.neq(a, b)` | `<>` | `_eq`, `_ord` / `_ord_ore`, `text_search` | +| `eql_v3.lt(a, b)` | `<` | `_ord` / `_ord_ore`, `text_search` | +| `eql_v3.lte(a, b)` | `<=` | `_ord` / `_ord_ore`, `text_search` | +| `eql_v3.gt(a, b)` | `>` | `_ord` / `_ord_ore`, `text_search` | +| `eql_v3.gte(a, b)` | `>=` | `_ord` / `_ord_ore`, `text_search` | +| `eql_v3.contains(a, b)` | `@>` | `text_match`, `text_search`, `eql_v3.json` | +| `eql_v3.contained_by(a, b)` | `<@` | `text_match`, `text_search`, `eql_v3.json` | + +```sql +SELECT * FROM users WHERE eql_v3.eq(email, $1::eql_v3.text_eq); +SELECT * FROM users WHERE eql_v3.lt(created_at, $1::eql_v3.timestamp_ord); +``` + +There are no `like` / `ilike` function forms — text matching is `eql_v3.contains` on a `text_match` value. See [Functions](/reference/eql/functions) for the full function surface, including `MIN` / `MAX`. + +## JSON operators + +`eql_v3.json` has its own operator surface — document containment (`@>` / `<@`), field access (`->` / `->>`), and comparisons on extracted leaves — and its own set of blocked native JSONB operators. See [JSON support](/reference/eql/json). diff --git a/content/docs/reference/eql/payload-format.mdx b/content/docs/reference/eql/payload-format.mdx new file mode 100644 index 0000000..24af439 --- /dev/null +++ b/content/docs/reference/eql/payload-format.mdx @@ -0,0 +1,123 @@ +--- +title: Payload format +description: "The wire format of every EQL encrypted value: the v/i/c envelope, the index-term keys, and the ste_vec document shape." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Every EQL encrypted value is a `jsonb` payload with a shared envelope plus the index terms that make it queryable. This page defines that wire format. Earlier CipherStash docs called this format the **CipherCell** — this page is the current definition of the same structure. + +Payloads are produced by the encryption clients — the [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) — and consumed by EQL's operators and functions inside Postgres. EQL never sees plaintext: it validates, stores, and compares these payloads; it cannot produce or decrypt them. + +## The envelope + +Every payload carries three envelope keys. Each `eql_v3` domain's `CHECK` constraint requires them, so a value missing any of these is rejected at write time: + +| Key | Contents | Notes | +| --- | --- | --- | +| `v` | Payload version | Always exactly `2` on the wire. The domain `CHECK`s assert it and raise on any other value. | +| `i` | Ident: `{"t": "<table>", "c": "<column>"}` | Binds the ciphertext to the table and column it was encrypted for. Both keys required. | +| `c` | Ciphertext | The opaque, non-deterministic encrypted blob (mp_base85-encoded). Never used in comparisons. | + +<Callout> +`eql_v3` names the **SQL schema generation**, not the payload version. The JSON envelope version is still `v: 2` — the wire field names are unchanged from EQL v2, and the domain `CHECK`s assert `v = 2`. +</Callout> + +A `k` discriminator (`"ct"` for a scalar ciphertext, `"sv"` for a JSON document) also appears on payloads emitted by the clients, distinguishing the two top-level shapes. + +## Index-term keys + +Alongside the envelope, a payload carries the index terms for its column's capability. On the wire, a payload is discriminated by *which term key is present* — the SQL domain name carries the rest. Each key is backed by a SEM (searchable encrypted metadata) type in the `eql_v3` schema: + +| Key | SEM type | Wire shape | Enables | Reveals | +| --- | --- | --- | --- | --- | +| `hm` | `eql_v3.hmac_256` (domain over `text`) | Hex string (HMAC-SHA-256) | `=`, `<>` on `_eq` and `text_search` domains | Whether two values are equal — nothing else | +| `ob` | `eql_v3.ore_block_256` (composite: array of `bytea` block terms) | Array of hex-encoded ORE blocks | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ore` domains — and `=` / `<>`, since ORE comparison collapses to equality | The relative order of two values | +| `bf` | `eql_v3.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit) | `@>` / `<@` token containment on `_match` domains | Probabilistic token overlap between values | + +Notes on the wire shapes: + +- **`ob` block count is width-agnostic**: 8 blocks for the int scalars, 12 for timestamp, 14 for numeric — the array just carries more block strings. +- **`bf` positions are signed**: EQL stores the filter as PostgreSQL `smallint[]`, and filters sized above 32768 emit upper-half bit positions as *negative* signed values. Consumers must use a signed 16-bit integer type. + +The capability is encoded as **required keys**: the payload for an `eql_v3.text_eq` column must carry `hm`; an `eql_v3.int4_ord` payload must carry `ob` (and only `ob` — equality on `_ord` domains compares ORE terms, so no `hm` is needed); a `text_match` payload must carry `bf`; a `text_search` payload carries all three. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. See [Types](/reference/eql/types) for the domain-to-capability mapping, and [Searchable encryption](/concepts/searchable-encryption) for what these terms do and don't leak. + +## JSON documents: the `sv` vector + +An [encrypted JSON document](/reference/eql/json) uses a different payload shape: no root ciphertext, and an `sv` array with one encrypted entry per path in the document. Each entry carries: + +| Key | Contents | +| --- | --- | +| `s` | Selector — a deterministic hash of the JSON path. Required; entry matching compares selectors first. | +| `c` | Ciphertext for the node at that path. | +| `hm` **or** `oc` | Exactly one, never both — the domain `CHECK` enforces the exclusivity. `hm` (HMAC-256) on Boolean/`null` leaves and Object/Array roots; `oc` (CLLW ORE, backed by `eql_v3.ore_cllw`) on String/Number leaves. | +| `a` | Optional array marker — `true` when the selector points at an array context. | + +The decoded `oc` value starts with a domain-tag byte (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext, so numeric and string values in one column keep a consistent total order. Earlier payload versions split this into two fields — `ocf` (fixed-width, numeric) and `ocv` (variable-width, string) — which consolidated into the single `oc` key; the tag byte now carries the distinction. + +A containment **query** payload (`eql_v3.ste_vec_query`) has the same `sv` shape but its entries carry no `c` — containment matches selectors and index terms, never ciphertexts. + +## Example payloads + +A scalar payload for an `eql_v3.text_search` column (lookup + ordering + free-text match, so all three terms are required): + +```json +{ + "v": 2, + "i": { "t": "users", "c": "email" }, + "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", + "hm": "9c8ec1d2f9932b979b1bf3f09f8a4e2f6a41f8de2f0c8b7a52e1f5c3d4b6a790", + "ob": ["7a1fd0c2...", "d24c9be1...", "03fa66b8..."], + "bf": [42, 1290, -8113, 30201] +} +``` + +- `v`, `i`, `c` — the envelope +- `hm` — equality term: `WHERE email = $1` compares this +- `ob` — ordering term: `ORDER BY` and range comparisons walk these blocks +- `bf` — bloom-filter term: `@>` token containment tests these bit positions + +A JSON document payload for an `eql_v3.json` column: + +```json +{ + "v": 2, + "k": "sv", + "i": { "t": "orders", "c": "metadata" }, + "sv": [ + { "s": "2517068c0d1f9d4d41d2c666211f785e", "c": "mBbKmM...", "hm": "b0e0..." }, + { "s": "f510853a4ab9d4f75f51a533ac264c5d", "c": "mBbKmQ...", "oc": "01a3f2..." }, + { "s": "33743aed3ae636f6bf05cff11ac4b519", "c": "mBbKmR...", "oc": "004e19..." } + ] +} +``` + +- First entry: an object root — `hm` only, equality/containment +- Second entry: a string leaf — `oc` starting with tag `01` +- Third entry: a numeric leaf — `oc` starting with tag `00` + +And the containment needle the client builds for a `@>` query — index terms, no ciphertexts: + +```json +{ + "sv": [ + { "s": "f510853a4ab9d4f75f51a533ac264c5d", "oc": "01a3f2..." } + ] +} +``` + +## Machine-readable schemas + +The [EQL repository](https://github.com/cipherstash/encrypt-query-language) publishes the format as JSON Schema in two places: + +- **`crates/eql-bindings/schema/`** — one schema per scalar domain (`$id`s under `https://schemas.cipherstash.com/eql/v3/`), generated from the canonical Rust wire types in the `eql-bindings` crate. TypeScript bindings are generated from the same definitions, so every producer and consumer shares one source of truth. +- **`docs/reference/schema/`** — full-payload schemas covering both the scalar and `sv` document shapes. These files are currently named for the v2.x payload releases (`eql-payload-v2.2.schema.json`, `eql-payload-v2.3.schema.json`) and reference `eql_v2` function names, even though the current SQL surface is `eql_v3` — the v2.3 schema is the applicable document-shape definition, matching the still-`v: 2` envelope. + +## Who produces and consumes this + +- **Produce:** the Stack SDK and CipherStash Proxy encrypt plaintext into these payloads — ciphertext, index terms, selectors — using keys the database never holds. +- **Consume:** EQL's domain `CHECK`s validate the shape on write, and its operators and extractor functions ([Operators](/reference/eql/operators), [Indexes](/reference/eql/indexes)) compare the term keys at query time. + +The division is strict: EQL never sees plaintext, and the clients never rely on the database for key material. diff --git a/content/docs/reference/eql/types.mdx b/content/docs/reference/eql/types.mdx new file mode 100644 index 0000000..e4eb005 --- /dev/null +++ b/content/docs/reference/eql/types.mdx @@ -0,0 +1,95 @@ +--- +title: Encrypted types +description: "The eql_v3 encrypted-domain type families: which domain variant to declare for each scalar type, and what each variant lets you query." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +EQL ships its searchable-encryption surface as PostgreSQL **domains in the `eql_v3` schema**. There are two kinds: + +- **Per-scalar encrypted-domain types** — `eql_v3.int4`, `eql_v3.text`, `eql_v3.timestamp`, and so on. One family of domain *variants* per scalar type. +- **An encrypted-JSON document type** — `eql_v3.json` — for structured encryption of whole JSONB documents. See [JSON support](/reference/eql/json). + +A column's query capability is fixed by the **domain variant you type it as**. There is no database-side configuration step: which index terms travel in a value's payload is decided by the encryption client (the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy)), and the column's domain variant is what makes the matching operators resolve. + +## The family model + +Every scalar type `<T>` generates a storage-only variant plus the query variants its capabilities allow. All variants are `jsonb`-backed domains. + +| Domain variant | Capability | Index term carried | +| --- | --- | --- | +| `eql_v3.<T>` | Storage and decryption only. Every comparison operator is blocked — only `IS NULL` / `IS NOT NULL` work. | none | +| `eql_v3.<T>_eq` | Equality: `=` and `<>` (plus `IN`, `GROUP BY`, `DISTINCT`, equijoins). | `hm` (`eql_v3.hmac_256`) | +| `eql_v3.<T>_ord` / `eql_v3.<T>_ord_ore` | Full comparison surface: `=` `<>` `<` `<=` `>` `>=`, `BETWEEN`, `ORDER BY`, and the `eql_v3.min` / `eql_v3.max` aggregates. | `ob` (`eql_v3.ore_block_256`) | +| `eql_v3.text_match` (text only) | Encrypted free-text token containment via `@>` / `<@`. No equality, no ordering. | `bf` (`eql_v3.bloom_filter`) | +| `eql_v3.text_search` (text only) | Everything: equality, ordering, and token containment combined. | `hm` + `ob` + `bf` | + +Two things worth calling out: + +- **The bare variant blocks everything.** `eql_v3.<T>` carries no index term. Querying it with any comparison operator raises an "operator not supported" exception. Use it for columns you only ever store and decrypt. If you later need to query, type the column as a query variant — or cast at the call site (`col::eql_v3.int4_ord`) if the payload already carries the term. +- **`_ord` and `_ord_ore` are twins.** They are byte-identical surfaces backed by the same ORE block term. Pick the name that documents intent — "ordered" versus "ordered via ORE block". Both support the full ordered surface and `MIN` / `MAX`. + +## Type matrix + +The scalar tokens that ship in EQL 3.0.0 are `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool`. + +| Scalar | `eql_v3.<T>` | `<T>_eq` | `<T>_ord` | `<T>_ord_ore` | `text_match` | `text_search` | +| --- | :---: | :---: | :---: | :---: | :---: | :---: | +| `int2` | ✅ | ✅ | ✅ | ✅ | — | — | +| `int4` | ✅ | ✅ | ✅ | ✅ | — | — | +| `int8` | ✅ | ✅ | ✅ | ✅ | — | — | +| `float4` | ✅ | ✅ | ✅ | ✅ | — | — | +| `float8` | ✅ | ✅ | ✅ | ✅ | — | — | +| `numeric` | ✅ | ✅ | ✅ | ✅ | — | — | +| `date` | ✅ | ✅ | ✅ | ✅ | — | — | +| `timestamp` | ✅ | ✅ | ✅ | ✅ | — | — | +| `text` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `bool` | ✅ | ❌ | ❌ | ❌ | — | — | + +<Callout type="warn"> +**`bool` is storage-only by design.** A two-value column has too little cardinality for any searchable index to be safe — an equality index over `true`/`false` would leak the value distribution outright. EQL ships only `eql_v3.bool`, with no `_eq` or `_ord` variants. Store and decrypt boolean columns; filter on them client-side. +</Callout> + +## Index terms + +Each query variant stores one or more encrypted index terms alongside the ciphertext: + +- **`hm`** — an HMAC-256 term (`eql_v3.hmac_256`). Supports exact equality. +- **`ob`** — an ORE block term (`eql_v3.ore_block_256`). Order-revealing: supports comparison and sorting. +- **`bf`** — a bloom filter term (`eql_v3.bloom_filter`). Supports probabilistic ngram token containment. + +The payload structure — envelope keys plus per-variant term keys — is documented in [Payload format](/reference/eql/payload-format). What each term mathematically reveals about the plaintext (and why you should only carry the terms you need) is covered in [Searchable encryption](/concepts/searchable-encryption). + +## Encrypted JSON: `eql_v3.json` + +`eql_v3.json` is the encrypted-JSON document domain, built on the structured-encryption ("ste_vec") model: a JSONB document is encrypted into a searchable vector of terms, one per path inside the document, supporting containment (`@>`), field access (`->` / `->>`), and path queries. It has its own operator and function surface — see [JSON support](/reference/eql/json). + +## Choosing a variant + +Declare only the capabilities you query on. Every index term a value carries is extra material stored in the database, and each term class reveals different structure to an observer — equality terms reveal value repetition, ORE terms reveal ordering, bloom terms reveal token overlap (see [Searchable encryption](/concepts/searchable-encryption)): + +- Never queried, only decrypted → bare `eql_v3.<T>` +- Exact lookup, `IN`, joins, `GROUP BY` → `<T>_eq` +- Ranges, `ORDER BY`, `MIN`/`MAX` → `<T>_ord` +- Free-text matching on text → `text_match` +- Text you need to look up, sort, *and* search → `text_search` + +The variant you declare must match the terms the client is configured to emit for that column — the domain makes the operator resolve, but the term in the payload is what makes it answer. + +## Example + +```sql +CREATE TABLE users ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email eql_v3.text_search, -- lookup, sort, and free-text match + name eql_v3.text_match, -- free-text match only + tax_id eql_v3.text_eq, -- exact lookup only + salary eql_v3.int8_ord, -- range queries, ORDER BY, MIN/MAX + is_active eql_v3.bool, -- storage only (by design) + created_at eql_v3.timestamp_ord +); +``` + +Once the table exists, add functional indexes on the term extractors so queries engage an index — see [Indexes](/reference/eql/indexes). The operators each variant supports are listed in [Operators](/reference/eql/operators). From dd2a8d68b431b40abad7e4aac811029a2b826586 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 19:56:24 +1000 Subject: [PATCH 09/38] refactor(v2): restructure EQL reference Tailwind-style (CIP-3326) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit EQL is an abstraction over SQL the way Tailwind is over CSS — the docs now follow the same shape: Install → Core concepts → type categories → Indexes → query patterns, increasing in complexity. Each type-category page is the complete reference for its types (variants, payload shape, operators/functions, example queries on one page). - index: trimmed to the Install page - core-concepts (new): the canonical home for shared mechanics — variant model, payload anatomy (v/i/c envelope + hm/ob/bf terms, absorbs payload-format/CipherCell), typed-operand rule, fail-loud blockers, ORE-equality on _ord, term-leakage pointer - numbers-and-dates, text, booleans (new) + json (reworked): category pages; text owns the no-LIKE treatment; json absorbs the sv payload shape; booleans framed as "every type has a storage-only variant — for bool it's the only one" - filtering, sorting, grouping-and-aggregates, joins (new): cross-type query patterns; joins headlines the same-keyset constraint - deleted: types.mdx, operators.mdx, functions.mdx, payload-format.mdx (content redistributed; URLs never shipped publicly, no redirect debt) - Anti-drift rule recorded in IA.md: mechanics live ONLY in core-concepts; category/query pages link, never restate - meta.json: flat URLs with ---Types---/---Indexes---/---Queries--- sidebar separators; legacy redirect map retargeted (queries → filtering, cipher-cell → core-concepts) Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- IA.md | 22 ++- content/docs/reference/eql/booleans.mdx | 62 +++++++ content/docs/reference/eql/core-concepts.mdx | 142 ++++++++++++++++ content/docs/reference/eql/filtering.mdx | 124 ++++++++++++++ content/docs/reference/eql/functions.mdx | 112 ------------- .../reference/eql/grouping-and-aggregates.mdx | 104 ++++++++++++ content/docs/reference/eql/index.mdx | 54 +++--- content/docs/reference/eql/indexes.mdx | 4 +- content/docs/reference/eql/joins.mdx | 112 +++++++++++++ content/docs/reference/eql/json.mdx | 60 +++++-- content/docs/reference/eql/meta.json | 17 +- .../docs/reference/eql/numbers-and-dates.mdx | 140 ++++++++++++++++ content/docs/reference/eql/operators.mdx | 153 ----------------- content/docs/reference/eql/payload-format.mdx | 123 -------------- content/docs/reference/eql/sorting.mdx | 96 +++++++++++ content/docs/reference/eql/text.mdx | 157 ++++++++++++++++++ content/docs/reference/eql/types.mdx | 95 ----------- v2-redirects.mjs | 4 +- 18 files changed, 1047 insertions(+), 534 deletions(-) create mode 100644 content/docs/reference/eql/booleans.mdx create mode 100644 content/docs/reference/eql/core-concepts.mdx create mode 100644 content/docs/reference/eql/filtering.mdx delete mode 100644 content/docs/reference/eql/functions.mdx create mode 100644 content/docs/reference/eql/grouping-and-aggregates.mdx create mode 100644 content/docs/reference/eql/joins.mdx create mode 100644 content/docs/reference/eql/numbers-and-dates.mdx delete mode 100644 content/docs/reference/eql/operators.mdx delete mode 100644 content/docs/reference/eql/payload-format.mdx create mode 100644 content/docs/reference/eql/sorting.mdx create mode 100644 content/docs/reference/eql/text.mdx delete mode 100644 content/docs/reference/eql/types.mdx diff --git a/IA.md b/IA.md index b51956c..6db1dee 100644 --- a/IA.md +++ b/IA.md @@ -130,14 +130,22 @@ live at `/docs/errors/<code>` — permanent, never restructured (CIP-3338). ## Reference - [x] Section scaffold 🚧 (eql, stack, auth, cli, proxy, workspace) -- **EQL (v3 rewrite — CIP-3326):** -- [x] `/reference/eql` — overview + install (single SQL file, permissions split, dbdev, Docker) -- [x] `/reference/eql/types` — 10 scalar families × variants + `eql_v3.json` -- [x] `/reference/eql/operators` — per-variant matrix incl. what RAISES; typed-operand rule +- **EQL (v3 rewrite — CIP-3326; Tailwind-shaped: install → core concepts → type + categories → indexes → query patterns). Anti-drift rule: shared mechanics + (typed operands, blockers, envelope, variant model, ORE-equality) live ONLY in + core-concepts — category/query pages link, never restate:** +- [x] `/reference/eql` — install (single SQL file, permissions split, dbdev, Docker) +- [x] `/reference/eql/core-concepts` — variant model, payload anatomy (absorbs + cipher-cell), typed-operand rule, fail-loud blockers, term leakage pointer +- [x] `/reference/eql/numbers-and-dates` — int*/float*/numeric/date/timestamp +- [x] `/reference/eql/text` — all six text variants; owns the no-LIKE treatment +- [x] `/reference/eql/json` — ste_vec + sv payload shape + containment/path queries +- [x] `/reference/eql/booleans` — storage-only variants (bool has only that one) - [x] `/reference/eql/indexes` — functional indexes on extractors; Supabase-compatible -- [x] `/reference/eql/json` — ste_vec, path queries -- [x] `/reference/eql/functions` — incl. aggregates (min/max only) -- [x] `/reference/eql/payload-format` — v/i/c envelope, hm/ob/bf (absorbs cipher-cell) +- [x] `/reference/eql/filtering` — =, IN, ranges, token match, containment +- [x] `/reference/eql/sorting` — ORDER BY, extractor sort-key form, pagination +- [x] `/reference/eql/grouping-and-aggregates` — GROUP BY/DISTINCT, min/max, no SUM/AVG +- [x] `/reference/eql/joins` — equijoins, the same-keyset constraint - **Stack SDK:** - [ ] `/reference/stack` — client + configuration (port encryption/* pages) - [ ] `/reference/stack/schema` diff --git a/content/docs/reference/eql/booleans.mdx b/content/docs/reference/eql/booleans.mdx new file mode 100644 index 0000000..390dc97 --- /dev/null +++ b/content/docs/reference/eql/booleans.mdx @@ -0,0 +1,62 @@ +--- +title: Booleans +description: "Encrypted booleans are storage-only by design: eql_v3.bool stores and decrypts, carries no index terms, and blocks every comparison." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Every scalar type has a storage-only variant — for `bool` it's the only one. EQL ships `eql_v3.bool` and nothing else: there is no `bool_eq` and no `bool_ord`. An encrypted boolean column can be stored, decrypted, and null-checked; it cannot be filtered, sorted, grouped, or joined on. + +## Why there are no query variants + +A two-value column has too little cardinality for any searchable index to be safe. An equality term over `true` / `false` would partition the table into two visible buckets — leaking the value distribution (and, with any outside knowledge, the values themselves) outright. Rather than ship an index term that can't keep its promise, EQL omits the query variants entirely. See [Searchable encryption](/concepts/searchable-encryption) for the general analysis of what index terms reveal. + +## What works, what raises + +`eql_v3.bool` follows the bare-variant contract described in [Core concepts](/reference/eql/core-concepts#variants-declare-capability): it carries no index terms, so `IS NULL` / `IS NOT NULL` are the only predicates that work. Every comparison operator routes to a blocker and raises — the [fail-loud behavior](/reference/eql/core-concepts#unsupported-operations-fail-loudly) shared by all encrypted variants: + +```sql +-- ❌ Raises: operator = is not supported for eql_v3.bool +SELECT * FROM users WHERE is_active = $1::eql_v3.bool; + +-- ✅ Works: NULL columns are not encrypted +SELECT * FROM users WHERE is_active IS NOT NULL; +``` + +## Filter client-side + +Query on other columns, decrypt the boolean in your application, and filter there: + +```sql +CREATE TABLE users ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email eql_v3.text_eq, -- exact lookup + created_at eql_v3.timestamp_ord, -- range queries, ORDER BY + is_active eql_v3.bool -- storage only (by design) +); +``` + +```sql +-- Narrow the result set with the columns that do carry index terms… +SELECT id, email, is_active FROM users +WHERE created_at >= $1::eql_v3.timestamp_ord; +-- …then decrypt is_active in the client and filter on the plaintext. +``` + +The [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) decrypt the payload back to a plain boolean on read, so the client-side filter is an ordinary `if`. + +If a boolean genuinely needs to be a server-side predicate, that is a data-modelling signal: consider whether the flag is actually sensitive. A non-sensitive flag can stay a plain PostgreSQL `boolean` column alongside your encrypted columns. + +## Storing without querying + +`bool` is the forced case of a pattern available to every scalar type: the bare variant `eql_v3.<T>` (for example `eql_v3.int4`, `eql_v3.text`, `eql_v3.timestamp`) is storage-and-decryption only. It carries no index terms, and every comparison operator raises — use it for columns you only ever store and decrypt, so the database holds no searchable material for them at all. + +For every type other than `bool`, storage-only is a choice you can walk back. If you later need to query, retype the column as a query variant — or, if the payloads already carry the needed term (the client decides which terms travel in the payload), cast at the call site: + +```sql +SELECT * FROM readings WHERE value::eql_v3.int4_ord > $1::eql_v3.int4_ord; +``` + +The variant families and what each one enables are covered in [Core concepts](/reference/eql/core-concepts); the per-type specifics live in [Numbers and dates](/reference/eql/numbers-and-dates) and [Text](/reference/eql/text). diff --git a/content/docs/reference/eql/core-concepts.mdx b/content/docs/reference/eql/core-concepts.mdx new file mode 100644 index 0000000..a262fe8 --- /dev/null +++ b/content/docs/reference/eql/core-concepts.mdx @@ -0,0 +1,142 @@ +--- +title: Core concepts +description: "The model behind every EQL page: domain variants that declare capability, the encrypted payload envelope, the typed-operand rule, and fail-loud blockers." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Everything in the EQL reference builds on four ideas: columns are typed as **domain variants** that declare what they can do, every value is a **`jsonb` payload** carrying encrypted index terms, **operands must be typed** for the encrypted operators to resolve, and anything a column can't do **fails loudly** instead of returning wrong rows. This page is the canonical home for all four — the per-type and per-query pages link back here rather than restating them. + +## Variants declare capability + +EQL ships its searchable-encryption surface as PostgreSQL **domains in the `eql_v3` schema**, all backed by `jsonb`. Each scalar type generates a *family* of domain variants, and the variant you type a column as fixes its query capability. Each domain carries a `CHECK` constraint that validates the encrypted payload on insert, so a malformed or wrong-version value is rejected at write time rather than surfacing at query time. + +There is no database-side configuration table. Earlier EQL versions tracked encryption config in the database (`config_add_table`, `config_add_column`, and friends) — those are gone in v3. The searchable surface of a column is fixed by the domain variant you type it as, and which index terms travel in a value's payload is decided by the encryption client (the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy)). The domain makes the matching operators resolve; the term in the payload is what makes them answer. + +For any scalar type `<T>`, the family looks like this: + +| Domain variant | Capability | Index term carried | +| --- | --- | --- | +| `eql_v3.<T>` | Storage and decryption only. Every comparison operator is blocked — only `IS NULL` / `IS NOT NULL` work. | none | +| `eql_v3.<T>_eq` | Equality: `=` and `<>` (plus `IN`, `GROUP BY`, `DISTINCT`, equijoins). | `hm` (`eql_v3.hmac_256`) | +| `eql_v3.<T>_ord` / `eql_v3.<T>_ord_ore` | Full comparison surface: `=` `<>` `<` `<=` `>` `>=`, `BETWEEN`, `ORDER BY`, and the `eql_v3.min` / `eql_v3.max` aggregates. | `ob` (`eql_v3.ore_block_256`) | +| `eql_v3.text_match` (text only) | Encrypted free-text token containment via `@>` / `<@`. No equality, no ordering. | `bf` (`eql_v3.bloom_filter`) | +| `eql_v3.text_search` (text only) | Everything: equality, ordering, and token containment combined. | `hm` + `ob` + `bf` | + +Two things worth calling out: + +- **The bare variant blocks everything.** `eql_v3.<T>` carries no index term. Querying it with any comparison operator raises an "operator not supported" exception. Use it for columns you only ever store and decrypt — [Booleans](/reference/eql/booleans) covers this pattern in full. +- **`_ord` and `_ord_ore` are twins.** They are byte-identical surfaces backed by the same ORE block term. Pick the name that documents intent — "ordered" versus "ordered via ORE block". Both support the full ordered surface and `MIN` / `MAX`. + +Declaring a table is just typing each column as the variant it needs: + +```sql +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email eql_v3.text_eq, -- equality only + salary eql_v3.int4_ord, -- equality + range + ORDER BY + created_at eql_v3.timestamp_ord +); +``` + +Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.0 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers and dates](/reference/eql/numbers-and-dates), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use a separate domain, `eql_v3.json`, with its own operator surface — see [JSON](/reference/eql/json). + +## Anatomy of an encrypted value + +Every EQL encrypted value is a `jsonb` payload with a shared envelope plus the index terms that make it queryable. Earlier CipherStash docs called this format the **CipherCell** — this section is the current definition of the same structure. + +Payloads are **produced** by the encryption clients — the [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) — and **consumed** by EQL's operators and functions inside Postgres. EQL never sees plaintext: it validates, stores, and compares these payloads; it cannot produce or decrypt them. The division is strict: the clients never rely on the database for key material. + +### The envelope + +Every payload carries three envelope keys. Each `eql_v3` domain's `CHECK` constraint requires them, so a value missing any of these is rejected at write time: + +| Key | Contents | Notes | +| --- | --- | --- | +| `v` | Payload version | Always exactly `2` on the wire. The domain `CHECK`s assert it and raise on any other value. | +| `i` | Ident: `{"t": "<table>", "c": "<column>"}` | Binds the ciphertext to the table and column it was encrypted for. Both keys required. | +| `c` | Ciphertext | The opaque, non-deterministic encrypted blob (mp_base85-encoded). Never used in comparisons. | + +<Callout> +`eql_v3` names the **SQL schema generation**, not the payload version. The JSON envelope version is still `v: 2` — the wire field names are unchanged from EQL v2, and the domain `CHECK`s assert `v = 2`. +</Callout> + +A `k` discriminator (`"ct"` for a scalar ciphertext, `"sv"` for a JSON document) also appears on payloads emitted by the clients, distinguishing the two top-level shapes. + +### Index-term keys + +Alongside the envelope, a payload carries the index terms for its column's capability. Each key is backed by a SEM (searchable encrypted metadata) type in the `eql_v3` schema: + +| Key | SEM type | Wire shape | Enables | Reveals | +| --- | --- | --- | --- | --- | +| `hm` | `eql_v3.hmac_256` (domain over `text`) | Hex string (HMAC-SHA-256) | `=`, `<>` on `_eq` and `text_search` domains | Whether two values are equal — nothing else | +| `ob` | `eql_v3.ore_block_256` (composite: array of `bytea` block terms) | Array of hex-encoded ORE blocks (block count varies by scalar width) | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ore` domains — and `=` / `<>`, since ORE comparison collapses to equality | The relative order of two values | +| `bf` | `eql_v3.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit — large filters emit negative positions) | `@>` / `<@` token containment on `_match` domains | Probabilistic token overlap between values | + +The capability is encoded as **required keys**: the payload for an `eql_v3.text_eq` column must carry `hm`; an `eql_v3.int4_ord` payload must carry `ob` (and only `ob`); a `text_match` payload must carry `bf`; a `text_search` payload carries all three. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. + +A scalar payload for an `eql_v3.text_search` column (lookup + ordering + free-text match, so all three terms are required): + +```json +{ + "v": 2, + "i": { "t": "users", "c": "email" }, + "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", + "hm": "9c8ec1d2f9932b979b1bf3f09f8a4e2f6a41f8de2f0c8b7a52e1f5c3d4b6a790", + "ob": ["7a1fd0c2...", "d24c9be1...", "03fa66b8..."], + "bf": [42, 1290, -8113, 30201] +} +``` + +- `v`, `i`, `c` — the envelope +- `hm` — equality term: `WHERE email = $1` compares this +- `ob` — ordering term: `ORDER BY` and range comparisons walk these blocks +- `bf` — bloom-filter term: `@>` token containment tests these bit positions + +Encrypted JSON documents use a different payload shape — an `sv` array with one encrypted entry per path in the document instead of a root ciphertext — defined in [JSON](/reference/eql/json). + +### Machine-readable schemas + +The [EQL repository](https://github.com/cipherstash/encrypt-query-language) publishes the format as JSON Schema in two places: + +- **`crates/eql-bindings/schema/`** — one schema per scalar domain (`$id`s under `https://schemas.cipherstash.com/eql/v3/`), generated from the canonical Rust wire types in the `eql-bindings` crate. TypeScript bindings are generated from the same definitions, so every producer and consumer shares one source of truth. +- **`docs/reference/schema/`** — full-payload schemas covering both the scalar and `sv` document shapes. These files are currently named for the v2.x payload releases (`eql-payload-v2.2.schema.json`, `eql-payload-v2.3.schema.json`) and reference `eql_v2` function names, even though the current SQL surface is `eql_v3` — the v2.3 schema is the applicable document-shape definition, matching the still-`v: 2` envelope. + +## The typed-operand rule + +The `eql_v3` domains are backed by `jsonb`. When an operand has no known type — a bare string literal, an untyped parameter — PostgreSQL reduces the domain to its `jsonb` base type and resolves the **native `jsonb` operator** instead of the encrypted one. The query doesn't fail; it silently returns native `jsonb` semantics, which are meaningless for encrypted payloads. + +```sql +-- ❌ Wrong: untyped parameter. PostgreSQL falls back to the native jsonb `=`, +-- which compares raw payloads — syntactically valid, semantically meaningless. +SELECT * FROM users WHERE email = $1; + +-- ✅ Right: typed operand — the encrypted `=` resolves. +SELECT * FROM users WHERE email = $1::eql_v3.text_eq; +``` + +Always type the operand: a typed parameter (`$1::eql_v3.text_eq`) or an explicit cast (`'…'::eql_v3.int4_ord`). The [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) type bound parameters automatically — raw SQL must do it by hand. + +This is the one place where a mistake is *silent*. Everything else fails loudly: + +## Unsupported operations fail loudly + +Unsupported operators are not silent no-ops. Every operator that a variant doesn't support is still *defined* — it routes to a blocker function that raises an `operator … is not supported` exception. A mis-typed query fails loudly instead of silently returning wrong results: + +```sql +-- salary is eql_v3.int8_eq (equality only) +SELECT * FROM users WHERE salary > $1::eql_v3.int8_eq; +-- ERROR: operator > is not supported for eql_v3.int8_eq +``` + +A `NULL` operand still raises — the blockers are deliberately not `STRICT`, so PostgreSQL can't skip the check. (A SQL `NULL` column value is not encrypted, so `IS NULL` / `IS NOT NULL` themselves always work, on every variant.) + +`LIKE` and `ILIKE` are blocked on **every** encrypted variant — pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter token containment instead; [Text](/reference/eql/text) covers it. + +One equality subtlety follows from the term table above: on `_ord` / `_ord_ore` columns, `=` and `<>` compare the **ORE (`ob`) term** — ORE comparison collapses to equality — so `_ord` payloads carry no `hm` term at all. On `_eq` and `text_search` columns, equality compares the HMAC (`hm`) term. + +## What the terms reveal + +Every index term a value carries is extra material stored in the database, and each term class reveals defined structure to an observer who can read the stored payloads: equality terms reveal *value repetition* (which rows share a value), ORE terms reveal *ordering* (which of two values is larger), and bloom terms reveal *probabilistic token overlap*. None of them reveal the plaintext — but you should only carry the terms you actually query on. The full analysis of what each term does and doesn't leak is in [Searchable encryption](/concepts/searchable-encryption). diff --git a/content/docs/reference/eql/filtering.mdx b/content/docs/reference/eql/filtering.mdx new file mode 100644 index 0000000..6f1b779 --- /dev/null +++ b/content/docs/reference/eql/filtering.mdx @@ -0,0 +1,124 @@ +--- +title: Filtering +description: "WHERE-clause patterns on encrypted columns: equality, IN lists, ranges and BETWEEN, text token matching, JSON containment, and combining encrypted and plaintext predicates." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Every filter below is ordinary SQL — the encrypted operators resolve from the column's domain variant, and a functional index on the matching term extractor serves the predicate. One rule applies throughout: **operands must be typed** (`$1::eql_v3.text_eq`, not a bare literal), or PostgreSQL falls through to native `jsonb` semantics. See [Core concepts](/reference/eql/core-concepts) for the typed-operand rule and how unsupported operators fail loudly instead of returning wrong rows. + +## Equality: `=` and `<>` + +Works on `_eq` and `_ord` / `_ord_ore` variants of every scalar, and on `text_search`: + +```sql +SELECT * FROM users WHERE email = $1::eql_v3.text_eq; +SELECT * FROM users WHERE tax_id <> $1::eql_v3.text_eq; +``` + +On `_eq` and `text_search` columns equality compares the HMAC (`hm`) term. On `_ord` variants there is no `hm` — equality compares the ORE (`ob`) term, which collapses to equality, so `_ord` columns get `=` and `<>` for free. See [Core concepts](/reference/eql/core-concepts) for the mechanism. + +```sql +-- salary is eql_v3.int8_ord: equality works without an hm term +SELECT * FROM users WHERE salary = $1::eql_v3.int8_ord; +``` + +Bare storage-only variants (`eql_v3.text`, `eql_v3.int4`, …) block every comparison — see the type pages for what each variant supports: [Numbers & dates](/reference/eql/numbers-and-dates), [Text](/reference/eql/text), [Booleans](/reference/eql/booleans). + +## `IN` lists + +`IN` desugars to `=`, so it needs the same equality-capable variants. Each list element is a separately encrypted, typed operand: + +```sql +SELECT * FROM users +WHERE email IN ($1::eql_v3.text_eq, $2::eql_v3.text_eq, $3::eql_v3.text_eq); +``` + +There is no way to encrypt a list as one value — the client encrypts each element and binds it as its own parameter. `IN (subquery)` also works, subject to the same-keyset rule covered in [Joins](/reference/eql/joins). + +## Ranges and `BETWEEN` + +`<`, `<=`, `>`, `>=` work on `_ord` / `_ord_ore` variants and `text_search` — the variants carrying an ORE (`ob`) term: + +```sql +SELECT * FROM users WHERE salary >= $1::eql_v3.int8_ord; + +-- BETWEEN desugars to >= and <= +SELECT * FROM users +WHERE created_at BETWEEN $1::eql_v3.timestamp_ord AND $2::eql_v3.timestamp_ord; +``` + +Half-open ranges compose the same way: + +```sql +SELECT * FROM events +WHERE occurred_at >= $1::eql_v3.timestamp_ord + AND occurred_at < $2::eql_v3.timestamp_ord; +``` + +## Text token matching: `@>` + +There is no `LIKE` on encrypted columns — encrypted free-text matching is bloom-filter token containment via `@>` on a `text_match` or `text_search` column: + +```sql +SELECT * FROM users WHERE name @> $1::eql_v3.text_match; +``` + +The client encrypts the search term into a bloom-filter query value; matching is probabilistic (false positives possible, false negatives not). For the full no-`LIKE` story and match-term tuning, see [Text](/reference/eql/text). + +## JSON containment and path filters + +Encrypted JSON documents (`eql_v3.json`) filter by containment and path existence: + +```sql +-- Does the document contain this (encrypted) structure? +SELECT * FROM orders WHERE metadata @> $1::eql_v3.ste_vec_query; + +-- Does this path exist in the document? +SELECT * FROM orders WHERE eql_v3.jsonb_path_exists(metadata, 'region_selector'); + +-- Equality on an extracted leaf +SELECT * FROM orders +WHERE metadata -> 'email_selector'::text = $1::eql_v3.ste_vec_entry; +``` + +Field access is by selector hash, not plaintext path. The full JSON surface — containment, field access, path queries, and range filters on extracted leaves — is in [JSON](/reference/eql/json). + +## Combining predicates + +Encrypted predicates compose with `AND`, `OR`, `NOT`, and parentheses like any other predicate — and plaintext columns filter normally alongside encrypted ones in the same `WHERE` clause: + +```sql +SELECT * FROM users +WHERE status = 'active' -- plaintext column, native operator + AND created_at >= $1::eql_v3.timestamp_ord -- encrypted range + AND (email = $2::eql_v3.text_eq -- encrypted equality + OR name @> $3::eql_v3.text_match); -- encrypted token match +``` + +The planner treats each encrypted predicate independently, so it can combine an index on a plaintext column with a functional index on an encrypted one (bitmap-AND, or whichever plan is cheapest). + +## `IS NULL` and `IS NOT NULL` + +A SQL `NULL` column value is never encrypted — there is no payload to encrypt — so null checks work on **every** variant, including storage-only ones: + +```sql +SELECT * FROM users WHERE tax_id IS NULL; +SELECT * FROM users WHERE tax_id IS NOT NULL; +``` + +Don't confuse this with a JSON `null` *inside* an encrypted document, which is an encrypted value like any other — see [JSON](/reference/eql/json). + +## Shape summary + +| Filter shape | Operators | Works on | Index | +| --- | --- | --- | --- | +| Equality | `=` `<>` `IN` | `_eq`, `_ord` / `_ord_ore`, `text_search` | hash (or btree) on `eql_v3.eq_term` — btree on `eql_v3.ord_term` for `_ord` | +| Range | `<` `<=` `>` `>=` `BETWEEN` | `_ord` / `_ord_ore`, `text_search` | btree on `eql_v3.ord_term` | +| Text token match | `@>` `<@` | `text_match`, `text_search` | GIN on `eql_v3.match_term` | +| JSON containment | `@>` `<@` | `eql_v3.json` | GIN on `eql_v3.to_ste_vec_query(col)::jsonb` | +| Null check | `IS NULL` / `IS NOT NULL` | every variant | — | + +Every one of these has a full index recipe — which method, which extractor, and how to confirm the index engages with `EXPLAIN` — in [Indexes](/reference/eql/indexes). diff --git a/content/docs/reference/eql/functions.mdx b/content/docs/reference/eql/functions.mdx deleted file mode 100644 index 210ca31..0000000 --- a/content/docs/reference/eql/functions.mdx +++ /dev/null @@ -1,112 +0,0 @@ ---- -title: Functions -description: "The eql_v3 function surface: comparison functions, index-term extractors, MIN/MAX aggregates, JSON functions, and version reporting." -type: reference -components: [eql] -verifiedAgainst: - eql: "3.0.0" ---- - -Everything EQL exposes lives in the `eql_v3` schema. Most functions are generated per [domain variant](/reference/eql/types), so PostgreSQL's overload resolution picks the right implementation from the argument type. As with operators, arguments must be typed — see [the typed-operand rule](/reference/eql/operators). - -## Comparison functions - -Function forms of the comparison operators, for platforms that disallow custom operators. Each is generated per capable domain variant, with overloads accepting the domain on either side and `jsonb` on the other: - -```sql -eql_v3.eq(a, b) RETURNS boolean -- = on _eq / _ord / _ord_ore / text_search -eql_v3.neq(a, b) RETURNS boolean -- <> -eql_v3.lt(a, b) RETURNS boolean -- < on _ord / _ord_ore / text_search -eql_v3.lte(a, b) RETURNS boolean -- <= -eql_v3.gt(a, b) RETURNS boolean -- > -eql_v3.gte(a, b) RETURNS boolean -- >= -eql_v3.contains(a, b) RETURNS boolean -- @> on text_match / text_search / eql_v3.json -eql_v3.contained_by(a, b) RETURNS boolean -- <@ -``` - -```sql -SELECT * FROM users WHERE eql_v3.eq(email, $1::eql_v3.text_eq); -SELECT * FROM users WHERE eql_v3.lt(created_at, $1::eql_v3.timestamp_ord); -``` - -Calling a comparison function a variant doesn't support resolves to a blocker that raises `operator … is not supported` — the same [fail-loud behavior](/reference/eql/operators) as the operators. There are no `like` / `ilike` functions: text matching is `eql_v3.contains` on a `text_match` value. - -## Index-term extractors - -These extract the encrypted index term from a domain value. They're generated per eq-, ord-, and match-capable variant of every scalar type, and they return the self-contained `eql_v3` index-term types: - -```sql --- Equality term (hm) -eql_v3.eq_term(a eql_v3.<T>_eq) RETURNS eql_v3.hmac_256 - --- Ordering term (ob) -eql_v3.ord_term(a eql_v3.<T>_ord) RETURNS eql_v3.ore_block_256 -eql_v3.ord_term(a eql_v3.<T>_ord_ore) RETURNS eql_v3.ore_block_256 - --- Text-match term (bf) -eql_v3.match_term(a eql_v3.text_match) RETURNS eql_v3.bloom_filter -``` - -`eql_v3.text_search` carries all three terms, so all three extractors work on it. - -The extractors exist for **indexing**: EQL indexes through a functional index on the extractor, never an operator class on the column. The extractors are inlinable, so bare-form predicates (`WHERE email = $1`) engage the index without rewriting. Sort keys are the exception — see [Range and ORDER BY](/reference/eql/indexes#range-and-order-by): - -```sql -CREATE INDEX users_email_eq ON users USING hash (eql_v3.eq_term(email)); -CREATE INDEX users_salary_ord ON users USING btree (eql_v3.ord_term(salary)); -CREATE INDEX users_name_match ON users USING gin (eql_v3.match_term(name)); -``` - -See [Indexes](/reference/eql/indexes) for the full recipes and performance guidance. - -## Aggregates: `eql_v3.min` and `eql_v3.max` - -`MIN` / `MAX` over encrypted values, defined per ord-capable variant of every scalar type. The input type selects the aggregate; the return type matches the input: - -```sql -eql_v3.min(eql_v3.<T>_ord) RETURNS eql_v3.<T>_ord -eql_v3.max(eql_v3.<T>_ord) RETURNS eql_v3.<T>_ord -eql_v3.min(eql_v3.<T>_ord_ore) RETURNS eql_v3.<T>_ord_ore -eql_v3.max(eql_v3.<T>_ord_ore) RETURNS eql_v3.<T>_ord_ore -``` - -Comparison routes through the variant's `<` / `>` operator, which uses the ORE block term — no decryption happens in the database. `NULL` inputs are skipped, and an all-`NULL` input set returns `NULL`. - -```sql -SELECT eql_v3.min(salary) FROM users; -SELECT eql_v3.max(salary) FROM users WHERE department = 'engineering'; - --- On a generic jsonb column, cast to the right domain at the call site -SELECT eql_v3.min(salary_jsonb::eql_v3.int8_ord) FROM users; -``` - -<Callout type="warn"> -**`SUM`, `AVG`, and other arithmetic aggregates are not supported** on encrypted columns — they would require homomorphic encryption. `MIN` / `MAX` work because they only need comparison. For sums and averages, decrypt at the application boundary and aggregate client-side. -</Callout> - -## JSON functions - -The encrypted-JSON document type `eql_v3.json` has its own function surface: - -- `eql_v3.jsonb_path_query(doc, selector)` — set-returning path query yielding encrypted entries; also `jsonb_path_query_first` and `jsonb_path_exists` -- `eql_v3.jsonb_array_length` / `jsonb_array_elements` / `jsonb_array_elements_text` — array helpers -- `eql_v3.to_ste_vec_query(doc)` — builds the GIN-indexable containment query form -- Entry-level term extractors: `eql_v3.eq_term(eql_v3.ste_vec_entry)` and `eql_v3.ore_cllw(eql_v3.ste_vec_entry)` - -These are documented with worked examples in [JSON support](/reference/eql/json). - -## `eql_v3.version()` - -Returns the installed EQL version string, baked in at build time: - -```sql -SELECT eql_v3.version(); --- '3.0.0' -``` - -The same version string is mirrored as a comment on the `eql_v3` schema, so you can read it without calling a function: - -```sql -SELECT obj_description('eql_v3'::regnamespace); --- '3.0.0' -``` diff --git a/content/docs/reference/eql/grouping-and-aggregates.mdx b/content/docs/reference/eql/grouping-and-aggregates.mdx new file mode 100644 index 0000000..544a91c --- /dev/null +++ b/content/docs/reference/eql/grouping-and-aggregates.mdx @@ -0,0 +1,104 @@ +--- +title: Grouping & aggregates +description: "GROUP BY, DISTINCT, COUNT, and eql_v3.min/max on encrypted columns — why to group on the extractor, and why SUM and AVG stay client-side." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Grouping and deduplication need an equality term, so they work on the same variants as `=`: `_eq`, `_ord` / `_ord_ore`, and `text_search`. `MIN` / `MAX` need an ordering term (`_ord` / `_ord_ore`, `text_search`). Arithmetic aggregates don't work at all — that's the last section. As everywhere, operands and call-site casts must be typed; see [Core concepts](/reference/eql/core-concepts). + +## `GROUP BY` and `DISTINCT` + +Both work in natural form on equality-capable variants: + +```sql +SELECT email, COUNT(*) FROM logins GROUP BY email; +SELECT DISTINCT email FROM logins; +``` + +Grouping compares equality terms, so rows encrypting the same plaintext land in the same group — but the group key that comes back is ciphertext. Decrypt it in the client if you need to display it. + +## Group on the extractor + +For anything beyond small tables, group on the equality-term extractor instead of the raw column: + +```sql +SELECT eql_v3.eq_term(email) AS email_term, COUNT(*) + FROM logins + GROUP BY eql_v3.eq_term(email); +``` + +The reason is planner economics. `GROUP BY email` uses the entire encrypted payload — 1–2 KB per row — as the hash key. Postgres estimates a hash table far larger than the default `work_mem` and falls back to a disk-spilling `GroupAggregate`. The extractor key is a small deterministic term: the hash table fits in `work_mem` and the planner picks `HashAggregate` reliably. If an ORM forces the raw-column form, raising `work_mem` is the rescue knob — but the extractor form is the design. The same reasoning, from the index-tuning angle, is in [Indexes](/reference/eql/indexes). + +Note the trade-off: grouping on `eq_term` returns the *term*, not the encrypted value — fine for counting, but the term itself can't be decrypted. If you need the group key's plaintext, join the grouped result back to the table on the term to recover a representative encrypted value, then decrypt that in the client. + +## `COUNT` and `COUNT(DISTINCT)` + +Plain `COUNT(col)` counts non-`NULL` rows — it never compares values, so it works on **any** variant, including storage-only ones: + +```sql +SELECT COUNT(tax_id) FROM users; -- works even on bare eql_v3.text +``` + +`COUNT(DISTINCT col)` deduplicates, so it needs an equality-capable variant — and the same extractor advice applies: + +```sql +SELECT COUNT(DISTINCT eql_v3.eq_term(email)) FROM logins; +``` + +## `MIN` and `MAX`: `eql_v3.min` / `eql_v3.max` + +EQL ships `min` / `max` aggregates per ord-capable variant of every scalar type. The input type selects the aggregate, and the return type matches the input: + +```sql +eql_v3.min(eql_v3.<T>_ord) RETURNS eql_v3.<T>_ord +eql_v3.max(eql_v3.<T>_ord) RETURNS eql_v3.<T>_ord +eql_v3.min(eql_v3.<T>_ord_ore) RETURNS eql_v3.<T>_ord_ore +eql_v3.max(eql_v3.<T>_ord_ore) RETURNS eql_v3.<T>_ord_ore +``` + +Comparison routes through the variant's `<` / `>` operator on the ORE term — no decryption happens in the database, and the result is an encrypted value the client decrypts. `NULL` inputs are skipped; an all-`NULL` input set returns `NULL`, matching native aggregate semantics. + +```sql +SELECT eql_v3.min(salary) FROM users; +SELECT eql_v3.max(salary) FROM users WHERE department = 'engineering'; + +-- Combined with grouping +SELECT eql_v3.eq_term(department_code) AS dept, eql_v3.max(salary) + FROM users + GROUP BY eql_v3.eq_term(department_code); +``` + +If the column is generic `jsonb` rather than a domain, cast to the right variant at the call site so overload resolution can pick the aggregate: + +```sql +SELECT eql_v3.min(salary_jsonb::eql_v3.int8_ord) FROM users; +``` + +A btree on `eql_v3.ord_term(col)` serves `MIN` / `MAX` — the [Indexes](/reference/eql/indexes) page has the recipe. + +## No `SUM`, no `AVG` + +<Callout type="warn"> +**`SUM`, `AVG`, and every other arithmetic aggregate are unsupported** on encrypted columns — they would require homomorphic encryption, which EQL does not do. `MIN` / `MAX` work because they only need *comparison*, which the ORE term provides. For sums and averages, select the rows (or `MIN`/`MAX`/`COUNT` server-side to narrow them) and aggregate client-side after decryption. +</Callout> + +## Grouping on extracted JSON leaves + +Leaves inside an encrypted JSON document group the same way — extract the entry by selector, then group on its equality term: + +```sql +SELECT eql_v3.eq_term(metadata -> 'region_selector'::text) AS region, COUNT(*) + FROM orders + GROUP BY eql_v3.eq_term(metadata -> 'region_selector'::text); +``` + +`eql_v3.eq_term` reads whichever term the entry carries, so this works on every JSON node type. String and Number leaves also support `eql_v3.min` / `eql_v3.max` via their CLLW ORE term. Selectors and node capabilities are in [JSON](/reference/eql/json). + +## Where to go next + +- [Indexes](/reference/eql/indexes) — the hash/btree recipes that back these shapes, and the full `work_mem` / `HashAggregate` story. +- [Joins](/reference/eql/joins) — equality terms across tables, and the same-keyset rule. +- [Filtering](/reference/eql/filtering) — the `WHERE` shapes that feed these aggregates. diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index 68b95c2..e3d7f0f 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -11,20 +11,7 @@ Encrypt Query Language (EQL) is a set of types, operators, and functions for sto EQL itself never encrypts anything. Encryption and decryption happen in the client, using the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). EQL provides the database-side surface those clients query against: encrypted column types, the operators that compare them, and the term-extractor functions that make indexes work. -## The v3 model - -Every encrypted column is a `jsonb`-backed **domain type** in the `eql_v3` schema. The domain variant you choose declares the column's searchable capability: `eql_v3.text_eq` supports equality (`=` / `<>`), `eql_v3.text_match` supports encrypted text containment (`@>` / `<@`), `eql_v3.int4_ord` adds range comparisons, `ORDER BY`, and `MIN` / `MAX`. Each domain carries a `CHECK` constraint that validates the encrypted payload on insert, so a malformed or wrong-version value is rejected at write time rather than surfacing at query time. - -There is no database-side configuration table. Earlier EQL versions tracked encryption config in the database (`config_add_table`, `config_add_column`, and friends) — those are gone in v3. The searchable surface of a column is fixed by the domain variant you type it as, and which index terms travel in a value's payload is decided by the encryption client. Operators that a variant doesn't support raise an "operator not supported" error rather than silently falling through to native `jsonb` semantics — and `LIKE` / `ILIKE` are blocked on every encrypted column. - -```sql -CREATE TABLE users ( - id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v3.text_eq, -- equality only - salary eql_v3.int4_ord, -- equality + range + ORDER BY - created_at eql_v3.timestamp_ord -); -``` +Every encrypted column is a `jsonb`-backed domain type in the `eql_v3` schema, and the domain variant you choose declares what the column can do — the full model is in [Core concepts](/reference/eql/core-concepts). ## Install @@ -113,25 +100,42 @@ Schema changes — adding or removing encrypted columns — always go through th EQL v3 is designed to install without superuser. There are no custom operator classes (which managed platforms typically block), no `postgresql.conf` changes, and no separate Supabase build — the single install script is the same artefact everywhere. Indexing works through ordinary functional indexes over EQL's term-extractor functions, which any user who can `CREATE INDEX` can build. See the [Supabase integration](/integrations/supabase) for platform-specific setup. -## In this section +## Understand <Cards> - <Card title="Types" href="/reference/eql/types"> - The encrypted domain type families and the capability each variant carries. + <Card title="Core concepts" href="/reference/eql/core-concepts"> + Domain variants, the encrypted payload, typed operands, and fail-loud blockers — the model every other page assumes. </Card> - <Card title="Operators" href="/reference/eql/operators"> - Which SQL operators resolve on which variant, and what raises. + <Card title="Numbers and dates" href="/reference/eql/numbers-and-dates"> + Encrypted integers, floats, numerics, dates, and timestamps. </Card> - <Card title="Indexes" href="/reference/eql/indexes"> - Functional-index recipes for equality, range, and text match. + <Card title="Text" href="/reference/eql/text"> + Encrypted text: equality, ordering, and free-text token matching — and why there is no `LIKE`. </Card> <Card title="JSON" href="/reference/eql/json"> Encrypted JSON documents: containment, field access, and GIN indexing. </Card> - <Card title="Functions" href="/reference/eql/functions"> - The function equivalents of every operator, extractors, and aggregates. + <Card title="Booleans" href="/reference/eql/booleans"> + Storage-only by design: why encrypted booleans carry no index terms. + </Card> + <Card title="Indexes" href="/reference/eql/indexes"> + Functional-index recipes over the term extractors, and what it takes for an index to engage. + </Card> +</Cards> + +## Use + +<Cards> + <Card title="Filtering" href="/reference/eql/filtering"> + `WHERE` clauses on encrypted columns: equality, ranges, and text containment. + </Card> + <Card title="Sorting" href="/reference/eql/sorting"> + `ORDER BY` on encrypted columns, and how to keep the sort in the index. + </Card> + <Card title="Grouping and aggregates" href="/reference/eql/grouping-and-aggregates"> + `GROUP BY`, `DISTINCT`, `COUNT`, and the `MIN` / `MAX` aggregates. </Card> - <Card title="Payload format" href="/reference/eql/payload-format"> - The encrypted payload envelope and index terms. + <Card title="Joins" href="/reference/eql/joins"> + Equijoins on encrypted columns and the same-keyset rule. </Card> </Cards> diff --git a/content/docs/reference/eql/indexes.mdx b/content/docs/reference/eql/indexes.mdx index 83f354e..3d1df3e 100644 --- a/content/docs/reference/eql/indexes.mdx +++ b/content/docs/reference/eql/indexes.mdx @@ -29,7 +29,7 @@ EQL v3 deliberately ships no operator class for encrypted columns. Operators res ## Index recipes -Type the column as the domain variant that carries the term ([Types](/reference/eql/types)), then index the matching extractor: +Type the column as the domain variant that carries the term (see [Core concepts](/reference/eql/core-concepts) for the variant model, and the per-type pages for specifics), then index the matching extractor: ```sql -- Equality: hash index on eq_term @@ -65,8 +65,8 @@ All three must hold: ```sql -- ✓ resolves the encrypted operator → uses the index -WHERE email = $1; WHERE email = $1::eql_v3.text_eq; +WHERE email = $1; -- only when the client (Stack SDK / Proxy) binds $1 typed -- ✗ falls through to native jsonb semantics WHERE email = '{"hm":"abc"}'::jsonb; diff --git a/content/docs/reference/eql/joins.mdx b/content/docs/reference/eql/joins.mdx new file mode 100644 index 0000000..fdcc0e4 --- /dev/null +++ b/content/docs/reference/eql/joins.mdx @@ -0,0 +1,112 @@ +--- +title: Joins +description: "Equijoins on encrypted columns: the same-keyset and matching-variant constraint, IN (subquery) and set operations, a worked example, and how to diagnose a join that returns nothing." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Equijoins work on equality-capable variants (`_eq`, `_ord` / `_ord_ore`, `text_search`) — the join condition is just encrypted equality. But there is one constraint that has no plaintext equivalent, and it is the single thing to internalize on this page: + +<Callout type="warn"> +**Both sides of the join must be encrypted with the same keyset and typed as a matching variant.** Encrypted equality compares deterministic index terms, and those terms are derived from the encryption keys. Two columns encrypted under different keysets produce different terms for the *same plaintext* — their terms can **never** match, and the join returns no rows. This is not an error the database can detect: the query is valid, the plan is fine, the result is simply empty. +</Callout> + +"Matching variant" means both sides compare the same term kind: `_eq` with `_eq` (or `text_search`, which carries an `hm` term too) compares HMAC terms; `_ord` with `_ord` compares ORE terms. An `_eq` column can't join an `_ord` column — one side has no `hm`, the other no `ob`, and the equality operator between mismatched variants doesn't resolve. See [Core concepts](/reference/eql/core-concepts) for the term model. + +## Equijoin + +```sql +SELECT u.*, o.total +FROM users u +JOIN orders o ON u.email = o.customer_email; +-- both columns eql_v3.text_eq, encrypted with the same keyset +``` + +No typed-operand cast is needed here — both operands are encrypted columns, so their domain types resolve the encrypted operator directly. All join types (`INNER`, `LEFT`, `RIGHT`, `FULL`) work; `LEFT JOIN` null-extension behaves normally because SQL `NULL`s are not encrypted. + +Index both sides for anything beyond small tables — a hash (or btree) index on `eql_v3.eq_term(col)` on each column. Recipes are in [Indexes](/reference/eql/indexes). + +## `IN (subquery)` and set operations + +Both follow the same rule, because both compare equality terms across two column sources: + +```sql +-- IN (subquery): users.email and orders.customer_email must share a keyset +SELECT * FROM users +WHERE email IN (SELECT customer_email FROM orders WHERE flagged); + +-- Set-operation dedup: UNION / INTERSECT / EXCEPT dedupe by equality term +SELECT email FROM users +UNION +SELECT customer_email FROM orders; +``` + +If the two columns are under different keysets, `IN (subquery)` matches nothing, `INTERSECT` is empty, `EXCEPT` returns everything, and `UNION` never merges duplicates — all silently. + +## Worked example + +Two tables sharing an encrypted customer identifier, both columns typed `eql_v3.text_eq` and encrypted by the same client configuration (same keyset): + +```sql +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email eql_v3.text_eq +); + +CREATE TABLE orders ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + customer_email eql_v3.text_eq, + total BIGINT NOT NULL +); + +CREATE INDEX users_email_eq ON users USING hash (eql_v3.eq_term(email)); +CREATE INDEX orders_cust_eq ON orders USING hash (eql_v3.eq_term(customer_email)); +ANALYZE users; ANALYZE orders; +``` + +Orders per user, filtered by an encrypted lookup on one side: + +```sql +SELECT u.id, COUNT(o.id) AS order_count +FROM users u +LEFT JOIN orders o ON u.email = o.customer_email +WHERE u.email = $1::eql_v3.text_eq +GROUP BY u.id; +``` + +The `WHERE` engages the hash index on `users`; the join condition engages the one on `orders`. The grouping key here is a plaintext `id`, so no extractor is needed — grouping on encrypted columns is covered in [Grouping & aggregates](/reference/eql/grouping-and-aggregates). + +## Anti-pattern: joining across keysets + +The failure mode is quiet. A join across keysets doesn't raise, doesn't warn, and produces a plan that looks healthy — the terms just never match, so it behaves exactly like a join where no rows happen to correlate: + +```sql +-- users encrypted by service A's keyset, partners by service B's: +SELECT * FROM users u JOIN partners p ON u.email = p.contact_email; +-- 0 rows. Always. Even when the plaintext emails overlap. +``` + +To diagnose a join that returns fewer rows than expected (or none): + +1. **Check the variants.** Both columns must be equality-capable and compare the same term kind. A blocked operator raises loudly, so if the query *runs*, the variants at least resolve — but confirm they compare the same term (`hm` vs `ob`). +2. **Compare terms for a known-matching pair.** Take one row from each table that you know holds the same plaintext and compare their equality terms: + + ```sql + SELECT eql_v3.eq_term(u.email) = eql_v3.eq_term(p.contact_email) AS terms_match + FROM users u, partners p + WHERE u.id = 42 AND p.id = 7; -- rows known to share a plaintext value + ``` + + `false` for plaintext-identical values means the terms were derived under different keysets (or different client configurations) — no SQL will make them join. +3. **Fix it at the encryption layer.** Configure both columns under the same keyset in the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy) and re-encrypt one side. Cross-keyset correlation otherwise has to happen in the client, after decryption. + +Treat shared keysets as part of your schema design: columns you intend to join are a unit, the same way a foreign key pair is. + +## Where to go next + +- [Filtering](/reference/eql/filtering) — the equality and `IN` shapes joins are built from. +- [Grouping & aggregates](/reference/eql/grouping-and-aggregates) — grouping joined results on encrypted keys. +- [Indexes](/reference/eql/indexes) — equality index recipes for both sides of a join. +- [Core concepts](/reference/eql/core-concepts) — index terms, variants, and why determinism makes joins possible at all. diff --git a/content/docs/reference/eql/json.mdx b/content/docs/reference/eql/json.mdx index 6ef2e60..4204ca8 100644 --- a/content/docs/reference/eql/json.mdx +++ b/content/docs/reference/eql/json.mdx @@ -1,6 +1,6 @@ --- -title: Encrypted JSON -description: "Store and query encrypted JSON documents with eql_v3.json — containment, field access, and path queries over ciphertext, with the native jsonb operators that don't apply blocked outright." +title: JSON +description: "The complete reference for encrypted JSON documents with eql_v3.json — the ste_vec payload shape, containment, field access, and path queries over ciphertext, with the native jsonb operators that don't apply blocked outright." type: reference components: [eql] verifiedAgainst: @@ -21,7 +21,47 @@ Three `jsonb`-backed domains make up the encrypted JSON surface: | `eql_v3.ste_vec_entry` | A single entry from the vector: a selector, a ciphertext, and exactly one index term. This is what `->` returns. | | `eql_v3.ste_vec_query` | A containment needle: entries with selectors and index terms but **no ciphertext**. This is what you cast a `@>` operand to. | -The full wire shape of each is documented in [Payload format](/reference/eql/payload-format). +## Payload shape + +An encrypted JSON document uses a different payload shape from the scalar types: the standard envelope keys are present (`v`, `i`, plus the `k: "sv"` discriminator — envelope anatomy is covered in [Core concepts](/reference/eql/core-concepts)), but there is no root ciphertext. Instead, an `sv` array carries one encrypted entry per path in the document. Each entry has: + +| Key | Contents | +| --- | --- | +| `s` | Selector — a deterministic hash of the JSON path. Required; entry matching compares selectors first. | +| `c` | Ciphertext for the node at that path. | +| `hm` **or** `oc` | Exactly one, never both — the domain `CHECK` enforces the exclusivity. `hm` (HMAC-256) on Boolean/`null` leaves and Object/Array roots; `oc` (CLLW ORE, backed by `eql_v3.ore_cllw`) on String/Number leaves. | +| `a` | Optional array marker — `true` when the selector points at an array context. | + +The decoded `oc` value starts with a domain-tag byte (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext, so numeric and string values in one column keep a consistent total order. Earlier payload versions split this into two fields — `ocf` (fixed-width, numeric) and `ocv` (variable-width, string) — which consolidated into the single `oc` key; the tag byte now carries the distinction. + +A document payload for an `eql_v3.json` column: + +```json +{ + "v": 2, + "k": "sv", + "i": { "t": "orders", "c": "metadata" }, + "sv": [ + { "s": "2517068c0d1f9d4d41d2c666211f785e", "c": "mBbKmM...", "hm": "b0e0..." }, + { "s": "f510853a4ab9d4f75f51a533ac264c5d", "c": "mBbKmQ...", "oc": "01a3f2..." }, + { "s": "33743aed3ae636f6bf05cff11ac4b519", "c": "mBbKmR...", "oc": "004e19..." } + ] +} +``` + +- First entry: an object root — `hm` only, equality/containment +- Second entry: a string leaf — `oc` starting with tag `01` +- Third entry: a numeric leaf — `oc` starting with tag `00` + +A containment **query** payload (`eql_v3.ste_vec_query`) has the same `sv` shape but its entries carry no `c` — containment matches selectors and index terms, never ciphertexts. This is the needle the client builds for a `@>` query: + +```json +{ + "sv": [ + { "s": "f510853a4ab9d4f75f51a533ac264c5d", "oc": "01a3f2..." } + ] +} +``` ## Storing encrypted JSON @@ -50,7 +90,7 @@ During encryption, the client flattens the document: each unique path gets a det | String | `oc` (CLLW ORE, string domain) | Yes | Yes | | Number | `oc` (CLLW ORE, numeric domain) | Yes | Yes | -Each entry carries exactly one of `hm` or `oc` — the domain `CHECK` enforces the exclusivity. `hm` is a deterministic hash, so it supports equality only. `oc` is a CLLW ORE term that reveals ordering and, being deterministic, collapses to equality on matching selectors — `eql_v3.eq_term` reads whichever term an entry carries, so equality works uniformly across all node types. Earlier payload versions split the ORE term into `ocf` (fixed-width, numeric) and `ocv` (variable-width, string); current payloads emit a single `oc` whose leading domain-tag byte carries the numeric/string distinction. +Each entry carries exactly one of `hm` or `oc` — the domain `CHECK` enforces the exclusivity. `hm` is a deterministic hash, so it supports equality only. `oc` is a CLLW ORE term that reveals ordering and, being deterministic, collapses to equality on matching selectors — `eql_v3.eq_term` reads whichever term an entry carries, so equality works uniformly across all node types. JSON `null` here means a `null` literal *inside* the document. A SQL `NULL` column value is not encrypted at all. @@ -66,7 +106,7 @@ These native PostgreSQL `jsonb` operators are **blocked** on `eql_v3.json`. They Use containment (`@>` / `<@`), field access (`->` / `->>`), or the `eql_v3.jsonb_path_*` functions instead. There is no server-side mutation of an encrypted document — updates re-encrypt in the client. <Callout type="warn"> -**Type your operands.** `eql_v3.json` is a domain over `jsonb`, and PostgreSQL resolves `domain OP untyped_literal` to the **native** `jsonb` operator — bypassing both the encrypted operator and the blockers. `WHERE doc -> 'email'` silently runs native `jsonb ->` and returns `NULL`; `WHERE doc -> 'email'::text` resolves the encrypted operator. This is the same rule as the [scalar operators](/reference/eql/operators). Queries through CipherStash Proxy always bind typed parameters, so this only bites hand-written ad-hoc SQL. +**Operands must be typed** (`doc -> 'email'::text`, not `doc -> 'email'`) — an untyped operand resolves the native `jsonb` operator, bypassing both the encrypted operator and the blockers. See [Core concepts](/reference/eql/core-concepts). </Callout> ## Containment: `@>` and `<@` @@ -213,16 +253,16 @@ The rows come back as ciphertext; decrypt them in the client. </Step> </Steps> -## In this section +## Where to next <Cards> - <Card title="Payload format" href="/reference/eql/payload-format"> - The wire shape of the ste_vec envelope and its entries. + <Card title="Core concepts" href="/reference/eql/core-concepts"> + The envelope anatomy, typed-operand rule, and fail-loud behavior shared by every EQL type. </Card> <Card title="Indexes" href="/reference/eql/indexes"> GIN containment and field-level functional index recipes. </Card> - <Card title="Operators" href="/reference/eql/operators"> - The full operator surface, including the typed-operand rule. + <Card title="Filtering" href="/reference/eql/filtering"> + WHERE-clause patterns across all encrypted types. </Card> </Cards> diff --git a/content/docs/reference/eql/meta.json b/content/docs/reference/eql/meta.json index 48fe2e7..3f4469e 100644 --- a/content/docs/reference/eql/meta.json +++ b/content/docs/reference/eql/meta.json @@ -1,11 +1,18 @@ { "title": "EQL", "pages": [ - "types", - "operators", - "indexes", + "core-concepts", + "---Types---", + "numbers-and-dates", + "text", "json", - "functions", - "payload-format" + "booleans", + "---Indexes---", + "indexes", + "---Queries---", + "filtering", + "sorting", + "grouping-and-aggregates", + "joins" ] } diff --git a/content/docs/reference/eql/numbers-and-dates.mdx b/content/docs/reference/eql/numbers-and-dates.mdx new file mode 100644 index 0000000..1d01dec --- /dev/null +++ b/content/docs/reference/eql/numbers-and-dates.mdx @@ -0,0 +1,140 @@ +--- +title: Numbers & dates +description: "The complete reference for encrypted numeric and date/time columns: the int, float, numeric, date, and timestamp domain variants, the ORE-backed payload they carry, and range, ORDER BY, and MIN/MAX queries." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Eight scalar types share one identical query surface: `int2`, `int4`, `int8`, `float4`, `float8`, `numeric`, `date`, and `timestamp`. These are the columns you filter by range, sort newest-first, and take a `MIN` / `MAX` over — salaries, totals, rates, hire dates, timestamps. Everything on this page applies to all eight; only the domain name changes. + +There is no free-text matching for these types — `_match` and `_search` are [text-only variants](/reference/eql/text). Boolean columns are a separate, storage-only story — see [Booleans](/reference/eql/booleans). + +## Variants + +Each of the eight scalar types generates the same four `jsonb`-backed domain variants: + +| Domain variant | Capability | Index term carried | +| --- | --- | --- | +| `eql_v3.<T>` | Storage and decryption only. Every comparison operator is blocked — only `IS NULL` / `IS NOT NULL` work. | none | +| `eql_v3.<T>_eq` | Equality: `=` and `<>` (plus `IN`, `GROUP BY`, `DISTINCT`, equijoins). | `hm` (HMAC-256) | +| `eql_v3.<T>_ord` | Full comparison surface: `=` `<>` `<` `<=` `>` `>=`, `BETWEEN`, `ORDER BY`, and `MIN` / `MAX`. | `ob` (ORE block) | +| `eql_v3.<T>_ord_ore` | Identical to `<T>_ord` — a twin name that documents intent. | `ob` (ORE block) | + +Declare only the capability you query on — each index term class reveals different structure to an observer (see [Searchable encryption](/concepts/searchable-encryption)), and the variant model itself is covered in [Core concepts](/reference/eql/core-concepts): + +```sql +CREATE TABLE employees ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + salary eql_v3.int8_ord, -- range queries, ORDER BY, MIN/MAX + tax_rate eql_v3.numeric_eq, -- exact lookup only + net_worth eql_v3.numeric, -- store and decrypt only, never queried + hired_on eql_v3.date_ord, + created_at eql_v3.timestamp_ord +); +``` + +## Payload + +A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus the `ob` ordering term. Here is a payload for the `eql_v3.int8_ord` `salary` column: + +```json +{ + "v": 2, + "i": { "t": "employees", "c": "salary" }, + "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", + "ob": [ + "7a1fd0c2...", "d24c9be1...", "03fa66b8...", "91b7e04d...", + "5c28aa19...", "e6f3071c...", "48d92ab5...", "0b64cf37..." + ] +} +``` + +- **`ob` is the only index term.** An `_ord` payload carries no `hm`: equality on `_ord` variants compares ORE terms, which collapse to equality — see [Core concepts](/reference/eql/core-concepts). Only `_eq` payloads carry `hm` (a single hex HMAC-SHA-256 string) instead of `ob`. +- **The `ob` block count varies with the plaintext width**: 8 blocks for the int scalars, 12 for `timestamp`, 14 for `numeric` — the array just carries more block strings. + +## Operators and functions + +The function forms exist for managed platforms that disallow custom operators — they take the same typed arguments and resolve identically. + +| SQL operator | Function form | `eql_v3.<T>` | `<T>_eq` | `<T>_ord` / `<T>_ord_ore` | +| --- | --- | :---: | :---: | :---: | +| `=` / `<>` | `eql_v3.eq(a, b)` / `eql_v3.neq(a, b)` | ❌ | ✅ | ✅ | +| `<` `<=` `>` `>=` | `eql_v3.lt` / `lte` / `gt` / `gte` | ❌ | ❌ | ✅ | +| `BETWEEN` | desugars to `>=` and `<=` | ❌ | ❌ | ✅ | +| `IN` | desugars to `=` | ❌ | ✅ | ✅ | +| `GROUP BY` / `DISTINCT` | — (needs an equality term) | ❌ | ✅ | ✅ | +| `ORDER BY` | sort key: `eql_v3.ord_term(col)` | ❌ | ❌ | ✅ | +| `MIN` / `MAX` | `eql_v3.min(col)` / `eql_v3.max(col)` | ❌ | ❌ | ✅ | +| `IS NULL` / `IS NOT NULL` | — | ✅ | ✅ | ✅ | + +Blocked cells raise an `operator … is not supported` exception — they never silently return wrong rows. Operands must be typed (`$1::eql_v3.int8_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). + +**`SUM`, `AVG`, and other arithmetic aggregates are not supported** on encrypted columns — they would require homomorphic encryption. `MIN` / `MAX` work because they only need comparison; for sums and averages, decrypt at the application boundary and aggregate client-side. + +## Example queries + +### Range filter + +```sql +SELECT * FROM employees +WHERE salary >= $1::eql_v3.int8_ord; + +SELECT * FROM employees +WHERE salary BETWEEN $1::eql_v3.int8_ord AND $2::eql_v3.int8_ord; +``` + +### Date window + +`BETWEEN` works the same on `date` and `timestamp` columns: + +```sql +SELECT * FROM employees +WHERE hired_on BETWEEN $1::eql_v3.date_ord AND $2::eql_v3.date_ord; +``` + +### Newest-first listing + +Bare `ORDER BY created_at` sorts correctly, but the planner doesn't rewrite sort keys, so it adds a `Sort` node even when a btree index exists. Write the sort key in extractor form to stream rows out of the index already ordered — at large row counts this is the difference between seconds and milliseconds (see [Sorting](/reference/eql/sorting)): + +```sql +SELECT * FROM employees +WHERE created_at >= $1::eql_v3.timestamp_ord +ORDER BY eql_v3.ord_term(created_at) DESC +LIMIT 10; +``` + +### MIN and MAX + +`eql_v3.min` / `eql_v3.max` compare ORE terms — no decryption happens in the database, and the encrypted result decrypts in the client. `NULL` inputs are skipped; an all-`NULL` input set returns `NULL`: + +```sql +SELECT eql_v3.min(salary) FROM employees; +SELECT eql_v3.max(created_at) FROM employees; +``` + +### Cast at the call site + +On a generic `jsonb` column whose payloads already carry the `ob` term, cast to the right domain in the query: + +```sql +SELECT eql_v3.min(salary_jsonb::eql_v3.int8_ord) FROM employees; +``` + +## Where to next + +<Cards> + <Card title="Indexes" href="/reference/eql/indexes"> + Btree recipes on `eql_v3.ord_term` for range, ORDER BY, and MIN/MAX. + </Card> + <Card title="Filtering" href="/reference/eql/filtering"> + WHERE-clause patterns across all encrypted types. + </Card> + <Card title="Sorting" href="/reference/eql/sorting"> + Why the extractor-form sort key matters, and how to verify with EXPLAIN. + </Card> + <Card title="Grouping & aggregates" href="/reference/eql/grouping-and-aggregates"> + GROUP BY, DISTINCT, and the aggregate surface on encrypted columns. + </Card> +</Cards> diff --git a/content/docs/reference/eql/operators.mdx b/content/docs/reference/eql/operators.mdx deleted file mode 100644 index 60a5fff..0000000 --- a/content/docs/reference/eql/operators.mdx +++ /dev/null @@ -1,153 +0,0 @@ ---- -title: Operators -description: "Which SQL operators work on each eql_v3 encrypted-domain variant, how unsupported operators fail, and why operands must be typed." -type: reference -components: [eql] -verifiedAgainst: - eql: "3.0.0" ---- - -EQL overloads standard PostgreSQL operators on the [encrypted-domain types](/reference/eql/types). Type the column as the variant that carries the right index term and the operator resolves — and engages a matching [functional index](/reference/eql/indexes). - -<Callout type="warn"> -**Operands must be typed.** The `eql_v3` domains are backed by `jsonb`. When an operand has no known type — a bare string literal, an untyped parameter — PostgreSQL reduces the domain to its `jsonb` base type and resolves the **native `jsonb` operator** instead of the encrypted one. The query doesn't fail; it silently returns native `jsonb` semantics, which are meaningless for encrypted payloads. - -Always type the operand: a typed parameter (`$1::eql_v3.text_eq`) or an explicit cast (`'…'::eql_v3.int4_ord`). The [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) type bound parameters automatically — raw SQL must do it by hand. -</Callout> - -## Operator support by variant - -A ✅ means the operator resolves on a column typed as that variant. A ❌ means it is blocked — it raises, it does not return wrong rows. - -| SQL operator | Meaning | `eql_v3.<T>` | `_eq` | `_ord` / `_ord_ore` | `text_match` | `text_search` | -| --- | --- | :---: | :---: | :---: | :---: | :---: | -| `=` | Equality | ❌ | ✅ | ✅ | ❌ | ✅ | -| `<>` / `!=` | Inequality | ❌ | ✅ | ✅ | ❌ | ✅ | -| `<` `<=` `>` `>=` | Ordered comparison | ❌ | ❌ | ✅ | ❌ | ✅ | -| `@>` / `<@` | Bloom-filter token containment | ❌ | ❌ | ❌ | ✅ | ✅ | -| `LIKE` / `ILIKE` (`~~` / `~~*`) | SQL pattern match | ❌ | ❌ | ❌ | ❌ | ❌ | -| `IS NULL` / `IS NOT NULL` | Null check | ✅ | ✅ | ✅ | ✅ | ✅ | - -A SQL `NULL` column value is not encrypted, so `IS NULL` / `IS NOT NULL` always work regardless of variant. - -## There is no `LIKE` - -`LIKE` and `ILIKE` (`~~` / `~~*`) raise on **every** encrypted-domain variant. SQL pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter token containment — `@>` on a `text_match` or `text_search` column: - -```sql --- ❌ Raises: operator not supported -SELECT * FROM users WHERE email LIKE '%alice%'; - --- ✅ Encrypted free-text match -SELECT * FROM users WHERE email @> $1::eql_v3.text_match; -``` - -`@>` / `<@` here is **probabilistic ngram-bloom containment** — it tests whether the encrypted text contains the (encrypted) search terms. It is not JSONB containment and not `LIKE`. The client encrypts the search term into a bloom-filter query value; false positives are possible, false negatives are not. - -## Unsupported operators fail loudly - -Unsupported operators are not silent no-ops. Every operator that a variant doesn't support is still *defined* — it routes to a blocker function that raises an `operator … is not supported` exception. A mis-typed query fails loudly instead of silently returning wrong results: - -```sql --- salary is eql_v3.int8_eq (equality only) -SELECT * FROM users WHERE salary > $1::eql_v3.int8_eq; --- ERROR: operator > is not supported for eql_v3.int8_eq -``` - -A `NULL` operand still raises — the blockers are deliberately not `STRICT`, so PostgreSQL can't skip the check. - -## Query shapes - -### Equality: `=` and `<>` - -Works on `_eq`, `_ord` / `_ord_ore`, and `text_search`. On `_eq` and `text_search`, equality compares the HMAC (`hm`) term; on `_ord` variants it compares the ORE (`ob`) term, which collapses to equality — so `_ord` columns get equality without carrying an `hm` term: - -```sql -SELECT * FROM users WHERE email = $1::eql_v3.text_eq; -SELECT * FROM users WHERE email <> $1::eql_v3.text_eq; -``` - -### Comparison, `BETWEEN`, and `ORDER BY` - -Works on `_ord` / `_ord_ore` and `text_search` (variants carrying an `ob` ORE term): - -```sql -SELECT * FROM users WHERE salary >= $1::eql_v3.int8_ord; - --- BETWEEN desugars to >= and <= -SELECT * FROM users -WHERE created_at BETWEEN $1::eql_v3.timestamp_ord AND $2::eql_v3.timestamp_ord; - --- ORDER BY is meaningful only with an ORE term -SELECT * FROM users ORDER BY salary DESC; -``` - -`ORDER BY` on a variant without an `ob` term won't produce a meaningful order — type the column as an `_ord` variant when ordering matters. - -Bare `ORDER BY col` sorts correctly, but the planner doesn't rewrite sort keys, so it adds a `Sort` node even when a btree index exists. To stream rows out of the index already ordered, write the sort key in extractor form (`ORDER BY eql_v3.ord_term(col)`) — see [Range and ORDER BY](/reference/eql/indexes#range-and-order-by). - -### Text containment: `@>` and `<@` - -Works on `text_match` and `text_search` only: - -```sql -SELECT * FROM users WHERE email @> $1::eql_v3.text_match; -``` - -### `IN` - -Desugars to `=`, so it needs an equality-capable variant (`_eq`, `_ord`, `text_search`): - -```sql -SELECT * FROM users -WHERE email IN ($1::eql_v3.text_eq, $2::eql_v3.text_eq); -``` - -### `GROUP BY` and `DISTINCT` - -Need an equality term (`_eq`, `_ord`, `text_search`): - -```sql -SELECT email, COUNT(*) FROM logins GROUP BY email; -SELECT DISTINCT email FROM logins; -``` - -Plain `COUNT(col)` needs no term and works on any variant; `COUNT(DISTINCT col)` needs an equality term. - -### Joins - -Equijoins work on equality-capable variants, with one extra constraint: **both sides must have been encrypted with the same keyset and typed as a matching variant** — otherwise the equality terms can never match: - -```sql -SELECT u.*, o.total -FROM users u -JOIN orders o ON u.email = o.customer_email; -- both eql_v3.text_eq, same keyset -``` - -The same rule applies to `IN (subquery)` and set-operation deduplication. - -## Function-form equivalents - -Some managed platforms disallow custom operators. Every operator has a function form, generated per domain variant, taking the same domain types: - -| Function | Operator | Available on | -| --- | --- | --- | -| `eql_v3.eq(a, b)` | `=` | `_eq`, `_ord` / `_ord_ore`, `text_search` | -| `eql_v3.neq(a, b)` | `<>` | `_eq`, `_ord` / `_ord_ore`, `text_search` | -| `eql_v3.lt(a, b)` | `<` | `_ord` / `_ord_ore`, `text_search` | -| `eql_v3.lte(a, b)` | `<=` | `_ord` / `_ord_ore`, `text_search` | -| `eql_v3.gt(a, b)` | `>` | `_ord` / `_ord_ore`, `text_search` | -| `eql_v3.gte(a, b)` | `>=` | `_ord` / `_ord_ore`, `text_search` | -| `eql_v3.contains(a, b)` | `@>` | `text_match`, `text_search`, `eql_v3.json` | -| `eql_v3.contained_by(a, b)` | `<@` | `text_match`, `text_search`, `eql_v3.json` | - -```sql -SELECT * FROM users WHERE eql_v3.eq(email, $1::eql_v3.text_eq); -SELECT * FROM users WHERE eql_v3.lt(created_at, $1::eql_v3.timestamp_ord); -``` - -There are no `like` / `ilike` function forms — text matching is `eql_v3.contains` on a `text_match` value. See [Functions](/reference/eql/functions) for the full function surface, including `MIN` / `MAX`. - -## JSON operators - -`eql_v3.json` has its own operator surface — document containment (`@>` / `<@`), field access (`->` / `->>`), and comparisons on extracted leaves — and its own set of blocked native JSONB operators. See [JSON support](/reference/eql/json). diff --git a/content/docs/reference/eql/payload-format.mdx b/content/docs/reference/eql/payload-format.mdx deleted file mode 100644 index 24af439..0000000 --- a/content/docs/reference/eql/payload-format.mdx +++ /dev/null @@ -1,123 +0,0 @@ ---- -title: Payload format -description: "The wire format of every EQL encrypted value: the v/i/c envelope, the index-term keys, and the ste_vec document shape." -type: reference -components: [eql] -verifiedAgainst: - eql: "3.0.0" ---- - -Every EQL encrypted value is a `jsonb` payload with a shared envelope plus the index terms that make it queryable. This page defines that wire format. Earlier CipherStash docs called this format the **CipherCell** — this page is the current definition of the same structure. - -Payloads are produced by the encryption clients — the [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) — and consumed by EQL's operators and functions inside Postgres. EQL never sees plaintext: it validates, stores, and compares these payloads; it cannot produce or decrypt them. - -## The envelope - -Every payload carries three envelope keys. Each `eql_v3` domain's `CHECK` constraint requires them, so a value missing any of these is rejected at write time: - -| Key | Contents | Notes | -| --- | --- | --- | -| `v` | Payload version | Always exactly `2` on the wire. The domain `CHECK`s assert it and raise on any other value. | -| `i` | Ident: `{"t": "<table>", "c": "<column>"}` | Binds the ciphertext to the table and column it was encrypted for. Both keys required. | -| `c` | Ciphertext | The opaque, non-deterministic encrypted blob (mp_base85-encoded). Never used in comparisons. | - -<Callout> -`eql_v3` names the **SQL schema generation**, not the payload version. The JSON envelope version is still `v: 2` — the wire field names are unchanged from EQL v2, and the domain `CHECK`s assert `v = 2`. -</Callout> - -A `k` discriminator (`"ct"` for a scalar ciphertext, `"sv"` for a JSON document) also appears on payloads emitted by the clients, distinguishing the two top-level shapes. - -## Index-term keys - -Alongside the envelope, a payload carries the index terms for its column's capability. On the wire, a payload is discriminated by *which term key is present* — the SQL domain name carries the rest. Each key is backed by a SEM (searchable encrypted metadata) type in the `eql_v3` schema: - -| Key | SEM type | Wire shape | Enables | Reveals | -| --- | --- | --- | --- | --- | -| `hm` | `eql_v3.hmac_256` (domain over `text`) | Hex string (HMAC-SHA-256) | `=`, `<>` on `_eq` and `text_search` domains | Whether two values are equal — nothing else | -| `ob` | `eql_v3.ore_block_256` (composite: array of `bytea` block terms) | Array of hex-encoded ORE blocks | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ore` domains — and `=` / `<>`, since ORE comparison collapses to equality | The relative order of two values | -| `bf` | `eql_v3.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit) | `@>` / `<@` token containment on `_match` domains | Probabilistic token overlap between values | - -Notes on the wire shapes: - -- **`ob` block count is width-agnostic**: 8 blocks for the int scalars, 12 for timestamp, 14 for numeric — the array just carries more block strings. -- **`bf` positions are signed**: EQL stores the filter as PostgreSQL `smallint[]`, and filters sized above 32768 emit upper-half bit positions as *negative* signed values. Consumers must use a signed 16-bit integer type. - -The capability is encoded as **required keys**: the payload for an `eql_v3.text_eq` column must carry `hm`; an `eql_v3.int4_ord` payload must carry `ob` (and only `ob` — equality on `_ord` domains compares ORE terms, so no `hm` is needed); a `text_match` payload must carry `bf`; a `text_search` payload carries all three. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. See [Types](/reference/eql/types) for the domain-to-capability mapping, and [Searchable encryption](/concepts/searchable-encryption) for what these terms do and don't leak. - -## JSON documents: the `sv` vector - -An [encrypted JSON document](/reference/eql/json) uses a different payload shape: no root ciphertext, and an `sv` array with one encrypted entry per path in the document. Each entry carries: - -| Key | Contents | -| --- | --- | -| `s` | Selector — a deterministic hash of the JSON path. Required; entry matching compares selectors first. | -| `c` | Ciphertext for the node at that path. | -| `hm` **or** `oc` | Exactly one, never both — the domain `CHECK` enforces the exclusivity. `hm` (HMAC-256) on Boolean/`null` leaves and Object/Array roots; `oc` (CLLW ORE, backed by `eql_v3.ore_cllw`) on String/Number leaves. | -| `a` | Optional array marker — `true` when the selector points at an array context. | - -The decoded `oc` value starts with a domain-tag byte (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext, so numeric and string values in one column keep a consistent total order. Earlier payload versions split this into two fields — `ocf` (fixed-width, numeric) and `ocv` (variable-width, string) — which consolidated into the single `oc` key; the tag byte now carries the distinction. - -A containment **query** payload (`eql_v3.ste_vec_query`) has the same `sv` shape but its entries carry no `c` — containment matches selectors and index terms, never ciphertexts. - -## Example payloads - -A scalar payload for an `eql_v3.text_search` column (lookup + ordering + free-text match, so all three terms are required): - -```json -{ - "v": 2, - "i": { "t": "users", "c": "email" }, - "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", - "hm": "9c8ec1d2f9932b979b1bf3f09f8a4e2f6a41f8de2f0c8b7a52e1f5c3d4b6a790", - "ob": ["7a1fd0c2...", "d24c9be1...", "03fa66b8..."], - "bf": [42, 1290, -8113, 30201] -} -``` - -- `v`, `i`, `c` — the envelope -- `hm` — equality term: `WHERE email = $1` compares this -- `ob` — ordering term: `ORDER BY` and range comparisons walk these blocks -- `bf` — bloom-filter term: `@>` token containment tests these bit positions - -A JSON document payload for an `eql_v3.json` column: - -```json -{ - "v": 2, - "k": "sv", - "i": { "t": "orders", "c": "metadata" }, - "sv": [ - { "s": "2517068c0d1f9d4d41d2c666211f785e", "c": "mBbKmM...", "hm": "b0e0..." }, - { "s": "f510853a4ab9d4f75f51a533ac264c5d", "c": "mBbKmQ...", "oc": "01a3f2..." }, - { "s": "33743aed3ae636f6bf05cff11ac4b519", "c": "mBbKmR...", "oc": "004e19..." } - ] -} -``` - -- First entry: an object root — `hm` only, equality/containment -- Second entry: a string leaf — `oc` starting with tag `01` -- Third entry: a numeric leaf — `oc` starting with tag `00` - -And the containment needle the client builds for a `@>` query — index terms, no ciphertexts: - -```json -{ - "sv": [ - { "s": "f510853a4ab9d4f75f51a533ac264c5d", "oc": "01a3f2..." } - ] -} -``` - -## Machine-readable schemas - -The [EQL repository](https://github.com/cipherstash/encrypt-query-language) publishes the format as JSON Schema in two places: - -- **`crates/eql-bindings/schema/`** — one schema per scalar domain (`$id`s under `https://schemas.cipherstash.com/eql/v3/`), generated from the canonical Rust wire types in the `eql-bindings` crate. TypeScript bindings are generated from the same definitions, so every producer and consumer shares one source of truth. -- **`docs/reference/schema/`** — full-payload schemas covering both the scalar and `sv` document shapes. These files are currently named for the v2.x payload releases (`eql-payload-v2.2.schema.json`, `eql-payload-v2.3.schema.json`) and reference `eql_v2` function names, even though the current SQL surface is `eql_v3` — the v2.3 schema is the applicable document-shape definition, matching the still-`v: 2` envelope. - -## Who produces and consumes this - -- **Produce:** the Stack SDK and CipherStash Proxy encrypt plaintext into these payloads — ciphertext, index terms, selectors — using keys the database never holds. -- **Consume:** EQL's domain `CHECK`s validate the shape on write, and its operators and extractor functions ([Operators](/reference/eql/operators), [Indexes](/reference/eql/indexes)) compare the term keys at query time. - -The division is strict: EQL never sees plaintext, and the clients never rely on the database for key material. diff --git a/content/docs/reference/eql/sorting.mdx b/content/docs/reference/eql/sorting.mdx new file mode 100644 index 0000000..b684f3e --- /dev/null +++ b/content/docs/reference/eql/sorting.mdx @@ -0,0 +1,96 @@ +--- +title: Sorting +description: "ORDER BY on encrypted columns: which variants sort, when to write the sort key in extractor form, keyset pagination, and the ::jsonb projection trap." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +`ORDER BY` on an encrypted column needs an ORE ordering term: it works on `_ord` / `_ord_ore` variants of every scalar and on `text_search`. ORE terms are order-preserving, so the database sorts ciphertext in exactly the order the plaintext would sort — without decrypting anything. Which variants carry the term is covered in [Numbers & dates](/reference/eql/numbers-and-dates) and [Text](/reference/eql/text); the variant model itself is in [Core concepts](/reference/eql/core-concepts). + +Sorting a variant *without* an ORE term (`_eq`, `text_match`, bare storage variants) won't raise — but the order is meaningless. Type the column as an `_ord` variant when ordering matters. + +## Bare form vs extractor form + +Both of these sort correctly: + +```sql +-- Bare form +SELECT * FROM users ORDER BY created_at DESC; + +-- Extractor form +SELECT * FROM users ORDER BY eql_v3.ord_term(created_at) DESC; +``` + +The difference is the plan. The planner inlines encrypted operators in *predicates*, so a `WHERE created_at < $1` matches a btree on `eql_v3.ord_term(created_at)` without rewriting — but it does **not** rewrite *sort keys*. Bare `ORDER BY created_at` therefore adds a `Sort` node above the scan, and that sort's cost scales linearly with the rows passing the filter. + +Writing the sort key in extractor form makes it textually match the index expression, so rows stream out of the btree already ordered — no `Sort` node at all: + +```sql +CREATE INDEX users_created_at_ord + ON users USING btree (eql_v3.ord_term(created_at)); +ANALYZE users; + +SELECT * FROM users + WHERE created_at < $1::eql_v3.timestamp_ord + ORDER BY eql_v3.ord_term(created_at) DESC + LIMIT 10; +-- Index Scan Backward using users_created_at_ord — no Sort node +``` + +At large row counts this is the difference between seconds and milliseconds, and it matters most for `LIMIT` queries: with a `Sort` node, Postgres must sort *every* matching row before it can return the top 10; streaming from the index, it stops after 10. + +Rule of thumb: bare form is fine for small result sets or when no ordering index exists; any hot query with `ORDER BY ... LIMIT` should use the extractor form. Confirm with `EXPLAIN (COSTS OFF)` — a `Sort` node above an `Index Scan` means the sort key didn't match the index. Full plan-reading guidance is in [Indexes](/reference/eql/indexes). + +## `ASC`, `DESC`, and `NULLS` + +`ASC` / `DESC` behave normally — a btree serves both directions (backward scans handle `DESC`). SQL `NULL` column values are not encrypted, so `NULLS FIRST` / `NULLS LAST` also behave normally: + +```sql +SELECT * FROM users +ORDER BY eql_v3.ord_term(last_login) DESC NULLS LAST; +``` + +## Keyset pagination + +`OFFSET` pagination degrades on encrypted columns the same way it does on plaintext ones — every page re-sorts and discards the rows before the offset. Keyset (cursor) pagination composes an encrypted range filter with an extractor-form sort: + +```sql +-- Page 1 +SELECT id, email, created_at FROM users + ORDER BY eql_v3.ord_term(created_at) DESC + LIMIT 20; + +-- Next page: pass the last row's created_at back, re-encrypted as the cursor +SELECT id, email, created_at FROM users + WHERE created_at < $1::eql_v3.timestamp_ord + ORDER BY eql_v3.ord_term(created_at) DESC + LIMIT 20; +``` + +Both the filter and the sort ride the same btree on `eql_v3.ord_term(created_at)`, so every page is an index scan that stops after 20 rows. The client re-encrypts the cursor value for the next request — the database only ever sees ciphertext. + +## The `::jsonb` projection trap + +<Callout type="warn"> +If you project the column with a cast and sort on it — `SELECT col::jsonb ... ORDER BY col` — Postgres folds the cast into the scan and uses `(col)::jsonb` as the sort key, which matches no index. Project the column raw and let the client decode it, or write the sort key as `eql_v3.ord_term(col)`, which sidesteps the problem entirely. +</Callout> + +## Sorting extracted JSON leaves + +String and Number leaves inside an encrypted JSON document carry a CLLW ORE term, so they sort too — the extractor is `eql_v3.ore_cllw` on the extracted entry: + +```sql +SELECT * FROM orders +ORDER BY eql_v3.ore_cllw(metadata -> 'total_selector'::text) DESC +LIMIT 10; +``` + +A btree on the same `eql_v3.ore_cllw(...)` expression streams this ordered, exactly like `ord_term` on a scalar column. Selectors, node types, and which leaves are orderable are covered in [JSON](/reference/eql/json). + +## Where to go next + +- [Indexes](/reference/eql/indexes) — the btree recipe behind every sort on this page, plus `EXPLAIN` verification and large-table build guidance. +- [Filtering](/reference/eql/filtering) — the range predicates that pair with these sorts. +- [Grouping & aggregates](/reference/eql/grouping-and-aggregates) — `MIN` / `MAX`, which use the same ordering term. diff --git a/content/docs/reference/eql/text.mdx b/content/docs/reference/eql/text.mdx new file mode 100644 index 0000000..c70ee13 --- /dev/null +++ b/content/docs/reference/eql/text.mdx @@ -0,0 +1,157 @@ +--- +title: Text +description: "The complete reference for encrypted text columns: all six text domain variants, the multi-term payload, why LIKE is gone everywhere, and bloom-filter token containment as the encrypted free-text match." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +Text is the richest encrypted scalar. Beyond the four variants every scalar type gets, `text` adds two of its own: `text_match` for encrypted free-text matching, and `text_search` for columns you need to look up, sort, *and* search. Emails, names, tax IDs, addresses — this page is the full surface for all of them. + +## Variants + +All six are `jsonb`-backed domains. Which one you declare fixes the column's query capability — the variant model itself is covered in [Core concepts](/reference/eql/core-concepts): + +| Domain variant | Capability | Index terms carried | +| --- | --- | --- | +| `eql_v3.text` | Storage and decryption only. Every comparison operator is blocked — only `IS NULL` / `IS NOT NULL` work. | none | +| `eql_v3.text_eq` | Equality: `=` and `<>` (plus `IN`, `GROUP BY`, `DISTINCT`, equijoins). | `hm` (HMAC-256) | +| `eql_v3.text_ord` | Full comparison surface: `=` `<>` `<` `<=` `>` `>=`, `BETWEEN`, `ORDER BY`, `MIN` / `MAX`. | `ob` (ORE block) | +| `eql_v3.text_ord_ore` | Identical to `text_ord` — a twin name that documents intent. | `ob` (ORE block) | +| `eql_v3.text_match` | Encrypted free-text token containment via `@>` / `<@`. No equality, no ordering. | `bf` (bloom filter) | +| `eql_v3.text_search` | Everything: equality, ordering, and token containment combined. | `hm` + `ob` + `bf` | + +Declare only the capabilities you query on — each term class reveals different structure to an observer: equality terms reveal value repetition, ORE terms reveal ordering, bloom terms reveal token overlap (see [Searchable encryption](/concepts/searchable-encryption)): + +```sql +CREATE TABLE users ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email eql_v3.text_search, -- lookup, sort, and free-text match + name eql_v3.text_match, -- free-text match only + tax_id eql_v3.text_eq, -- exact lookup only + notes eql_v3.text -- store and decrypt only +); +``` + +## Payload + +A value for a `text_search` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus all three index terms: + +```json +{ + "v": 2, + "i": { "t": "users", "c": "email" }, + "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", + "hm": "9c8ec1d2f9932b979b1bf3f09f8a4e2f6a41f8de2f0c8b7a52e1f5c3d4b6a790", + "ob": ["7a1fd0c2...", "d24c9be1...", "03fa66b8..."], + "bf": [42, 1290, -8113, 30201] +} +``` + +- `hm` — equality term: `WHERE email = $1` compares this +- `ob` — ordering term: `ORDER BY` and range comparisons walk these blocks +- `bf` — bloom-filter term: `@>` token containment tests these bit positions + +The narrower variants carry only their own term: a `text_eq` payload carries `hm` only, `text_match` carries `bf` only, and `text_ord` / `text_ord_ore` carry `ob` only (no `hm` — equality on `_ord` variants compares ORE terms, see [Core concepts](/reference/eql/core-concepts)). A payload missing its variant's required term fails the domain `CHECK` at write time. + +**`bf` positions are signed**: EQL stores the filter as PostgreSQL `smallint[]`, and filters sized above 32768 emit upper-half bit positions as *negative* signed values. Consumers must use a signed 16-bit integer type. + +## Operators and functions + +The function forms exist for managed platforms that disallow custom operators — they take the same typed arguments and resolve identically. + +| SQL operator | Function form | `eql_v3.text` | `text_eq` | `text_ord` / `text_ord_ore` | `text_match` | `text_search` | +| --- | --- | :---: | :---: | :---: | :---: | :---: | +| `=` / `<>` | `eql_v3.eq(a, b)` / `eql_v3.neq(a, b)` | ❌ | ✅ | ✅ | ❌ | ✅ | +| `<` `<=` `>` `>=` | `eql_v3.lt` / `lte` / `gt` / `gte` | ❌ | ❌ | ✅ | ❌ | ✅ | +| `@>` / `<@` | `eql_v3.contains(a, b)` / `eql_v3.contained_by(a, b)` | ❌ | ❌ | ❌ | ✅ | ✅ | +| `LIKE` / `ILIKE` (`~~` / `~~*`) | none | ❌ | ❌ | ❌ | ❌ | ❌ | +| `IN` / `GROUP BY` / `DISTINCT` | desugar to `=` / need an equality term | ❌ | ✅ | ✅ | ❌ | ✅ | +| `ORDER BY`, `MIN` / `MAX` | `eql_v3.min(col)` / `eql_v3.max(col)` | ❌ | ❌ | ✅ | ❌ | ✅ | +| `IS NULL` / `IS NOT NULL` | — | ✅ | ✅ | ✅ | ✅ | ✅ | + +Blocked cells raise an `operator … is not supported` exception — they never silently return wrong rows. Operands must be typed (`$1::eql_v3.text_eq`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). + +## There is no `LIKE` + +`LIKE` and `ILIKE` (`~~` / `~~*`) raise on **every** encrypted-domain variant — including `text_match` and `text_search`. SQL pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter token containment — `@>` on a `text_match` or `text_search` column: + +```sql +-- ❌ Raises: operator not supported +SELECT * FROM users WHERE email LIKE '%alice%'; + +-- ✅ Encrypted free-text match +SELECT * FROM users WHERE email @> $1::eql_v3.text_match; +``` + +`@>` / `<@` here is **probabilistic ngram-bloom containment** — it tests whether the encrypted text contains the (encrypted) search terms. It is not JSONB containment and not `LIKE`. The client encrypts the search term into a bloom-filter query value; false positives are possible, false negatives are not. There are no `like` / `ilike` function forms either — text matching is `eql_v3.contains` on a `text_match` value. + +## Example queries + +### Exact lookup + +Equality on a `text_eq` column compares HMAC terms. `IN` desugars to `=`: + +```sql +SELECT * FROM users WHERE tax_id = $1::eql_v3.text_eq; + +SELECT * FROM users +WHERE tax_id IN ($1::eql_v3.text_eq, $2::eql_v3.text_eq); +``` + +### Free-text match + +The client encrypts the search term into the bloom-filter needle: + +```sql +SELECT * FROM users WHERE name @> $1::eql_v3.text_match; + +-- Function form, for platforms without custom operators +SELECT * FROM users WHERE eql_v3.contains(name, $1::eql_v3.text_match); +``` + +### The works: `text_search` + +A `text_search` column answers exact lookup, free-text match, and ordering — here, all three in one query: + +```sql +SELECT id, email FROM users +WHERE email @> $1::eql_v3.text_match -- token containment on bf + AND email <> $2::eql_v3.text_eq -- exclude an exact value via hm +ORDER BY eql_v3.ord_term(email) -- sort on ob +LIMIT 20; +``` + +### Sorting text + +ORE terms are order-preserving, so `ORDER BY` sorts encrypted text correctly. Write the sort key in extractor form so a btree index can do the ordering instead of a `Sort` node — see [Sorting](/reference/eql/sorting): + +```sql +SELECT * FROM users +ORDER BY eql_v3.ord_term(email) +LIMIT 50; +``` + +`MIN` / `MAX` work on any ord-capable text column too: + +```sql +SELECT eql_v3.min(email) FROM users; +``` + +## Where to next + +<Cards> + <Card title="Indexes" href="/reference/eql/indexes"> + Hash on `eq_term`, btree on `ord_term`, GIN on `match_term`. + </Card> + <Card title="Filtering" href="/reference/eql/filtering"> + WHERE-clause patterns across all encrypted types. + </Card> + <Card title="Sorting" href="/reference/eql/sorting"> + Extractor-form sort keys and index-backed ordering. + </Card> + <Card title="Joins" href="/reference/eql/joins"> + Equijoins on encrypted text columns, and the same-keyset rule. + </Card> +</Cards> diff --git a/content/docs/reference/eql/types.mdx b/content/docs/reference/eql/types.mdx deleted file mode 100644 index e4eb005..0000000 --- a/content/docs/reference/eql/types.mdx +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: Encrypted types -description: "The eql_v3 encrypted-domain type families: which domain variant to declare for each scalar type, and what each variant lets you query." -type: reference -components: [eql] -verifiedAgainst: - eql: "3.0.0" ---- - -EQL ships its searchable-encryption surface as PostgreSQL **domains in the `eql_v3` schema**. There are two kinds: - -- **Per-scalar encrypted-domain types** — `eql_v3.int4`, `eql_v3.text`, `eql_v3.timestamp`, and so on. One family of domain *variants* per scalar type. -- **An encrypted-JSON document type** — `eql_v3.json` — for structured encryption of whole JSONB documents. See [JSON support](/reference/eql/json). - -A column's query capability is fixed by the **domain variant you type it as**. There is no database-side configuration step: which index terms travel in a value's payload is decided by the encryption client (the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy)), and the column's domain variant is what makes the matching operators resolve. - -## The family model - -Every scalar type `<T>` generates a storage-only variant plus the query variants its capabilities allow. All variants are `jsonb`-backed domains. - -| Domain variant | Capability | Index term carried | -| --- | --- | --- | -| `eql_v3.<T>` | Storage and decryption only. Every comparison operator is blocked — only `IS NULL` / `IS NOT NULL` work. | none | -| `eql_v3.<T>_eq` | Equality: `=` and `<>` (plus `IN`, `GROUP BY`, `DISTINCT`, equijoins). | `hm` (`eql_v3.hmac_256`) | -| `eql_v3.<T>_ord` / `eql_v3.<T>_ord_ore` | Full comparison surface: `=` `<>` `<` `<=` `>` `>=`, `BETWEEN`, `ORDER BY`, and the `eql_v3.min` / `eql_v3.max` aggregates. | `ob` (`eql_v3.ore_block_256`) | -| `eql_v3.text_match` (text only) | Encrypted free-text token containment via `@>` / `<@`. No equality, no ordering. | `bf` (`eql_v3.bloom_filter`) | -| `eql_v3.text_search` (text only) | Everything: equality, ordering, and token containment combined. | `hm` + `ob` + `bf` | - -Two things worth calling out: - -- **The bare variant blocks everything.** `eql_v3.<T>` carries no index term. Querying it with any comparison operator raises an "operator not supported" exception. Use it for columns you only ever store and decrypt. If you later need to query, type the column as a query variant — or cast at the call site (`col::eql_v3.int4_ord`) if the payload already carries the term. -- **`_ord` and `_ord_ore` are twins.** They are byte-identical surfaces backed by the same ORE block term. Pick the name that documents intent — "ordered" versus "ordered via ORE block". Both support the full ordered surface and `MIN` / `MAX`. - -## Type matrix - -The scalar tokens that ship in EQL 3.0.0 are `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool`. - -| Scalar | `eql_v3.<T>` | `<T>_eq` | `<T>_ord` | `<T>_ord_ore` | `text_match` | `text_search` | -| --- | :---: | :---: | :---: | :---: | :---: | :---: | -| `int2` | ✅ | ✅ | ✅ | ✅ | — | — | -| `int4` | ✅ | ✅ | ✅ | ✅ | — | — | -| `int8` | ✅ | ✅ | ✅ | ✅ | — | — | -| `float4` | ✅ | ✅ | ✅ | ✅ | — | — | -| `float8` | ✅ | ✅ | ✅ | ✅ | — | — | -| `numeric` | ✅ | ✅ | ✅ | ✅ | — | — | -| `date` | ✅ | ✅ | ✅ | ✅ | — | — | -| `timestamp` | ✅ | ✅ | ✅ | ✅ | — | — | -| `text` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| `bool` | ✅ | ❌ | ❌ | ❌ | — | — | - -<Callout type="warn"> -**`bool` is storage-only by design.** A two-value column has too little cardinality for any searchable index to be safe — an equality index over `true`/`false` would leak the value distribution outright. EQL ships only `eql_v3.bool`, with no `_eq` or `_ord` variants. Store and decrypt boolean columns; filter on them client-side. -</Callout> - -## Index terms - -Each query variant stores one or more encrypted index terms alongside the ciphertext: - -- **`hm`** — an HMAC-256 term (`eql_v3.hmac_256`). Supports exact equality. -- **`ob`** — an ORE block term (`eql_v3.ore_block_256`). Order-revealing: supports comparison and sorting. -- **`bf`** — a bloom filter term (`eql_v3.bloom_filter`). Supports probabilistic ngram token containment. - -The payload structure — envelope keys plus per-variant term keys — is documented in [Payload format](/reference/eql/payload-format). What each term mathematically reveals about the plaintext (and why you should only carry the terms you need) is covered in [Searchable encryption](/concepts/searchable-encryption). - -## Encrypted JSON: `eql_v3.json` - -`eql_v3.json` is the encrypted-JSON document domain, built on the structured-encryption ("ste_vec") model: a JSONB document is encrypted into a searchable vector of terms, one per path inside the document, supporting containment (`@>`), field access (`->` / `->>`), and path queries. It has its own operator and function surface — see [JSON support](/reference/eql/json). - -## Choosing a variant - -Declare only the capabilities you query on. Every index term a value carries is extra material stored in the database, and each term class reveals different structure to an observer — equality terms reveal value repetition, ORE terms reveal ordering, bloom terms reveal token overlap (see [Searchable encryption](/concepts/searchable-encryption)): - -- Never queried, only decrypted → bare `eql_v3.<T>` -- Exact lookup, `IN`, joins, `GROUP BY` → `<T>_eq` -- Ranges, `ORDER BY`, `MIN`/`MAX` → `<T>_ord` -- Free-text matching on text → `text_match` -- Text you need to look up, sort, *and* search → `text_search` - -The variant you declare must match the terms the client is configured to emit for that column — the domain makes the operator resolve, but the term in the payload is what makes it answer. - -## Example - -```sql -CREATE TABLE users ( - id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v3.text_search, -- lookup, sort, and free-text match - name eql_v3.text_match, -- free-text match only - tax_id eql_v3.text_eq, -- exact lookup only - salary eql_v3.int8_ord, -- range queries, ORDER BY, MIN/MAX - is_active eql_v3.bool, -- storage only (by design) - created_at eql_v3.timestamp_ord -); -``` - -Once the table exists, add functional indexes on the term extractors so queries engage an index — see [Indexes](/reference/eql/indexes). The operators each variant supports are listed in [Operators](/reference/eql/operators). diff --git a/v2-redirects.mjs b/v2-redirects.mjs index 1f068b1..a4d8129 100644 --- a/v2-redirects.mjs +++ b/v2-redirects.mjs @@ -73,7 +73,7 @@ export const v2Redirects = [ }, { source: "/stack/cipherstash/encryption/queries", - destination: "/reference/eql/operators", + destination: "/reference/eql/filtering", permanent: true, }, // configuration, encrypt-decrypt, bulk-operations, models, schema, storing-data @@ -286,7 +286,7 @@ export const v2Redirects = [ }, { source: "/stack/reference/cipher-cell", - destination: "/reference/eql/payload-format", + destination: "/reference/eql/core-concepts", permanent: true, }, { From 9cecb45f043a5c430fa318e262370fa59965cf21 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 20:18:03 +1000 Subject: [PATCH 10/38] refactor(v2): SEM specifiers, Tailwind-style variant enumeration, payload v:3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback on the EQL section: - Variant tables: generic form first, then full enumeration of every concrete domain name (Tailwind-style); capability column made concise; "index term carried" column dropped — term internals live in core-concepts' payload anatomy - SEM specifiers documented as a concept in core-concepts: a trailing mechanism suffix (_ord_ore) pins WHICH searchable-encryption mechanism implements a capability; _ord tracks the default (currently ORE). Replaces the "twins" framing. Each orderable type page lists its specifiers under an "SEM specifiers" heading, noting the OPE specifier arriving for all orderable types (incl. text) in the v3 release - Payload `v` field documented as the EQL version (3) per team decision 2026-07-02; all payload examples updated from v:2 Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- content/docs/reference/eql/core-concepts.mdx | 34 +++++++++----- content/docs/reference/eql/json.mdx | 2 +- .../docs/reference/eql/numbers-and-dates.mdx | 46 ++++++++++++++----- content/docs/reference/eql/text.mdx | 31 +++++++++---- 4 files changed, 79 insertions(+), 34 deletions(-) diff --git a/content/docs/reference/eql/core-concepts.mdx b/content/docs/reference/eql/core-concepts.mdx index a262fe8..11b6a0f 100644 --- a/content/docs/reference/eql/core-concepts.mdx +++ b/content/docs/reference/eql/core-concepts.mdx @@ -17,18 +17,28 @@ There is no database-side configuration table. Earlier EQL versions tracked encr For any scalar type `<T>`, the family looks like this: -| Domain variant | Capability | Index term carried | -| --- | --- | --- | -| `eql_v3.<T>` | Storage and decryption only. Every comparison operator is blocked — only `IS NULL` / `IS NOT NULL` work. | none | -| `eql_v3.<T>_eq` | Equality: `=` and `<>` (plus `IN`, `GROUP BY`, `DISTINCT`, equijoins). | `hm` (`eql_v3.hmac_256`) | -| `eql_v3.<T>_ord` / `eql_v3.<T>_ord_ore` | Full comparison surface: `=` `<>` `<` `<=` `>` `>=`, `BETWEEN`, `ORDER BY`, and the `eql_v3.min` / `eql_v3.max` aggregates. | `ob` (`eql_v3.ore_block_256`) | -| `eql_v3.text_match` (text only) | Encrypted free-text token containment via `@>` / `<@`. No equality, no ordering. | `bf` (`eql_v3.bloom_filter`) | -| `eql_v3.text_search` (text only) | Everything: equality, ordering, and token containment combined. | `hm` + `ob` + `bf` | +| Domain variant | Capability | +| --- | --- | +| `eql_v3.<T>` | Storage and decryption only. | +| `eql_v3.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | +| `eql_v3.<T>_ord` | Comparisons (`<` … `>=`), `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | +| `eql_v3.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | +| `eql_v3.text_match` (text only) | Free-text token containment: `@>` / `<@`. | +| `eql_v3.text_search` (text only) | Equality + ordering + token containment. | Two things worth calling out: - **The bare variant blocks everything.** `eql_v3.<T>` carries no index term. Querying it with any comparison operator raises an "operator not supported" exception. Use it for columns you only ever store and decrypt — [Booleans](/reference/eql/booleans) covers this pattern in full. -- **`_ord` and `_ord_ore` are twins.** They are byte-identical surfaces backed by the same ORE block term. Pick the name that documents intent — "ordered" versus "ordered via ORE block". Both support the full ordered surface and `MIN` / `MAX`. +- **Which index term backs each capability** is an implementation detail of the payload — covered in [Anatomy of an encrypted value](#anatomy-of-an-encrypted-value) below. + +### SEM specifiers + +A trailing mechanism suffix — the `_ore` in `_ord_ore` — is a **SEM specifier**: it pins *which* searchable-encryption mechanism implements the capability, rather than just declaring the capability itself. + +- `eql_v3.<T>_ord` declares *orderable* and leaves the mechanism to EQL's default — currently ORE (order-revealing encryption). +- `eql_v3.<T>_ord_ore` declares *orderable via ORE, explicitly*. Today the two are byte-identical surfaces backed by the same term. + +The distinction earns its keep as mechanisms multiply: the EQL v3 release adds an **OPE** (order-preserving encryption) specifier for every orderable type — including `text` — at which point pinning a specifier documents and freezes a column's mechanism choice, while unspecified variants track the default. Each type page lists its available specifiers under an "SEM specifiers" heading. Declaring a table is just typing each column as the variant it needs: @@ -55,12 +65,12 @@ Every payload carries three envelope keys. Each `eql_v3` domain's `CHECK` constr | Key | Contents | Notes | | --- | --- | --- | -| `v` | Payload version | Always exactly `2` on the wire. The domain `CHECK`s assert it and raise on any other value. | +| `v` | The EQL version | `3` — the payload version matches the EQL major version. The domain `CHECK`s assert it and raise on any other value. | | `i` | Ident: `{"t": "<table>", "c": "<column>"}` | Binds the ciphertext to the table and column it was encrypted for. Both keys required. | | `c` | Ciphertext | The opaque, non-deterministic encrypted blob (mp_base85-encoded). Never used in comparisons. | <Callout> -`eql_v3` names the **SQL schema generation**, not the payload version. The JSON envelope version is still `v: 2` — the wire field names are unchanged from EQL v2, and the domain `CHECK`s assert `v = 2`. +Payloads produced by EQL v2 clients carried `v: 2`; from 3.0.0 the payload version and the EQL version move together. </Callout> A `k` discriminator (`"ct"` for a scalar ciphertext, `"sv"` for a JSON document) also appears on payloads emitted by the clients, distinguishing the two top-level shapes. @@ -81,7 +91,7 @@ A scalar payload for an `eql_v3.text_search` column (lookup + ordering + free-te ```json { - "v": 2, + "v": 3, "i": { "t": "users", "c": "email" }, "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", "hm": "9c8ec1d2f9932b979b1bf3f09f8a4e2f6a41f8de2f0c8b7a52e1f5c3d4b6a790", @@ -102,7 +112,7 @@ Encrypted JSON documents use a different payload shape — an `sv` array with on The [EQL repository](https://github.com/cipherstash/encrypt-query-language) publishes the format as JSON Schema in two places: - **`crates/eql-bindings/schema/`** — one schema per scalar domain (`$id`s under `https://schemas.cipherstash.com/eql/v3/`), generated from the canonical Rust wire types in the `eql-bindings` crate. TypeScript bindings are generated from the same definitions, so every producer and consumer shares one source of truth. -- **`docs/reference/schema/`** — full-payload schemas covering both the scalar and `sv` document shapes. These files are currently named for the v2.x payload releases (`eql-payload-v2.2.schema.json`, `eql-payload-v2.3.schema.json`) and reference `eql_v2` function names, even though the current SQL surface is `eql_v3` — the v2.3 schema is the applicable document-shape definition, matching the still-`v: 2` envelope. +- **`docs/reference/schema/`** — full-payload schemas covering both the scalar and `sv` document shapes. These files are still named for the v2.x payload releases (`eql-payload-v2.2.schema.json`, `eql-payload-v2.3.schema.json`); the v2.3 schema describes the document shape, with the payload version field moving to `3` alongside the EQL 3.0.0 release. ## The typed-operand rule diff --git a/content/docs/reference/eql/json.mdx b/content/docs/reference/eql/json.mdx index 4204ca8..93eb920 100644 --- a/content/docs/reference/eql/json.mdx +++ b/content/docs/reference/eql/json.mdx @@ -38,7 +38,7 @@ A document payload for an `eql_v3.json` column: ```json { - "v": 2, + "v": 3, "k": "sv", "i": { "t": "orders", "c": "metadata" }, "sv": [ diff --git a/content/docs/reference/eql/numbers-and-dates.mdx b/content/docs/reference/eql/numbers-and-dates.mdx index 1d01dec..a78052f 100644 --- a/content/docs/reference/eql/numbers-and-dates.mdx +++ b/content/docs/reference/eql/numbers-and-dates.mdx @@ -13,16 +13,29 @@ There is no free-text matching for these types — `_match` and `_search` are [t ## Variants -Each of the eight scalar types generates the same four `jsonb`-backed domain variants: - -| Domain variant | Capability | Index term carried | -| --- | --- | --- | -| `eql_v3.<T>` | Storage and decryption only. Every comparison operator is blocked — only `IS NULL` / `IS NOT NULL` work. | none | -| `eql_v3.<T>_eq` | Equality: `=` and `<>` (plus `IN`, `GROUP BY`, `DISTINCT`, equijoins). | `hm` (HMAC-256) | -| `eql_v3.<T>_ord` | Full comparison surface: `=` `<>` `<` `<=` `>` `>=`, `BETWEEN`, `ORDER BY`, and `MIN` / `MAX`. | `ob` (ORE block) | -| `eql_v3.<T>_ord_ore` | Identical to `<T>_ord` — a twin name that documents intent. | `ob` (ORE block) | - -Declare only the capability you query on — each index term class reveals different structure to an observer (see [Searchable encryption](/concepts/searchable-encryption)), and the variant model itself is covered in [Core concepts](/reference/eql/core-concepts): +Each of the eight scalar types generates the same `jsonb`-backed domain variants. The generic form: + +| Domain variant | Capability | +| --- | --- | +| `eql_v3.<T>` | Storage and decryption only. | +| `eql_v3.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | +| `eql_v3.<T>_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | +| `eql_v3.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | + +And every concrete domain this page covers: + +| Type | Variants | +| --- | --- | +| `int2` | `eql_v3.int2` · `eql_v3.int2_eq` · `eql_v3.int2_ord` · `eql_v3.int2_ord_ore` | +| `int4` | `eql_v3.int4` · `eql_v3.int4_eq` · `eql_v3.int4_ord` · `eql_v3.int4_ord_ore` | +| `int8` | `eql_v3.int8` · `eql_v3.int8_eq` · `eql_v3.int8_ord` · `eql_v3.int8_ord_ore` | +| `float4` | `eql_v3.float4` · `eql_v3.float4_eq` · `eql_v3.float4_ord` · `eql_v3.float4_ord_ore` | +| `float8` | `eql_v3.float8` · `eql_v3.float8_eq` · `eql_v3.float8_ord` · `eql_v3.float8_ord_ore` | +| `numeric` | `eql_v3.numeric` · `eql_v3.numeric_eq` · `eql_v3.numeric_ord` · `eql_v3.numeric_ord_ore` | +| `date` | `eql_v3.date` · `eql_v3.date_eq` · `eql_v3.date_ord` · `eql_v3.date_ord_ore` | +| `timestamp` | `eql_v3.timestamp` · `eql_v3.timestamp_eq` · `eql_v3.timestamp_ord` · `eql_v3.timestamp_ord_ore` | + +Declare only the capability you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)), and the variant model itself is covered in [Core concepts](/reference/eql/core-concepts): ```sql CREATE TABLE employees ( @@ -35,13 +48,24 @@ CREATE TABLE employees ( ); ``` +### SEM specifiers + +All eight types take the same mechanism specifiers on their orderable variant (the concept is defined in [Core concepts](/reference/eql/core-concepts#sem-specifiers)): + +| Specifier | Meaning | +| --- | --- | +| `_ord` | Orderable, using EQL's default mechanism (currently ORE). | +| `_ord_ore` | Orderable via ORE, pinned explicitly. | + +The EQL v3 release adds an OPE specifier for every orderable type; unspecified `_ord` columns keep tracking the default. + ## Payload A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus the `ob` ordering term. Here is a payload for the `eql_v3.int8_ord` `salary` column: ```json { - "v": 2, + "v": 3, "i": { "t": "employees", "c": "salary" }, "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", "ob": [ diff --git a/content/docs/reference/eql/text.mdx b/content/docs/reference/eql/text.mdx index c70ee13..25228c9 100644 --- a/content/docs/reference/eql/text.mdx +++ b/content/docs/reference/eql/text.mdx @@ -13,16 +13,16 @@ Text is the richest encrypted scalar. Beyond the four variants every scalar type All six are `jsonb`-backed domains. Which one you declare fixes the column's query capability — the variant model itself is covered in [Core concepts](/reference/eql/core-concepts): -| Domain variant | Capability | Index terms carried | -| --- | --- | --- | -| `eql_v3.text` | Storage and decryption only. Every comparison operator is blocked — only `IS NULL` / `IS NOT NULL` work. | none | -| `eql_v3.text_eq` | Equality: `=` and `<>` (plus `IN`, `GROUP BY`, `DISTINCT`, equijoins). | `hm` (HMAC-256) | -| `eql_v3.text_ord` | Full comparison surface: `=` `<>` `<` `<=` `>` `>=`, `BETWEEN`, `ORDER BY`, `MIN` / `MAX`. | `ob` (ORE block) | -| `eql_v3.text_ord_ore` | Identical to `text_ord` — a twin name that documents intent. | `ob` (ORE block) | -| `eql_v3.text_match` | Encrypted free-text token containment via `@>` / `<@`. No equality, no ordering. | `bf` (bloom filter) | -| `eql_v3.text_search` | Everything: equality, ordering, and token containment combined. | `hm` + `ob` + `bf` | +| Domain variant | Capability | +| --- | --- | +| `eql_v3.text` | Storage and decryption only. | +| `eql_v3.text_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | +| `eql_v3.text_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | +| `eql_v3.text_ord_ore` | As `text_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | +| `eql_v3.text_match` | Free-text token containment: `@>` / `<@`. | +| `eql_v3.text_search` | Equality + ordering + token containment. | -Declare only the capabilities you query on — each term class reveals different structure to an observer: equality terms reveal value repetition, ORE terms reveal ordering, bloom terms reveal token overlap (see [Searchable encryption](/concepts/searchable-encryption)): +Declare only the capabilities you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)): ```sql CREATE TABLE users ( @@ -34,13 +34,24 @@ CREATE TABLE users ( ); ``` +### SEM specifiers + +Text takes the same mechanism specifiers as the other orderable types (the concept is defined in [Core concepts](/reference/eql/core-concepts#sem-specifiers)): + +| Specifier | Meaning | +| --- | --- | +| `_ord` | Orderable, using EQL's default mechanism (currently ORE). | +| `_ord_ore` | Orderable via ORE, pinned explicitly. | + +The EQL v3 release adds an OPE specifier for every orderable type — including `text` — so lexicographic ordering can be pinned to either mechanism; unspecified `_ord` columns keep tracking the default. + ## Payload A value for a `text_search` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus all three index terms: ```json { - "v": 2, + "v": 3, "i": { "t": "users", "c": "email" }, "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", "hm": "9c8ec1d2f9932b979b1bf3f09f8a4e2f6a41f8de2f0c8b7a52e1f5c3d4b6a790", From 127f0a6ce3d55ae6bdb1ffc27c55f75ebdafa4ec Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 21:27:44 +1000 Subject: [PATCH 11/38] refactor(v2): split numbers/dates pages, Example headings, separate Operators and Functions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback: - Dates & times split out of Numbers — same traits, distinct semantics; each page's examples now match its domain (payroll vs audit-event time windows / retention cutoffs / newest-first) - CREATE TABLE examples get an explicit "Example" sub-heading + lead-in - Operators and Functions are separate sections on every type page — operators as the per-variant support matrix, functions as the form-equivalents table (+ MIN/MAX, which only exist as functions) - IA.md: split reflected; query-performance follow-up added (CIP-3351 — the v3 branch already folded the v2 perf guide into database-indexes.md, which our indexes page absorbed) Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- IA.md | 7 +- content/docs/reference/eql/booleans.mdx | 2 +- content/docs/reference/eql/core-concepts.mdx | 2 +- .../docs/reference/eql/dates-and-times.mdx | 151 ++++++++++++++++++ content/docs/reference/eql/filtering.mdx | 2 +- content/docs/reference/eql/index.mdx | 7 +- content/docs/reference/eql/meta.json | 3 +- .../{numbers-and-dates.mdx => numbers.mdx} | 89 +++++------ content/docs/reference/eql/sorting.mdx | 2 +- content/docs/reference/eql/text.mdx | 41 +++-- 10 files changed, 239 insertions(+), 67 deletions(-) create mode 100644 content/docs/reference/eql/dates-and-times.mdx rename content/docs/reference/eql/{numbers-and-dates.mdx => numbers.mdx} (59%) diff --git a/IA.md b/IA.md index 6db1dee..d5736a9 100644 --- a/IA.md +++ b/IA.md @@ -137,7 +137,9 @@ live at `/docs/errors/<code>` — permanent, never restructured (CIP-3338). - [x] `/reference/eql` — install (single SQL file, permissions split, dbdev, Docker) - [x] `/reference/eql/core-concepts` — variant model, payload anatomy (absorbs cipher-cell), typed-operand rule, fail-loud blockers, term leakage pointer -- [x] `/reference/eql/numbers-and-dates` — int*/float*/numeric/date/timestamp +- [x] `/reference/eql/numbers` — int*/float*/numeric +- [x] `/reference/eql/dates-and-times` — date/timestamp (same traits as numbers, + distinct semantics) - [x] `/reference/eql/text` — all six text variants; owns the no-LIKE treatment - [x] `/reference/eql/json` — ste_vec + sv payload shape + containment/path queries - [x] `/reference/eql/booleans` — storage-only variants (bool has only that one) @@ -146,6 +148,9 @@ live at `/docs/errors/<code>` — permanent, never restructured (CIP-3338). - [x] `/reference/eql/sorting` — ORDER BY, extractor sort-key form, pagination - [x] `/reference/eql/grouping-and-aggregates` — GROUP BY/DISTINCT, min/max, no SUM/AVG - [x] `/reference/eql/joins` — equijoins, the same-keyset constraint +- [ ] ⛔ `/reference/eql/query-performance` — port the EQL repo performance guide once + rewritten for v3 upstream (v3 branch folded it into database-indexes.md; verify + nothing from the v2 guide on main was lost) — see CIP-3351 - **Stack SDK:** - [ ] `/reference/stack` — client + configuration (port encryption/* pages) - [ ] `/reference/stack/schema` diff --git a/content/docs/reference/eql/booleans.mdx b/content/docs/reference/eql/booleans.mdx index 390dc97..5403fc9 100644 --- a/content/docs/reference/eql/booleans.mdx +++ b/content/docs/reference/eql/booleans.mdx @@ -59,4 +59,4 @@ For every type other than `bool`, storage-only is a choice you can walk back. If SELECT * FROM readings WHERE value::eql_v3.int4_ord > $1::eql_v3.int4_ord; ``` -The variant families and what each one enables are covered in [Core concepts](/reference/eql/core-concepts); the per-type specifics live in [Numbers and dates](/reference/eql/numbers-and-dates) and [Text](/reference/eql/text). +The variant families and what each one enables are covered in [Core concepts](/reference/eql/core-concepts); the per-type specifics live in [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), and [Text](/reference/eql/text). diff --git a/content/docs/reference/eql/core-concepts.mdx b/content/docs/reference/eql/core-concepts.mdx index 11b6a0f..427bc7b 100644 --- a/content/docs/reference/eql/core-concepts.mdx +++ b/content/docs/reference/eql/core-concepts.mdx @@ -51,7 +51,7 @@ CREATE TABLE users ( ); ``` -Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.0 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers and dates](/reference/eql/numbers-and-dates), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use a separate domain, `eql_v3.json`, with its own operator surface — see [JSON](/reference/eql/json). +Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.0 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use a separate domain, `eql_v3.json`, with its own operator surface — see [JSON](/reference/eql/json). ## Anatomy of an encrypted value diff --git a/content/docs/reference/eql/dates-and-times.mdx b/content/docs/reference/eql/dates-and-times.mdx new file mode 100644 index 0000000..1014599 --- /dev/null +++ b/content/docs/reference/eql/dates-and-times.mdx @@ -0,0 +1,151 @@ +--- +title: Dates & times +description: "The complete reference for encrypted date and timestamp columns: the domain variants, the ORE-backed payload, and time-window, newest-first, and MIN/MAX queries." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0" +--- + +`date` and `timestamp` columns carry the same capabilities as [encrypted numbers](/reference/eql/numbers) — equality, ranges, ordering, `MIN` / `MAX` — but the queries they serve are temporal: time windows, newest-first listings, retention cutoffs, "when did this last happen". + +## Variants + +Both types generate the same `jsonb`-backed domain variants. The generic form: + +| Domain variant | Capability | +| --- | --- | +| `eql_v3.<T>` | Storage and decryption only. | +| `eql_v3.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | +| `eql_v3.<T>_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | +| `eql_v3.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | + +And every concrete domain this page covers: + +| Type | Variants | +| --- | --- | +| `date` | `eql_v3.date` · `eql_v3.date_eq` · `eql_v3.date_ord` · `eql_v3.date_ord_ore` | +| `timestamp` | `eql_v3.timestamp` · `eql_v3.timestamp_eq` · `eql_v3.timestamp_ord` · `eql_v3.timestamp_ord_ore` | + +Time columns are nearly always ranged and sorted, so `_ord` is the usual choice. Declare only the capability you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)), and the variant model itself is covered in [Core concepts](/reference/eql/core-concepts). + +### Example + +An audit-events table where the timestamps drive time-window queries and sorting: + +```sql +CREATE TABLE audit_events ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + occurred_at eql_v3.timestamp_ord, -- time windows, newest-first, MIN/MAX + review_due eql_v3.date_ord, -- range filters + sealed_on eql_v3.date -- store and decrypt only +); +``` + +### SEM specifiers + +Both types take the same mechanism specifiers on their orderable variant (the concept is defined in [Core concepts](/reference/eql/core-concepts#sem-specifiers)): + +| Specifier | Meaning | +| --- | --- | +| `_ord` | Orderable, using EQL's default mechanism (currently ORE). | +| `_ord_ore` | Orderable via ORE, pinned explicitly. | + +The EQL v3 release adds an OPE specifier for every orderable type; unspecified `_ord` columns keep tracking the default. + +## Payload + +A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus the `ob` ordering term. Here is a payload for the `eql_v3.timestamp_ord` `occurred_at` column: + +```json +{ + "v": 3, + "i": { "t": "audit_events", "c": "occurred_at" }, + "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", + "ob": [ + "7a1fd0c2...", "d24c9be1...", "03fa66b8...", "91b7e04d...", + "5c28aa19...", "e6f3071c...", "48d92ab5...", "0b64cf37...", + "2ce8b1f4...", "a90d57e2...", "6f13c8ba...", "d4720e95..." + ] +} +``` + +- **`ob` is the only index term.** An `_ord` payload carries no `hm`: equality on `_ord` variants compares ORE terms, which collapse to equality — see [Core concepts](/reference/eql/core-concepts). +- **The `ob` block count varies with the plaintext width** — `timestamp` values carry 12 blocks. + +## Operators + +| SQL operator | `eql_v3.<T>` | `<T>_eq` | `<T>_ord` / `<T>_ord_ore` | +| --- | :---: | :---: | :---: | +| `=` / `<>` | ❌ | ✅ | ✅ | +| `<` `<=` `>` `>=` | ❌ | ❌ | ✅ | +| `BETWEEN` (desugars to `>=` and `<=`) | ❌ | ❌ | ✅ | +| `IN` (desugars to `=`) | ❌ | ✅ | ✅ | +| `GROUP BY` / `DISTINCT` | ❌ | ✅ | ✅ | +| `ORDER BY` | ❌ | ❌ | ✅ | +| `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | + +Blocked cells raise an `operator … is not supported` exception — they never silently return wrong rows. Operands must be typed (`$1::eql_v3.timestamp_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). + +## Functions + +Every operator has a function form, for managed platforms that disallow custom operators — same typed arguments, identical resolution. The `MIN` / `MAX` aggregates only exist as functions: + +| Function | Equivalent | Available on | +| --- | --- | --- | +| `eql_v3.eq(a, b)` / `eql_v3.neq(a, b)` | `=` / `<>` | `_eq`, `_ord` / `_ord_ore` | +| `eql_v3.lt` / `lte` / `gt` / `gte` | `<` `<=` `>` `>=` | `_ord` / `_ord_ore` | +| `eql_v3.min(col)` / `eql_v3.max(col)` | aggregate `MIN` / `MAX` | `_ord` / `_ord_ore` | + +## Example queries + +### Time window + +```sql +SELECT * FROM audit_events +WHERE occurred_at BETWEEN $1::eql_v3.timestamp_ord AND $2::eql_v3.timestamp_ord; + +SELECT * FROM audit_events +WHERE review_due BETWEEN $1::eql_v3.date_ord AND $2::eql_v3.date_ord; +``` + +### Retention cutoff + +```sql +SELECT id FROM audit_events +WHERE occurred_at < $1::eql_v3.timestamp_ord; +``` + +### Newest-first listing + +Write the sort key in extractor form to stream rows out of the index already ordered — at large row counts this is the difference between seconds and milliseconds (see [Sorting](/reference/eql/sorting)): + +```sql +SELECT * FROM audit_events +WHERE occurred_at >= $1::eql_v3.timestamp_ord +ORDER BY eql_v3.ord_term(occurred_at) DESC +LIMIT 10; +``` + +### First and last event + +```sql +SELECT eql_v3.min(occurred_at), eql_v3.max(occurred_at) FROM audit_events; +``` + +## Where to next + +<Cards> + <Card title="Numbers" href="/reference/eql/numbers"> + The same capabilities on int, float, and numeric columns. + </Card> + <Card title="Indexes" href="/reference/eql/indexes"> + Btree recipes on `eql_v3.ord_term` for range, ORDER BY, and MIN/MAX. + </Card> + <Card title="Sorting" href="/reference/eql/sorting"> + Why the extractor-form sort key matters, and how to verify with EXPLAIN. + </Card> + <Card title="Filtering" href="/reference/eql/filtering"> + WHERE-clause patterns across all encrypted types. + </Card> +</Cards> diff --git a/content/docs/reference/eql/filtering.mdx b/content/docs/reference/eql/filtering.mdx index 6f1b779..9fe45a2 100644 --- a/content/docs/reference/eql/filtering.mdx +++ b/content/docs/reference/eql/filtering.mdx @@ -25,7 +25,7 @@ On `_eq` and `text_search` columns equality compares the HMAC (`hm`) term. On `_ SELECT * FROM users WHERE salary = $1::eql_v3.int8_ord; ``` -Bare storage-only variants (`eql_v3.text`, `eql_v3.int4`, …) block every comparison — see the type pages for what each variant supports: [Numbers & dates](/reference/eql/numbers-and-dates), [Text](/reference/eql/text), [Booleans](/reference/eql/booleans). +Bare storage-only variants (`eql_v3.text`, `eql_v3.int4`, …) block every comparison — see the type pages for what each variant supports: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), [Booleans](/reference/eql/booleans). ## `IN` lists diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index e3d7f0f..6b9ce9e 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -106,8 +106,11 @@ EQL v3 is designed to install without superuser. There are no custom operator cl <Card title="Core concepts" href="/reference/eql/core-concepts"> Domain variants, the encrypted payload, typed operands, and fail-loud blockers — the model every other page assumes. </Card> - <Card title="Numbers and dates" href="/reference/eql/numbers-and-dates"> - Encrypted integers, floats, numerics, dates, and timestamps. + <Card title="Numbers" href="/reference/eql/numbers"> + Encrypted integers, floats, and numerics. + </Card> + <Card title="Dates & times" href="/reference/eql/dates-and-times"> + Encrypted dates and timestamps: time windows, newest-first, retention cutoffs. </Card> <Card title="Text" href="/reference/eql/text"> Encrypted text: equality, ordering, and free-text token matching — and why there is no `LIKE`. diff --git a/content/docs/reference/eql/meta.json b/content/docs/reference/eql/meta.json index 3f4469e..f4268ce 100644 --- a/content/docs/reference/eql/meta.json +++ b/content/docs/reference/eql/meta.json @@ -3,7 +3,8 @@ "pages": [ "core-concepts", "---Types---", - "numbers-and-dates", + "numbers", + "dates-and-times", "text", "json", "booleans", diff --git a/content/docs/reference/eql/numbers-and-dates.mdx b/content/docs/reference/eql/numbers.mdx similarity index 59% rename from content/docs/reference/eql/numbers-and-dates.mdx rename to content/docs/reference/eql/numbers.mdx index a78052f..cda9471 100644 --- a/content/docs/reference/eql/numbers-and-dates.mdx +++ b/content/docs/reference/eql/numbers.mdx @@ -1,19 +1,19 @@ --- -title: Numbers & dates -description: "The complete reference for encrypted numeric and date/time columns: the int, float, numeric, date, and timestamp domain variants, the ORE-backed payload they carry, and range, ORDER BY, and MIN/MAX queries." +title: Numbers +description: "The complete reference for encrypted numeric columns: the int, float, and numeric domain variants, the ORE-backed payload they carry, and range, ORDER BY, and MIN/MAX queries." type: reference components: [eql] verifiedAgainst: eql: "3.0.0" --- -Eight scalar types share one identical query surface: `int2`, `int4`, `int8`, `float4`, `float8`, `numeric`, `date`, and `timestamp`. These are the columns you filter by range, sort newest-first, and take a `MIN` / `MAX` over — salaries, totals, rates, hire dates, timestamps. Everything on this page applies to all eight; only the domain name changes. +Six numeric types share one identical query surface: `int2`, `int4`, `int8`, `float4`, `float8`, and `numeric`. These are the columns you filter by range, sort, and take a `MIN` / `MAX` over — salaries, totals, rates, quantities. -There is no free-text matching for these types — `_match` and `_search` are [text-only variants](/reference/eql/text). Boolean columns are a separate, storage-only story — see [Booleans](/reference/eql/booleans). +Date and time columns have the same capabilities but their own semantics — see [Dates & times](/reference/eql/dates-and-times). There is no free-text matching for numeric types — `_match` and `_search` are [text-only variants](/reference/eql/text). ## Variants -Each of the eight scalar types generates the same `jsonb`-backed domain variants. The generic form: +Each numeric type generates the same `jsonb`-backed domain variants. The generic form: | Domain variant | Capability | | --- | --- | @@ -32,25 +32,25 @@ And every concrete domain this page covers: | `float4` | `eql_v3.float4` · `eql_v3.float4_eq` · `eql_v3.float4_ord` · `eql_v3.float4_ord_ore` | | `float8` | `eql_v3.float8` · `eql_v3.float8_eq` · `eql_v3.float8_ord` · `eql_v3.float8_ord_ore` | | `numeric` | `eql_v3.numeric` · `eql_v3.numeric_eq` · `eql_v3.numeric_ord` · `eql_v3.numeric_ord_ore` | -| `date` | `eql_v3.date` · `eql_v3.date_eq` · `eql_v3.date_ord` · `eql_v3.date_ord_ore` | -| `timestamp` | `eql_v3.timestamp` · `eql_v3.timestamp_eq` · `eql_v3.timestamp_ord` · `eql_v3.timestamp_ord_ore` | -Declare only the capability you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)), and the variant model itself is covered in [Core concepts](/reference/eql/core-concepts): +Declare only the capability you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)), and the variant model itself is covered in [Core concepts](/reference/eql/core-concepts). + +### Example + +A payroll table mixing the variants by how each column is queried: ```sql CREATE TABLE employees ( id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, salary eql_v3.int8_ord, -- range queries, ORDER BY, MIN/MAX tax_rate eql_v3.numeric_eq, -- exact lookup only - net_worth eql_v3.numeric, -- store and decrypt only, never queried - hired_on eql_v3.date_ord, - created_at eql_v3.timestamp_ord + net_worth eql_v3.numeric -- store and decrypt only, never queried ); ``` ### SEM specifiers -All eight types take the same mechanism specifiers on their orderable variant (the concept is defined in [Core concepts](/reference/eql/core-concepts#sem-specifiers)): +All six types take the same mechanism specifiers on their orderable variant (the concept is defined in [Core concepts](/reference/eql/core-concepts#sem-specifiers)): | Specifier | Meaning | | --- | --- | @@ -76,25 +76,32 @@ A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — ``` - **`ob` is the only index term.** An `_ord` payload carries no `hm`: equality on `_ord` variants compares ORE terms, which collapse to equality — see [Core concepts](/reference/eql/core-concepts). Only `_eq` payloads carry `hm` (a single hex HMAC-SHA-256 string) instead of `ob`. -- **The `ob` block count varies with the plaintext width**: 8 blocks for the int scalars, 12 for `timestamp`, 14 for `numeric` — the array just carries more block strings. - -## Operators and functions +- **The `ob` block count varies with the plaintext width**: 8 blocks for the int types, 14 for `numeric`. -The function forms exist for managed platforms that disallow custom operators — they take the same typed arguments and resolve identically. +## Operators -| SQL operator | Function form | `eql_v3.<T>` | `<T>_eq` | `<T>_ord` / `<T>_ord_ore` | -| --- | --- | :---: | :---: | :---: | -| `=` / `<>` | `eql_v3.eq(a, b)` / `eql_v3.neq(a, b)` | ❌ | ✅ | ✅ | -| `<` `<=` `>` `>=` | `eql_v3.lt` / `lte` / `gt` / `gte` | ❌ | ❌ | ✅ | -| `BETWEEN` | desugars to `>=` and `<=` | ❌ | ❌ | ✅ | -| `IN` | desugars to `=` | ❌ | ✅ | ✅ | -| `GROUP BY` / `DISTINCT` | — (needs an equality term) | ❌ | ✅ | ✅ | -| `ORDER BY` | sort key: `eql_v3.ord_term(col)` | ❌ | ❌ | ✅ | -| `MIN` / `MAX` | `eql_v3.min(col)` / `eql_v3.max(col)` | ❌ | ❌ | ✅ | -| `IS NULL` / `IS NOT NULL` | — | ✅ | ✅ | ✅ | +| SQL operator | `eql_v3.<T>` | `<T>_eq` | `<T>_ord` / `<T>_ord_ore` | +| --- | :---: | :---: | :---: | +| `=` / `<>` | ❌ | ✅ | ✅ | +| `<` `<=` `>` `>=` | ❌ | ❌ | ✅ | +| `BETWEEN` (desugars to `>=` and `<=`) | ❌ | ❌ | ✅ | +| `IN` (desugars to `=`) | ❌ | ✅ | ✅ | +| `GROUP BY` / `DISTINCT` | ❌ | ✅ | ✅ | +| `ORDER BY` | ❌ | ❌ | ✅ | +| `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | Blocked cells raise an `operator … is not supported` exception — they never silently return wrong rows. Operands must be typed (`$1::eql_v3.int8_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). +## Functions + +Every operator has a function form, for managed platforms that disallow custom operators — same typed arguments, identical resolution. The `MIN` / `MAX` aggregates only exist as functions: + +| Function | Equivalent | Available on | +| --- | --- | --- | +| `eql_v3.eq(a, b)` / `eql_v3.neq(a, b)` | `=` / `<>` | `_eq`, `_ord` / `_ord_ore` | +| `eql_v3.lt` / `lte` / `gt` / `gte` | `<` `<=` `>` `>=` | `_ord` / `_ord_ore` | +| `eql_v3.min(col)` / `eql_v3.max(col)` | aggregate `MIN` / `MAX` | `_ord` / `_ord_ore` | + **`SUM`, `AVG`, and other arithmetic aggregates are not supported** on encrypted columns — they would require homomorphic encryption. `MIN` / `MAX` work because they only need comparison; for sums and averages, decrypt at the application boundary and aggregate client-side. ## Example queries @@ -109,35 +116,25 @@ SELECT * FROM employees WHERE salary BETWEEN $1::eql_v3.int8_ord AND $2::eql_v3.int8_ord; ``` -### Date window +### MIN and MAX -`BETWEEN` works the same on `date` and `timestamp` columns: +`eql_v3.min` / `eql_v3.max` compare ORE terms — no decryption happens in the database, and the encrypted result decrypts in the client. `NULL` inputs are skipped; an all-`NULL` input set returns `NULL`: ```sql -SELECT * FROM employees -WHERE hired_on BETWEEN $1::eql_v3.date_ord AND $2::eql_v3.date_ord; +SELECT eql_v3.min(salary) FROM employees; +SELECT eql_v3.max(salary) FROM employees; ``` -### Newest-first listing +### Sorted listing -Bare `ORDER BY created_at` sorts correctly, but the planner doesn't rewrite sort keys, so it adds a `Sort` node even when a btree index exists. Write the sort key in extractor form to stream rows out of the index already ordered — at large row counts this is the difference between seconds and milliseconds (see [Sorting](/reference/eql/sorting)): +Write the sort key in extractor form to stream rows out of the index already ordered (see [Sorting](/reference/eql/sorting) for why): ```sql SELECT * FROM employees -WHERE created_at >= $1::eql_v3.timestamp_ord -ORDER BY eql_v3.ord_term(created_at) DESC +ORDER BY eql_v3.ord_term(salary) DESC LIMIT 10; ``` -### MIN and MAX - -`eql_v3.min` / `eql_v3.max` compare ORE terms — no decryption happens in the database, and the encrypted result decrypts in the client. `NULL` inputs are skipped; an all-`NULL` input set returns `NULL`: - -```sql -SELECT eql_v3.min(salary) FROM employees; -SELECT eql_v3.max(created_at) FROM employees; -``` - ### Cast at the call site On a generic `jsonb` column whose payloads already carry the `ob` term, cast to the right domain in the query: @@ -149,15 +146,15 @@ SELECT eql_v3.min(salary_jsonb::eql_v3.int8_ord) FROM employees; ## Where to next <Cards> + <Card title="Dates & times" href="/reference/eql/dates-and-times"> + The same capabilities on date and timestamp columns. + </Card> <Card title="Indexes" href="/reference/eql/indexes"> Btree recipes on `eql_v3.ord_term` for range, ORDER BY, and MIN/MAX. </Card> <Card title="Filtering" href="/reference/eql/filtering"> WHERE-clause patterns across all encrypted types. </Card> - <Card title="Sorting" href="/reference/eql/sorting"> - Why the extractor-form sort key matters, and how to verify with EXPLAIN. - </Card> <Card title="Grouping & aggregates" href="/reference/eql/grouping-and-aggregates"> GROUP BY, DISTINCT, and the aggregate surface on encrypted columns. </Card> diff --git a/content/docs/reference/eql/sorting.mdx b/content/docs/reference/eql/sorting.mdx index b684f3e..eb34f12 100644 --- a/content/docs/reference/eql/sorting.mdx +++ b/content/docs/reference/eql/sorting.mdx @@ -7,7 +7,7 @@ verifiedAgainst: eql: "3.0.0" --- -`ORDER BY` on an encrypted column needs an ORE ordering term: it works on `_ord` / `_ord_ore` variants of every scalar and on `text_search`. ORE terms are order-preserving, so the database sorts ciphertext in exactly the order the plaintext would sort — without decrypting anything. Which variants carry the term is covered in [Numbers & dates](/reference/eql/numbers-and-dates) and [Text](/reference/eql/text); the variant model itself is in [Core concepts](/reference/eql/core-concepts). +`ORDER BY` on an encrypted column needs an ORE ordering term: it works on `_ord` / `_ord_ore` variants of every scalar and on `text_search`. ORE terms are order-preserving, so the database sorts ciphertext in exactly the order the plaintext would sort — without decrypting anything. Which variants carry the term is covered in [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), and [Text](/reference/eql/text); the variant model itself is in [Core concepts](/reference/eql/core-concepts). Sorting a variant *without* an ORE term (`_eq`, `text_match`, bare storage variants) won't raise — but the order is meaningless. Type the column as an `_ord` variant when ordering matters. diff --git a/content/docs/reference/eql/text.mdx b/content/docs/reference/eql/text.mdx index 25228c9..9fe995f 100644 --- a/content/docs/reference/eql/text.mdx +++ b/content/docs/reference/eql/text.mdx @@ -22,7 +22,11 @@ All six are `jsonb`-backed domains. Which one you declare fixes the column's que | `eql_v3.text_match` | Free-text token containment: `@>` / `<@`. | | `eql_v3.text_search` | Equality + ordering + token containment. | -Declare only the capabilities you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)): +Declare only the capabilities you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)). + +### Example + +A users table mixing the variants by how each column is queried: ```sql CREATE TABLE users ( @@ -68,22 +72,33 @@ The narrower variants carry only their own term: a `text_eq` payload carries `hm **`bf` positions are signed**: EQL stores the filter as PostgreSQL `smallint[]`, and filters sized above 32768 emit upper-half bit positions as *negative* signed values. Consumers must use a signed 16-bit integer type. -## Operators and functions +## Operators -The function forms exist for managed platforms that disallow custom operators — they take the same typed arguments and resolve identically. - -| SQL operator | Function form | `eql_v3.text` | `text_eq` | `text_ord` / `text_ord_ore` | `text_match` | `text_search` | -| --- | --- | :---: | :---: | :---: | :---: | :---: | -| `=` / `<>` | `eql_v3.eq(a, b)` / `eql_v3.neq(a, b)` | ❌ | ✅ | ✅ | ❌ | ✅ | -| `<` `<=` `>` `>=` | `eql_v3.lt` / `lte` / `gt` / `gte` | ❌ | ❌ | ✅ | ❌ | ✅ | -| `@>` / `<@` | `eql_v3.contains(a, b)` / `eql_v3.contained_by(a, b)` | ❌ | ❌ | ❌ | ✅ | ✅ | -| `LIKE` / `ILIKE` (`~~` / `~~*`) | none | ❌ | ❌ | ❌ | ❌ | ❌ | -| `IN` / `GROUP BY` / `DISTINCT` | desugar to `=` / need an equality term | ❌ | ✅ | ✅ | ❌ | ✅ | -| `ORDER BY`, `MIN` / `MAX` | `eql_v3.min(col)` / `eql_v3.max(col)` | ❌ | ❌ | ✅ | ❌ | ✅ | -| `IS NULL` / `IS NOT NULL` | — | ✅ | ✅ | ✅ | ✅ | ✅ | +| SQL operator | `eql_v3.text` | `text_eq` | `text_ord` / `text_ord_ore` | `text_match` | `text_search` | +| --- | :---: | :---: | :---: | :---: | :---: | +| `=` / `<>` | ❌ | ✅ | ✅ | ❌ | ✅ | +| `<` `<=` `>` `>=` | ❌ | ❌ | ✅ | ❌ | ✅ | +| `@>` / `<@` | ❌ | ❌ | ❌ | ✅ | ✅ | +| `LIKE` / `ILIKE` (`~~` / `~~*`) | ❌ | ❌ | ❌ | ❌ | ❌ | +| `IN` / `GROUP BY` / `DISTINCT` | ❌ | ✅ | ✅ | ❌ | ✅ | +| `ORDER BY` | ❌ | ❌ | ✅ | ❌ | ✅ | +| `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | ✅ | ✅ | Blocked cells raise an `operator … is not supported` exception — they never silently return wrong rows. Operands must be typed (`$1::eql_v3.text_eq`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). +## Functions + +Every operator has a function form, for managed platforms that disallow custom operators — same typed arguments, identical resolution. The `MIN` / `MAX` aggregates only exist as functions: + +| Function | Equivalent | Available on | +| --- | --- | --- | +| `eql_v3.eq(a, b)` / `eql_v3.neq(a, b)` | `=` / `<>` | `text_eq`, `text_ord` / `text_ord_ore`, `text_search` | +| `eql_v3.lt` / `lte` / `gt` / `gte` | `<` `<=` `>` `>=` | `text_ord` / `text_ord_ore`, `text_search` | +| `eql_v3.contains(a, b)` / `eql_v3.contained_by(a, b)` | `@>` / `<@` | `text_match`, `text_search` | +| `eql_v3.min(col)` / `eql_v3.max(col)` | aggregate `MIN` / `MAX` | `text_ord` / `text_ord_ore`, `text_search` | + +There are no `like` / `ilike` function forms — encrypted text matching is `eql_v3.contains` on a `text_match` value. + ## There is no `LIKE` `LIKE` and `ILIKE` (`~~` / `~~*`) raise on **every** encrypted-domain variant — including `text_match` and `text_search`. SQL pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter token containment — `@>` on a `text_match` or `text_search` column: From 123a544c36d09d456ae86c1477710db5f743f003 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 21:39:31 +1000 Subject: [PATCH 12/38] docs(v2): track EQL 3.0.0 release-alignment gate in IA.md (CIP-3352) Claude-Session: https://claude.ai/code/session_01ACPpFPHvKtrV48nbEYuv7P --- IA.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/IA.md b/IA.md index d5736a9..6a13da8 100644 --- a/IA.md +++ b/IA.md @@ -184,5 +184,9 @@ live at `/docs/errors/<code>` — permanent, never restructured (CIP-3338). - [ ] OG images for v2 pages (route only covers legacy tree) - [ ] Correctness CI: snippet type-checking, SQL-vs-EQL-Docker, terminology lint (CIP-3337) - [ ] llms.txt curation + Cloudflare AI crawl policy + md-degradation check (CIP-3339) +- [ ] ⛔ EQL 3.0.0 release alignment (CIP-3352, blocks CIP-3335) — the EQL reference + documents the release as decided, ahead of the eql_v3 branch: payload `v: 3`, + OPE SEM specifier, Docker tag `:17-3.0.0`, `version()` output, schema files. + Each must land upstream or be walked back in the docs before merge - [ ] Flip `ENABLE_V2_REDIRECTS=1`, delete `content/stack` + `/stack` routes + legacy loader (CIP-3335) - [ ] Consistency sweep + Supabase listing v3 revision (CIP-3335) From 896eae51206add738221ed869cb8a5e0eec7321f Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 22:07:13 +1000 Subject: [PATCH 13/38] docs(v2): address Copilot review on EQL reference (PR #38) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - indexes.mdx: cast query-shape example params to their EQL domain types, consistent with the typed-operand rule - numbers/dates-and-times/text: the fail-loud note now scopes to operators — ORDER BY on a variant without an ordering term doesn't raise, it silently returns a meaningless order (links Sorting) --- content/docs/reference/eql/dates-and-times.mdx | 2 +- content/docs/reference/eql/indexes.mdx | 6 +++--- content/docs/reference/eql/numbers.mdx | 2 +- content/docs/reference/eql/text.mdx | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/content/docs/reference/eql/dates-and-times.mdx b/content/docs/reference/eql/dates-and-times.mdx index 1014599..3af2b9c 100644 --- a/content/docs/reference/eql/dates-and-times.mdx +++ b/content/docs/reference/eql/dates-and-times.mdx @@ -85,7 +85,7 @@ A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — | `ORDER BY` | ❌ | ❌ | ✅ | | `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | -Blocked cells raise an `operator … is not supported` exception — they never silently return wrong rows. Operands must be typed (`$1::eql_v3.timestamp_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). +Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::eql_v3.timestamp_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). ## Functions diff --git a/content/docs/reference/eql/indexes.mdx b/content/docs/reference/eql/indexes.mdx index 3d1df3e..ff4b1fe 100644 --- a/content/docs/reference/eql/indexes.mdx +++ b/content/docs/reference/eql/indexes.mdx @@ -77,7 +77,7 @@ WHERE email = '{"hm":"abc"}'::jsonb; ### Equality ```sql -SELECT * FROM users WHERE email = $1; +SELECT * FROM users WHERE email = $1::eql_v3.text_eq; -- Index Scan using users_email_eq -- Index Cond: (eql_v3.eq_term(email) = eql_v3.eq_term($1)) ``` @@ -87,14 +87,14 @@ SELECT * FROM users WHERE email = $1; The `<`, `<=`, `>`, `>=` operators inline to comparisons on `eql_v3.ord_term`, so natural-form range predicates match the btree: ```sql -SELECT * FROM users WHERE created_at < $1; +SELECT * FROM users WHERE created_at < $1::eql_v3.timestamp_ord; ``` `ORDER BY` needs care. The planner inlines operators in *predicates* but does not rewrite *sort keys*: `ORDER BY created_at` uses the index for the `WHERE` clause but still adds a `Sort` node, which scales linearly with the rows passing the filter. To stream rows out of the btree already ordered, write the sort key in extractor form: ```sql SELECT * FROM users - WHERE created_at < $1 + WHERE created_at < $1::eql_v3.timestamp_ord ORDER BY eql_v3.ord_term(created_at) DESC LIMIT 10; ``` diff --git a/content/docs/reference/eql/numbers.mdx b/content/docs/reference/eql/numbers.mdx index cda9471..2d9a2a0 100644 --- a/content/docs/reference/eql/numbers.mdx +++ b/content/docs/reference/eql/numbers.mdx @@ -90,7 +90,7 @@ A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — | `ORDER BY` | ❌ | ❌ | ✅ | | `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | -Blocked cells raise an `operator … is not supported` exception — they never silently return wrong rows. Operands must be typed (`$1::eql_v3.int8_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). +Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::eql_v3.int8_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). ## Functions diff --git a/content/docs/reference/eql/text.mdx b/content/docs/reference/eql/text.mdx index 9fe995f..f08eafc 100644 --- a/content/docs/reference/eql/text.mdx +++ b/content/docs/reference/eql/text.mdx @@ -84,7 +84,7 @@ The narrower variants carry only their own term: a `text_eq` payload carries `hm | `ORDER BY` | ❌ | ❌ | ✅ | ❌ | ✅ | | `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | ✅ | ✅ | -Blocked cells raise an `operator … is not supported` exception — they never silently return wrong rows. Operands must be typed (`$1::eql_v3.text_eq`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). +Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::eql_v3.text_eq`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). ## Functions From d0ccd7e68ae0c8a4b12304a9c21886d7b2cc1ec5 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Thu, 2 Jul 2026 22:08:23 +1000 Subject: [PATCH 14/38] docs(v2): address PR #37 review comments - v2-redirects.mjs: all entries permanent: false while the IA settles (per review); flip to permanent post-merge soak (CIP-3335) - IA.md: fix unmatched bold around 'Moving a page' - add placeholder pages for /concepts/searchable-encryption (CIP-3333) and /guides/troubleshooting/query-performance (CIP-3351) so the EQL reference's forward links resolve instead of 404ing --- IA.md | 2 +- .../docs/concepts/searchable-encryption.mdx | 9 ++ .../troubleshooting/query-performance.mdx | 9 ++ v2-redirects.mjs | 147 +++++++++--------- 4 files changed, 95 insertions(+), 72 deletions(-) create mode 100644 content/docs/concepts/searchable-encryption.mdx create mode 100644 content/docs/guides/troubleshooting/query-performance.mdx diff --git a/IA.md b/IA.md index f9c6f9e..cbcabec 100644 --- a/IA.md +++ b/IA.md @@ -17,7 +17,7 @@ on a product decision (see CIP-3307 checklist). enforces that every legacy page has a mapping. - Frontmatter facets (`type`, `components`, `audience`, `integration`, `verifiedAgainst`, `reviewBy`) are defined in `source.config.ts` (`v2docs`). -- **Moving a page = ** move the file into `content/docs`, update its facets, +- **Moving a page** = move the file into `content/docs`, update its facets, fix inbound links, confirm its `v2-redirects.mjs` entry, tick it here. ## URL conventions diff --git a/content/docs/concepts/searchable-encryption.mdx b/content/docs/concepts/searchable-encryption.mdx new file mode 100644 index 0000000..689d1fd --- /dev/null +++ b/content/docs/concepts/searchable-encryption.mdx @@ -0,0 +1,9 @@ +--- +title: Searchable encryption +description: "How querying encrypted data works, and exactly what each index term reveals." +type: concept +--- + +This page is being rewritten as part of the docs V2 overhaul ([CIP-3333](https://linear.app/cipherstash/issue/CIP-3333)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, the current version lives in the [existing docs](/stack/cipherstash/encryption/searchable-encryption). diff --git a/content/docs/guides/troubleshooting/query-performance.mdx b/content/docs/guides/troubleshooting/query-performance.mdx new file mode 100644 index 0000000..a128a95 --- /dev/null +++ b/content/docs/guides/troubleshooting/query-performance.mdx @@ -0,0 +1,9 @@ +--- +title: Query performance +description: "Diagnosing and fixing slow queries on encrypted columns." +type: guide +--- + +This page is being built as part of the docs V2 overhaul ([CIP-3351](https://linear.app/cipherstash/issue/CIP-3351)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). + +Until it lands, the EQL reference covers the essentials: [Indexes](/reference/eql/indexes) walks through `EXPLAIN` verification and large-table index builds, and [Sorting](/reference/eql/sorting) covers extractor-form sort keys. diff --git a/v2-redirects.mjs b/v2-redirects.mjs index 1f068b1..3b0f78e 100644 --- a/v2-redirects.mjs +++ b/v2-redirects.mjs @@ -10,366 +10,371 @@ // // Conventions (matching next.config.mjs): sources/destinations omit the // "/docs" basePath. Order matters — specific entries before wildcards. +// +// All entries are `permanent: false` (307) while the IA settles — browsers +// and crawlers cache 308s aggressively, and a mis-cached destination is hard +// to walk back. Flip to permanent once the map has soaked post-merge +// (CIP-3335). export const v2Redirects = [ // === Roots === - { source: "/stack", destination: "/", permanent: true }, + { source: "/stack", destination: "/", permanent: false }, { source: "/stack/quickstart", destination: "/get-started/quickstart", - permanent: true, + permanent: false, }, - { source: "/stack/cipherstash", destination: "/", permanent: true }, + { source: "/stack/cipherstash", destination: "/", permanent: false }, { source: "/stack/cipherstash/postgres", destination: "/reference/eql", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/supabase", destination: "/integrations/supabase", - permanent: true, + permanent: false, }, // === Encryption SDK section → Reference/stack + new homes === { source: "/stack/cipherstash/encryption", destination: "/reference/stack", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/encryption/searchable-encryption", destination: "/concepts/searchable-encryption", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/encryption/identity", destination: "/concepts/identity-aware-encryption", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/encryption/drizzle", destination: "/integrations/drizzle", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/encryption/prisma-next", destination: "/integrations/prisma-next", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/encryption/dynamodb", destination: "/integrations/aws/dynamodb", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/encryption/supabase", destination: "/reference/stack/supabase", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/encryption/indexes", destination: "/reference/eql/indexes", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/encryption/queries", destination: "/reference/eql/operators", - permanent: true, + permanent: false, }, // configuration, encrypt-decrypt, bulk-operations, models, schema, storing-data { source: "/stack/cipherstash/encryption/:path*", destination: "/reference/stack/:path*", - permanent: true, + permanent: false, }, // === KMS section → Security + Reference/auth + Concepts === { source: "/stack/cipherstash/kms", destination: "/security/zerokms", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/kms/cts", destination: "/security/cts", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/kms/oidc", destination: "/reference/auth/oidc-configuration", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/kms/access-keys", destination: "/reference/auth/access-keys", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/kms/clients", destination: "/reference/auth/clients", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/kms/disaster-recovery", destination: "/security/availability-and-continuity", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/kms/keysets", destination: "/concepts/key-management", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/kms/regions", destination: "/security/zerokms", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/kms/configuration", destination: "/reference/workspace/configuration", - permanent: true, + permanent: false, }, // === Proxy section → Reference/proxy + new homes === { source: "/stack/cipherstash/proxy", destination: "/reference/proxy", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/proxy/audit", destination: "/security/audit-logging", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/proxy/getting-started", destination: "/integrations/aws/rds-aurora", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/proxy/encrypt-tool", destination: "/guides/migration/encrypt-existing-data", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/proxy/searchable-json", destination: "/reference/eql/json", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/proxy/troubleshooting", destination: "/guides/troubleshooting/proxy", - permanent: true, + permanent: false, }, // configuration, message-flow, multitenant { source: "/stack/cipherstash/proxy/:path*", destination: "/reference/proxy/:path*", - permanent: true, + permanent: false, }, // === CLI section → Reference/cli === { source: "/stack/cipherstash/cli", destination: "/reference/cli", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/cli/troubleshooting", destination: "/guides/troubleshooting/cli", - permanent: true, + permanent: false, }, { source: "/stack/cipherstash/cli/:path*", destination: "/reference/cli/:path*", - permanent: true, + permanent: false, }, // === Deploy section → Guides === { source: "/stack/deploy", destination: "/guides/deployment", - permanent: true, + permanent: false, }, { source: "/stack/deploy/going-to-production", destination: "/guides/deployment/going-to-production", - permanent: true, + permanent: false, }, { source: "/stack/deploy/aws-ecs", destination: "/guides/deployment/proxy-deployment", - permanent: true, + permanent: false, }, { source: "/stack/deploy/bundling", destination: "/guides/deployment/serverless-and-bundling", - permanent: true, + permanent: false, }, { source: "/stack/deploy/sst", destination: "/guides/deployment/serverless-and-bundling", - permanent: true, + permanent: false, }, { source: "/stack/deploy/testing", destination: "/guides/development/testing-and-ci", - permanent: true, + permanent: false, }, { source: "/stack/deploy/team-onboarding", destination: "/guides/development/team-onboarding", - permanent: true, + permanent: false, }, { source: "/stack/deploy/troubleshooting", destination: "/guides/troubleshooting", - permanent: true, + permanent: false, }, // === Reference section === - { source: "/stack/reference", destination: "/reference", permanent: true }, + { source: "/stack/reference", destination: "/reference", permanent: false }, { source: "/stack/reference/what-is-cipherstash", destination: "/get-started/what-is-cipherstash", - permanent: true, + permanent: false, }, { source: "/stack/reference/security-architecture", destination: "/security/architecture", - permanent: true, + permanent: false, }, { source: "/stack/reference/compliance", destination: "/security/compliance", - permanent: true, + permanent: false, }, { source: "/stack/reference/comparisons", destination: "/compare", - permanent: true, + permanent: false, }, { source: "/stack/reference/comparisons/:path*", destination: "/compare/:path*", - permanent: true, + permanent: false, }, { source: "/stack/reference/use-cases", destination: "/solutions", - permanent: true, + permanent: false, }, { source: "/stack/reference/use-cases/ai-rag", destination: "/solutions/ai-and-rag", - permanent: true, + permanent: false, }, { source: "/stack/reference/use-cases/compliance", destination: "/security/compliance", - permanent: true, + permanent: false, }, { source: "/stack/reference/use-cases/:path*", destination: "/solutions/:path*", - permanent: true, + permanent: false, }, { source: "/stack/reference/billing", destination: "/reference/workspace/billing", - permanent: true, + permanent: false, }, { source: "/stack/reference/members", destination: "/reference/workspace/members", - permanent: true, + permanent: false, }, { source: "/stack/reference/cipher-cell", destination: "/reference/eql/payload-format", - permanent: true, + permanent: false, }, { source: "/stack/reference/eql-guide", destination: "/reference/eql", - permanent: true, + permanent: false, }, { source: "/stack/reference/eql", destination: "/reference/eql", - permanent: true, + permanent: false, }, { source: "/stack/reference/eql/:path*", destination: "/reference/eql/:path*", - permanent: true, + permanent: false, }, { source: "/stack/reference/encryption-sdk", destination: "/reference/stack", - permanent: true, + permanent: false, }, { source: "/stack/reference/error-handling", destination: "/reference/stack/errors", - permanent: true, + permanent: false, }, // NOTE: legacy "migration" page is the @cipherstash/protect→stack package // rename guide, NOT data migration (see IA.md). { source: "/stack/reference/migration", destination: "/reference/stack/upgrading-from-protect", - permanent: true, + permanent: false, }, { source: "/stack/reference/proxy-errors", destination: "/reference/proxy/errors", - permanent: true, + permanent: false, }, { source: "/stack/reference/proxy-reference", destination: "/reference/proxy/configuration", - permanent: true, + permanent: false, }, { source: "/stack/reference/drizzle", destination: "/integrations/drizzle", - permanent: true, + permanent: false, }, { source: "/stack/reference/dashboard-supabase-integration", destination: "/integrations/supabase", - permanent: true, + permanent: false, }, { source: "/stack/reference/discovery-session", destination: "/get-started/choose-your-stack", - permanent: true, + permanent: false, }, { source: "/stack/reference/planning-guide", destination: "/get-started/choose-your-stack", - permanent: true, + permanent: false, }, { source: "/stack/reference/supported-solutions", destination: "/integrations", - permanent: true, + permanent: false, }, { source: "/stack/reference/agent-skills", destination: "/reference/agent-skills", - permanent: true, + permanent: false, }, { source: "/stack/reference/glossary", destination: "/reference/glossary", - permanent: true, + permanent: false, }, // Generated TypeDoc API reference (scripts/generate-docs.ts output) { source: "/stack/reference/stack/:path*", destination: "/reference/stack/:path*", - permanent: true, + permanent: false, }, ]; From 4901b0f23e277379a00081af93780c52fdb2c82f Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Sun, 5 Jul 2026 22:13:15 +1000 Subject: [PATCH 15/38] docs(v2): add the two-axis content model to IA.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Slots the content-model spec in above the migration checklist: mode vs component axes, hubs at /components/<component>, the placement test, invariants, the mode-first sidebar (compare folded into concepts, hubs anchored under Get started), the hub-page template, and the auth worked example. Page placement is now decided by mode + components facet rather than re-litigated per PR. Note: this documents the target model; reconciling the migration checklist below (Comparisons → Concepts) and the components facet enum in source.config.ts (auth/zerokms → platform) are follow-ups. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- IA.md | 260 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+) diff --git a/IA.md b/IA.md index 4be54ce..3fb2645 100644 --- a/IA.md +++ b/IA.md @@ -28,6 +28,266 @@ live at `/docs/errors/<code>` — permanent, never restructured (CIP-3338). --- +## Content model — two axes + +The migration checklist below tracks _what_ moves; this section fixes _where +things go and why_, so placement stops being re-litigated per PR. + +### Terms + +- **Mode** — what the reader is trying to do. The four Diátaxis kinds + (tutorial, how-to, reference, explanation), plus our three audience doors + (integrations, solutions, security) which serve the same placement role. + A page has exactly one mode; the mode decides its sidebar home. +- **Component** — a layer of the product stack, drawn from the `components` + facet enum: `encryption` (Stack SDK), `platform` (CTS + ZeroKMS), `eql`, + `proxy`, `cli`. A page can touch several components at once. +- **Facet** — structured frontmatter that classifies a page along one axis + (`type`, `components`, `audience`, `integration`). Queryable; never shown + raw to users; independent of nav position. +- **Hub** — a per-component page at `/components/<component>` that gathers + every page tagged with that component. A generated view, not a subtree. + Not every component has a hub: hubs are for _layers_ of the stack + (`encryption`, `platform`, `eql`, `proxy`). `cli` is a component but a + cross-cutting tool, not a layer, so it has no hub. +- **Locator** — the one hand-written paragraph (plus mini stack diagram) at the + top of a hub, orienting the reader: what the component is, what sits above and + below it. Prose, nothing else. + +### The rule + +**The sidebar is organized only by mode. The by-component view lives only on the hubs.** + +CipherStash is a dependency stack, not a set of independent product pillars: +integrations sit on the Stack SDK and Proxy, both consume EQL, and EQL sits on +the Platform (CTS + ZeroKMS). That is a graph. A sidebar is a tree. So we split +the two axes rather than forcing the graph into the tree. + +- **Mode axis (the tree).** Every page has exactly one home, chosen by _what the + reader is doing_: Diátaxis mode (`get-started`, `guides`, `reference`, + `concepts`) plus three audience doors (`integrations`, `solutions`, + `security`). The sidebar is built from `meta.json` and nothing else. +- **Component axis (the graph).** Which layers a page touches is the `components` + facet — never a sidebar section. The dependency graph is expressed by facets + and surfaced by hub pages, so a shared layer like EQL is documented once and + referenced everywhere. + +Why the pillar layouts (Supabase, Clerk, GitHub) don't transfer: those are +feature-pillar products whose parts are independent, so each pillar can be its +own subtree. Our parts are layers in a dependency stack — a shared layer has no +single natural parent, so it cannot be a tree node without duplication. + +### Placement test + +Two questions, in this order, for every page: + +1. _Which mode is this?_ → decides the folder (its one sidebar home). +2. _Which components does it touch?_ → sets the `components` facet. + +Never the reverse. If you are tempted to file a page _under_ a component, that is +the signal to file it by mode and tag the component instead. + +### Invariants + +- The sidebar tree is `meta.json` only. Facets never affect nav position + (already asserted by the schema comment in `source.config.ts`). +- A page appears in exactly one place in the tree. Cross-cutting reuse happens + through links and hubs, never by duplicating a page under a second parent. +- Shared mechanics have exactly one canonical page; every other page links, + never restates. This generalises the existing EQL rule (mechanics live in + `/reference/eql/core-concepts`; category pages link to it) to every component. +- Each component has exactly one hub at `/components/<component>`. A hub is a + facet-driven _view_: it links to canonical pages and owns no mechanics. +- `/get-started/what-is-cipherstash` renders the components map; its nodes link + to the hubs. The hubs are the _only_ component-first surface, and they hold no + primary content — so they do not recreate the legacy component-first tree + (the old `root: true` Encryption/KMS/Proxy/Platform tabs). + +#### A note on hub URLs + +Hubs live at the top-level path `/components/<component>` for a short, memorable +front door, but are anchored in the sidebar under Get started (Fumadocs takes +nav position from `meta.json` independently of the URL). If you would rather the +URL mirror the nav, use `/get-started/components/<component>` instead — either +works; pick one and add it to `v2-redirects.mjs`. + +### Sidebar + +Mode-first, seven top-level sections. `compare` is folded into `concepts` (each +comparison already belongs to a door: aws-kms and vault are security-review +material, fhe and rls-and-tde are concepts). Component hubs are anchored under +Get started, not given their own top-level tab. + +``` +Get started +├─ What is CipherStash mental model · components map · audience router +├─ Quickstart +├─ Choose your stack platform × ORM × auth matrix +├─ Examples +└─ Components ← hub views (facet-driven, link-only) + ├─ Stack SDK + ├─ Proxy + ├─ EQL (shared) + └─ Platform (CTS + ZeroKMS) +Integrations flat; category grid on the index +├─ Supabase (Database · Auth · Dashboard) +├─ Drizzle · Prisma · Next.js · TypeScript +├─ Clerk · Auth0 · Okta +├─ AWS (RDS/Aurora · DynamoDB) +└─ Serverless · Docker +Concepts +├─ Privacy-first design +├─ Application-level encryption +├─ Searchable encryption canonical leakage model +├─ EQL typed-column model +├─ Key management +├─ Identity-aware encryption +├─ Threat modelling +└─ Compare aws-kms · fhe · rls-and-tde · hashicorp-vault +Guides +├─ Development local setup · schema design · testing & CI · onboarding +├─ Migration encrypt existing data · adopt incrementally · key rotation +├─ Deployment production · serverless & bundling · proxy deployment +└─ Troubleshooting query performance · runtime errors · cli · proxy +Security +├─ Architecture one reconciled ZeroKMS story +├─ ZeroKMS · CTS · Stack SDK · Proxy +├─ Threat scenarios +├─ Availability · Audit logging · Key ownership +└─ Compliance HIPAA · SOC 2 · GDPR +Solutions +└─ Protecting PII · Healthcare/HIPAA · AI & RAG · Data residency · Provable access +Reference +├─ EQL core-concepts · numbers · dates · text · json · … · joins +├─ Stack SDK client · schema · encrypt-decrypt · supabase · drizzle-operators · errors +├─ Auth lock-contexts · cts-tokens · oidc · access-keys +├─ CLI · Proxy · Workspace +└─ Benchmarks · Agent skills · Glossary +``` + +### Hub pages + +One per component, at `/components/<component>`. Same template every time: + +1. **Locator** — one paragraph plus the stack diagram with this layer + highlighted: what it is, what it sits on, what sits on it. +2. **Start here** — the one or two canonical entry pages. +3. **By mode** — grouped links (Concepts · Guides · Reference · Security · + Integrations that use it). Every entry links to its canonical mode-home. +4. Nothing else. No mechanics, no examples that live elsewhere. + +The "By mode" lists are generated, not hand-maintained: filter +`source.getPages()` to pages whose `components` facet includes this component, +then group by `type`. "Integrations that use it" is the subset whose frontmatter +also carries an `integration` block. This is why the facet earns its keep — the +hubs cost nothing to maintain once a page is correctly tagged. + +**Tagging rule — `eql` is conditional, not automatic.** Stack encrypts values +in three modes: general-purpose (a value in your app), Postgres columns, and +non-Postgres stores like DynamoDB. Only the Postgres path involves EQL. So tag +`eql` only when the page is actually about queryable-in-Postgres ciphertext. +General-purpose and DynamoDB pages are `components: [encryption, platform]` with +_no_ `eql` — DynamoDB (`encryptedDynamoDB`) is the canonical no-EQL example and +the reason "Stack depends on EQL" is wrong. EQL is the Postgres searchability +layer, not a layer every Stack page sits on. + +#### Stack SDK — `/components/stack-sdk` (facet: `encryption`) + +- Locator: the TypeScript SDK; encrypt and decrypt values in your app. Sits on + the Platform. Add EQL when the data lives in Postgres and you want the + ciphertext queryable there — but Stack also encrypts general-purpose values + and non-Postgres stores (e.g. DynamoDB) with no EQL at all. +- Start here: Quickstart · Reference → Stack SDK (client + configuration) +- Concepts: application-level encryption · searchable encryption · identity-aware encryption +- Guides: schema design · encrypt existing data · testing & CI · serverless & bundling +- Reference: `/reference/stack/*` (schema · encrypt-decrypt · supabase · drizzle-operators · errors) +- Security: `/security/stack-sdk` +- Integrations (auto): Supabase · Drizzle · Prisma · DynamoDB · Next.js … + +#### Proxy — `/components/proxy` (facet: `proxy`) + +- Locator: the no-app-changes path; sits in front of Postgres and speaks EQL for you. +- Start here: Reference → Proxy (configuration) · Guides → Proxy deployment +- Concepts: application-level encryption · searchable encryption +- Guides: proxy deployment · going to production +- Reference: `/reference/proxy/*` (configuration · message-flow · multitenant · errors) +- Security: `/security/proxy` +- Integrations (auto): AWS RDS/Aurora · Docker +- See also: EQL hub (shared dependency) + +#### EQL — `/components/eql` (facet: `eql`) ← the shared spine + +- Locator: the Postgres searchability layer. Makes ciphertext queryable in + Postgres by declaring an encrypted column's capability in the schema. The + Proxy always speaks EQL; the Stack SDK uses it only on its Postgres path (not + for general-purpose or DynamoDB encryption). +- Start here: Reference → EQL (install) · EQL core-concepts +- Concepts: EQL (typed-column model) · searchable encryption (leakage model) +- Reference: the whole `/reference/eql/*` tree +- Guides: troubleshooting → query performance +- Consumed by: Proxy hub (always) · Stack SDK hub (Postgres path only) +- Anti-drift: mechanics live in `/reference/eql/core-concepts`; this hub links only. + +#### Platform — `/components/platform` (facet: see vocab note) ← the foundation + +- Locator: the base everything relies on — key management (ZeroKMS) and identity-bound access (CTS). +- Start here: Security → Architecture · Concepts → Key management +- Concepts: key management · identity-aware encryption +- Reference: `/reference/auth/*` (lock-contexts · cts-tokens · oidc · access-keys) +- Security: architecture · zerokms · cts · availability · audit logging · key ownership +- Integrations (auto): Clerk · Auth0 · Okta + +### Worked example: how auth fits + +Auth is the concern most likely to feel like it needs its own section — it spans +the SDK, the Proxy, local dev, the Platform, and every auth provider. It doesn't +get one. It's the case that shows the model absorbing a cross-cutting concern +without bending: auth isn't a _layer_, it's a concern that shows up _on_ layers, +so it lives as facets and links, never as a tree section or a hub. + +"Auth" names several distinct things. Keep them separate: + +| Thing | What it is | `components` | Canonical home | +|---|---|---|---| +| CTS | Identity service in the Platform | `platform` | `/security/cts`, `/reference/auth/*` | +| `@cipherstash/auth` | Stack package (identity-aware encryption) | `[encryption, platform]` | `/reference/stack/auth` | +| Proxy stack-auth | How auth works inside the Proxy | `[proxy, platform]` | `/reference/proxy/*`, `/security/proxy` | +| Next.js adapter | Framework integration | `[encryption, platform]` | `/integrations/nextjs` | +| Clerk / Auth0 / Okta | Auth-provider integrations | `[platform]` | `/integrations/*` (`category: auth-provider`) | + +None of these is filed under an "auth" section, because there isn't one. Each is +filed by _mode_ (service → Security/Reference, package → Reference, providers → +Integrations) and tagged by _component_. The through-line is reassembled by +queries, not by a subtree: + +- The Platform hub gathers all of it — everything above carries `platform`. +- Faceted search on `integration.category: auth-provider` gives the providers. +- `/concepts/identity-aware-encryption` is the one explanatory page that ties the + concept together in prose; everything else links to it (anti-drift rule). + +Naming caution: `/reference/auth/*` means the CTS _service_; the `@cipherstash/auth` +_package_ lives at `/reference/stack/auth`. Two different things both called +"auth" — keep the paths distinct so contributors don't merge them. + +The `components` facet enum is `[encryption, platform, eql, proxy, cli]` +(collapsing the former `auth` and `zerokms` into a single `platform`, matching +the product story "Platform = CTS + ZeroKMS"). This maps 1:1 onto the hubs: +`encryption` → Stack SDK, `platform`, `eql`, `proxy`. + +Two notes: + +- The `encryption` facet value and its hub title **Stack SDK** differ on + purpose — the facet names the capability, the hub names the product. Document + once, move on. +- Collapsing `auth`/`zerokms` into `platform` means you can't isolate the + CTS/identity pages by facet query anymore — a `platform` query returns all of + Platform. That distinction still lives in nav location (`/security/cts` vs + `/security/zerokms`) and in `integration.category: auth-provider` for the + Clerk/Auth0/Okta pages, so it isn't lost — it just isn't on the `components` + axis. + +--- + ## Get started — CIP-3327 - [x] Section scaffold 🚧 From 299c1cb4f9a8541edd6f6ebd18aaaf3656e15b9c Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Mon, 6 Jul 2026 23:42:21 +1000 Subject: [PATCH 16/38] docs(v2): reconcile structure with the two-axis IA Closes the two follow-ups the IA content-model commit (4901b0f) flagged: - Comparisons -> Concepts: move content/docs/compare into content/docs/concepts/compare, drop `compare` from the root meta.json, sort the Compare group last under Concepts, and repoint the /stack/reference/comparisons redirects to /concepts/compare. The IA had already folded compare into Concepts in the sidebar spec; this makes the tree match. IA.md Comparisons checklist repointed to /concepts/compare/*. - components facet enum: collapse `auth`/`zerokms` into `platform` (source.config.ts), matching "Platform = CTS + ZeroKMS". Update the one current-content user of the old value (integrations/supabase/index.mdx). Component hubs (/components/<component>) remain future todos; not built here. Verified: types:check, validate-redirects (194/194), and next build all pass. --- IA.md | 15 ++++++++++----- content/docs/{ => concepts}/compare/index.mdx | 0 content/docs/{ => concepts}/compare/meta.json | 0 content/docs/concepts/meta.json | 2 +- content/docs/integrations/supabase/index.mdx | 2 +- content/docs/meta.json | 1 - source.config.ts | 2 +- v2-redirects.mjs | 4 ++-- 8 files changed, 15 insertions(+), 11 deletions(-) rename content/docs/{ => concepts}/compare/index.mdx (100%) rename content/docs/{ => concepts}/compare/meta.json (100%) diff --git a/IA.md b/IA.md index 3fb2645..a56a5cc 100644 --- a/IA.md +++ b/IA.md @@ -335,11 +335,16 @@ Two notes: ## Comparisons — CIP-3333 -- [x] Section scaffold 🚧 -- [ ] `/compare/aws-kms` (port) -- [ ] `/compare/fhe` (port) -- [ ] `/compare/rls-and-tde` (new — expand the Supabase-listing RLS contrast) -- [ ] `/compare/hashicorp-vault` (in flight on `docs/vault-comparison` branch — land there or here, then port) +Folded into Concepts (see the sidebar spec above): the comparison pages live at +`/concepts/compare/*`, not a top-level `/compare` tab. `/stack/reference/comparisons` +and the old `/compare/*` paths redirect there (`v2-redirects.mjs`). + +- [x] Section scaffold 🚧 (moved under `concepts/`) +- [ ] `/concepts/compare/aws-kms` (port) +- [ ] `/concepts/compare/fhe` (port) +- [ ] `/concepts/compare/zerokms-vs-hsm` (ZeroKMS vs hardware security modules) +- [ ] `/concepts/compare/rls-and-tde` (new — expand the Supabase-listing RLS contrast) +- [ ] `/concepts/compare/hashicorp-vault` (in flight on `docs/vault-comparison` branch — land there or here, then port) ## Guides diff --git a/content/docs/compare/index.mdx b/content/docs/concepts/compare/index.mdx similarity index 100% rename from content/docs/compare/index.mdx rename to content/docs/concepts/compare/index.mdx diff --git a/content/docs/compare/meta.json b/content/docs/concepts/compare/meta.json similarity index 100% rename from content/docs/compare/meta.json rename to content/docs/concepts/compare/meta.json diff --git a/content/docs/concepts/meta.json b/content/docs/concepts/meta.json index 521f756..49d9122 100644 --- a/content/docs/concepts/meta.json +++ b/content/docs/concepts/meta.json @@ -1,5 +1,5 @@ { "title": "Concepts", "icon": "Lightbulb", - "pages": ["..."] + "pages": ["...", "compare"] } diff --git a/content/docs/integrations/supabase/index.mdx b/content/docs/integrations/supabase/index.mdx index 86e2ef6..5ce53db 100644 --- a/content/docs/integrations/supabase/index.mdx +++ b/content/docs/integrations/supabase/index.mdx @@ -2,7 +2,7 @@ title: Supabase description: "Searchable, application-level encryption for your Supabase project — encrypt in your app, query in Postgres." type: tutorial -components: [encryption, eql, auth] +components: [encryption, eql, platform] audience: [developer] integration: category: platform diff --git a/content/docs/meta.json b/content/docs/meta.json index 74f486e..5d7ad44 100644 --- a/content/docs/meta.json +++ b/content/docs/meta.json @@ -4,7 +4,6 @@ "get-started", "integrations", "concepts", - "compare", "guides", "security", "solutions", diff --git a/source.config.ts b/source.config.ts index 18ad3f5..a95efc4 100644 --- a/source.config.ts +++ b/source.config.ts @@ -39,7 +39,7 @@ export const v2docs = defineDocs({ // tailored-quickstart picker (CIP-3339). Nav position never depends on // these — the sidebar tree comes from meta.json alone. components: z - .array(z.enum(["encryption", "auth", "zerokms", "eql", "proxy", "cli"])) + .array(z.enum(["encryption", "platform", "eql", "proxy", "cli"])) .optional(), audience: z.array(z.enum(["developer", "cto", "ciso"])).optional(), integration: z diff --git a/v2-redirects.mjs b/v2-redirects.mjs index dec811a..8779a7b 100644 --- a/v2-redirects.mjs +++ b/v2-redirects.mjs @@ -251,12 +251,12 @@ export const v2Redirects = [ }, { source: "/stack/reference/comparisons", - destination: "/compare", + destination: "/concepts/compare", permanent: false, }, { source: "/stack/reference/comparisons/:path*", - destination: "/compare/:path*", + destination: "/concepts/compare/:path*", permanent: false, }, { From 9bbdcfb91f40b4b8895b6048fc5108380ad7c806 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Tue, 7 Jul 2026 11:03:18 +1000 Subject: [PATCH 17/38] chore(v2): fix Biome lint errors so `bun run lint` passes clean MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v2 branch had a failing `biome check` (12 errors, 19 warnings), all pre-existing and unrelated to content. Cleared them: - a11y/noSvgWithoutTitle (5 errors): add role="img" + aria-label to the Prisma/Drizzle/Supabase/DynamoDB logo SVGs and the Supabase icon (empty/ missing accessible name). - Biome can't parse Tailwind v4 at-rules (@custom-variant) in src/app/global.css — it raised parse+format errors and 18 noImportantStyles warnings on intentional theme overrides. Exclude that one file from Biome (Tailwind owns it); nothing else authors CSS. - format (4): reformat scripts/generate-eql-docs.ts + three legacy content/stack meta.json files (whitespace only). - performance/noImgElement (2 warnings): documented biome-ignore on the two static SVG logos in layout.shared.tsx (next/image gives SVGs no benefit). - Bump the biome.json $schema to the installed 2.3.15. `bun run lint` now exits 0; types:check still passes. --- biome.json | 12 ++++++++++-- content/stack/meta.json | 7 +------ content/stack/reference/eql/meta.json | 6 ++---- content/stack/reference/stack/meta.json | 6 ++---- scripts/generate-eql-docs.ts | 18 ++++++++++-------- src/components/icons/supabase.tsx | 2 ++ src/components/integration-logos.tsx | 7 +++++++ src/lib/layout.shared.tsx | 2 ++ 8 files changed, 36 insertions(+), 24 deletions(-) diff --git a/biome.json b/biome.json index 95df3a1..ebbb858 100644 --- a/biome.json +++ b/biome.json @@ -1,5 +1,5 @@ { - "$schema": "https://biomejs.dev/schemas/2.2.0/schema.json", + "$schema": "https://biomejs.dev/schemas/2.3.15/schema.json", "vcs": { "enabled": true, "clientKind": "git", @@ -7,7 +7,15 @@ }, "files": { "ignoreUnknown": true, - "includes": ["**", "!node_modules", "!.next", "!dist", "!build", "!.source"] + "includes": [ + "**", + "!node_modules", + "!.next", + "!dist", + "!build", + "!.source", + "!src/app/global.css" + ] }, "formatter": { "enabled": true, diff --git a/content/stack/meta.json b/content/stack/meta.json index 340df56..34e13fc 100644 --- a/content/stack/meta.json +++ b/content/stack/meta.json @@ -1,8 +1,3 @@ { - "pages": [ - "quickstart", - "cipherstash", - "deploy", - "reference" - ] + "pages": ["quickstart", "cipherstash", "deploy", "reference"] } diff --git a/content/stack/reference/eql/meta.json b/content/stack/reference/eql/meta.json index 7568e54..d20d140 100644 --- a/content/stack/reference/eql/meta.json +++ b/content/stack/reference/eql/meta.json @@ -1,5 +1,3 @@ { - "pages": [ - "index" - ] -} \ No newline at end of file + "pages": ["index"] +} diff --git a/content/stack/reference/stack/meta.json b/content/stack/reference/stack/meta.json index 6f40473..dbb256d 100644 --- a/content/stack/reference/stack/meta.json +++ b/content/stack/reference/stack/meta.json @@ -1,5 +1,3 @@ { - "pages": [ - "latest" - ] -} \ No newline at end of file + "pages": ["latest"] +} diff --git a/scripts/generate-eql-docs.ts b/scripts/generate-eql-docs.ts index 4c85527..0625226 100644 --- a/scripts/generate-eql-docs.ts +++ b/scripts/generate-eql-docs.ts @@ -116,14 +116,16 @@ function escapeMdxSpecials(content: string): string { const escaped = parts .map((part, i) => { if (i % 2 === 1) return part; - return part - .replace(/\{/g, "\\{") - .replace(/\}/g, "\\}") - // Escape `<` unless it begins a real JSX/HTML tag, a closing - // tag, or an autolink (followed by a lowercase letter, `_`, `$`, - // or `/`). Uppercase-led tokens like `<T>` are type placeholders - // in the API reference, not JSX, so they must be escaped too. - .replace(/<(?![a-z_$/])/g, "\\<"); + return ( + part + .replace(/\{/g, "\\{") + .replace(/\}/g, "\\}") + // Escape `<` unless it begins a real JSX/HTML tag, a closing + // tag, or an autolink (followed by a lowercase letter, `_`, `$`, + // or `/`). Uppercase-led tokens like `<T>` are type placeholders + // in the API reference, not JSX, so they must be escaped too. + .replace(/<(?![a-z_$/])/g, "\\<") + ); }) .join(""); result.push(escaped); diff --git a/src/components/icons/supabase.tsx b/src/components/icons/supabase.tsx index 492ac61..1b9b9bb 100644 --- a/src/components/icons/supabase.tsx +++ b/src/components/icons/supabase.tsx @@ -4,6 +4,8 @@ export function SupabaseIcon(props: React.SVGProps<SVGSVGElement>) { viewBox="0 0 109 113" fill="none" xmlns="http://www.w3.org/2000/svg" + role="img" + aria-label="Supabase" {...props} > <path diff --git a/src/components/integration-logos.tsx b/src/components/integration-logos.tsx index 2ea7c85..e679d23 100644 --- a/src/components/integration-logos.tsx +++ b/src/components/integration-logos.tsx @@ -5,6 +5,7 @@ export function PrismaLogo({ className }: { className?: string }) { fill="currentColor" xmlns="http://www.w3.org/2000/svg" className={className} + role="img" aria-label="Prisma" > <path d="M254.313 235.519 148.057 4.5a8.288 8.288 0 0 0-7.215-4.486 8.534 8.534 0 0 0-7.499 4.084L1.296 212.55a8.521 8.521 0 0 0 .1 9.282l54.915 82.731a8.518 8.518 0 0 0 9.687 3.396l181.79-59.7a8.518 8.518 0 0 0 5.282-4.79 8.524 8.524 0 0 0 1.243-7.95Zm-23.077 5.769-156.642 51.45c-2.871.943-5.704-1.689-4.876-4.573l54.45-189.751a3.453 3.453 0 0 1 6.541-.252l103.013 137.317a3.454 3.454 0 0 1-2.486 5.809Z" /> @@ -19,6 +20,8 @@ export function DrizzleLogo({ className }: { className?: string }) { fill="currentColor" xmlns="http://www.w3.org/2000/svg" className={className} + role="img" + aria-label="Drizzle" > <rect width="5.25365" @@ -65,6 +68,8 @@ export function SupabaseLogo({ className }: { className?: string }) { fill="none" xmlns="http://www.w3.org/2000/svg" className={className} + role="img" + aria-label="Supabase" > {/* Wordmark — uses currentColor */} <path @@ -121,6 +126,8 @@ export function DynamoDBLogo({ className }: { className?: string }) { height="446" viewBox="0 0 561 446" className={className} + role="img" + aria-label="DynamoDB" > <g fill="none" transform="translate(.311)"> {/* Wordmark — uses currentColor */} diff --git a/src/lib/layout.shared.tsx b/src/lib/layout.shared.tsx index cba3da9..f0505ad 100644 --- a/src/lib/layout.shared.tsx +++ b/src/lib/layout.shared.tsx @@ -9,11 +9,13 @@ export const gitConfig = { function Logo() { return ( <div className="flex items-center gap-3"> + {/* biome-ignore lint/performance/noImgElement: static SVG logo; next/image gives no benefit for SVGs and would need extra config */} <img src="/docs/images/cipherstash-logo-dark.svg" alt="CipherStash" className="h-5 w-5 hidden dark:block" /> + {/* biome-ignore lint/performance/noImgElement: static SVG logo; next/image gives no benefit for SVGs and would need extra config */} <img src="/docs/images/cipherstash-logo-light.svg" alt="CipherStash" From a900369106956397b390f79c81851ec3d6f387b1 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Tue, 7 Jul 2026 12:02:43 +1000 Subject: [PATCH 18/38] chore: ignore .serena/ (local MCP tooling state) --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 1520277..7d245b9 100644 --- a/.gitignore +++ b/.gitignore @@ -39,3 +39,6 @@ content/stack/reference/drizzle/latest/ content/stack/reference/drizzle/v*/ .tmp-* .env.local + +# serena (local MCP tooling state) +.serena/ From 064c176669ce16c97d3c2a9d0f42b1ab44dc268a Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 16:48:13 +1000 Subject: [PATCH 19/38] fix(eql-docs): strip Doxygen <tt> tags so a stray one can't break the build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The EQL reference generator fetches the latest release's API.md at build time and passes lowercase-led tags (e.g. `<tt>`) through as real HTML. MDX requires every tag to be balanced, so a mangled SQL-comment source that emits an unclosed `<tt>` fails the whole Vercel build — which is exactly what eql-3.0.0-alpha.3 did (index.mdx:2153, `Expected a closing tag for <tt>`). `<tt>` is Doxygen's teletype tag; MDX doesn't need it. Strip `<tt>`/`</tt>` (non-code text only) before escaping, so malformed upstream docs degrade gracefully instead of taking the deploy down. Verified: regenerated content/stack/reference/eql/index.mdx from the live eql-3.0.0-alpha.3 release and `next build` compiles all 329 pages. --- scripts/generate-eql-docs.ts | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/generate-eql-docs.ts b/scripts/generate-eql-docs.ts index 0625226..975b9c1 100644 --- a/scripts/generate-eql-docs.ts +++ b/scripts/generate-eql-docs.ts @@ -120,6 +120,12 @@ function escapeMdxSpecials(content: string): string { part .replace(/\{/g, "\\{") .replace(/\}/g, "\\}") + // Strip Doxygen's `<tt>` teletype tags. They're a lowercase-led + // "tag" so the `<` rule below leaves them intact, but MDX requires + // every tag to be balanced — and mangled SQL-comment source can + // emit a stray, unclosed `<tt>` (e.g. eql-3.0.0-alpha.3's API.md), + // which fails the whole build. MDX doesn't need the tag, so drop it. + .replace(/<\/?tt\b[^>]*>/gi, "") // Escape `<` unless it begins a real JSX/HTML tag, a closing // tag, or an autolink (followed by a lowercase letter, `_`, `$`, // or `/`). Uppercase-led tokens like `<T>` are type placeholders From ccca2e5c80bda9986435d86893d7a8e676c8ddab Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Sun, 5 Jul 2026 23:03:19 +1000 Subject: [PATCH 20/38] =?UTF-8?q?docs(v2):=20prototype=20=E2=80=94=20gener?= =?UTF-8?q?ate=20the=20CLI=20reference=20from=20`stash=20--help`?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PROTOTYPE for review. Generates /reference/cli from the shipped `stash` CLI so the reference can't drift from the actual command surface, and stamps every page with the CLI version it was generated from. - scripts/generate-cli-docs.ts — parses `stash --help` (captured to a fixture) into a manifest, then renders one page per top-level command (grouped Setup/Auth/Database/Schema/Encrypt/Deployment) with a "Generated from stash v0.16.0" banner, a verifiedAgainst.cli facet, and [cli] / [cli, eql] components per the content-model tagging rule. - `bun run generate-docs:cli` (package.json). - scripts/fixtures/stash-help-0.16.0.txt — captured input. - content/docs/reference/cli/* — generated output (10 command pages + index). Bootstrap note: `stash` 0.16.0 is a hand-rolled TS CLI with no machine-readable output and no per-command --help, so we parse the single top-level --help. Target: add `stash manifest --json` to the CLI, then swap loadManifest() to read it and delete the parser — the page format stays identical. Today's --help is thin (no args, no per-command examples, auth/encrypt subcommands undetailed); those gaps are exactly what the JSON manifest fills. Surfaces the drift too: 0.16.0 uses `db install` (not `eql install`) and has ~15 commands the hand-written pages never documented. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/cli/auth.mdx | 26 ++ content/docs/reference/cli/db.mdx | 163 ++++++++++ content/docs/reference/cli/encrypt.mdx | 56 ++++ content/docs/reference/cli/env.mdx | 20 ++ content/docs/reference/cli/impl.mdx | 36 +++ content/docs/reference/cli/index.mdx | 65 +++- content/docs/reference/cli/init.mdx | 39 +++ content/docs/reference/cli/meta.json | 19 +- content/docs/reference/cli/plan.mdx | 34 ++ content/docs/reference/cli/schema.mdx | 30 ++ content/docs/reference/cli/status.mdx | 35 +++ content/docs/reference/cli/wizard.mdx | 26 ++ package.json | 1 + scripts/fixtures/stash-help-0.16.0.txt | 98 ++++++ scripts/generate-cli-docs.ts | 410 +++++++++++++++++++++++++ 15 files changed, 1054 insertions(+), 4 deletions(-) create mode 100644 content/docs/reference/cli/auth.mdx create mode 100644 content/docs/reference/cli/db.mdx create mode 100644 content/docs/reference/cli/encrypt.mdx create mode 100644 content/docs/reference/cli/env.mdx create mode 100644 content/docs/reference/cli/impl.mdx create mode 100644 content/docs/reference/cli/init.mdx create mode 100644 content/docs/reference/cli/plan.mdx create mode 100644 content/docs/reference/cli/schema.mdx create mode 100644 content/docs/reference/cli/status.mdx create mode 100644 content/docs/reference/cli/wizard.mdx create mode 100644 scripts/fixtures/stash-help-0.16.0.txt create mode 100644 scripts/generate-cli-docs.ts diff --git a/content/docs/reference/cli/auth.mdx b/content/docs/reference/cli/auth.mdx new file mode 100644 index 0000000..338e854 --- /dev/null +++ b/content/docs/reference/cli/auth.mdx @@ -0,0 +1,26 @@ +--- +title: stash auth +description: "Authenticate with CipherStash" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +Authenticate with CipherStash + +```bash +npx stash auth +``` + +## Examples + +```bash +npx stash auth login +``` diff --git a/content/docs/reference/cli/db.mdx b/content/docs/reference/cli/db.mdx new file mode 100644 index 0000000..82ffcd7 --- /dev/null +++ b/content/docs/reference/cli/db.mdx @@ -0,0 +1,163 @@ +--- +title: stash db +description: "Reference for the `stash db` commands." +type: reference +components: [cli, eql] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +The `stash db` command group. + +### `db install` + +Scaffold stash.config.ts (if missing) and install EQL extensions + +```bash +npx stash db install [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--force` | Reinstall / overwrite even if already installed | +| `--dry-run` | Show what would happen without making changes | +| `--supabase` | Use Supabase-compatible mode (auto-detected from DATABASE_URL) | +| `--drizzle` | Generate a Drizzle migration instead of direct install (auto-detected from project) | +| `--migration` | (requires --supabase) Write a Supabase migration file instead of running SQL directly | +| `--direct` | (requires --supabase) Run the SQL directly against the database (mutually exclusive with --migration) | +| `--migrations-dir <path>` | (requires --supabase) Override the Supabase migrations directory (default: supabase/migrations) | +| `--exclude-operator-family` | Skip operator family creation | +| `--latest` | Fetch the latest EQL from GitHub | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | + + +#### Examples + +```bash +npx stash db install +``` + +### `db upgrade` + +Upgrade EQL extensions to the latest version + +```bash +npx stash db upgrade [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--dry-run` | Show what would happen without making changes | +| `--supabase` | Use Supabase-compatible mode (auto-detected from DATABASE_URL) | +| `--exclude-operator-family` | Skip operator family creation | +| `--latest` | Fetch the latest EQL from GitHub | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | + + +### `db push` + +Push encryption schema (writes pending if active config already exists) + +```bash +npx stash db push [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--dry-run` | Show what would happen without making changes | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | + + +#### Examples + +```bash +npx stash db push +``` + +### `db activate` + +Promote pending → active without renames (use after additive db push) + +```bash +npx stash db activate [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | + + +### `db validate` + +Validate encryption schema + +```bash +npx stash db validate [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--supabase` | Use Supabase-compatible mode (auto-detected from DATABASE_URL) | +| `--exclude-operator-family` | Skip operator family creation | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | + + +### `db migrate` + +Run pending encrypt config migrations + +```bash +npx stash db migrate [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | + + +### `db status` + +Show EQL installation status + +```bash +npx stash db status [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | + + +### `db test-connection` + +Test database connectivity + +```bash +npx stash db test-connection [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | diff --git a/content/docs/reference/cli/encrypt.mdx b/content/docs/reference/cli/encrypt.mdx new file mode 100644 index 0000000..887983e --- /dev/null +++ b/content/docs/reference/cli/encrypt.mdx @@ -0,0 +1,56 @@ +--- +title: stash encrypt +description: "Reference for the `stash encrypt` commands." +type: reference +components: [cli, eql] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +The `stash encrypt` command group. + +### `encrypt status` + +Show per-column migration status (phase, progress, drift) + +```bash +npx stash encrypt status +``` + +### `encrypt plan` + +Diff intent (.cipherstash/migrations.json) vs observed state + +```bash +npx stash encrypt plan +``` + +### `encrypt backfill` + +Resumably encrypt plaintext into the encrypted column + +```bash +npx stash encrypt backfill +``` + +### `encrypt cutover` + +Rename swap encrypted → primary column + +```bash +npx stash encrypt cutover +``` + +### `encrypt drop` + +Generate a migration to drop the plaintext column + +```bash +npx stash encrypt drop +``` diff --git a/content/docs/reference/cli/env.mdx b/content/docs/reference/cli/env.mdx new file mode 100644 index 0000000..f30dc09 --- /dev/null +++ b/content/docs/reference/cli/env.mdx @@ -0,0 +1,20 @@ +--- +title: stash env +description: "(experimental) Print production env vars for deployment" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +(experimental) Print production env vars for deployment + +```bash +npx stash env +``` diff --git a/content/docs/reference/cli/impl.mdx b/content/docs/reference/cli/impl.mdx new file mode 100644 index 0000000..5604c50 --- /dev/null +++ b/content/docs/reference/cli/impl.mdx @@ -0,0 +1,36 @@ +--- +title: stash impl +description: "Execute the plan with a local agent" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +Execute the plan with a local agent + +```bash +npx stash impl [flags] +``` + +### Flags + +| Flag | Description | +| --- | --- | +| `--continue-without-plan` | Skip planning and go straight to implementation (interactively confirms before proceeding) | +| `--target <name>` | Skip the agent-target picker and hand off directly to one of claude-code \| codex \| agents-md \| wizard. Safe to call from non-TTY contexts (CI, pipes). Without --target in non-TTY, the command prints a hint and exits cleanly instead of hanging. | + + +## Examples + +```bash +npx stash impl +npx stash impl --continue-without-plan +npx stash impl --target claude-code +``` diff --git a/content/docs/reference/cli/index.mdx b/content/docs/reference/cli/index.mdx index 8eeffd3..8b4ab15 100644 --- a/content/docs/reference/cli/index.mdx +++ b/content/docs/reference/cli/index.mdx @@ -1,8 +1,67 @@ --- title: CLI -description: "CLI documentation — being built as part of the docs V2 overhaul." +description: "Command reference for the stash CLI, generated from v0.16.0." +type: reference +components: [cli] +verifiedAgainst: + cli: "0.16.0" --- -This section is being built as part of the docs V2 overhaul ([CIP-3307](https://linear.app/cipherstash/issue/CIP-3307)). Track progress in [IA.md](https://github.com/cipherstash/docs/blob/v2/IA.md). +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} -Until it lands, current documentation lives in the [existing docs](/stack). +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +The `stash` CLI. Install with `npx stash@0.16.0`. Every command accepts `--help` and `--version`. + +### Setup & workflow + +| Command | Description | +| --- | --- | +| [`init`](/reference/cli/init) | Initialize CipherStash for your project | +| [`plan`](/reference/cli/plan) | Draft a reviewable encryption plan at .cipherstash/plan.md | +| [`impl`](/reference/cli/impl) | Execute the plan with a local agent | +| [`status`](/reference/cli/status) | Displays implementation status | +| [`wizard`](/reference/cli/wizard) | AI-guided encryption setup (reads your codebase) | + +### Auth + +| Command | Description | +| --- | --- | +| [`auth`](/reference/cli/auth) | Authenticate with CipherStash | + +### Database + +| Command | Description | +| --- | --- | +| [`db install`](/reference/cli/db#db-install) | Scaffold stash.config.ts (if missing) and install EQL extensions | +| [`db upgrade`](/reference/cli/db#db-upgrade) | Upgrade EQL extensions to the latest version | +| [`db push`](/reference/cli/db#db-push) | Push encryption schema (writes pending if active config already exists) | +| [`db activate`](/reference/cli/db#db-activate) | Promote pending → active without renames (use after additive db push) | +| [`db validate`](/reference/cli/db#db-validate) | Validate encryption schema | +| [`db migrate`](/reference/cli/db#db-migrate) | Run pending encrypt config migrations | +| [`db status`](/reference/cli/db#db-status) | Show EQL installation status | +| [`db test-connection`](/reference/cli/db#db-test-connection) | Test database connectivity | + +### Schema + +| Command | Description | +| --- | --- | +| [`schema build`](/reference/cli/schema#schema-build) | Build an encryption schema from your database | + +### Encrypt + +| Command | Description | +| --- | --- | +| [`encrypt status`](/reference/cli/encrypt#encrypt-status) | Show per-column migration status (phase, progress, drift) | +| [`encrypt plan`](/reference/cli/encrypt#encrypt-plan) | Diff intent (.cipherstash/migrations.json) vs observed state | +| [`encrypt backfill`](/reference/cli/encrypt#encrypt-backfill) | Resumably encrypt plaintext into the encrypted column | +| [`encrypt cutover`](/reference/cli/encrypt#encrypt-cutover) | Rename swap encrypted → primary column | +| [`encrypt drop`](/reference/cli/encrypt#encrypt-drop) | Generate a migration to drop the plaintext column | + +### Deployment + +| Command | Description | +| --- | --- | +| [`env`](/reference/cli/env) | (experimental) Print production env vars for deployment | diff --git a/content/docs/reference/cli/init.mdx b/content/docs/reference/cli/init.mdx new file mode 100644 index 0000000..0e843c3 --- /dev/null +++ b/content/docs/reference/cli/init.mdx @@ -0,0 +1,39 @@ +--- +title: stash init +description: "Initialize CipherStash for your project" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +Initialize CipherStash for your project + +```bash +npx stash init [flags] +``` + +### Flags + +| Flag | Description | +| --- | --- | +| `--supabase` | Use Supabase-specific setup flow | +| `--drizzle` | Use Drizzle-specific setup flow | +| `--prisma-next` | Use Prisma Next-specific setup flow (EQL bundle installed via prisma-next migration apply) | +| `--proxy` | Query encrypted data via CipherStash Proxy | +| `--no-proxy` | Query encrypted data directly via the SDK (default) | + + +## Examples + +```bash +npx stash init +npx stash init --supabase +npx stash init --prisma-next +``` diff --git a/content/docs/reference/cli/meta.json b/content/docs/reference/cli/meta.json index 0a67892..5e139d7 100644 --- a/content/docs/reference/cli/meta.json +++ b/content/docs/reference/cli/meta.json @@ -1,4 +1,21 @@ { "title": "CLI", - "pages": ["..."] + "pages": [ + "---Setup & workflow---", + "init", + "plan", + "impl", + "status", + "wizard", + "---Auth---", + "auth", + "---Database---", + "db", + "---Schema---", + "schema", + "---Encrypt---", + "encrypt", + "---Deployment---", + "env" + ] } diff --git a/content/docs/reference/cli/plan.mdx b/content/docs/reference/cli/plan.mdx new file mode 100644 index 0000000..0c536df --- /dev/null +++ b/content/docs/reference/cli/plan.mdx @@ -0,0 +1,34 @@ +--- +title: stash plan +description: "Draft a reviewable encryption plan at .cipherstash/plan.md" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +Draft a reviewable encryption plan at .cipherstash/plan.md + +```bash +npx stash plan [flags] +``` + +### Flags + +| Flag | Description | +| --- | --- | +| `--complete-rollout` | Plan the entire encryption lifecycle (schema-add through drop) in one document. Skips the production-deploy gate that normally separates rollout from cutover. Only safe when this database is not backing a deployed application (local dev, sandbox, freshly seeded test environment). | +| `--target <name>` | Skip the agent-target picker and hand off directly to one of claude-code \| codex \| agents-md \| wizard. Safe to call from non-TTY contexts (CI, pipes). Without --target in non-TTY, the command prints a hint and exits cleanly instead of hanging. | + + +## Examples + +```bash +npx stash plan +``` diff --git a/content/docs/reference/cli/schema.mdx b/content/docs/reference/cli/schema.mdx new file mode 100644 index 0000000..bd8400c --- /dev/null +++ b/content/docs/reference/cli/schema.mdx @@ -0,0 +1,30 @@ +--- +title: stash schema +description: "Reference for the `stash schema` commands." +type: reference +components: [cli, eql] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +The `stash schema` command group. + +### `schema build` + +Build an encryption schema from your database + +```bash +npx stash schema build +``` + +#### Examples + +```bash +npx stash schema build +``` diff --git a/content/docs/reference/cli/status.mdx b/content/docs/reference/cli/status.mdx new file mode 100644 index 0000000..8fc84b6 --- /dev/null +++ b/content/docs/reference/cli/status.mdx @@ -0,0 +1,35 @@ +--- +title: stash status +description: "Displays implementation status" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +Displays implementation status + +```bash +npx stash status [flags] +``` + +### Flags + +| Flag | Description | +| --- | --- | +| `--quest` | Force the quest-log output (emoji + progress bars) even in non-TTY contexts. Default is auto: fancy in a terminal, plain in CI / pipes / agents. | +| `--plain` | Force the plain-text output even in TTY contexts. | +| `--json` | Emit a structured JSON document instead. | + + +## Examples + +```bash +npx stash status +``` diff --git a/content/docs/reference/cli/wizard.mdx b/content/docs/reference/cli/wizard.mdx new file mode 100644 index 0000000..e79f82d --- /dev/null +++ b/content/docs/reference/cli/wizard.mdx @@ -0,0 +1,26 @@ +--- +title: stash wizard +description: "AI-guided encryption setup (reads your codebase)" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.16.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} + +<Callout type="info"> +Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +</Callout> + +AI-guided encryption setup (reads your codebase) + +```bash +npx stash wizard +``` + +## Examples + +```bash +npx stash wizard +``` diff --git a/package.json b/package.json index 3b4681a..210cfd6 100644 --- a/package.json +++ b/package.json @@ -13,6 +13,7 @@ "format": "biome format --write", "generate-docs": "tsx scripts/generate-docs.ts", "generate-docs:eql": "tsx scripts/generate-eql-docs.ts", + "generate-docs:cli": "tsx scripts/generate-cli-docs.ts", "validate-links": "tsx scripts/validate-links.ts", "validate-redirects": "tsx scripts/validate-v2-redirects.ts" }, diff --git a/scripts/fixtures/stash-help-0.16.0.txt b/scripts/fixtures/stash-help-0.16.0.txt new file mode 100644 index 0000000..ad4f17f --- /dev/null +++ b/scripts/fixtures/stash-help-0.16.0.txt @@ -0,0 +1,98 @@ +◇ injected env (0) from .env.local // tip: ⌘ multiple files { path: ['.env.local', '.env'] } +◇ injected env (0) from .env.development.local // tip: ⌘ override existing { override: true } +◇ injected env (0) from .env.development // tip: ⌘ enable debugging { debug: true } +◇ injected env (0) from .env // tip: ⌘ enable debugging { debug: true } +CipherStash CLI v0.16.0 + +Usage: bunx stash <command> [options] + +Commands: + init Initialize CipherStash for your project + plan Draft a reviewable encryption plan at .cipherstash/plan.md + impl Execute the plan with a local agent + status Displays implementation status + auth <subcommand> Authenticate with CipherStash + wizard AI-guided encryption setup (reads your codebase) + + db install Scaffold stash.config.ts (if missing) and install EQL extensions + db upgrade Upgrade EQL extensions to the latest version + db push Push encryption schema (writes pending if active config already exists) + db activate Promote pending → active without renames (use after additive db push) + db validate Validate encryption schema + db migrate Run pending encrypt config migrations + db status Show EQL installation status + db test-connection Test database connectivity + + schema build Build an encryption schema from your database + + encrypt status Show per-column migration status (phase, progress, drift) + encrypt plan Diff intent (.cipherstash/migrations.json) vs observed state + encrypt backfill Resumably encrypt plaintext into the encrypted column + encrypt cutover Rename swap encrypted → primary column + encrypt drop Generate a migration to drop the plaintext column + + env (experimental) Print production env vars for deployment + +Options: + --help, -h Show help + --version, -v Show version + +Init Flags: + --supabase Use Supabase-specific setup flow + --drizzle Use Drizzle-specific setup flow + --prisma-next Use Prisma Next-specific setup flow (EQL bundle installed via prisma-next migration apply) + --proxy Query encrypted data via CipherStash Proxy + --no-proxy Query encrypted data directly via the SDK (default) + +Plan Flags: + --complete-rollout Plan the entire encryption lifecycle (schema-add through drop) + in one document. Skips the production-deploy gate that + normally separates rollout from cutover. Only safe when this + database is not backing a deployed application (local dev, + sandbox, freshly seeded test environment). + --target <name> Skip the agent-target picker and hand off directly to one of + claude-code | codex | agents-md | wizard. Safe to call from + non-TTY contexts (CI, pipes). Without --target in non-TTY, + the command prints a hint and exits cleanly instead of hanging. + +Status Flags: + --quest Force the quest-log output (emoji + progress bars) + even in non-TTY contexts. Default is auto: fancy + in a terminal, plain in CI / pipes / agents. + --plain Force the plain-text output even in TTY contexts. + --json Emit a structured JSON document instead. + +Impl Flags: + --continue-without-plan Skip planning and go straight to implementation + (interactively confirms before proceeding) + --target <name> Skip the agent-target picker and hand off directly to one of + claude-code | codex | agents-md | wizard. Safe to call from + non-TTY contexts (CI, pipes). Without --target in non-TTY, + the command prints a hint and exits cleanly instead of hanging. + +DB Flags: + --force (install) Reinstall / overwrite even if already installed + --dry-run (install, push, upgrade) Show what would happen without making changes + --supabase (install, upgrade, validate) Use Supabase-compatible mode (auto-detected from DATABASE_URL) + --drizzle (install) Generate a Drizzle migration instead of direct install (auto-detected from project) + --migration (install, requires --supabase) Write a Supabase migration file instead of running SQL directly + --direct (install, requires --supabase) Run the SQL directly against the database (mutually exclusive with --migration) + --migrations-dir <path> (install, requires --supabase) Override the Supabase migrations directory (default: supabase/migrations) + --exclude-operator-family (install, upgrade, validate) Skip operator family creation + --latest (install, upgrade) Fetch the latest EQL from GitHub + --database-url <url> (all db / schema commands) Override DATABASE_URL for this run only — never written to disk + +Examples: + bunx stash init + bunx stash init --supabase + bunx stash init --prisma-next + bunx stash plan + bunx stash impl + bunx stash impl --continue-without-plan + bunx stash impl --target claude-code + bunx stash status + bunx stash auth login + bunx stash wizard + bunx stash db install + bunx stash db push + bunx stash schema build diff --git a/scripts/generate-cli-docs.ts b/scripts/generate-cli-docs.ts new file mode 100644 index 0000000..73e21bd --- /dev/null +++ b/scripts/generate-cli-docs.ts @@ -0,0 +1,410 @@ +#!/usr/bin/env tsx +/** + * CLI reference generator — PROTOTYPE (CIP-33xx). + * + * Generates the `/reference/cli` pages from the `stash` CLI itself, so the + * reference can never drift from the shipped command surface. Every page is + * stamped with the CLI version it was generated from. + * + * ── Data source ─────────────────────────────────────────────────────────── + * TODAY (bootstrap): we parse `stash --help`, captured to a fixture. `stash` + * is a hand-rolled TS CLI with no machine-readable output yet, and no + * per-command `--help` (every command prints the top-level help), so the + * single top-level help is the whole surface. It is thin — no args, no + * per-command examples, `auth`/`encrypt` subcommands undetailed. + * + * TARGET: add `stash manifest --json` to the CLI (it already has the command + * registry it prints `--help` from). Then replace `loadManifest()` with: + * + * JSON.parse(execSync(`npx stash@${CLI_VERSION} manifest --json`)) + * + * and delete the parser below. The renderer and page format stay identical — + * that is the point of this prototype. + * + * ── Versioning ──────────────────────────────────────────────────────────── + * CLI_VERSION is pinned. Bump it (Renovate can watch the `stash` npm dep), + * re-run `bun run generate-docs:cli`, commit the regenerated pages. Every page + * carries `verifiedAgainst.cli` and a visible banner, so readers and agents + * always know which version the docs describe. + */ +import { execSync } from "node:child_process"; +import fs from "node:fs"; +import path from "node:path"; + +const CLI_NAME = "stash"; +const CLI_VERSION = "0.16.0"; // pinned; bump + regenerate on each release +const RUNNER = "npx"; // normalized invocation shown in docs +const FIXTURE = path.join( + process.cwd(), + "scripts/fixtures", + `stash-help-${CLI_VERSION}.txt`, +); +const OUT_DIR = path.join(process.cwd(), "content/docs/reference/cli"); + +// ── Types (this shape is the spec for `stash manifest --json`) ────────────── +interface Flag { + name: string; // "--supabase" + value?: string; // "<path>" + description: string; + appliesTo?: string[]; // db-flag applicability: ["install", "upgrade"] or ["all"] +} +interface Command { + path: string; // "db install" + base: string; // "db" + sub?: string; // "install" + summary: string; + flags: Flag[]; + examples: string[]; +} +interface Manifest { + name: string; + version: string; + usage: string; + globalFlags: Flag[]; + commands: Command[]; +} + +// Which nav group each top-level command belongs to, and the group order. +const GROUPS: Record<string, string> = { + init: "Setup & workflow", + plan: "Setup & workflow", + impl: "Setup & workflow", + status: "Setup & workflow", + wizard: "Setup & workflow", + auth: "Auth", + db: "Database", + schema: "Schema", + encrypt: "Encrypt", + env: "Deployment", +}; +const GROUP_ORDER = [ + "Setup & workflow", + "Auth", + "Database", + "Schema", + "Encrypt", + "Deployment", +]; +// Known db/schema subcommand names, used to resolve db-flag applicability. +const DB_SUBCOMMANDS = new Set([ + "install", + "upgrade", + "push", + "activate", + "validate", + "migrate", + "status", + "test-connection", +]); + +// EQL/Postgres commands get the `eql` component facet too (content-model rule: +// tag `eql` only for queryable-in-Postgres ciphertext). +const componentsFor = (base: string): string[] => + ["db", "schema", "encrypt"].includes(base) ? ["cli", "eql"] : ["cli"]; + +// ── Source ────────────────────────────────────────────────────────────────── +function loadHelp(): string { + if (fs.existsSync(FIXTURE)) return fs.readFileSync(FIXTURE, "utf8"); + // Bootstrap: fetch and cache. (Target: `stash manifest --json`.) + const out = execSync(`npx --yes ${CLI_NAME}@${CLI_VERSION} --help`, { + encoding: "utf8", + cwd: require("node:os").tmpdir(), + }); + fs.mkdirSync(path.dirname(FIXTURE), { recursive: true }); + fs.writeFileSync(FIXTURE, out); + return out; +} + +// Drop dotenvx's env-injection tips and blank leading noise. +const stripNoise = (text: string): string[] => + text + .split("\n") + .filter((l) => !/^\s*◇|injected env|dotenvx|www\.(dotenvx|vestauth)/.test(l)); + +// ── Parser (delete once `stash manifest --json` exists) ───────────────────── +function parseHelp(text: string): Manifest { + const lines = stripNoise(text); + const joined = lines.join("\n"); + + const version = joined.match(/CipherStash CLI v([0-9]+\.[0-9]+\.[0-9]+)/)?.[1] ?? CLI_VERSION; + + // Section boundaries: a line like "Commands:", "Options:", "DB Flags:", "Examples:". + const sections: Record<string, string[]> = {}; + let current = ""; + for (const line of lines) { + const header = line.match(/^([A-Za-z][A-Za-z ]*):\s*$/); + if (header && !line.startsWith(" ")) { + current = header[1].trim(); + sections[current] = []; + } else if (current) { + sections[current].push(line); + } + } + + // Commands: " db install Scaffold ..." (name is non-greedy up to 2+ spaces) + const commands: Command[] = []; + for (const line of sections.Commands ?? []) { + const m = line.match(/^ {2}(\S.*?) {2,}(.+)$/); + if (!m) continue; + const rawName = m[1].replace(/\s*<subcommand>\s*/, "").trim(); + const [base, ...rest] = rawName.split(/\s+/); + commands.push({ + path: rawName, + base, + sub: rest.length ? rest.join(" ") : undefined, + summary: m[2].trim(), + flags: [], + examples: [], + }); + } + + // Global options. + const globalFlags = parseFlagBlock(sections.Options ?? []); + + // Per-command flag sections: "Init Flags", "Plan Flags", "DB Flags", … + for (const [name, body] of Object.entries(sections)) { + const fm = name.match(/^(.*) Flags$/); + if (!fm) continue; + const label = fm[1].toLowerCase(); // "init", "plan", "db" + const flags = parseFlagBlock(body); + if (label === "db") { + // DB flags carry applicability annotations; resolve onto each subcommand. + for (const cmd of commands.filter((c) => c.base === "db")) { + cmd.flags = flags.filter( + (f) => + !f.appliesTo || + f.appliesTo.includes("all") || + (cmd.sub ? f.appliesTo.includes(cmd.sub) : false), + ); + } + } else { + const cmd = commands.find((c) => c.path === label); + if (cmd) cmd.flags = flags; + } + } + + // Examples: " npx stash db install" → attach to the longest matching command. + const byLength = [...commands].sort((a, b) => b.path.length - a.path.length); + for (const line of sections.Examples ?? []) { + const inv = line.trim(); + const m = inv.match(/^(?:npx|bunx|pnpm dlx|stash)\s+(?:stash\s+)?(.+)$/); + const argPart = m ? m[1] : inv; + const cmd = byLength.find( + (c) => argPart === c.path || argPart.startsWith(`${c.path} `), + ); + if (cmd) cmd.examples.push(`${RUNNER} ${CLI_NAME} ${argPart}`); + } + + return { + name: CLI_NAME, + version, + usage: `${RUNNER} ${CLI_NAME} <command> [options]`, + globalFlags, + commands, + }; +} + +// Parse an indented flag block, folding continuation lines into descriptions. +function parseFlagBlock(body: string[]): Flag[] { + const flags: Flag[] = []; + for (const line of body) { + if (!line.trim()) continue; + const m = line.match(/^ {2}(--[\w-]+)(?:,\s*-\w)?(?: +(<[^>]+>))? {2,}(.+)$/); + if (m) { + let description = m[3].trim(); + let appliesTo: string[] | undefined; + // Leading "(install, push, …)" on DB flags = applicability (+ conditions). + const paren = description.match(/^\(([^)]+)\)\s*(.*)$/); + if (paren) { + const inner = paren[1].trim(); + if (/^all\b/.test(inner)) { + // "(all db / schema commands)" — applies everywhere; drop the note. + appliesTo = ["all"]; + description = paren[2]; + } else { + const tokens = inner.split(/[,/]/).map((t) => t.trim()).filter(Boolean); + const applic = tokens.filter((t) => DB_SUBCOMMANDS.has(t)); + const conditions = tokens.filter((t) => !DB_SUBCOMMANDS.has(t)); + if (applic.length) appliesTo = applic; + description = + (conditions.length ? `(${conditions.join(", ")}) ` : "") + paren[2]; + } + } + flags.push({ name: m[1], value: m[2], description: description.trim(), appliesTo }); + } else { + const cont = line.trim(); + if (flags.length && cont) flags[flags.length - 1].description += ` ${cont}`; + } + } + return flags; +} + +// ── Render ─────────────────────────────────────────────────────────────────── +const GENERATED = `{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from \`${CLI_NAME} --help\` (v${CLI_VERSION}). Bump CLI_VERSION and re-run \`bun run generate-docs:cli\`. */}`; + +function banner(): string { + return `<Callout type="info"> +Generated from **\`${CLI_NAME}\` v${CLI_VERSION}**. Run \`${RUNNER} ${CLI_NAME}@${CLI_VERSION} --help\` to see the live command surface. +</Callout>`; +} + +function flagsTable(flags: Flag[]): string { + if (!flags.length) return ""; + const rows = flags + .map((f) => { + const opt = `\`${f.name}${f.value ? ` ${f.value}` : ""}\``; + return `| ${opt} | ${f.description.replace(/\|/g, "\\|")} |`; + }) + .join("\n"); + return `\n### Flags\n\n| Flag | Description |\n| --- | --- |\n${rows}\n`; +} + +function commandSection(cmd: Command, level: "##" | "###"): string { + const synopsis = `${RUNNER} ${CLI_NAME} ${cmd.path}${cmd.flags.length ? " [flags]" : ""}`; + const parts = [ + `${level} \`${cmd.path}\``, + "", + cmd.summary, + "", + "```bash", + synopsis, + "```", + ]; + if (cmd.flags.length) + parts.push(flagsTable(cmd.flags).replace("### Flags", `${level}# Flags`)); + if (cmd.examples.length) { + parts.push(`\n${level}# Examples\n`, "```bash", cmd.examples.join("\n"), "```"); + } + return parts.join("\n"); +} + +function renderPage(base: string, cmds: Command[]): { slug: string; body: string } { + const isGroup = cmds.some((c) => c.sub) || cmds.length > 1; + const title = base; + const components = componentsFor(base); + const description = isGroup + ? `Reference for the \`${CLI_NAME} ${base}\` commands.` + : cmds[0].summary; + + const frontmatter = [ + "---", + `title: ${CLI_NAME} ${title}`, + `description: ${JSON.stringify(description)}`, + "type: reference", + `components: [${components.join(", ")}]`, + "verifiedAgainst:", + ` cli: "${CLI_VERSION}"`, + "---", + ].join("\n"); + + const parts = [frontmatter, "", GENERATED, "", banner(), ""]; + + if (isGroup) { + parts.push( + `The \`${CLI_NAME} ${base}\` command group.`, + "", + cmds.map((c) => commandSection(c, "###")).join("\n\n"), + ); + } else { + const c = cmds[0]; + parts.push(c.summary, "", "```bash", `${RUNNER} ${CLI_NAME} ${c.path}${c.flags.length ? " [flags]" : ""}`, "```"); + if (c.flags.length) parts.push(flagsTable(c.flags)); + if (c.examples.length) parts.push("\n## Examples\n", "```bash", c.examples.join("\n"), "```"); + } + + return { slug: base, body: `${parts.join("\n").trimEnd()}\n` }; +} + +function renderIndex(manifest: Manifest, groups: Map<string, string[]>): string { + const frontmatter = [ + "---", + "title: CLI", + `description: "Command reference for the ${CLI_NAME} CLI, generated from v${CLI_VERSION}."`, + "type: reference", + "components: [cli]", + "verifiedAgainst:", + ` cli: "${CLI_VERSION}"`, + "---", + ].join("\n"); + + const sections = GROUP_ORDER.filter((g) => groups.has(g)) + .map((g) => { + const rows = groups + .get(g)! + .flatMap((base) => + manifest.commands + .filter((c) => c.base === base) + .map((c) => { + const anchor = c.sub ? `#${c.path.replace(/\s+/g, "-")}` : ""; + return `| [\`${c.path}\`](/reference/cli/${base}${anchor}) | ${c.summary} |`; + }), + ) + .join("\n"); + return `### ${g}\n\n| Command | Description |\n| --- | --- |\n${rows}`; + }) + .join("\n\n"); + + return `${frontmatter} + +${GENERATED} + +${banner()} + +The \`${CLI_NAME}\` CLI. Install with \`${RUNNER} ${CLI_NAME}@${CLI_VERSION}\`. Every command accepts \`--help\` and \`--version\`. + +${sections} +`; +} + +function renderMeta(groups: Map<string, string[]>): string { + const pages: string[] = []; + for (const g of GROUP_ORDER) { + if (!groups.has(g)) continue; + pages.push(`---${g}---`); + pages.push(...groups.get(g)!); + } + return `${JSON.stringify({ title: "CLI", pages }, null, 2)}\n`; +} + +// ── Main ───────────────────────────────────────────────────────────────────── +function loadManifest(): Manifest { + // Swap point: return JSON.parse(execSync(`npx ${CLI_NAME}@${CLI_VERSION} manifest --json`)). + return parseHelp(loadHelp()); +} + +function main() { + const manifest = loadManifest(); + + // Group top-level commands by base, preserving discovery order. + const bases: string[] = []; + for (const c of manifest.commands) if (!bases.includes(c.base)) bases.push(c.base); + + const groups = new Map<string, string[]>(); + for (const base of bases) { + const g = GROUPS[base] ?? "Other"; + if (!groups.has(g)) groups.set(g, []); + groups.get(g)!.push(base); + } + + // Clean previously generated pages, then write fresh. + fs.mkdirSync(OUT_DIR, { recursive: true }); + for (const f of fs.readdirSync(OUT_DIR)) { + if (f.endsWith(".mdx") || f === "meta.json") fs.rmSync(path.join(OUT_DIR, f)); + } + + let count = 0; + for (const base of bases) { + const cmds = manifest.commands.filter((c) => c.base === base); + const { slug, body } = renderPage(base, cmds); + fs.writeFileSync(path.join(OUT_DIR, `${slug}.mdx`), body); + count++; + } + fs.writeFileSync(path.join(OUT_DIR, "index.mdx"), renderIndex(manifest, groups)); + fs.writeFileSync(path.join(OUT_DIR, "meta.json"), renderMeta(groups)); + + console.log( + `✓ Generated ${count} CLI reference page(s) for ${CLI_NAME} v${manifest.version} → ${path.relative(process.cwd(), OUT_DIR)}`, + ); +} + +main(); From 73712570c838ec306b527434b0a1ad673cb4c1ae Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Sun, 5 Jul 2026 23:23:02 +1000 Subject: [PATCH 21/38] docs(v2): CLI generator tracks latest published stash + wire into prebuild - Resolve the latest published version via `npm view stash version` and run that exact version live, so a new `stash` release refreshes the docs the next time the script runs (offline: falls back to the cached fixture). Removes the pinned CLI_VERSION constant. - Wire `generate-docs:cli` into `prebuild`, next to the EQL/TypeDoc generators, so every production build regenerates the CLI reference from the latest CLI. - Rename the fixture to a version-agnostic cache (scripts/fixtures/stash-help.txt). Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/cli/auth.mdx | 2 +- content/docs/reference/cli/db.mdx | 2 +- content/docs/reference/cli/encrypt.mdx | 2 +- content/docs/reference/cli/env.mdx | 2 +- content/docs/reference/cli/impl.mdx | 2 +- content/docs/reference/cli/index.mdx | 2 +- content/docs/reference/cli/init.mdx | 2 +- content/docs/reference/cli/plan.mdx | 2 +- content/docs/reference/cli/schema.mdx | 2 +- content/docs/reference/cli/status.mdx | 2 +- content/docs/reference/cli/wizard.mdx | 2 +- package.json | 2 +- .../{stash-help-0.16.0.txt => stash-help.txt} | 8 +- scripts/generate-cli-docs.ts | 73 +++++++++++++------ 14 files changed, 65 insertions(+), 40 deletions(-) rename scripts/fixtures/{stash-help-0.16.0.txt => stash-help.txt} (93%) diff --git a/content/docs/reference/cli/auth.mdx b/content/docs/reference/cli/auth.mdx index 338e854..5e1ec0a 100644 --- a/content/docs/reference/cli/auth.mdx +++ b/content/docs/reference/cli/auth.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/db.mdx b/content/docs/reference/cli/db.mdx index 82ffcd7..d5924d0 100644 --- a/content/docs/reference/cli/db.mdx +++ b/content/docs/reference/cli/db.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/encrypt.mdx b/content/docs/reference/cli/encrypt.mdx index 887983e..421feb5 100644 --- a/content/docs/reference/cli/encrypt.mdx +++ b/content/docs/reference/cli/encrypt.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/env.mdx b/content/docs/reference/cli/env.mdx index f30dc09..bcf8961 100644 --- a/content/docs/reference/cli/env.mdx +++ b/content/docs/reference/cli/env.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/impl.mdx b/content/docs/reference/cli/impl.mdx index 5604c50..9190888 100644 --- a/content/docs/reference/cli/impl.mdx +++ b/content/docs/reference/cli/impl.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/index.mdx b/content/docs/reference/cli/index.mdx index 8b4ab15..8ee4efa 100644 --- a/content/docs/reference/cli/index.mdx +++ b/content/docs/reference/cli/index.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/init.mdx b/content/docs/reference/cli/init.mdx index 0e843c3..57f4171 100644 --- a/content/docs/reference/cli/init.mdx +++ b/content/docs/reference/cli/init.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/plan.mdx b/content/docs/reference/cli/plan.mdx index 0c536df..70cbc0a 100644 --- a/content/docs/reference/cli/plan.mdx +++ b/content/docs/reference/cli/plan.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/schema.mdx b/content/docs/reference/cli/schema.mdx index bd8400c..11ae03c 100644 --- a/content/docs/reference/cli/schema.mdx +++ b/content/docs/reference/cli/schema.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/status.mdx b/content/docs/reference/cli/status.mdx index 8fc84b6..74e2834 100644 --- a/content/docs/reference/cli/status.mdx +++ b/content/docs/reference/cli/status.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/content/docs/reference/cli/wizard.mdx b/content/docs/reference/cli/wizard.mdx index e79f82d..081dc9e 100644 --- a/content/docs/reference/cli/wizard.mdx +++ b/content/docs/reference/cli/wizard.mdx @@ -7,7 +7,7 @@ verifiedAgainst: cli: "0.16.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Bump CLI_VERSION and re-run `bun run generate-docs:cli`. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. diff --git a/package.json b/package.json index 210cfd6..d3d8965 100644 --- a/package.json +++ b/package.json @@ -3,7 +3,7 @@ "version": "0.0.0", "private": true, "scripts": { - "prebuild": "bun run generate-docs && bun run generate-docs:eql && bun run validate-links && bun run validate-redirects", + "prebuild": "bun run generate-docs && bun run generate-docs:eql && bun run generate-docs:cli && bun run validate-links && bun run validate-redirects", "build": "next build", "dev": "next dev -p 3001", "start": "next start", diff --git a/scripts/fixtures/stash-help-0.16.0.txt b/scripts/fixtures/stash-help.txt similarity index 93% rename from scripts/fixtures/stash-help-0.16.0.txt rename to scripts/fixtures/stash-help.txt index ad4f17f..fa27406 100644 --- a/scripts/fixtures/stash-help-0.16.0.txt +++ b/scripts/fixtures/stash-help.txt @@ -1,7 +1,7 @@ -◇ injected env (0) from .env.local // tip: ⌘ multiple files { path: ['.env.local', '.env'] } -◇ injected env (0) from .env.development.local // tip: ⌘ override existing { override: true } -◇ injected env (0) from .env.development // tip: ⌘ enable debugging { debug: true } -◇ injected env (0) from .env // tip: ⌘ enable debugging { debug: true } +◇ injected env (0) from .env.local // tip: ⌁ auth for agents [www.vestauth.com] +◇ injected env (0) from .env.development.local // tip: ◈ secrets for agents [www.dotenvx.com] +◇ injected env (0) from .env.development // tip: ◈ encrypted .env [www.dotenvx.com] +◇ injected env (0) from .env // tip: ◈ encrypted .env [www.dotenvx.com] CipherStash CLI v0.16.0 Usage: bunx stash <command> [options] diff --git a/scripts/generate-cli-docs.ts b/scripts/generate-cli-docs.ts index 73e21bd..215cb45 100644 --- a/scripts/generate-cli-docs.ts +++ b/scripts/generate-cli-docs.ts @@ -22,23 +22,22 @@ * that is the point of this prototype. * * ── Versioning ──────────────────────────────────────────────────────────── - * CLI_VERSION is pinned. Bump it (Renovate can watch the `stash` npm dep), - * re-run `bun run generate-docs:cli`, commit the regenerated pages. Every page - * carries `verifiedAgainst.cli` and a visible banner, so readers and agents - * always know which version the docs describe. + * Always generated from the LATEST published `stash` on npm (resolved via + * `npm view stash version`), so a new release plus a run of this script — it + * runs in `prebuild` — refreshes the docs automatically. Every page carries + * `verifiedAgainst.cli` and a visible banner, so readers and agents always + * know which version the docs describe. Offline, it falls back to the cached + * `scripts/fixtures/stash-help.txt`. */ import { execSync } from "node:child_process"; import fs from "node:fs"; +import os from "node:os"; import path from "node:path"; const CLI_NAME = "stash"; -const CLI_VERSION = "0.16.0"; // pinned; bump + regenerate on each release +let CLI_VERSION = ""; // resolved to the latest published npm version at run time const RUNNER = "npx"; // normalized invocation shown in docs -const FIXTURE = path.join( - process.cwd(), - "scripts/fixtures", - `stash-help-${CLI_VERSION}.txt`, -); +const FIXTURE = path.join(process.cwd(), "scripts/fixtures", "stash-help.txt"); const OUT_DIR = path.join(process.cwd(), "content/docs/reference/cli"); // ── Types (this shape is the spec for `stash manifest --json`) ────────────── @@ -103,16 +102,40 @@ const componentsFor = (base: string): string[] => ["db", "schema", "encrypt"].includes(base) ? ["cli", "eql"] : ["cli"]; // ── Source ────────────────────────────────────────────────────────────────── -function loadHelp(): string { - if (fs.existsSync(FIXTURE)) return fs.readFileSync(FIXTURE, "utf8"); - // Bootstrap: fetch and cache. (Target: `stash manifest --json`.) - const out = execSync(`npx --yes ${CLI_NAME}@${CLI_VERSION} --help`, { - encoding: "utf8", - cwd: require("node:os").tmpdir(), - }); - fs.mkdirSync(path.dirname(FIXTURE), { recursive: true }); - fs.writeFileSync(FIXTURE, out); - return out; +// Resolve the latest published version so the docs track releases automatically. +function latestVersion(): string { + try { + return execSync(`npm view ${CLI_NAME} version`, { encoding: "utf8" }).trim(); + } catch { + const cached = fs.existsSync(FIXTURE) + ? fs.readFileSync(FIXTURE, "utf8").match(/CipherStash CLI v([0-9.]+)/)?.[1] + : undefined; + if (cached) { + console.warn(`⚠ npm unreachable; using cached stash v${cached}.`); + return cached; + } + throw new Error("Cannot resolve latest stash version (offline, no fixture)."); + } +} + +// Run the resolved CLI version and cache its help. (Target: `stash manifest --json`.) +function loadHelp(version: string): string { + try { + const out = execSync(`npx --yes ${CLI_NAME}@${version} --help`, { + encoding: "utf8", + cwd: os.tmpdir(), + stdio: ["ignore", "pipe", "ignore"], + }); + fs.mkdirSync(path.dirname(FIXTURE), { recursive: true }); + fs.writeFileSync(FIXTURE, out); + return out; + } catch { + if (fs.existsSync(FIXTURE)) { + console.warn(`⚠ Could not run stash@${version}; using cached fixture.`); + return fs.readFileSync(FIXTURE, "utf8"); + } + throw new Error(`Could not run stash@${version} and no cached fixture exists.`); + } } // Drop dotenvx's env-injection tips and blank leading noise. @@ -240,7 +263,8 @@ function parseFlagBlock(body: string[]): Flag[] { } // ── Render ─────────────────────────────────────────────────────────────────── -const GENERATED = `{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from \`${CLI_NAME} --help\` (v${CLI_VERSION}). Bump CLI_VERSION and re-run \`bun run generate-docs:cli\`. */}`; +const generatedMarker = (): string => + `{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from \`${CLI_NAME} --help\` (v${CLI_VERSION}). Re-run \`bun run generate-docs:cli\` to refresh from the latest published CLI. */}`; function banner(): string { return `<Callout type="info"> @@ -297,7 +321,7 @@ function renderPage(base: string, cmds: Command[]): { slug: string; body: string "---", ].join("\n"); - const parts = [frontmatter, "", GENERATED, "", banner(), ""]; + const parts = [frontmatter, "", generatedMarker(), "", banner(), ""]; if (isGroup) { parts.push( @@ -346,7 +370,7 @@ function renderIndex(manifest: Manifest, groups: Map<string, string[]>): string return `${frontmatter} -${GENERATED} +${generatedMarker()} ${banner()} @@ -369,10 +393,11 @@ function renderMeta(groups: Map<string, string[]>): string { // ── Main ───────────────────────────────────────────────────────────────────── function loadManifest(): Manifest { // Swap point: return JSON.parse(execSync(`npx ${CLI_NAME}@${CLI_VERSION} manifest --json`)). - return parseHelp(loadHelp()); + return parseHelp(loadHelp(CLI_VERSION)); } function main() { + CLI_VERSION = latestVersion(); const manifest = loadManifest(); // Group top-level commands by base, preserving discovery order. From 22dd036f218aff65d1157f40502d69dd13f7ae37 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Sun, 5 Jul 2026 23:30:08 +1000 Subject: [PATCH 22/38] =?UTF-8?q?docs(v2):=20hybrid=20CLI=20reference=20?= =?UTF-8?q?=E2=80=94=20merge=20hand-authored=20supplements=20+=20examples?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a supplement hook to the CLI generator: an optional scripts/cli-supplements/<command>.md is merged after the generated reference (synopsis + flags), so the rich per-command narrative and curated examples the thin `stash --help` can't provide can be authored now, without waiting on the CLI. Supplements live outside content/ so they're never treated as pages or wiped by the clean step. Demonstrated on `auth` (device-code flow + keyset binding, grounded in the CLI source). Content-model direction: per-command reference detail (examples, flag help) should migrate into the CLI itself (long-help + examples, GitHub-CLI/cobra style) and be generated from there; cross-command conceptual narrative (the profile / workspace model) belongs in a linked CLI concept page. The supplement hook is the interim bridge and the seam for both. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/cli/auth.mdx | 24 ++++++++++++++++++++++++ scripts/cli-supplements/auth.md | 23 +++++++++++++++++++++++ scripts/generate-cli-docs.ts | 17 ++++++++++++++++- 3 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 scripts/cli-supplements/auth.md diff --git a/content/docs/reference/cli/auth.mdx b/content/docs/reference/cli/auth.mdx index 5e1ec0a..4ae27d0 100644 --- a/content/docs/reference/cli/auth.mdx +++ b/content/docs/reference/cli/auth.mdx @@ -24,3 +24,27 @@ npx stash auth ```bash npx stash auth login ``` + +## How authentication works + +`stash auth login` runs the OAuth 2.0 **device authorization flow**: + +1. You pick a **region** for your CipherStash workspace. +2. The CLI opens your browser to a verification URL — and prints it, so it also + works over SSH or in a headless/agent environment — where you approve the + request. +3. Meanwhile the CLI polls CipherStash until you approve, then receives a + short-lived access token (it reports the token's expiry). +4. Your device is **bound to the workspace's default keyset**, so later commands + (`stash eql install`, `stash db push`, …) authenticate without a fresh login. + +The `--supabase` / `--drizzle` flags only tag the referrer for onboarding +analytics; they don't change the flow. + +> **Good to know**: login is device- and workspace-scoped. Authenticating from a +> new machine — or for a different workspace — re-runs the device flow. + +{/* TODO(verify with product): profiles, multiple workspaces, and switching +between them — where they're stored and how they're selected — belong here (or +in a linked CLI concept page). The CLI currently exposes only `auth login`; +confirm the profile / workspace-switching surface before documenting it. */} diff --git a/scripts/cli-supplements/auth.md b/scripts/cli-supplements/auth.md new file mode 100644 index 0000000..4d13348 --- /dev/null +++ b/scripts/cli-supplements/auth.md @@ -0,0 +1,23 @@ +## How authentication works + +`stash auth login` runs the OAuth 2.0 **device authorization flow**: + +1. You pick a **region** for your CipherStash workspace. +2. The CLI opens your browser to a verification URL — and prints it, so it also + works over SSH or in a headless/agent environment — where you approve the + request. +3. Meanwhile the CLI polls CipherStash until you approve, then receives a + short-lived access token (it reports the token's expiry). +4. Your device is **bound to the workspace's default keyset**, so later commands + (`stash eql install`, `stash db push`, …) authenticate without a fresh login. + +The `--supabase` / `--drizzle` flags only tag the referrer for onboarding +analytics; they don't change the flow. + +> **Good to know**: login is device- and workspace-scoped. Authenticating from a +> new machine — or for a different workspace — re-runs the device flow. + +{/* TODO(verify with product): profiles, multiple workspaces, and switching +between them — where they're stored and how they're selected — belong here (or +in a linked CLI concept page). The CLI currently exposes only `auth login`; +confirm the profile / workspace-switching surface before documenting it. */} diff --git a/scripts/generate-cli-docs.ts b/scripts/generate-cli-docs.ts index 215cb45..30a0427 100644 --- a/scripts/generate-cli-docs.ts +++ b/scripts/generate-cli-docs.ts @@ -39,6 +39,12 @@ let CLI_VERSION = ""; // resolved to the latest published npm version at run tim const RUNNER = "npx"; // normalized invocation shown in docs const FIXTURE = path.join(process.cwd(), "scripts/fixtures", "stash-help.txt"); const OUT_DIR = path.join(process.cwd(), "content/docs/reference/cli"); +// Hand-authored per-command prose merged into the generated page (hybrid model): +// the generated skeleton (synopsis + flags) stays drift-free; a supplement adds +// rich narrative + curated examples the thin `--help` can't provide. Lives +// outside content/ so it's never treated as a page or wiped by the clean step. +// Long-term, migrate these into the CLI's own long-help/examples (see the PR). +const SUPPLEMENTS_DIR = path.join(process.cwd(), "scripts/cli-supplements"); // ── Types (this shape is the spec for `stash manifest --json`) ────────────── interface Flag { @@ -336,7 +342,16 @@ function renderPage(base: string, cmds: Command[]): { slug: string; body: string if (c.examples.length) parts.push("\n## Examples\n", "```bash", c.examples.join("\n"), "```"); } - return { slug: base, body: `${parts.join("\n").trimEnd()}\n` }; + const supplement = readSupplement(base); + const body = `${parts.join("\n").trimEnd()}${supplement ? `\n\n${supplement}` : ""}\n`; + return { slug: base, body }; +} + +// Optional hand-authored prose for a command, merged after its generated +// reference. Returns "" when no supplement exists. +function readSupplement(slug: string): string { + const file = path.join(SUPPLEMENTS_DIR, `${slug}.md`); + return fs.existsSync(file) ? fs.readFileSync(file, "utf8").trim() : ""; } function renderIndex(manifest: Manifest, groups: Map<string, string[]>): string { From 6a7ac7b817fc561a3ed3753d25028de1ca718da8 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 16:33:39 +1000 Subject: [PATCH 23/38] docs(cli): generate the CLI reference from `stash manifest --json` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit stash CLI 0.17 ships the `manifest` command (structured, versioned command surface), so swap the generator off `--help` scraping onto `stash manifest --json` and delete the text parser — the realization of this PR's stated target. - loadManifest() now runs `npx stash@<version> manifest --json` and projects it onto the renderer's model; nav groups + order come straight from the CLI. - Per-command flags carry their `default` and `env` (folded into the flag description); examples come from the manifest. Pipes in flag values (`--eql-version <2|3>`) are escaped so the GFM table renders. - Fixture swapped from stash-help.txt to stash-manifest.json. - Regenerated against 0.17.0: new `eql` group (eql install/upgrade/status), new `doctor` + `manifest` pages, `db` no longer lists install/upgrade. types:check passes; generator lints clean (bar pre-existing noNonNullAssertion style warnings shared with the original). --- content/docs/reference/cli/auth.mdx | 52 ++- content/docs/reference/cli/db.mdx | 101 +---- content/docs/reference/cli/doctor.mdx | 20 + content/docs/reference/cli/encrypt.mdx | 44 +- content/docs/reference/cli/env.mdx | 14 +- content/docs/reference/cli/eql.mdx | 77 ++++ content/docs/reference/cli/impl.mdx | 10 +- content/docs/reference/cli/index.mdx | 30 +- content/docs/reference/cli/init.mdx | 18 +- content/docs/reference/cli/manifest.mdx | 34 ++ content/docs/reference/cli/meta.json | 6 +- content/docs/reference/cli/plan.mdx | 11 +- content/docs/reference/cli/schema.mdx | 17 +- content/docs/reference/cli/status.mdx | 15 +- content/docs/reference/cli/wizard.mdx | 12 +- scripts/fixtures/stash-help.txt | 98 ----- scripts/fixtures/stash-manifest.json | 511 ++++++++++++++++++++++++ scripts/generate-cli-docs.ts | 370 ++++++++--------- 18 files changed, 967 insertions(+), 473 deletions(-) create mode 100644 content/docs/reference/cli/doctor.mdx create mode 100644 content/docs/reference/cli/eql.mdx create mode 100644 content/docs/reference/cli/manifest.mdx delete mode 100644 scripts/fixtures/stash-help.txt create mode 100644 scripts/fixtures/stash-manifest.json diff --git a/content/docs/reference/cli/auth.mdx b/content/docs/reference/cli/auth.mdx index 4ae27d0..09f0a18 100644 --- a/content/docs/reference/cli/auth.mdx +++ b/content/docs/reference/cli/auth.mdx @@ -1,28 +1,68 @@ --- title: stash auth -description: "Authenticate with CipherStash" +description: "Reference for the `stash auth` commands." type: reference components: [cli] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> +The `stash auth` command group. + +### `auth login` + Authenticate with CipherStash ```bash -npx stash auth +npx stash auth login [flags] ``` -## Examples +#### Flags + +| Flag | Description | +| --- | --- | +| `--region <slug>` | Region to authenticate against (e.g. us-east-1). Skips the interactive region picker. (env: `STASH_REGION`) | +| `--json` | Emit newline-delimited JSON events instead of prose. The first event (authorization_required) carries the device verification URL for a human to open. Implies no prompt and no browser auto-open. | +| `--no-open` | Don't auto-open the verification URL in a browser (already implied by --json). | +| `--supabase` | Track Supabase as the referrer. | +| `--drizzle` | Track Drizzle as the referrer. | + + +#### Examples ```bash npx stash auth login +npx stash auth login --region us-east-1 +npx stash auth login --supabase +npx stash auth login --region us-east-1 --json +``` + +### `auth regions` + +List the regions you can authenticate against + +```bash +npx stash auth regions [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--json` | Emit machine-readable [{ slug, label }] instead of a text list. | + + +#### Examples + +```bash +npx stash auth regions +npx stash auth regions --json ``` ## How authentication works diff --git a/content/docs/reference/cli/db.mdx b/content/docs/reference/cli/db.mdx index d5924d0..e0c26b2 100644 --- a/content/docs/reference/cli/db.mdx +++ b/content/docs/reference/cli/db.mdx @@ -4,66 +4,17 @@ description: "Reference for the `stash db` commands." type: reference components: [cli, eql] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> The `stash db` command group. -### `db install` - -Scaffold stash.config.ts (if missing) and install EQL extensions - -```bash -npx stash db install [flags] -``` - -#### Flags - -| Flag | Description | -| --- | --- | -| `--force` | Reinstall / overwrite even if already installed | -| `--dry-run` | Show what would happen without making changes | -| `--supabase` | Use Supabase-compatible mode (auto-detected from DATABASE_URL) | -| `--drizzle` | Generate a Drizzle migration instead of direct install (auto-detected from project) | -| `--migration` | (requires --supabase) Write a Supabase migration file instead of running SQL directly | -| `--direct` | (requires --supabase) Run the SQL directly against the database (mutually exclusive with --migration) | -| `--migrations-dir <path>` | (requires --supabase) Override the Supabase migrations directory (default: supabase/migrations) | -| `--exclude-operator-family` | Skip operator family creation | -| `--latest` | Fetch the latest EQL from GitHub | -| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | - - -#### Examples - -```bash -npx stash db install -``` - -### `db upgrade` - -Upgrade EQL extensions to the latest version - -```bash -npx stash db upgrade [flags] -``` - -#### Flags - -| Flag | Description | -| --- | --- | -| `--dry-run` | Show what would happen without making changes | -| `--supabase` | Use Supabase-compatible mode (auto-detected from DATABASE_URL) | -| `--exclude-operator-family` | Skip operator family creation | -| `--latest` | Fetch the latest EQL from GitHub | -| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | - - ### `db push` Push encryption schema (writes pending if active config already exists) @@ -76,15 +27,9 @@ npx stash db push [flags] | Flag | Description | | --- | --- | -| `--dry-run` | Show what would happen without making changes | -| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | - +| `--dry-run` | Show what would happen without making changes. | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk. (env: `DATABASE_URL`) | -#### Examples - -```bash -npx stash db push -``` ### `db activate` @@ -98,7 +43,7 @@ npx stash db activate [flags] | Flag | Description | | --- | --- | -| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk. (env: `DATABASE_URL`) | ### `db validate` @@ -113,41 +58,19 @@ npx stash db validate [flags] | Flag | Description | | --- | --- | -| `--supabase` | Use Supabase-compatible mode (auto-detected from DATABASE_URL) | -| `--exclude-operator-family` | Skip operator family creation | -| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | +| `--supabase` | Use Supabase-compatible mode. | +| `--exclude-operator-family` | Skip operator family creation. | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk. (env: `DATABASE_URL`) | ### `db migrate` -Run pending encrypt config migrations +Run pending encrypt config migrations (not yet implemented) ```bash -npx stash db migrate [flags] +npx stash db migrate ``` -#### Flags - -| Flag | Description | -| --- | --- | -| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | - - -### `db status` - -Show EQL installation status - -```bash -npx stash db status [flags] -``` - -#### Flags - -| Flag | Description | -| --- | --- | -| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | - - ### `db test-connection` Test database connectivity @@ -160,4 +83,4 @@ npx stash db test-connection [flags] | Flag | Description | | --- | --- | -| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk. (env: `DATABASE_URL`) | diff --git a/content/docs/reference/cli/doctor.mdx b/content/docs/reference/cli/doctor.mdx new file mode 100644 index 0000000..1c6efb8 --- /dev/null +++ b/content/docs/reference/cli/doctor.mdx @@ -0,0 +1,20 @@ +--- +title: stash doctor +description: "Diagnose install problems (native binaries, runtime)" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.17.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} + +<Callout type="info"> +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. +</Callout> + +Diagnose install problems (native binaries, runtime) + +```bash +npx stash doctor +``` diff --git a/content/docs/reference/cli/encrypt.mdx b/content/docs/reference/cli/encrypt.mdx index 421feb5..9c03ae8 100644 --- a/content/docs/reference/cli/encrypt.mdx +++ b/content/docs/reference/cli/encrypt.mdx @@ -4,13 +4,13 @@ description: "Reference for the `stash encrypt` commands." type: reference components: [cli, eql] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> The `stash encrypt` command group. @@ -36,21 +36,53 @@ npx stash encrypt plan Resumably encrypt plaintext into the encrypted column ```bash -npx stash encrypt backfill +npx stash encrypt backfill [flags] ``` +#### Flags + +| Flag | Description | +| --- | --- | +| `--table <name>` | Target table. | +| `--column <name>` | Target column. | +| `--pk-column <name>` | Primary-key column used to page through rows. | +| `--chunk-size <n>` | Rows encrypted per batch. | +| `--encrypted-column <name>` | Destination encrypted column. | +| `--schema-column-key <key>` | Schema key identifying the column config. | +| `--confirm-dual-writes-deployed` | Assert the app is dual-writing before backfilling (safety gate). | +| `--force` | Proceed past non-fatal safety checks. | + + ### `encrypt cutover` Rename swap encrypted → primary column ```bash -npx stash encrypt cutover +npx stash encrypt cutover [flags] ``` +#### Flags + +| Flag | Description | +| --- | --- | +| `--table <name>` | Target table. | +| `--column <name>` | Target column. | +| `--proxy-url <url>` | Proxy URL to verify against. | +| `--migrations-dir <path>` | Directory to write the rename migration into. | + + ### `encrypt drop` Generate a migration to drop the plaintext column ```bash -npx stash encrypt drop +npx stash encrypt drop [flags] ``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--table <name>` | Target table. | +| `--column <name>` | Target column. | +| `--migrations-dir <path>` | Directory to write the drop migration into. | diff --git a/content/docs/reference/cli/env.mdx b/content/docs/reference/cli/env.mdx index bcf8961..fe277a9 100644 --- a/content/docs/reference/cli/env.mdx +++ b/content/docs/reference/cli/env.mdx @@ -4,17 +4,23 @@ description: "(experimental) Print production env vars for deployment" type: reference components: [cli] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> (experimental) Print production env vars for deployment ```bash -npx stash env +npx stash env [flags] ``` + +### Flags + +| Flag | Description | +| --- | --- | +| `--write` | Write the vars to a file instead of printing them. | diff --git a/content/docs/reference/cli/eql.mdx b/content/docs/reference/cli/eql.mdx new file mode 100644 index 0000000..63bf8be --- /dev/null +++ b/content/docs/reference/cli/eql.mdx @@ -0,0 +1,77 @@ +--- +title: stash eql +description: "Reference for the `stash eql` commands." +type: reference +components: [cli, eql] +verifiedAgainst: + cli: "0.17.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} + +<Callout type="info"> +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. +</Callout> + +The `stash eql` command group. + +### `eql install` + +Scaffold stash.config.ts (if missing) and install EQL extensions + +```bash +npx stash eql install [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--force` | Reinstall / overwrite even if already installed. | +| `--dry-run` | Show what would happen without making changes. | +| `--supabase` | Use Supabase-compatible mode (auto-detected from DATABASE_URL). | +| `--drizzle` | Generate a Drizzle migration instead of direct install (auto-detected from project). | +| `--migration` | Write a Supabase migration file instead of running SQL directly (requires --supabase). | +| `--direct` | Run the SQL directly against the database (requires --supabase; mutually exclusive with --migration). | +| `--migrations-dir <path>` | Override the Supabase migrations directory (requires --supabase). (default: `supabase/migrations`) | +| `--exclude-operator-family` | Skip operator family creation. | +| `--eql-version <2\|3>` | EQL generation to target. v3 is the native eql_v3.* domain schema (direct install only for now). (default: `2`) | +| `--latest` | Fetch the latest EQL from GitHub (v2 only). | +| `--name <name>` | With --drizzle: name for the generated migration (defaults to a scaffold name). | +| `--out <path>` | With --drizzle: directory to write the generated migration into. | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk. (env: `DATABASE_URL`) | + + +### `eql upgrade` + +Upgrade EQL extensions to the latest version + +```bash +npx stash eql upgrade [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--dry-run` | Show what would happen without making changes. | +| `--supabase` | Use Supabase-compatible mode. | +| `--exclude-operator-family` | Skip operator family creation. | +| `--eql-version <2\|3>` | EQL generation to target. (default: `2`) | +| `--latest` | Fetch the latest EQL from GitHub (v2 only). | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk. (env: `DATABASE_URL`) | + + +### `eql status` + +Show EQL installation status + +```bash +npx stash eql status [flags] +``` + +#### Flags + +| Flag | Description | +| --- | --- | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk. (env: `DATABASE_URL`) | diff --git a/content/docs/reference/cli/impl.mdx b/content/docs/reference/cli/impl.mdx index 9190888..b3ce114 100644 --- a/content/docs/reference/cli/impl.mdx +++ b/content/docs/reference/cli/impl.mdx @@ -4,13 +4,13 @@ description: "Execute the plan with a local agent" type: reference components: [cli] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> Execute the plan with a local agent @@ -23,8 +23,8 @@ npx stash impl [flags] | Flag | Description | | --- | --- | -| `--continue-without-plan` | Skip planning and go straight to implementation (interactively confirms before proceeding) | -| `--target <name>` | Skip the agent-target picker and hand off directly to one of claude-code \| codex \| agents-md \| wizard. Safe to call from non-TTY contexts (CI, pipes). Without --target in non-TTY, the command prints a hint and exits cleanly instead of hanging. | +| `--continue-without-plan` | Skip planning and go straight to implementation (interactively confirms before proceeding). | +| `--target <name>` | Skip the agent-target picker and hand off directly to one of claude-code \| codex \| agents-md \| wizard. Safe in non-TTY contexts. | ## Examples diff --git a/content/docs/reference/cli/index.mdx b/content/docs/reference/cli/index.mdx index 8ee4efa..88764db 100644 --- a/content/docs/reference/cli/index.mdx +++ b/content/docs/reference/cli/index.mdx @@ -1,19 +1,19 @@ --- title: CLI -description: "Command reference for the stash CLI, generated from v0.16.0." +description: "Command reference for the stash CLI, generated from v0.17.0." type: reference components: [cli] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> -The `stash` CLI. Install with `npx stash@0.16.0`. Every command accepts `--help` and `--version`. +The `stash` CLI. Install with `npx stash@0.17.0`. Every command accepts `--help` and `--version`. ### Setup & workflow @@ -24,24 +24,32 @@ The `stash` CLI. Install with `npx stash@0.16.0`. Every command accepts `--help` | [`impl`](/reference/cli/impl) | Execute the plan with a local agent | | [`status`](/reference/cli/status) | Displays implementation status | | [`wizard`](/reference/cli/wizard) | AI-guided encryption setup (reads your codebase) | +| [`doctor`](/reference/cli/doctor) | Diagnose install problems (native binaries, runtime) | +| [`manifest`](/reference/cli/manifest) | Print the structured, versioned command surface | ### Auth | Command | Description | | --- | --- | -| [`auth`](/reference/cli/auth) | Authenticate with CipherStash | +| [`auth login`](/reference/cli/auth#auth-login) | Authenticate with CipherStash | +| [`auth regions`](/reference/cli/auth#auth-regions) | List the regions you can authenticate against | + +### EQL + +| Command | Description | +| --- | --- | +| [`eql install`](/reference/cli/eql#eql-install) | Scaffold stash.config.ts (if missing) and install EQL extensions | +| [`eql upgrade`](/reference/cli/eql#eql-upgrade) | Upgrade EQL extensions to the latest version | +| [`eql status`](/reference/cli/eql#eql-status) | Show EQL installation status | ### Database | Command | Description | | --- | --- | -| [`db install`](/reference/cli/db#db-install) | Scaffold stash.config.ts (if missing) and install EQL extensions | -| [`db upgrade`](/reference/cli/db#db-upgrade) | Upgrade EQL extensions to the latest version | | [`db push`](/reference/cli/db#db-push) | Push encryption schema (writes pending if active config already exists) | | [`db activate`](/reference/cli/db#db-activate) | Promote pending → active without renames (use after additive db push) | | [`db validate`](/reference/cli/db#db-validate) | Validate encryption schema | -| [`db migrate`](/reference/cli/db#db-migrate) | Run pending encrypt config migrations | -| [`db status`](/reference/cli/db#db-status) | Show EQL installation status | +| [`db migrate`](/reference/cli/db#db-migrate) | Run pending encrypt config migrations (not yet implemented) | | [`db test-connection`](/reference/cli/db#db-test-connection) | Test database connectivity | ### Schema @@ -60,7 +68,7 @@ The `stash` CLI. Install with `npx stash@0.16.0`. Every command accepts `--help` | [`encrypt cutover`](/reference/cli/encrypt#encrypt-cutover) | Rename swap encrypted → primary column | | [`encrypt drop`](/reference/cli/encrypt#encrypt-drop) | Generate a migration to drop the plaintext column | -### Deployment +### Experimental | Command | Description | | --- | --- | diff --git a/content/docs/reference/cli/init.mdx b/content/docs/reference/cli/init.mdx index 57f4171..518544b 100644 --- a/content/docs/reference/cli/init.mdx +++ b/content/docs/reference/cli/init.mdx @@ -4,13 +4,13 @@ description: "Initialize CipherStash for your project" type: reference components: [cli] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> Initialize CipherStash for your project @@ -23,11 +23,12 @@ npx stash init [flags] | Flag | Description | | --- | --- | -| `--supabase` | Use Supabase-specific setup flow | -| `--drizzle` | Use Drizzle-specific setup flow | -| `--prisma-next` | Use Prisma Next-specific setup flow (EQL bundle installed via prisma-next migration apply) | -| `--proxy` | Query encrypted data via CipherStash Proxy | -| `--no-proxy` | Query encrypted data directly via the SDK (default) | +| `--supabase` | Use Supabase-specific setup flow. | +| `--drizzle` | Use Drizzle-specific setup flow. | +| `--prisma-next` | Use Prisma Next-specific setup flow (EQL bundle installed via prisma-next migration apply). | +| `--proxy` | Query encrypted data via CipherStash Proxy. | +| `--no-proxy` | Query encrypted data directly via the SDK. (default: `true`) | +| `--region <slug>` | Region to authenticate against (e.g. us-east-1). Skips the interactive region picker. Required for non-interactive init when not already logged in. (env: `STASH_REGION`) | ## Examples @@ -36,4 +37,5 @@ npx stash init [flags] npx stash init npx stash init --supabase npx stash init --prisma-next +npx stash init --region us-east-1 ``` diff --git a/content/docs/reference/cli/manifest.mdx b/content/docs/reference/cli/manifest.mdx new file mode 100644 index 0000000..4c7578d --- /dev/null +++ b/content/docs/reference/cli/manifest.mdx @@ -0,0 +1,34 @@ +--- +title: stash manifest +description: "Print the structured, versioned command surface" +type: reference +components: [cli] +verifiedAgainst: + cli: "0.17.0" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} + +<Callout type="info"> +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. +</Callout> + +Print the structured, versioned command surface + +```bash +npx stash manifest [flags] +``` + +### Flags + +| Flag | Description | +| --- | --- | +| `--json` | Emit the structured JSON manifest instead of a text list. | + + +## Examples + +```bash +npx stash manifest --json +npx stash manifest +``` diff --git a/content/docs/reference/cli/meta.json b/content/docs/reference/cli/meta.json index 5e139d7..6b90890 100644 --- a/content/docs/reference/cli/meta.json +++ b/content/docs/reference/cli/meta.json @@ -7,15 +7,19 @@ "impl", "status", "wizard", + "doctor", + "manifest", "---Auth---", "auth", + "---EQL---", + "eql", "---Database---", "db", "---Schema---", "schema", "---Encrypt---", "encrypt", - "---Deployment---", + "---Experimental---", "env" ] } diff --git a/content/docs/reference/cli/plan.mdx b/content/docs/reference/cli/plan.mdx index 70cbc0a..4904f7a 100644 --- a/content/docs/reference/cli/plan.mdx +++ b/content/docs/reference/cli/plan.mdx @@ -4,13 +4,13 @@ description: "Draft a reviewable encryption plan at .cipherstash/plan.md" type: reference components: [cli] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> Draft a reviewable encryption plan at .cipherstash/plan.md @@ -23,12 +23,13 @@ npx stash plan [flags] | Flag | Description | | --- | --- | -| `--complete-rollout` | Plan the entire encryption lifecycle (schema-add through drop) in one document. Skips the production-deploy gate that normally separates rollout from cutover. Only safe when this database is not backing a deployed application (local dev, sandbox, freshly seeded test environment). | -| `--target <name>` | Skip the agent-target picker and hand off directly to one of claude-code \| codex \| agents-md \| wizard. Safe to call from non-TTY contexts (CI, pipes). Without --target in non-TTY, the command prints a hint and exits cleanly instead of hanging. | +| `--complete-rollout` | Plan the entire encryption lifecycle (schema-add through drop) in one document. Skips the production-deploy gate; only safe when this database is not backing a deployed application. | +| `--target <name>` | Skip the agent-target picker and hand off directly to one of claude-code \| codex \| agents-md \| wizard. Safe in non-TTY contexts. | ## Examples ```bash npx stash plan +npx stash plan --target claude-code ``` diff --git a/content/docs/reference/cli/schema.mdx b/content/docs/reference/cli/schema.mdx index 11ae03c..9b5e360 100644 --- a/content/docs/reference/cli/schema.mdx +++ b/content/docs/reference/cli/schema.mdx @@ -4,13 +4,13 @@ description: "Reference for the `stash schema` commands." type: reference components: [cli, eql] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> The `stash schema` command group. @@ -20,11 +20,12 @@ The `stash schema` command group. Build an encryption schema from your database ```bash -npx stash schema build +npx stash schema build [flags] ``` -#### Examples +#### Flags -```bash -npx stash schema build -``` +| Flag | Description | +| --- | --- | +| `--supabase` | Use Supabase-compatible mode. | +| `--database-url <url>` | Override DATABASE_URL for this run only — never written to disk. (env: `DATABASE_URL`) | diff --git a/content/docs/reference/cli/status.mdx b/content/docs/reference/cli/status.mdx index 74e2834..ba033d7 100644 --- a/content/docs/reference/cli/status.mdx +++ b/content/docs/reference/cli/status.mdx @@ -4,13 +4,13 @@ description: "Displays implementation status" type: reference components: [cli] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> Displays implementation status @@ -23,13 +23,6 @@ npx stash status [flags] | Flag | Description | | --- | --- | -| `--quest` | Force the quest-log output (emoji + progress bars) even in non-TTY contexts. Default is auto: fancy in a terminal, plain in CI / pipes / agents. | +| `--quest` | Force the quest-log output (emoji + progress bars) even in non-TTY contexts. | | `--plain` | Force the plain-text output even in TTY contexts. | | `--json` | Emit a structured JSON document instead. | - - -## Examples - -```bash -npx stash status -``` diff --git a/content/docs/reference/cli/wizard.mdx b/content/docs/reference/cli/wizard.mdx index 081dc9e..05678ed 100644 --- a/content/docs/reference/cli/wizard.mdx +++ b/content/docs/reference/cli/wizard.mdx @@ -4,13 +4,13 @@ description: "AI-guided encryption setup (reads your codebase)" type: reference components: [cli] verifiedAgainst: - cli: "0.16.0" + cli: "0.17.0" --- -{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash --help` (v0.16.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} +{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from `stash manifest --json` (v0.17.0). Re-run `bun run generate-docs:cli` to refresh from the latest published CLI. */} <Callout type="info"> -Generated from **`stash` v0.16.0**. Run `npx stash@0.16.0 --help` to see the live command surface. +Generated from **`stash` v0.17.0** via `npx stash@0.17.0 manifest --json`. Run `npx stash@0.17.0 --help` to see the live command surface. </Callout> AI-guided encryption setup (reads your codebase) @@ -18,9 +18,3 @@ AI-guided encryption setup (reads your codebase) ```bash npx stash wizard ``` - -## Examples - -```bash -npx stash wizard -``` diff --git a/scripts/fixtures/stash-help.txt b/scripts/fixtures/stash-help.txt deleted file mode 100644 index fa27406..0000000 --- a/scripts/fixtures/stash-help.txt +++ /dev/null @@ -1,98 +0,0 @@ -◇ injected env (0) from .env.local // tip: ⌁ auth for agents [www.vestauth.com] -◇ injected env (0) from .env.development.local // tip: ◈ secrets for agents [www.dotenvx.com] -◇ injected env (0) from .env.development // tip: ◈ encrypted .env [www.dotenvx.com] -◇ injected env (0) from .env // tip: ◈ encrypted .env [www.dotenvx.com] -CipherStash CLI v0.16.0 - -Usage: bunx stash <command> [options] - -Commands: - init Initialize CipherStash for your project - plan Draft a reviewable encryption plan at .cipherstash/plan.md - impl Execute the plan with a local agent - status Displays implementation status - auth <subcommand> Authenticate with CipherStash - wizard AI-guided encryption setup (reads your codebase) - - db install Scaffold stash.config.ts (if missing) and install EQL extensions - db upgrade Upgrade EQL extensions to the latest version - db push Push encryption schema (writes pending if active config already exists) - db activate Promote pending → active without renames (use after additive db push) - db validate Validate encryption schema - db migrate Run pending encrypt config migrations - db status Show EQL installation status - db test-connection Test database connectivity - - schema build Build an encryption schema from your database - - encrypt status Show per-column migration status (phase, progress, drift) - encrypt plan Diff intent (.cipherstash/migrations.json) vs observed state - encrypt backfill Resumably encrypt plaintext into the encrypted column - encrypt cutover Rename swap encrypted → primary column - encrypt drop Generate a migration to drop the plaintext column - - env (experimental) Print production env vars for deployment - -Options: - --help, -h Show help - --version, -v Show version - -Init Flags: - --supabase Use Supabase-specific setup flow - --drizzle Use Drizzle-specific setup flow - --prisma-next Use Prisma Next-specific setup flow (EQL bundle installed via prisma-next migration apply) - --proxy Query encrypted data via CipherStash Proxy - --no-proxy Query encrypted data directly via the SDK (default) - -Plan Flags: - --complete-rollout Plan the entire encryption lifecycle (schema-add through drop) - in one document. Skips the production-deploy gate that - normally separates rollout from cutover. Only safe when this - database is not backing a deployed application (local dev, - sandbox, freshly seeded test environment). - --target <name> Skip the agent-target picker and hand off directly to one of - claude-code | codex | agents-md | wizard. Safe to call from - non-TTY contexts (CI, pipes). Without --target in non-TTY, - the command prints a hint and exits cleanly instead of hanging. - -Status Flags: - --quest Force the quest-log output (emoji + progress bars) - even in non-TTY contexts. Default is auto: fancy - in a terminal, plain in CI / pipes / agents. - --plain Force the plain-text output even in TTY contexts. - --json Emit a structured JSON document instead. - -Impl Flags: - --continue-without-plan Skip planning and go straight to implementation - (interactively confirms before proceeding) - --target <name> Skip the agent-target picker and hand off directly to one of - claude-code | codex | agents-md | wizard. Safe to call from - non-TTY contexts (CI, pipes). Without --target in non-TTY, - the command prints a hint and exits cleanly instead of hanging. - -DB Flags: - --force (install) Reinstall / overwrite even if already installed - --dry-run (install, push, upgrade) Show what would happen without making changes - --supabase (install, upgrade, validate) Use Supabase-compatible mode (auto-detected from DATABASE_URL) - --drizzle (install) Generate a Drizzle migration instead of direct install (auto-detected from project) - --migration (install, requires --supabase) Write a Supabase migration file instead of running SQL directly - --direct (install, requires --supabase) Run the SQL directly against the database (mutually exclusive with --migration) - --migrations-dir <path> (install, requires --supabase) Override the Supabase migrations directory (default: supabase/migrations) - --exclude-operator-family (install, upgrade, validate) Skip operator family creation - --latest (install, upgrade) Fetch the latest EQL from GitHub - --database-url <url> (all db / schema commands) Override DATABASE_URL for this run only — never written to disk - -Examples: - bunx stash init - bunx stash init --supabase - bunx stash init --prisma-next - bunx stash plan - bunx stash impl - bunx stash impl --continue-without-plan - bunx stash impl --target claude-code - bunx stash status - bunx stash auth login - bunx stash wizard - bunx stash db install - bunx stash db push - bunx stash schema build diff --git a/scripts/fixtures/stash-manifest.json b/scripts/fixtures/stash-manifest.json new file mode 100644 index 0000000..0de5ec4 --- /dev/null +++ b/scripts/fixtures/stash-manifest.json @@ -0,0 +1,511 @@ +{ + "name": "stash", + "version": "0.17.0", + "groups": [ + { + "title": "Setup & workflow", + "commands": [ + { + "name": "init", + "summary": "Initialize CipherStash for your project", + "long": "Set up CipherStash end-to-end: authenticate, introspect your database,\ninstall dependencies, install EQL, and hand off the rest to your local\ncoding agent. Every prompt has a non-interactive escape hatch, so init\nnever blocks waiting on a TTY (CI, agents, pipes).", + "examples": [ + "init", + "init --supabase", + "init --prisma-next", + "init --region us-east-1" + ], + "flags": [ + { + "name": "--supabase", + "description": "Use Supabase-specific setup flow." + }, + { + "name": "--drizzle", + "description": "Use Drizzle-specific setup flow." + }, + { + "name": "--prisma-next", + "description": "Use Prisma Next-specific setup flow (EQL bundle installed via prisma-next migration apply)." + }, + { + "name": "--proxy", + "description": "Query encrypted data via CipherStash Proxy." + }, + { + "name": "--no-proxy", + "description": "Query encrypted data directly via the SDK.", + "default": "true" + }, + { + "name": "--region", + "value": "<slug>", + "description": "Region to authenticate against (e.g. us-east-1). Skips the interactive region picker. Required for non-interactive init when not already logged in.", + "env": "STASH_REGION" + } + ] + }, + { + "name": "plan", + "summary": "Draft a reviewable encryption plan at .cipherstash/plan.md", + "examples": [ + "plan", + "plan --target claude-code" + ], + "flags": [ + { + "name": "--complete-rollout", + "description": "Plan the entire encryption lifecycle (schema-add through drop) in one document. Skips the production-deploy gate; only safe when this database is not backing a deployed application." + }, + { + "name": "--target", + "value": "<name>", + "description": "Skip the agent-target picker and hand off directly to one of claude-code | codex | agents-md | wizard. Safe in non-TTY contexts." + } + ] + }, + { + "name": "impl", + "summary": "Execute the plan with a local agent", + "examples": [ + "impl", + "impl --continue-without-plan", + "impl --target claude-code" + ], + "flags": [ + { + "name": "--continue-without-plan", + "description": "Skip planning and go straight to implementation (interactively confirms before proceeding)." + }, + { + "name": "--target", + "value": "<name>", + "description": "Skip the agent-target picker and hand off directly to one of claude-code | codex | agents-md | wizard. Safe in non-TTY contexts." + } + ] + }, + { + "name": "status", + "summary": "Displays implementation status", + "flags": [ + { + "name": "--quest", + "description": "Force the quest-log output (emoji + progress bars) even in non-TTY contexts." + }, + { + "name": "--plain", + "description": "Force the plain-text output even in TTY contexts." + }, + { + "name": "--json", + "description": "Emit a structured JSON document instead." + } + ] + }, + { + "name": "wizard", + "summary": "AI-guided encryption setup (reads your codebase)" + }, + { + "name": "doctor", + "summary": "Diagnose install problems (native binaries, runtime)" + }, + { + "name": "manifest", + "summary": "Print the structured, versioned command surface", + "long": "Emit the CLI command surface as data. `--json` produces the machine-\nreadable manifest the docs generator and agents consume; without it a\ngrouped command list is printed. The manifest is stamped with the CLI\nversion, so a page generated from it always names the version it describes.", + "examples": [ + "manifest --json", + "manifest" + ], + "flags": [ + { + "name": "--json", + "description": "Emit the structured JSON manifest instead of a text list." + } + ] + } + ] + }, + { + "title": "Auth", + "commands": [ + { + "name": "auth login", + "summary": "Authenticate with CipherStash", + "long": "Runs the OAuth 2.0 device authorization flow:\n1. Pick a region for your workspace.\n2. Approve in the browser — the URL is printed, so it works over SSH/headless.\n3. The CLI polls until you approve, then stores a short-lived token.\n4. Your device is bound to the workspace's default keyset, so later\n commands authenticate without a fresh login.", + "examples": [ + "auth login", + "auth login --region us-east-1", + "auth login --supabase", + "auth login --region us-east-1 --json" + ], + "flags": [ + { + "name": "--region", + "value": "<slug>", + "description": "Region to authenticate against (e.g. us-east-1). Skips the interactive region picker.", + "env": "STASH_REGION" + }, + { + "name": "--json", + "description": "Emit newline-delimited JSON events instead of prose. The first event (authorization_required) carries the device verification URL for a human to open. Implies no prompt and no browser auto-open." + }, + { + "name": "--no-open", + "description": "Don't auto-open the verification URL in a browser (already implied by --json)." + }, + { + "name": "--supabase", + "description": "Track Supabase as the referrer." + }, + { + "name": "--drizzle", + "description": "Track Drizzle as the referrer." + } + ] + }, + { + "name": "auth regions", + "summary": "List the regions you can authenticate against", + "examples": [ + "auth regions", + "auth regions --json" + ], + "flags": [ + { + "name": "--json", + "description": "Emit machine-readable [{ slug, label }] instead of a text list." + } + ] + } + ] + }, + { + "title": "EQL", + "commands": [ + { + "name": "eql install", + "summary": "Scaffold stash.config.ts (if missing) and install EQL extensions", + "flags": [ + { + "name": "--force", + "description": "Reinstall / overwrite even if already installed." + }, + { + "name": "--dry-run", + "description": "Show what would happen without making changes." + }, + { + "name": "--supabase", + "description": "Use Supabase-compatible mode (auto-detected from DATABASE_URL)." + }, + { + "name": "--drizzle", + "description": "Generate a Drizzle migration instead of direct install (auto-detected from project)." + }, + { + "name": "--migration", + "description": "Write a Supabase migration file instead of running SQL directly (requires --supabase)." + }, + { + "name": "--direct", + "description": "Run the SQL directly against the database (requires --supabase; mutually exclusive with --migration)." + }, + { + "name": "--migrations-dir", + "value": "<path>", + "description": "Override the Supabase migrations directory (requires --supabase).", + "default": "supabase/migrations" + }, + { + "name": "--exclude-operator-family", + "description": "Skip operator family creation." + }, + { + "name": "--eql-version", + "value": "<2|3>", + "description": "EQL generation to target. v3 is the native eql_v3.* domain schema (direct install only for now).", + "default": "2" + }, + { + "name": "--latest", + "description": "Fetch the latest EQL from GitHub (v2 only)." + }, + { + "name": "--name", + "value": "<name>", + "description": "With --drizzle: name for the generated migration (defaults to a scaffold name)." + }, + { + "name": "--out", + "value": "<path>", + "description": "With --drizzle: directory to write the generated migration into." + }, + { + "name": "--database-url", + "value": "<url>", + "description": "Override DATABASE_URL for this run only — never written to disk.", + "env": "DATABASE_URL" + } + ] + }, + { + "name": "eql upgrade", + "summary": "Upgrade EQL extensions to the latest version", + "flags": [ + { + "name": "--dry-run", + "description": "Show what would happen without making changes." + }, + { + "name": "--supabase", + "description": "Use Supabase-compatible mode." + }, + { + "name": "--exclude-operator-family", + "description": "Skip operator family creation." + }, + { + "name": "--eql-version", + "value": "<2|3>", + "description": "EQL generation to target.", + "default": "2" + }, + { + "name": "--latest", + "description": "Fetch the latest EQL from GitHub (v2 only)." + }, + { + "name": "--database-url", + "value": "<url>", + "description": "Override DATABASE_URL for this run only — never written to disk.", + "env": "DATABASE_URL" + } + ] + }, + { + "name": "eql status", + "summary": "Show EQL installation status", + "flags": [ + { + "name": "--database-url", + "value": "<url>", + "description": "Override DATABASE_URL for this run only — never written to disk.", + "env": "DATABASE_URL" + } + ] + } + ] + }, + { + "title": "Database", + "commands": [ + { + "name": "db push", + "summary": "Push encryption schema (writes pending if active config already exists)", + "flags": [ + { + "name": "--dry-run", + "description": "Show what would happen without making changes." + }, + { + "name": "--database-url", + "value": "<url>", + "description": "Override DATABASE_URL for this run only — never written to disk.", + "env": "DATABASE_URL" + } + ] + }, + { + "name": "db activate", + "summary": "Promote pending → active without renames (use after additive db push)", + "flags": [ + { + "name": "--database-url", + "value": "<url>", + "description": "Override DATABASE_URL for this run only — never written to disk.", + "env": "DATABASE_URL" + } + ] + }, + { + "name": "db validate", + "summary": "Validate encryption schema", + "flags": [ + { + "name": "--supabase", + "description": "Use Supabase-compatible mode." + }, + { + "name": "--exclude-operator-family", + "description": "Skip operator family creation." + }, + { + "name": "--database-url", + "value": "<url>", + "description": "Override DATABASE_URL for this run only — never written to disk.", + "env": "DATABASE_URL" + } + ] + }, + { + "name": "db migrate", + "summary": "Run pending encrypt config migrations (not yet implemented)" + }, + { + "name": "db test-connection", + "summary": "Test database connectivity", + "flags": [ + { + "name": "--database-url", + "value": "<url>", + "description": "Override DATABASE_URL for this run only — never written to disk.", + "env": "DATABASE_URL" + } + ] + } + ] + }, + { + "title": "Schema", + "commands": [ + { + "name": "schema build", + "summary": "Build an encryption schema from your database", + "flags": [ + { + "name": "--supabase", + "description": "Use Supabase-compatible mode." + }, + { + "name": "--database-url", + "value": "<url>", + "description": "Override DATABASE_URL for this run only — never written to disk.", + "env": "DATABASE_URL" + } + ] + } + ] + }, + { + "title": "Encrypt", + "commands": [ + { + "name": "encrypt status", + "summary": "Show per-column migration status (phase, progress, drift)" + }, + { + "name": "encrypt plan", + "summary": "Diff intent (.cipherstash/migrations.json) vs observed state" + }, + { + "name": "encrypt backfill", + "summary": "Resumably encrypt plaintext into the encrypted column", + "flags": [ + { + "name": "--table", + "value": "<name>", + "description": "Target table." + }, + { + "name": "--column", + "value": "<name>", + "description": "Target column." + }, + { + "name": "--pk-column", + "value": "<name>", + "description": "Primary-key column used to page through rows." + }, + { + "name": "--chunk-size", + "value": "<n>", + "description": "Rows encrypted per batch." + }, + { + "name": "--encrypted-column", + "value": "<name>", + "description": "Destination encrypted column." + }, + { + "name": "--schema-column-key", + "value": "<key>", + "description": "Schema key identifying the column config." + }, + { + "name": "--confirm-dual-writes-deployed", + "description": "Assert the app is dual-writing before backfilling (safety gate)." + }, + { + "name": "--force", + "description": "Proceed past non-fatal safety checks." + } + ] + }, + { + "name": "encrypt cutover", + "summary": "Rename swap encrypted → primary column", + "flags": [ + { + "name": "--table", + "value": "<name>", + "description": "Target table." + }, + { + "name": "--column", + "value": "<name>", + "description": "Target column." + }, + { + "name": "--proxy-url", + "value": "<url>", + "description": "Proxy URL to verify against." + }, + { + "name": "--migrations-dir", + "value": "<path>", + "description": "Directory to write the rename migration into." + } + ] + }, + { + "name": "encrypt drop", + "summary": "Generate a migration to drop the plaintext column", + "flags": [ + { + "name": "--table", + "value": "<name>", + "description": "Target table." + }, + { + "name": "--column", + "value": "<name>", + "description": "Target column." + }, + { + "name": "--migrations-dir", + "value": "<path>", + "description": "Directory to write the drop migration into." + } + ] + } + ] + }, + { + "title": "Experimental", + "commands": [ + { + "name": "env", + "summary": "(experimental) Print production env vars for deployment", + "flags": [ + { + "name": "--write", + "description": "Write the vars to a file instead of printing them." + } + ] + } + ] + } + ] +} diff --git a/scripts/generate-cli-docs.ts b/scripts/generate-cli-docs.ts index 30a0427..dd0d88b 100644 --- a/scripts/generate-cli-docs.ts +++ b/scripts/generate-cli-docs.ts @@ -1,25 +1,17 @@ #!/usr/bin/env tsx /** - * CLI reference generator — PROTOTYPE (CIP-33xx). + * CLI reference generator (CIP-33xx). * * Generates the `/reference/cli` pages from the `stash` CLI itself, so the * reference can never drift from the shipped command surface. Every page is * stamped with the CLI version it was generated from. * * ── Data source ─────────────────────────────────────────────────────────── - * TODAY (bootstrap): we parse `stash --help`, captured to a fixture. `stash` - * is a hand-rolled TS CLI with no machine-readable output yet, and no - * per-command `--help` (every command prints the top-level help), so the - * single top-level help is the whole surface. It is thin — no args, no - * per-command examples, `auth`/`encrypt` subcommands undetailed. - * - * TARGET: add `stash manifest --json` to the CLI (it already has the command - * registry it prints `--help` from). Then replace `loadManifest()` with: - * - * JSON.parse(execSync(`npx stash@${CLI_VERSION} manifest --json`)) - * - * and delete the parser below. The renderer and page format stay identical — - * that is the point of this prototype. + * We consume `stash manifest --json` (shipped in stash CLI 0.17) — the + * structured, versioned command surface the CLI builds from its own command + * registry. Groups, summaries, per-command flags (with defaults + env vars), + * and curated examples all come straight from the CLI, so the docs are a + * projection of the real command set rather than a scrape of `--help`. * * ── Versioning ──────────────────────────────────────────────────────────── * Always generated from the LATEST published `stash` on npm (resolved via @@ -27,7 +19,7 @@ * runs in `prebuild` — refreshes the docs automatically. Every page carries * `verifiedAgainst.cli` and a visible banner, so readers and agents always * know which version the docs describe. Offline, it falls back to the cached - * `scripts/fixtures/stash-help.txt`. + * `scripts/fixtures/stash-manifest.json`. */ import { execSync } from "node:child_process"; import fs from "node:fs"; @@ -37,26 +29,58 @@ import path from "node:path"; const CLI_NAME = "stash"; let CLI_VERSION = ""; // resolved to the latest published npm version at run time const RUNNER = "npx"; // normalized invocation shown in docs -const FIXTURE = path.join(process.cwd(), "scripts/fixtures", "stash-help.txt"); +const FIXTURE = path.join( + process.cwd(), + "scripts/fixtures", + "stash-manifest.json", +); const OUT_DIR = path.join(process.cwd(), "content/docs/reference/cli"); // Hand-authored per-command prose merged into the generated page (hybrid model): -// the generated skeleton (synopsis + flags) stays drift-free; a supplement adds -// rich narrative + curated examples the thin `--help` can't provide. Lives -// outside content/ so it's never treated as a page or wiped by the clean step. -// Long-term, migrate these into the CLI's own long-help/examples (see the PR). +// the generated skeleton (synopsis + flags + examples) stays drift-free; a +// supplement adds rich narrative the manifest doesn't carry. Lives outside +// content/ so it's never treated as a page or wiped by the clean step. Where +// the CLI grows per-command long-help, that prose can migrate into the CLI and +// this hook retires. const SUPPLEMENTS_DIR = path.join(process.cwd(), "scripts/cli-supplements"); -// ── Types (this shape is the spec for `stash manifest --json`) ────────────── -interface Flag { +// ── The `stash manifest --json` contract ──────────────────────────────────── +// Mirrors packages/cli/src/cli/manifest.ts in the stack repo. Command `name` +// is the full path ("eql install"); flags are already resolved per-command. +interface CliFlag { name: string; // "--supabase" - value?: string; // "<path>" + value?: string; // "<slug>" + description: string; + default?: string; // surfaced default, when worth showing + env?: string; // env var that also sets this, e.g. DATABASE_URL +} +interface CliCommand { + name: string; + summary: string; + long?: string; + examples?: string[]; + flags?: CliFlag[]; +} +interface CliGroup { + title: string; + commands: CliCommand[]; +} +interface CliManifest { + name: string; + version: string; + groups: CliGroup[]; +} + +// ── Internal model (what the renderer consumes) ───────────────────────────── +interface Flag { + name: string; + value?: string; description: string; - appliesTo?: string[]; // db-flag applicability: ["install", "upgrade"] or ["all"] } interface Command { - path: string; // "db install" - base: string; // "db" + path: string; // "eql install" + base: string; // "eql" sub?: string; // "install" + group: string; // nav group title, from the manifest summary: string; flags: Flag[]; examples: string[]; @@ -64,217 +88,104 @@ interface Command { interface Manifest { name: string; version: string; - usage: string; - globalFlags: Flag[]; commands: Command[]; + groupOrder: string[]; // nav group order, as the CLI declares it } -// Which nav group each top-level command belongs to, and the group order. -const GROUPS: Record<string, string> = { - init: "Setup & workflow", - plan: "Setup & workflow", - impl: "Setup & workflow", - status: "Setup & workflow", - wizard: "Setup & workflow", - auth: "Auth", - db: "Database", - schema: "Schema", - encrypt: "Encrypt", - env: "Deployment", -}; -const GROUP_ORDER = [ - "Setup & workflow", - "Auth", - "Database", - "Schema", - "Encrypt", - "Deployment", -]; -// Known db/schema subcommand names, used to resolve db-flag applicability. -const DB_SUBCOMMANDS = new Set([ - "install", - "upgrade", - "push", - "activate", - "validate", - "migrate", - "status", - "test-connection", -]); - -// EQL/Postgres commands get the `eql` component facet too (content-model rule: -// tag `eql` only for queryable-in-Postgres ciphertext). +// EQL/Postgres command groups get the `eql` component facet too (content-model +// rule: tag `eql` for queryable-in-Postgres ciphertext). const componentsFor = (base: string): string[] => - ["db", "schema", "encrypt"].includes(base) ? ["cli", "eql"] : ["cli"]; + ["eql", "db", "schema", "encrypt"].includes(base) ? ["cli", "eql"] : ["cli"]; // ── Source ────────────────────────────────────────────────────────────────── // Resolve the latest published version so the docs track releases automatically. function latestVersion(): string { try { - return execSync(`npm view ${CLI_NAME} version`, { encoding: "utf8" }).trim(); + return execSync(`npm view ${CLI_NAME} version`, { + encoding: "utf8", + }).trim(); } catch { const cached = fs.existsSync(FIXTURE) - ? fs.readFileSync(FIXTURE, "utf8").match(/CipherStash CLI v([0-9.]+)/)?.[1] + ? (JSON.parse(fs.readFileSync(FIXTURE, "utf8")) as CliManifest).version : undefined; if (cached) { console.warn(`⚠ npm unreachable; using cached stash v${cached}.`); return cached; } - throw new Error("Cannot resolve latest stash version (offline, no fixture)."); + throw new Error( + "Cannot resolve latest stash version (offline, no fixture).", + ); } } -// Run the resolved CLI version and cache its help. (Target: `stash manifest --json`.) -function loadHelp(version: string): string { +// Run the resolved CLI and read its `manifest --json`, caching to a fixture for +// offline builds. dotenvx (the CLI's launcher) may print tips before the JSON, +// so slice from the first `{` to the last `}` defensively. +function loadRawManifest(version: string): CliManifest { try { - const out = execSync(`npx --yes ${CLI_NAME}@${version} --help`, { + const out = execSync(`npx --yes ${CLI_NAME}@${version} manifest --json`, { encoding: "utf8", cwd: os.tmpdir(), stdio: ["ignore", "pipe", "ignore"], }); + const json = out.slice(out.indexOf("{"), out.lastIndexOf("}") + 1); + const manifest = JSON.parse(json) as CliManifest; fs.mkdirSync(path.dirname(FIXTURE), { recursive: true }); - fs.writeFileSync(FIXTURE, out); - return out; + fs.writeFileSync(FIXTURE, `${JSON.stringify(manifest, null, 2)}\n`); + return manifest; } catch { if (fs.existsSync(FIXTURE)) { console.warn(`⚠ Could not run stash@${version}; using cached fixture.`); - return fs.readFileSync(FIXTURE, "utf8"); - } - throw new Error(`Could not run stash@${version} and no cached fixture exists.`); - } -} - -// Drop dotenvx's env-injection tips and blank leading noise. -const stripNoise = (text: string): string[] => - text - .split("\n") - .filter((l) => !/^\s*◇|injected env|dotenvx|www\.(dotenvx|vestauth)/.test(l)); - -// ── Parser (delete once `stash manifest --json` exists) ───────────────────── -function parseHelp(text: string): Manifest { - const lines = stripNoise(text); - const joined = lines.join("\n"); - - const version = joined.match(/CipherStash CLI v([0-9]+\.[0-9]+\.[0-9]+)/)?.[1] ?? CLI_VERSION; - - // Section boundaries: a line like "Commands:", "Options:", "DB Flags:", "Examples:". - const sections: Record<string, string[]> = {}; - let current = ""; - for (const line of lines) { - const header = line.match(/^([A-Za-z][A-Za-z ]*):\s*$/); - if (header && !line.startsWith(" ")) { - current = header[1].trim(); - sections[current] = []; - } else if (current) { - sections[current].push(line); - } - } - - // Commands: " db install Scaffold ..." (name is non-greedy up to 2+ spaces) - const commands: Command[] = []; - for (const line of sections.Commands ?? []) { - const m = line.match(/^ {2}(\S.*?) {2,}(.+)$/); - if (!m) continue; - const rawName = m[1].replace(/\s*<subcommand>\s*/, "").trim(); - const [base, ...rest] = rawName.split(/\s+/); - commands.push({ - path: rawName, - base, - sub: rest.length ? rest.join(" ") : undefined, - summary: m[2].trim(), - flags: [], - examples: [], - }); - } - - // Global options. - const globalFlags = parseFlagBlock(sections.Options ?? []); - - // Per-command flag sections: "Init Flags", "Plan Flags", "DB Flags", … - for (const [name, body] of Object.entries(sections)) { - const fm = name.match(/^(.*) Flags$/); - if (!fm) continue; - const label = fm[1].toLowerCase(); // "init", "plan", "db" - const flags = parseFlagBlock(body); - if (label === "db") { - // DB flags carry applicability annotations; resolve onto each subcommand. - for (const cmd of commands.filter((c) => c.base === "db")) { - cmd.flags = flags.filter( - (f) => - !f.appliesTo || - f.appliesTo.includes("all") || - (cmd.sub ? f.appliesTo.includes(cmd.sub) : false), - ); - } - } else { - const cmd = commands.find((c) => c.path === label); - if (cmd) cmd.flags = flags; + return JSON.parse(fs.readFileSync(FIXTURE, "utf8")) as CliManifest; } - } - - // Examples: " npx stash db install" → attach to the longest matching command. - const byLength = [...commands].sort((a, b) => b.path.length - a.path.length); - for (const line of sections.Examples ?? []) { - const inv = line.trim(); - const m = inv.match(/^(?:npx|bunx|pnpm dlx|stash)\s+(?:stash\s+)?(.+)$/); - const argPart = m ? m[1] : inv; - const cmd = byLength.find( - (c) => argPart === c.path || argPart.startsWith(`${c.path} `), + throw new Error( + `Could not run stash@${version} manifest --json and no cached fixture exists.`, ); - if (cmd) cmd.examples.push(`${RUNNER} ${CLI_NAME} ${argPart}`); } +} - return { - name: CLI_NAME, - version, - usage: `${RUNNER} ${CLI_NAME} <command> [options]`, - globalFlags, - commands, - }; +// Fold the manifest's richer flag metadata (default + env) into the description +// column so the page format (Flag | Description) stays a single table. +function mapFlag(f: CliFlag): Flag { + const notes: string[] = []; + if (f.default !== undefined) notes.push(`default: \`${f.default}\``); + if (f.env) notes.push(`env: \`${f.env}\``); + const description = notes.length + ? `${f.description} (${notes.join("; ")})` + : f.description; + return { name: f.name, value: f.value, description }; } -// Parse an indented flag block, folding continuation lines into descriptions. -function parseFlagBlock(body: string[]): Flag[] { - const flags: Flag[] = []; - for (const line of body) { - if (!line.trim()) continue; - const m = line.match(/^ {2}(--[\w-]+)(?:,\s*-\w)?(?: +(<[^>]+>))? {2,}(.+)$/); - if (m) { - let description = m[3].trim(); - let appliesTo: string[] | undefined; - // Leading "(install, push, …)" on DB flags = applicability (+ conditions). - const paren = description.match(/^\(([^)]+)\)\s*(.*)$/); - if (paren) { - const inner = paren[1].trim(); - if (/^all\b/.test(inner)) { - // "(all db / schema commands)" — applies everywhere; drop the note. - appliesTo = ["all"]; - description = paren[2]; - } else { - const tokens = inner.split(/[,/]/).map((t) => t.trim()).filter(Boolean); - const applic = tokens.filter((t) => DB_SUBCOMMANDS.has(t)); - const conditions = tokens.filter((t) => !DB_SUBCOMMANDS.has(t)); - if (applic.length) appliesTo = applic; - description = - (conditions.length ? `(${conditions.join(", ")}) ` : "") + paren[2]; - } - } - flags.push({ name: m[1], value: m[2], description: description.trim(), appliesTo }); - } else { - const cont = line.trim(); - if (flags.length && cont) flags[flags.length - 1].description += ` ${cont}`; +// Project the CLI manifest onto the internal model the renderer consumes. +function toManifest(m: CliManifest): Manifest { + const commands: Command[] = []; + const groupOrder: string[] = []; + for (const group of m.groups) { + if (!group.commands.length) continue; + if (!groupOrder.includes(group.title)) groupOrder.push(group.title); + for (const c of group.commands) { + const [base, ...rest] = c.name.split(/\s+/); + commands.push({ + path: c.name, + base, + sub: rest.length ? rest.join(" ") : undefined, + group: group.title, + summary: c.summary, + flags: (c.flags ?? []).map(mapFlag), + examples: (c.examples ?? []).map((e) => `${RUNNER} ${CLI_NAME} ${e}`), + }); } } - return flags; + return { name: m.name, version: m.version, commands, groupOrder }; } // ── Render ─────────────────────────────────────────────────────────────────── const generatedMarker = (): string => - `{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from \`${CLI_NAME} --help\` (v${CLI_VERSION}). Re-run \`bun run generate-docs:cli\` to refresh from the latest published CLI. */}`; + `{/* GENERATED — do not edit. Produced by scripts/generate-cli-docs.ts from \`${CLI_NAME} manifest --json\` (v${CLI_VERSION}). Re-run \`bun run generate-docs:cli\` to refresh from the latest published CLI. */}`; function banner(): string { return `<Callout type="info"> -Generated from **\`${CLI_NAME}\` v${CLI_VERSION}**. Run \`${RUNNER} ${CLI_NAME}@${CLI_VERSION} --help\` to see the live command surface. +Generated from **\`${CLI_NAME}\` v${CLI_VERSION}** via \`${RUNNER} ${CLI_NAME}@${CLI_VERSION} manifest --json\`. Run \`${RUNNER} ${CLI_NAME}@${CLI_VERSION} --help\` to see the live command surface. </Callout>`; } @@ -282,7 +193,12 @@ function flagsTable(flags: Flag[]): string { if (!flags.length) return ""; const rows = flags .map((f) => { - const opt = `\`${f.name}${f.value ? ` ${f.value}` : ""}\``; + // Escape pipes (e.g. the `<2|3>` in `--eql-version`) so they don't read + // as table-column separators, even inside the code span. + const opt = `\`${f.name}${f.value ? ` ${f.value}` : ""}\``.replace( + /\|/g, + "\\|", + ); return `| ${opt} | ${f.description.replace(/\|/g, "\\|")} |`; }) .join("\n"); @@ -303,12 +219,20 @@ function commandSection(cmd: Command, level: "##" | "###"): string { if (cmd.flags.length) parts.push(flagsTable(cmd.flags).replace("### Flags", `${level}# Flags`)); if (cmd.examples.length) { - parts.push(`\n${level}# Examples\n`, "```bash", cmd.examples.join("\n"), "```"); + parts.push( + `\n${level}# Examples\n`, + "```bash", + cmd.examples.join("\n"), + "```", + ); } return parts.join("\n"); } -function renderPage(base: string, cmds: Command[]): { slug: string; body: string } { +function renderPage( + base: string, + cmds: Command[], +): { slug: string; body: string } { const isGroup = cmds.some((c) => c.sub) || cmds.length > 1; const title = base; const components = componentsFor(base); @@ -337,9 +261,16 @@ function renderPage(base: string, cmds: Command[]): { slug: string; body: string ); } else { const c = cmds[0]; - parts.push(c.summary, "", "```bash", `${RUNNER} ${CLI_NAME} ${c.path}${c.flags.length ? " [flags]" : ""}`, "```"); + parts.push( + c.summary, + "", + "```bash", + `${RUNNER} ${CLI_NAME} ${c.path}${c.flags.length ? " [flags]" : ""}`, + "```", + ); if (c.flags.length) parts.push(flagsTable(c.flags)); - if (c.examples.length) parts.push("\n## Examples\n", "```bash", c.examples.join("\n"), "```"); + if (c.examples.length) + parts.push("\n## Examples\n", "```bash", c.examples.join("\n"), "```"); } const supplement = readSupplement(base); @@ -354,7 +285,10 @@ function readSupplement(slug: string): string { return fs.existsSync(file) ? fs.readFileSync(file, "utf8").trim() : ""; } -function renderIndex(manifest: Manifest, groups: Map<string, string[]>): string { +function renderIndex( + manifest: Manifest, + groups: Map<string, string[]>, +): string { const frontmatter = [ "---", "title: CLI", @@ -366,7 +300,8 @@ function renderIndex(manifest: Manifest, groups: Map<string, string[]>): string "---", ].join("\n"); - const sections = GROUP_ORDER.filter((g) => groups.has(g)) + const sections = manifest.groupOrder + .filter((g) => groups.has(g)) .map((g) => { const rows = groups .get(g)! @@ -375,7 +310,8 @@ function renderIndex(manifest: Manifest, groups: Map<string, string[]>): string .filter((c) => c.base === base) .map((c) => { const anchor = c.sub ? `#${c.path.replace(/\s+/g, "-")}` : ""; - return `| [\`${c.path}\`](/reference/cli/${base}${anchor}) | ${c.summary} |`; + const summary = c.summary.replace(/\|/g, "\\|"); + return `| [\`${c.path}\`](/reference/cli/${base}${anchor}) | ${summary} |`; }), ) .join("\n"); @@ -395,9 +331,9 @@ ${sections} `; } -function renderMeta(groups: Map<string, string[]>): string { +function renderMeta(manifest: Manifest, groups: Map<string, string[]>): string { const pages: string[] = []; - for (const g of GROUP_ORDER) { + for (const g of manifest.groupOrder) { if (!groups.has(g)) continue; pages.push(`---${g}---`); pages.push(...groups.get(g)!); @@ -407,8 +343,7 @@ function renderMeta(groups: Map<string, string[]>): string { // ── Main ───────────────────────────────────────────────────────────────────── function loadManifest(): Manifest { - // Swap point: return JSON.parse(execSync(`npx ${CLI_NAME}@${CLI_VERSION} manifest --json`)). - return parseHelp(loadHelp(CLI_VERSION)); + return toManifest(loadRawManifest(CLI_VERSION)); } function main() { @@ -417,19 +352,24 @@ function main() { // Group top-level commands by base, preserving discovery order. const bases: string[] = []; - for (const c of manifest.commands) if (!bases.includes(c.base)) bases.push(c.base); + for (const c of manifest.commands) + if (!bases.includes(c.base)) bases.push(c.base); + // Nav groups, in the order the CLI declares them; a base inherits the group + // of its commands. const groups = new Map<string, string[]>(); + for (const g of manifest.groupOrder) groups.set(g, []); for (const base of bases) { - const g = GROUPS[base] ?? "Other"; - if (!groups.has(g)) groups.set(g, []); + const g = manifest.commands.find((c) => c.base === base)!.group; groups.get(g)!.push(base); } + for (const [g, list] of groups) if (!list.length) groups.delete(g); // Clean previously generated pages, then write fresh. fs.mkdirSync(OUT_DIR, { recursive: true }); for (const f of fs.readdirSync(OUT_DIR)) { - if (f.endsWith(".mdx") || f === "meta.json") fs.rmSync(path.join(OUT_DIR, f)); + if (f.endsWith(".mdx") || f === "meta.json") + fs.rmSync(path.join(OUT_DIR, f)); } let count = 0; @@ -439,8 +379,14 @@ function main() { fs.writeFileSync(path.join(OUT_DIR, `${slug}.mdx`), body); count++; } - fs.writeFileSync(path.join(OUT_DIR, "index.mdx"), renderIndex(manifest, groups)); - fs.writeFileSync(path.join(OUT_DIR, "meta.json"), renderMeta(groups)); + fs.writeFileSync( + path.join(OUT_DIR, "index.mdx"), + renderIndex(manifest, groups), + ); + fs.writeFileSync( + path.join(OUT_DIR, "meta.json"), + renderMeta(manifest, groups), + ); console.log( `✓ Generated ${count} CLI reference page(s) for ${CLI_NAME} v${manifest.version} → ${path.relative(process.cwd(), OUT_DIR)}`, From a10e6b7d4097e91f955ff89eeaa743738156500e Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Sun, 5 Jul 2026 15:14:10 +1000 Subject: [PATCH 24/38] docs(v2): add nested EQL v2 reference for existing v2.2 deployments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Some customers still run EQL v2.2 (v2.3 is being withdrawn). The new /reference/eql section documents EQL v3 only, so this adds a durable, nested EQL v2 reference sourced from the v2.2 docs, surviving the CIP-3335 deletion of the legacy /stack tree: content/docs/reference/eql/v2/ index.mdx — EQL v2 overview (from reference/eql-guide.mdx) indexes.mdx — index recipes, incl. the Supabase query forms queries.mdx — equality / match / range query patterns payload.mdx — the CipherCell v2.2 payload (b3, ocf/ocv) Content is the v2.2 legacy docs, NOT the EQL 2.3 rewrites from #21/#22 (which no longer ship). Two grounded doc-bug fixes are carried in: the eql-guide's fabricated index-extraction function names are corrected to the real eql_v2.hmac_256 / bloom_filter / ore_block_u64_8_256 (verified against the index recipes), and a stale cs_ste_vec_v2 name is aligned to eql_v2.ste_vec. Each page carries v2 facets (type: reference, components: [eql], verifiedAgainst eql 2.2) and a version banner routing new projects to the v3 reference. Wired into the EQL nav under a "Previous versions" separator. Supersedes #21 and #22 for the v2 branch. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/eql/meta.json | 4 +- content/docs/reference/eql/v2/index.mdx | 122 +++++++++ content/docs/reference/eql/v2/indexes.mdx | 154 +++++++++++ content/docs/reference/eql/v2/meta.json | 4 + content/docs/reference/eql/v2/payload.mdx | 309 ++++++++++++++++++++++ content/docs/reference/eql/v2/queries.mdx | 277 +++++++++++++++++++ 6 files changed, 869 insertions(+), 1 deletion(-) create mode 100644 content/docs/reference/eql/v2/index.mdx create mode 100644 content/docs/reference/eql/v2/indexes.mdx create mode 100644 content/docs/reference/eql/v2/meta.json create mode 100644 content/docs/reference/eql/v2/payload.mdx create mode 100644 content/docs/reference/eql/v2/queries.mdx diff --git a/content/docs/reference/eql/meta.json b/content/docs/reference/eql/meta.json index f4268ce..76a855b 100644 --- a/content/docs/reference/eql/meta.json +++ b/content/docs/reference/eql/meta.json @@ -14,6 +14,8 @@ "filtering", "sorting", "grouping-and-aggregates", - "joins" + "joins", + "---Previous versions---", + "v2" ] } diff --git a/content/docs/reference/eql/v2/index.mdx b/content/docs/reference/eql/v2/index.mdx new file mode 100644 index 0000000..d25131a --- /dev/null +++ b/content/docs/reference/eql/v2/index.mdx @@ -0,0 +1,122 @@ +--- +title: EQL v2 +description: PostgreSQL types, operators, and functions for querying encrypted data with EQL v2 (v2.2) — the eql_v2_encrypted type and searchable index types. +type: reference +components: [eql] +verifiedAgainst: + eql: "2.2" +--- + +<Callout type="info" title="EQL v2 reference"> +This section documents **EQL v2** (v2.2), the release existing CipherStash deployments run. Starting a new project? Use **EQL v3** — see the [EQL reference](/reference/eql). +</Callout> + +**Encrypt Query Language (EQL)** is a set of PostgreSQL types, operators, and functions that enable queries on encrypted data without decryption. EQL works seamlessly with the [CipherCell](/reference/eql/v2/payload) format to provide searchable encryption capabilities directly in PostgreSQL. + +## What is EQL? + +EQL provides the database-side components needed to query encrypted data. Unlike traditional PostgreSQL extensions, EQL is implemented as a collection of types, operators, and functions, making it compatible with managed database providers like AWS RDS that restrict extension installation. + +When combined with the [Encryption SDK](/stack/cipherstash/encryption) or [CipherStash Proxy](/stack/cipherstash/proxy), EQL enables: + +- **Exact match queries** using encrypted equality operators +- **Range queries** with order-preserving encryption +- **Pattern matching** using encrypted Bloom filters +- **Unique constraints** on encrypted columns +- **JSON/JSONB operations** on encrypted structured data + +## Core components + +### The `eql_v2_encrypted` type + +The foundation of EQL is the `eql_v2_encrypted` data type, which stores [CipherCells](/reference/eql/v2/payload) containing encrypted data and searchable encrypted metadata. + +```sql +CREATE TABLE users ( + id SERIAL PRIMARY KEY, + email eql_v2_encrypted, + name eql_v2_encrypted +); +``` + +<Callout type="warn"> +The `eql_v2_encrypted` type is required for searchable encryption in PostgreSQL. Regular `JSON` or `JSONB` types can store CipherCells but do not support encrypted queries. +</Callout> + +### Operators + +EQL provides PostgreSQL operators that work directly with encrypted data: + +```sql +-- Exact match +SELECT * FROM users WHERE email = 'encrypted_search_value'::eql_v2_encrypted; + +-- Range queries +SELECT * FROM products WHERE price > 'encrypted_value'::eql_v2_encrypted; + +-- Pattern matching +SELECT * FROM documents WHERE content LIKE '%encrypted_pattern%'; +``` + +### Functions + +EQL ships in the `eql_v2` schema, with functions in three groups: + +- **Configuration** — register tables, columns, and searchable indexes in the EQL configuration. +- **Index-term extraction** — `eql_v2.hmac_256()` (exact match), `eql_v2.bloom_filter()` (pattern matching), and `eql_v2.ore_block_u64_8_256()` (range) extract a searchable term from an encrypted value. These back the functional indexes — see [Setting up indexes](/reference/eql/v2/indexes). +- **Comparison** — operators (`=`, `<`, `LIKE`, `@>`, …) are the query surface over encrypted columns. + +For the complete, per-version function reference — every signature, parameter, and return type — see the [EQL API reference](/stack/reference/eql/), generated from each EQL release. + +## Index types + +EQL supports multiple searchable encryption index types. Each index type enables different query patterns: + +### `unique`: Exact match + +Enables exact equality queries and unique constraints using HMAC-SHA256. + +[Learn more about exact indexes](/stack/cipherstash/encryption/searchable-encryption#exact-match) + +### `ore`: Range queries + +Enables range comparisons (`<`, `>`, `BETWEEN`) and ordering (`ORDER BY`) using Order Revealing Encryption. + +[Learn more about range indexes](/stack/cipherstash/encryption/searchable-encryption#range--order) + +### `match`: Pattern matching + +Enables substring and full-text search (`LIKE`, `ILIKE`) using encrypted Bloom filters with trigrams. + +[Learn more about match indexes](/stack/cipherstash/encryption/searchable-encryption#match-pattern) + +### `ste_vec`: Structured data + +Enables containment queries and JSON-style operations on encrypted arrays and JSONB data. + +## How it works + +EQL leverages PostgreSQL's native indexing capabilities to enable efficient queries on encrypted data. The searchable encrypted metadata within [CipherCells](/reference/eql/v2/payload) is indexed using standard PostgreSQL index types (B-tree for exact/range, GIN for pattern matching). + +When a query is executed: + +1. **Client-side**: The application encrypts the search value using the same encryption scheme, producing a CipherCell with the appropriate searchable encrypted metadata +2. **Database-side**: EQL operators extract and compare the searchable encrypted metadata from both the stored CipherCells and the search CipherCell +3. **Result**: Matching rows are returned without ever decrypting the data in the database + +## Compatibility + +EQL is designed to work with: + +- **PostgreSQL 14+**: Full support for all EQL features +- **Managed databases**: Works with AWS RDS, Azure Database, Google Cloud SQL, and other managed PostgreSQL providers +- **CipherStash SDKs**: Integrates with all CipherStash SDKs as well as CipherStash Proxy + +<Callout type="info"> +Unlike PostgreSQL extensions that require `CREATE EXTENSION`, EQL types and functions are installed directly into your database schema, making it compatible with managed database environments that restrict extension installation. +</Callout> + +## Related documentation + +- [CipherCell format](/reference/eql/v2/payload): The data structure used by EQL +- [Supported queries](/stack/cipherstash/encryption/searchable-encryption): Available searchable encryption schemes diff --git a/content/docs/reference/eql/v2/indexes.mdx b/content/docs/reference/eql/v2/indexes.mdx new file mode 100644 index 0000000..6a84ca4 --- /dev/null +++ b/content/docs/reference/eql/v2/indexes.mdx @@ -0,0 +1,154 @@ +--- +title: Setting up indexes +description: Create PostgreSQL indexes for encrypted columns. Index syntax differs between self-hosted PostgreSQL and managed databases like Supabase. +type: reference +components: [eql] +verifiedAgainst: + eql: "2.2" +--- + +<Callout type="info"> +EQL v2 reference, for existing v2.2 deployments. New projects use [EQL v3](/reference/eql). +</Callout> + +Encrypted columns need PostgreSQL indexes for fast queries. Without an index, the database performs a sequential scan: correct but slow at scale. + +Index syntax differs between deployment types. Self-hosted PostgreSQL with full EQL installed supports custom operator classes and can use B-tree indexes directly on `eql_v2_encrypted` columns. Managed databases like Supabase cannot install operator families (they require superuser), so indexes must use extraction functions instead. + +## Deployment matrix + +| Query type | Self-hosted (full EQL) | Supabase | +|---|---|---| +| Equality | `USING btree (col)` with opclass, or `USING hash (eql_v2.hmac_256(col))` | `USING hash (eql_v2.hmac_256(col))` only | +| Range / ORDER BY | `USING btree (col)` with opclass | None (OPE-index work in progress) | +| Pattern match | `USING gin (eql_v2.bloom_filter(col))` | Same | +| JSONB containment | `USING gin (eql_v2.ste_vec(col))` | Same | + +<Callout type="info"> + Range filters (`>`, `>=`, `<`, `<=`) work on Supabase without a range index (they use a sequential scan). `ORDER BY` on encrypted columns is not supported on Supabase at all. Sort application-side after decrypting results. Operator family support for Supabase is in development. +</Callout> + +--- + +## Equality + +Equality indexes speed up `WHERE col = $1` queries and `IN` lists. + +**Self-hosted (B-tree with operator class):** + +```sql +CREATE INDEX ON users USING btree (email); +``` + +This works because the full EQL install registers a B-tree operator class for `eql_v2_encrypted` that compares HMAC terms. + +**Self-hosted or Supabase (hash on extraction function):** + +```sql +CREATE INDEX ON users USING hash (eql_v2.hmac_256(email)); +``` + +This form works on both deployment types. Use it when you want one index that works everywhere, or when you are on Supabase. + +See queries: [Equality queries](/reference/eql/v2/queries#equality) + +--- + +## Match + +Match indexes speed up `WHERE col LIKE $1` and `ILIKE` queries. They use a GIN index on the Bloom filter extracted from each encrypted value. + +```sql +CREATE INDEX ON users USING gin (eql_v2.bloom_filter(name)); +``` + +This form is identical for self-hosted and Supabase. + +See queries: [Match queries](/reference/eql/v2/queries#match-free-text) + +--- + +## Range and order + +Range indexes support `>`, `>=`, `<`, `<=`, `BETWEEN`, and `ORDER BY` on encrypted columns. + +**Self-hosted (B-tree with operator class):** + +```sql +CREATE INDEX ON users USING btree (age); +``` + +Requires the EQL operator family (`CREATE OPERATOR FAMILY`) to be installed. The full EQL install includes this. The `--exclude-operator-family` install flag omits it. + +**Supabase:** + +Functional range indexes for Supabase are not yet available. Range _filters_ work without an index (sequential scan). `ORDER BY` on encrypted columns is not supported on Supabase. + +See queries: [Range queries](/reference/eql/v2/queries#range-and-ordering) + +--- + +## JSONB + +JSONB indexes support path existence and containment queries on encrypted JSON columns. + +```sql +CREATE INDEX ON documents USING gin (eql_v2.ste_vec(metadata)); +``` + +This form is identical for self-hosted and Supabase. + +See queries: [JSONB queries](/reference/eql/v2/queries#jsonb-queries) + +--- + +## Supabase query forms + +This is the most common source of silent performance problems with encrypted columns on Supabase. + +A functional index on `eql_v2.hmac_256(email)` is only engaged when the query uses the same extraction function. A bare `WHERE email = $1` query does not use the index, even if the index exists. The database falls back to a sequential scan: your query returns correct results, but it scans every row. + +**Wrong (does not use functional index):** + +```sql +SELECT * FROM users WHERE email = $1::eql_v2_encrypted; +``` + +**Right (engages the functional index):** + +```sql +SELECT * FROM users WHERE eql_v2.hmac_256(email) = eql_v2.hmac_256($1::eql_v2_encrypted); +``` + +<Callout type="warn"> + SDK wrappers (Drizzle adapter, Supabase wrapper) generate the correct query form automatically. This only matters when you write raw SQL queries against Supabase encrypted columns. If you are using the Drizzle adapter or Supabase wrapper, no action is needed. +</Callout> + +The same principle applies to `eql_v2.bloom_filter` and `eql_v2.ste_vec` indexes: the extraction function must appear in both the index definition and the query predicate. + +--- + +## Complete example + +```sql filename="migrations/add_encrypted_indexes.sql" +-- Equality index (Supabase-compatible form) +CREATE INDEX users_email_eq_idx ON users USING hash (eql_v2.hmac_256(email)); + +-- Match index +CREATE INDEX users_name_match_idx ON users USING gin (eql_v2.bloom_filter(name)); + +-- JSONB index +CREATE INDEX documents_metadata_ste_idx ON documents USING gin (eql_v2.ste_vec(metadata)); + +-- Range index (self-hosted only — requires operator family) +CREATE INDEX users_age_range_idx ON users USING btree (age); +``` + +--- + +## Related + +- [Searchable encryption queries](/reference/eql/v2/queries): Query patterns for each index type +- [Searchable encryption overview](/stack/cipherstash/encryption/searchable-encryption): How searchable indexes work +- [Supabase integration](/stack/cipherstash/supabase): Supabase-specific setup and limitations +- [EQL guide](/reference/eql/v2): Full reference for EQL types and functions diff --git a/content/docs/reference/eql/v2/meta.json b/content/docs/reference/eql/v2/meta.json new file mode 100644 index 0000000..68333fd --- /dev/null +++ b/content/docs/reference/eql/v2/meta.json @@ -0,0 +1,4 @@ +{ + "title": "EQL v2", + "pages": ["indexes", "queries", "payload"] +} diff --git a/content/docs/reference/eql/v2/payload.mdx b/content/docs/reference/eql/v2/payload.mdx new file mode 100644 index 0000000..953264d --- /dev/null +++ b/content/docs/reference/eql/v2/payload.mdx @@ -0,0 +1,309 @@ +--- +title: The CipherCell +description: Understand the CipherCell, the CipherStash JSON format that stores ciphertext, searchable encrypted metadata, and fields for querying encrypted data via EQL. +type: reference +components: [eql] +verifiedAgainst: + eql: "2.2" +--- + +<Callout type="info"> +EQL v2 reference, for existing v2.2 deployments. New projects use [EQL v3](/reference/eql). +</Callout> + +The **CipherCell** is CipherStash's standard format for storing encrypted data in a database. It is a JSON-based structure that combines **encrypted values**, **searchable encrypted metadata**, and **non-sensitive metadata** into a single, self-contained record. +CipherCells are designed to make encrypted data practical to work with in real applications. They can be stored in existing databases (such as PostgreSQL jsonb columns), indexed, queried, and audited without exposing plaintext. + +## What a CipherCell contains + +A CipherCell typically includes: + +- **Encrypted data** + - The ciphertext for one or more sensitive values. + - Each value is encrypted independently using strong authenticated encryption and unique per-value keys. +- **Searchable Encrypted Metadata (SEM)** + - Additional cryptographic material derived from the plaintext that enables secure querying using searchable encryption. + - This metadata allows operations such as equality checks, range queries, or text search to be performed **without decrypting the data**. + - The database can evaluate queries over this metadata, but cannot recover the original values. +- **Non-sensitive metadata** + - Plaintext fields that are safe to expose, such as schema identifiers, versioning information, timestamps, or application-level IDs. + - Keeping this metadata unencrypted allows efficient filtering, indexing, and integration with existing tooling. + +## How CipherCells are used + +CipherCells are stored directly in the database, usually in a JSON-compatible column. +[Encrypt Query Language (EQL)](/reference/eql/v2) understands this structure and provides database functions and operators that work over CipherCells, enabling encrypted search and filtering while preserving strong security guarantees. + +From an application's perspective, a CipherCell behaves like a regular database value: + +- Applications write encrypted data as JSON +- Databases store and index it +- Queries operate on Searchable Encrypted Metadata (SEM) +- Decryption happens only in trusted application code with the right keys and claims + +## Why CipherCells exist + +The CipherCell format solves a common problem with encryption at rest: traditional encryption makes data opaque and hard to query. CipherCells retain the benefits of encryption while enabling: + +- Fine-grained, **per-value** protection +- Searchable encryption over structured data +- Compatibility with existing databases and ORMs +- Clear separation between sensitive and non-sensitive information + +A CipherCell is the **unit of encrypted storage** in CipherStash: a portable, self-describing JSON record that makes encrypted data usable, searchable, and auditable by default. + +## Structure + +<Callout type="warn"> +**Required fields**: Only `i` (identifier) and `v` (version) are required. + +**Payload requirement**: Either `c` (ciphertext) or `sv` (structured encryption vector) must be present, but never both. + +**Optional fields**: All searchable encrypted metadata (SEM) fields are optional and only included when the corresponding index types are configured. +</Callout> + +A CipherCell is stored as a JSON object with the following top-level structure: + +```json +{ + "i": { + "t": "table_name", + "c": "column_name" + }, + "v": 2, + "c": "encrypted_data_in_messagepack_base85", + "hm": "2e182f0c444d1d51f5f70f32d778b2eaa854f5921a4a2acaa4446c44055cb777", + "ob": ["ore_block_1", "ore_block_2"], + "bf": [1234, 5678, 9012] +} +``` + +## Top-level fields + +### `i` - Identifier + +The table and column identifier for this encrypted data. + +**Type**: Object with `t` (table) and `c` (column) properties + +**Required**: Yes + +```json +{ + "i": { + "t": "users", + "c": "email" + } +} +``` + +This field identifies which table and column the encrypted data belongs to, enabling proper decryption and index usage. + +### `v` - Version + +The encryption version used for this CipherCell. + +**Type**: Integer + +**Required**: Yes + +```json +{ + "v": 2 +} +``` + +The version field allows for cryptographic algorithm upgrades over time while maintaining backward compatibility. + +### `c` - Ciphertext (required unless `sv` is present) + +The encrypted record containing the actual plaintext data. + +**Type**: String (MessagePack encoded and Base85 encoded) + +**Required**: Yes (unless `sv` field is present) + +```json +{ + "c": "Xk}0>Z*pVbW@%*8a%F0@" +} +``` + +<Callout type="warn"> +Either `c` or `sv` must be present in every CipherCell, but never both. Use `c` for standard encrypted values and `sv` for structured encryption vectors (arrays or JSON structures). +</Callout> + +### `a` - Array item flag + +Indicates whether this CipherCell represents an item within an array. + +**Type**: Boolean + +**Required**: No + +```json +{ + "a": true +} +``` + +## Searchable Encrypted Metadata (SEM) + +The CipherCell can contain various types of searchable encrypted metadata, each enabling different query capabilities. All SEM fields are optional and only included when the corresponding index type is configured. + +### `hm` - HMAC-SHA256 + +Enables exact match queries using HMAC-SHA256. + +**Type**: Hex-encoded string (64 characters) + +**Index Type**: [Exact](/stack/cipherstash/encryption/searchable-encryption#exact-match) + +```json +{ + "hm": "2e182f0c444d1d51f5f70f32d778b2eaa854f5921a4a2acaa4446c44055cb777" +} +``` + +### `ob` - ORE Block + +Enables range queries and ordering using Order Revealing Encryption. + +**Type**: Array of strings + +**Index Type**: [Order / Range](/stack/cipherstash/encryption/searchable-encryption#range--order) + +```json +{ + "ob": [ + "01a2b3c4d5e6f7g8h9i0", + "j1k2l3m4n5o6p7q8r9s0" + ] +} +``` + +### `bf` - Bloom Filter + +Enables substring and pattern matching queries using encrypted Bloom filters with trigrams. + +**Type**: Array of integers + +**Index Type**: [Match](/stack/cipherstash/encryption/searchable-encryption#match-pattern) + +```json +{ + "bf": [1234, 5678, 9012, 3456, 7890] +} +``` + +### `b3` - Blake3 + +Blake3 hash for exact matches in structured encryption vectors. + +**Type**: Hex-encoded string + +**Used in**: SteVec (Structured Encryption Vector) subfield + +### `s` - Selector + +Selector value for field selection in structured encryption vectors. + +**Type**: String + +**Used in**: SteVec (Structured Encryption Vector) subfield + +### `ocf` - ORE CLWW Fixed-Width + +ORE CLWW (Chenette-Lewi-Weis-Wu) fixed-width scheme for 64-bit integer values in structured encryption vectors. + +**Type**: String + +**Used in**: SteVec (Structured Encryption Vector) subfield + +### `ocv` - ORE CLWW Variable-Width + +ORE CLWW variable-width scheme for string comparison in structured encryption vectors. + +**Type**: String + +**Used in**: SteVec (Structured Encryption Vector) subfield + +### `sv` - Structured Encryption Vector (SteVec) (required unless `c` is present) + +Nested array of CipherCells for supporting containment queries and JSON-style operations. + +**Type**: Array of CipherCell objects + +**Required**: Yes (unless `c` attribute is present) + +```json +{ + "sv": [ + { + "c": "Xk}0>Z*pVbW@%*8a%F0@", + "hm": "hash1...", + "s": "selector1" + }, + { + "c": "Yl~1?A+qWcX#&+9b&G1#", + "hm": "hash2...", + "s": "selector2" + } + ] +} +``` + +SteVec enables queries on array elements and JSON document structures while maintaining encryption. Each element in the `sv` array is itself a CipherCell that can contain SEM fields like `b3`, `s`, `ocf`, and `ocv`. + +## Complete example + +Here's a complete CipherCell with multiple index types enabled: + +```json +{ + "i": { + "t": "products", + "c": "price" + }, + "v": 2, + "c": "Xk}0>Z*pVbW@%*8a%F0@Yl~1?A+qWcX#&+9b&G1#", + "hm": "2e182f0c444d1d51f5f70f32d778b2eaa854f5921a4a2acaa4446c44055cb777", + "ob": [ + "01a2b3c4d5e6f7g8h9i0", + "j1k2l3m4n5o6p7q8r9s0", + "t1u2v3w4x5y6z7a8b9c0" + ], + "bf": [1234, 5678, 9012, 3456, 7890, 2345, 6789] +} +``` + +This CipherCell: +- Belongs to the `price` column of the `products` table +- Uses encryption version 2 +- Contains the encrypted plaintext value +- Supports exact match queries via `hm` +- Supports range queries and ordering via `ob` +- Supports pattern matching via `bf` + +## Design principles + +### Minimal storage + +Only the index types configured for a column are included in the CipherCell. This minimizes storage overhead and ensures optimal performance. + +### Composable indexes + +Multiple index types can be combined on a single column, enabling both exact matches and range queries, or exact matches and pattern matching, depending on application needs. + +### Forward compatibility + +The version field (`v`) enables cryptographic algorithm upgrades without requiring full database re-encryption. Older versions can coexist with newer versions during migration. + +### Standardized format + +The CipherCell format is consistent across all CipherStash SDKs and tools, ensuring interoperability and portability of encrypted data. + +## Database storage + +CipherCells can be stored as JSON in any database that supports JSON data types. +However, for search to be supported using the [Encryption SDK](/stack/cipherstash/encryption) or [CipherStash Proxy](/stack/cipherstash/proxy), the `eql_v2.encrypted` database type must be used which is available when the Encrypt Query Language (EQL) helpers have been installed. diff --git a/content/docs/reference/eql/v2/queries.mdx b/content/docs/reference/eql/v2/queries.mdx new file mode 100644 index 0000000..cbf0f81 --- /dev/null +++ b/content/docs/reference/eql/v2/queries.mdx @@ -0,0 +1,277 @@ +--- +title: Searchable encryption queries +description: Equality, match, and range query patterns for encrypted PostgreSQL columns, with SDK predicates and raw SQL forms. +type: reference +components: [eql] +verifiedAgainst: + eql: "2.2" +--- + +<Callout type="info"> +EQL v2 reference, for existing v2.2 deployments. New projects use [EQL v3](/reference/eql). +</Callout> + +This page covers the three query families available for encrypted columns: equality, match (free-text), and range/order. Each section shows the SDK predicate, the raw SQL form, the underlying EQL index, and links to the corresponding index setup. + +For index creation (the `CREATE INDEX` statements your database needs), see [Setting up indexes](/reference/eql/v2/indexes). + +For a conceptual overview of how searchable encryption works, see [Searchable encryption](/stack/cipherstash/encryption/searchable-encryption). + +## Equality + +Exact match on an encrypted column. Uses the `unique` (HMAC-SHA256) index. + +**Schema:** + +```typescript filename="src/schema.ts" +import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" + +const users = encryptedTable("users", { + email: encryptedColumn("email").equality(), +}) +``` + +**SDK (single value):** + +```typescript filename="src/queries.ts" +const term = await client.encryptQuery("alice@example.com", { + column: users.email, + table: users, + queryType: "equality", +}) + +const result = await pgClient.query( + "SELECT * FROM users WHERE email = $1", + [term.data], +) +``` + +**SDK (IN list):** + +```typescript filename="src/queries.ts" +const terms = await client.encryptQuery([ + { value: "alice@example.com", column: users.email, table: users, queryType: "equality" as const }, + { value: "bob@example.com", column: users.email, table: users, queryType: "equality" as const }, +]) + +// Use each term.data as a separate parameter, or build an ANY($1) query. +``` + +**Drizzle:** + +```typescript filename="src/queries.ts" +const results = await db + .select() + .from(usersTable) + .where(await encryptionOps.eq(usersTable.email, "alice@example.com")) +``` + +**Supabase wrapper:** + +```typescript filename="src/queries.ts" +const { data } = await eSupabase + .from("users", users) + .select("id, email") + .eq("email", "alice@example.com") +``` + +**Raw SQL (self-hosted with EQL operator classes):** + +```sql +SELECT * FROM users WHERE email = $1::eql_v2_encrypted; +``` + +**Raw SQL (Supabase / functional index form):** + +```sql +SELECT * FROM users WHERE eql_v2.hmac_256(email) = eql_v2.hmac_256($1::eql_v2_encrypted); +``` + +<Callout type="warn"> + On Supabase, bare `WHERE email = $1` does not use the functional index. Wrap both sides with `eql_v2.hmac_256()` to engage the hash index. The SDK wrappers (Drizzle, Supabase wrapper) handle this automatically. See [Index setup: Supabase callout](/reference/eql/v2/indexes#supabase-query-forms). +</Callout> + +**Underlying index:** [Equality index setup](/reference/eql/v2/indexes#equality) + +--- + +## Match (free-text) + +Substring and full-text search on an encrypted column. Uses the `match` (Bloom filter) index. Corresponds to `LIKE` / `ILIKE` semantics. + +**Schema:** + +```typescript filename="src/schema.ts" +const users = encryptedTable("users", { + name: encryptedColumn("name").freeTextSearch(), +}) +``` + +**SDK:** + +```typescript filename="src/queries.ts" +const term = await client.encryptQuery("alice", { + column: users.name, + table: users, + queryType: "freeTextSearch", +}) + +const result = await pgClient.query( + "SELECT * FROM users WHERE name LIKE $1", + [term.data], +) +``` + +**Drizzle:** + +```typescript filename="src/queries.ts" +const results = await db + .select() + .from(usersTable) + .where(await encryptionOps.ilike(usersTable.name, "%alice%")) +``` + +**Supabase wrapper:** + +```typescript filename="src/queries.ts" +const { data } = await eSupabase + .from("users", users) + .select("id, name") + .ilike("name", "%alice%") +``` + +**Raw SQL:** + +```sql +SELECT * FROM users WHERE name LIKE $1; +``` + +The Bloom filter index uses a GIN index on the extracted filter term. See [Match index setup](/reference/eql/v2/indexes#match). + +**Underlying index:** [Match index setup](/reference/eql/v2/indexes#match) + +--- + +## Range and ordering + +Comparison (`>`, `>=`, `<`, `<=`, `BETWEEN`) and `ORDER BY` on an encrypted column. Uses the `ore` (Order Revealing Encryption) index. + +**Schema:** + +```typescript filename="src/schema.ts" +const users = encryptedTable("users", { + age: encryptedColumn("age").dataType("number").orderAndRange(), +}) +``` + +**SDK (range filter):** + +```typescript filename="src/queries.ts" +const term = await client.encryptQuery(21, { + column: users.age, + table: users, + queryType: "orderAndRange", +}) + +const result = await pgClient.query( + "SELECT * FROM users WHERE age > $1", + [term.data], +) +``` + +**SDK, ORDER BY (self-hosted only):** + +```typescript filename="src/queries.ts" +// Self-hosted PostgreSQL with EQL operator families installed: +const result = await pgClient.query( + "SELECT * FROM users ORDER BY age ASC", +) + +// Without operator family support (Supabase, or --exclude-operator-family): +const result = await pgClient.query( + "SELECT * FROM users ORDER BY eql_v2.ore_block_u64_8_256(age) ASC", +) +``` + +**Drizzle:** + +```typescript filename="src/queries.ts" +// Range +const results = await db + .select() + .from(usersTable) + .where(await encryptionOps.gte(usersTable.age, 18)) + +// Sort (requires operator family support; not available on Supabase) +const results = await db + .select() + .from(usersTable) + .orderBy(encryptionOps.asc(usersTable.age)) +``` + +**Supabase wrapper:** + +```typescript filename="src/queries.ts" +// Range filter works +const { data } = await eSupabase + .from("users", users) + .select("id, age") + .gte("age", 18) + +// ORDER BY on encrypted columns is not supported on Supabase. +// Sort application-side after decrypting. +``` + +<Callout type="warn"> + `ORDER BY` on encrypted columns requires EQL operator families, which need superuser access to install. Supabase does not grant superuser. Range _filters_ (`>`, `>=`, `<`, `<=`) work on both self-hosted and Supabase. Sorting on encrypted columns is not currently supported on Supabase. Sort application-side after decrypting results. Operator family support for Supabase is being developed in collaboration with the Supabase and CipherStash teams. +</Callout> + +**Underlying index:** [Range index setup](/reference/eql/v2/indexes#range-and-order) + +--- + +## JSONB queries + +Query encrypted JSON columns using path existence or containment. Uses the `ste_vec` index. + +**Schema:** + +```typescript filename="src/schema.ts" +const documents = encryptedTable("documents", { + metadata: encryptedColumn("metadata").searchableJson(), +}) +``` + +**SDK (path existence):** + +```typescript filename="src/queries.ts" +const term = await client.encryptQuery("$.user.role", { + column: documents.metadata, + table: documents, +}) + +const result = await pgClient.query( + "SELECT * FROM documents WHERE eql_v2.ste_vec(metadata) @> $1", + [term.data], +) +``` + +**SDK (containment):** + +```typescript filename="src/queries.ts" +const term = await client.encryptQuery({ role: "admin" }, { + column: documents.metadata, + table: documents, +}) +``` + +**Drizzle:** + +```typescript filename="src/queries.ts" +const results = await db + .select() + .from(documentsTable) + .where(await encryptionOps.jsonbPathExists(documentsTable.metadata, "$.user.role")) +``` + +**Underlying index:** [JSONB index setup](/reference/eql/v2/indexes#jsonb) From 40ee2216ae3430c364b72d532df9e76e6a87797a Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 17:30:54 +1000 Subject: [PATCH 25/38] fix(cli-docs): escape MDX braces in manifest-derived prose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `auth regions --json` flag description is "Emit machine-readable [{ slug, label }] ...". MDX parsed `{ slug, label }` as a JS expression, so prerendering /reference/cli/auth threw `ReferenceError: slug is not defined` and failed the build. (The earlier `<tt>` failure masked this; it surfaced once #52 landed via rebase.) Escape `{`/`}` (and stray `<`) in manifest-derived prose — flag descriptions and summaries. Flag names/values stay in code spans, which are literal, so they're untouched. Verified: `next build` prerenders all 355 pages, including /reference/cli/auth. --- content/docs/reference/cli/auth.mdx | 2 +- scripts/generate-cli-docs.ts | 16 ++++++++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/content/docs/reference/cli/auth.mdx b/content/docs/reference/cli/auth.mdx index 09f0a18..482ae39 100644 --- a/content/docs/reference/cli/auth.mdx +++ b/content/docs/reference/cli/auth.mdx @@ -55,7 +55,7 @@ npx stash auth regions [flags] | Flag | Description | | --- | --- | -| `--json` | Emit machine-readable [{ slug, label }] instead of a text list. | +| `--json` | Emit machine-readable [\{ slug, label \}] instead of a text list. | #### Examples diff --git a/scripts/generate-cli-docs.ts b/scripts/generate-cli-docs.ts index dd0d88b..e80ca6d 100644 --- a/scripts/generate-cli-docs.ts +++ b/scripts/generate-cli-docs.ts @@ -189,6 +189,13 @@ Generated from **\`${CLI_NAME}\` v${CLI_VERSION}** via \`${RUNNER} ${CLI_NAME}@$ </Callout>`; } +// Escape characters MDX parses as JSX inside prose: `{`/`}` (expression braces — +// e.g. the `auth regions` flag description "[{ slug, label }]" would otherwise +// evaluate `slug` and crash the prerender) and stray `<` (tags). Flag names and +// values render inside code spans, which are literal, so this only applies to +// manifest-derived prose (descriptions, summaries). +const escapeMdxText = (s: string): string => s.replace(/([{}<])/g, "\\$1"); + function flagsTable(flags: Flag[]): string { if (!flags.length) return ""; const rows = flags @@ -199,7 +206,8 @@ function flagsTable(flags: Flag[]): string { /\|/g, "\\|", ); - return `| ${opt} | ${f.description.replace(/\|/g, "\\|")} |`; + const description = escapeMdxText(f.description).replace(/\|/g, "\\|"); + return `| ${opt} | ${description} |`; }) .join("\n"); return `\n### Flags\n\n| Flag | Description |\n| --- | --- |\n${rows}\n`; @@ -210,7 +218,7 @@ function commandSection(cmd: Command, level: "##" | "###"): string { const parts = [ `${level} \`${cmd.path}\``, "", - cmd.summary, + escapeMdxText(cmd.summary), "", "```bash", synopsis, @@ -262,7 +270,7 @@ function renderPage( } else { const c = cmds[0]; parts.push( - c.summary, + escapeMdxText(c.summary), "", "```bash", `${RUNNER} ${CLI_NAME} ${c.path}${c.flags.length ? " [flags]" : ""}`, @@ -310,7 +318,7 @@ function renderIndex( .filter((c) => c.base === base) .map((c) => { const anchor = c.sub ? `#${c.path.replace(/\s+/g, "-")}` : ""; - const summary = c.summary.replace(/\|/g, "\\|"); + const summary = escapeMdxText(c.summary).replace(/\|/g, "\\|"); return `| [\`${c.path}\`](/reference/cli/${base}${anchor}) | ${summary} |`; }), ) From 261b69ca1b221fce85a7d9d8a99c9a348c03ec47 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 18:03:22 +1000 Subject: [PATCH 26/38] docs(cli): address auth-page review feedback Vercel preview feedback on /reference/cli/auth: - Remove the "--supabase / --drizzle flags only tag the referrer" paragraph. - Convert the "Good to know" blockquote into a proper <Callout title=...>. Edited the auth supplement (the page is generated from it) and recast the em-dashes in the touched prose. Verified: next build prerenders all 355 pages. --- content/docs/reference/cli/auth.mdx | 16 +++++++--------- scripts/cli-supplements/auth.md | 16 +++++++--------- 2 files changed, 14 insertions(+), 18 deletions(-) diff --git a/content/docs/reference/cli/auth.mdx b/content/docs/reference/cli/auth.mdx index 482ae39..2ba2a00 100644 --- a/content/docs/reference/cli/auth.mdx +++ b/content/docs/reference/cli/auth.mdx @@ -70,21 +70,19 @@ npx stash auth regions --json `stash auth login` runs the OAuth 2.0 **device authorization flow**: 1. You pick a **region** for your CipherStash workspace. -2. The CLI opens your browser to a verification URL — and prints it, so it also - works over SSH or in a headless/agent environment — where you approve the +2. The CLI opens your browser to a verification URL (and prints it, so it also + works over SSH or in a headless/agent environment) where you approve the request. 3. Meanwhile the CLI polls CipherStash until you approve, then receives a short-lived access token (it reports the token's expiry). 4. Your device is **bound to the workspace's default keyset**, so later commands (`stash eql install`, `stash db push`, …) authenticate without a fresh login. -The `--supabase` / `--drizzle` flags only tag the referrer for onboarding -analytics; they don't change the flow. - -> **Good to know**: login is device- and workspace-scoped. Authenticating from a -> new machine — or for a different workspace — re-runs the device flow. +<Callout title="Good to know"> +Login is device- and workspace-scoped. Authenticating from a new machine, or for a different workspace, re-runs the device flow. +</Callout> {/* TODO(verify with product): profiles, multiple workspaces, and switching -between them — where they're stored and how they're selected — belong here (or -in a linked CLI concept page). The CLI currently exposes only `auth login`; +between them (where they're stored and how they're selected) belong here, or +in a linked CLI concept page. The CLI currently exposes only `auth login`; confirm the profile / workspace-switching surface before documenting it. */} diff --git a/scripts/cli-supplements/auth.md b/scripts/cli-supplements/auth.md index 4d13348..b17fc57 100644 --- a/scripts/cli-supplements/auth.md +++ b/scripts/cli-supplements/auth.md @@ -3,21 +3,19 @@ `stash auth login` runs the OAuth 2.0 **device authorization flow**: 1. You pick a **region** for your CipherStash workspace. -2. The CLI opens your browser to a verification URL — and prints it, so it also - works over SSH or in a headless/agent environment — where you approve the +2. The CLI opens your browser to a verification URL (and prints it, so it also + works over SSH or in a headless/agent environment) where you approve the request. 3. Meanwhile the CLI polls CipherStash until you approve, then receives a short-lived access token (it reports the token's expiry). 4. Your device is **bound to the workspace's default keyset**, so later commands (`stash eql install`, `stash db push`, …) authenticate without a fresh login. -The `--supabase` / `--drizzle` flags only tag the referrer for onboarding -analytics; they don't change the flow. - -> **Good to know**: login is device- and workspace-scoped. Authenticating from a -> new machine — or for a different workspace — re-runs the device flow. +<Callout title="Good to know"> +Login is device- and workspace-scoped. Authenticating from a new machine, or for a different workspace, re-runs the device flow. +</Callout> {/* TODO(verify with product): profiles, multiple workspaces, and switching -between them — where they're stored and how they're selected — belong here (or -in a linked CLI concept page). The CLI currently exposes only `auth login`; +between them (where they're stored and how they're selected) belong here, or +in a linked CLI concept page. The CLI currently exposes only `auth login`; confirm the profile / workspace-switching surface before documenting it. */} From 7d1dcffa2f3ff8900bc79af4252bfeaed4e1a2e8 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 18:08:41 +1000 Subject: [PATCH 27/38] fix(cli-docs): harden manifest loading (Copilot review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - loadRawManifest: validate the JSON delimiters before slicing, throwing a clear error (which falls back to the fixture) instead of feeding garbage to JSON.parse. - Reconcile CLI_VERSION to the loaded manifest's version after loading, so pages are stamped with the version of the data actually used — including the fixture-fallback path, where npm-latest and the cached manifest can differ. --- scripts/generate-cli-docs.ts | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/scripts/generate-cli-docs.ts b/scripts/generate-cli-docs.ts index e80ca6d..07effd9 100644 --- a/scripts/generate-cli-docs.ts +++ b/scripts/generate-cli-docs.ts @@ -128,8 +128,14 @@ function loadRawManifest(version: string): CliManifest { cwd: os.tmpdir(), stdio: ["ignore", "pipe", "ignore"], }); - const json = out.slice(out.indexOf("{"), out.lastIndexOf("}") + 1); - const manifest = JSON.parse(json) as CliManifest; + const start = out.indexOf("{"); + const end = out.lastIndexOf("}"); + if (start === -1 || end < start) { + throw new Error( + `\`${CLI_NAME}@${version} manifest --json\` did not emit a JSON object (got: ${out.trim().slice(0, 120)}…)`, + ); + } + const manifest = JSON.parse(out.slice(start, end + 1)) as CliManifest; fs.mkdirSync(path.dirname(FIXTURE), { recursive: true }); fs.writeFileSync(FIXTURE, `${JSON.stringify(manifest, null, 2)}\n`); return manifest; @@ -355,8 +361,13 @@ function loadManifest(): Manifest { } function main() { + // latestVersion() picks which published CLI to invoke; the manifest we get + // back is authoritative for what to stamp. Reconcile CLI_VERSION to it so + // pages never claim a version different from the data they were built from + // (e.g. when the live run fails and we fall back to an older cached fixture). CLI_VERSION = latestVersion(); const manifest = loadManifest(); + CLI_VERSION = manifest.version; // Group top-level commands by base, preserving discovery order. const bases: string[] = []; From 089645d9394b62cbda61cb4d7c914248b403bbbd Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 18:15:47 +1000 Subject: [PATCH 28/38] chore(cli-docs): drop the internal tracker ref from the generator header The header comment carried an internal issue id; remove it so no private tracker reference lives in this public repo. Comment-only. --- scripts/generate-cli-docs.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/generate-cli-docs.ts b/scripts/generate-cli-docs.ts index 07effd9..89a59ce 100644 --- a/scripts/generate-cli-docs.ts +++ b/scripts/generate-cli-docs.ts @@ -1,6 +1,6 @@ #!/usr/bin/env tsx /** - * CLI reference generator (CIP-33xx). + * CLI reference generator. * * Generates the `/reference/cli` pages from the `stash` CLI itself, so the * reference can never drift from the shipped command surface. Every page is From 1dd4b98e1fcd25b60e1a2776d07d63a1f7c8891d Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Mon, 6 Jul 2026 00:34:53 +1000 Subject: [PATCH 29/38] docs(v2): generate the EQL function catalog from the manifest + drift-lint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consumes the eql-manifest.json the EQL repo emits from its Doxygen'd SQL (cipherstash/encrypt-query-language#364) to: - generate content/docs/reference/eql/functions.mdx — a version-stamped, drift-proof catalog of the eql_v3 function surface that the hand-written pedagogical pages link to (added to the EQL nav under "Reference"); - drift-lint the hand-written pages: every `eql_v3.<fn>(...)` they reference should exist in the manifest for the shipped EQL version. This is what would have caught the fabricated function names / stale payload that slipped into the v2 docs. Wired into prebuild. Reads an illustrative sample fixture today (so the format and lint are reviewable now); swaps to the real json/eql-manifest.json from the eql-docs release asset once #364 ships — generate-eql-docs.ts already downloads that tarball. The lint is report-only against the sample and becomes a failing gate once wired to the real manifest (STRICT). Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/eql/functions.mdx | 55 +++++++ content/docs/reference/eql/meta.json | 2 + package.json | 3 +- scripts/fixtures/eql-manifest.sample.json | 43 ++++++ scripts/generate-eql-api-docs.ts | 177 ++++++++++++++++++++++ 5 files changed, 279 insertions(+), 1 deletion(-) create mode 100644 content/docs/reference/eql/functions.mdx create mode 100644 scripts/fixtures/eql-manifest.sample.json create mode 100644 scripts/generate-eql-api-docs.ts diff --git a/content/docs/reference/eql/functions.mdx b/content/docs/reference/eql/functions.mdx new file mode 100644 index 0000000..5f1eca2 --- /dev/null +++ b/content/docs/reference/eql/functions.mdx @@ -0,0 +1,55 @@ +--- +title: Functions +description: "Generated catalog of EQL SQL functions and operators (EQL 3.0.0-sample)." +type: reference +components: [eql] +verifiedAgainst: + eql: "3.0.0-sample" +--- + +{/* GENERATED — do not edit. Produced by scripts/generate-eql-api-docs.ts from the EQL manifest (v3.0.0-sample). */} + +<Callout type="info"> +Generated from the **EQL 3.0.0-sample** manifest (the Doxygen'd SQL is the source of truth). For the model behind these — variants, terms, typed operands — see [Core concepts](/reference/eql/core-concepts). +</Callout> + +The complete function surface of the `eql_v3` schema. The type and query pages explain *when* to use these; this page is the exhaustive signature reference they link to. + +## Functions + +### `hmac_256(jsonb)` + +Extract the HMAC-SHA-256 equality term from an encrypted value. + +Backs equality (`=`, `IN`, joins) on `_eq` and `text_search` columns. + +| Parameter | Type | Description | +| --- | --- | --- | +| `val` | `jsonb` | the encrypted value | + + +**Returns:** `text` — the HMAC term +### `bloom_filter(jsonb)` + +Extract the bloom-filter match term from an encrypted value. + +Backs token containment (`@>`) on `text_match` / `text_search` columns. + +| Parameter | Type | Description | +| --- | --- | --- | +| `val` | `jsonb` | the encrypted value | + + +**Returns:** `smallint[]` — the set bit positions +### `ore_block_256(jsonb)` + +Extract the ORE ordering term from an encrypted value. + +Backs range and ordering (`<`, `>`, `ORDER BY`) on `_ord` columns. + +| Parameter | Type | Description | +| --- | --- | --- | +| `val` | `jsonb` | the encrypted value | + + +**Returns:** `eql_v3.ore_block_256` — the ORE term diff --git a/content/docs/reference/eql/meta.json b/content/docs/reference/eql/meta.json index 76a855b..6d448c7 100644 --- a/content/docs/reference/eql/meta.json +++ b/content/docs/reference/eql/meta.json @@ -15,6 +15,8 @@ "sorting", "grouping-and-aggregates", "joins", + "---Reference---", + "functions", "---Previous versions---", "v2" ] diff --git a/package.json b/package.json index d3d8965..215bc8b 100644 --- a/package.json +++ b/package.json @@ -3,7 +3,7 @@ "version": "0.0.0", "private": true, "scripts": { - "prebuild": "bun run generate-docs && bun run generate-docs:eql && bun run generate-docs:cli && bun run validate-links && bun run validate-redirects", + "prebuild": "bun run generate-docs && bun run generate-docs:eql && bun run generate-docs:eql-api && bun run generate-docs:cli && bun run validate-links && bun run validate-redirects", "build": "next build", "dev": "next dev -p 3001", "start": "next start", @@ -13,6 +13,7 @@ "format": "biome format --write", "generate-docs": "tsx scripts/generate-docs.ts", "generate-docs:eql": "tsx scripts/generate-eql-docs.ts", + "generate-docs:eql-api": "tsx scripts/generate-eql-api-docs.ts", "generate-docs:cli": "tsx scripts/generate-cli-docs.ts", "validate-links": "tsx scripts/validate-links.ts", "validate-redirects": "tsx scripts/validate-v2-redirects.ts" diff --git a/scripts/fixtures/eql-manifest.sample.json b/scripts/fixtures/eql-manifest.sample.json new file mode 100644 index 0000000..99ce5d5 --- /dev/null +++ b/scripts/fixtures/eql-manifest.sample.json @@ -0,0 +1,43 @@ +{ + "_note": "ILLUSTRATIVE SAMPLE — shape only. Replaced at build time by the real eql-manifest.json from the eql-docs release asset once cipherstash/encrypt-query-language#364 ships. Do not treat these signatures as authoritative.", + "$schema": "https://schemas.cipherstash.com/eql/manifest/v1.json", + "name": "eql", + "version": "3.0.0-sample", + "generatedFrom": "doxygen-xml", + "counts": { "functions": 3, "public": 3, "private": 0 }, + "functions": [ + { + "name": "hmac_256", + "signature": "hmac_256(jsonb)", + "visibility": "public", + "brief": "Extract the HMAC-SHA-256 equality term from an encrypted value.", + "description": "Backs equality (`=`, `IN`, joins) on `_eq` and `text_search` columns.", + "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], + "returns": { "type": "text", "description": "the HMAC term" }, + "throws": [], "notes": "", "warnings": "", "seeAlso": "", + "source": { "file": "src/hmac_256/functions.sql", "line": 12 } + }, + { + "name": "bloom_filter", + "signature": "bloom_filter(jsonb)", + "visibility": "public", + "brief": "Extract the bloom-filter match term from an encrypted value.", + "description": "Backs token containment (`@>`) on `text_match` / `text_search` columns.", + "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], + "returns": { "type": "smallint[]", "description": "the set bit positions" }, + "throws": [], "notes": "", "warnings": "", "seeAlso": "", + "source": { "file": "src/bloom_filter/functions.sql", "line": 20 } + }, + { + "name": "ore_block_256", + "signature": "ore_block_256(jsonb)", + "visibility": "public", + "brief": "Extract the ORE ordering term from an encrypted value.", + "description": "Backs range and ordering (`<`, `>`, `ORDER BY`) on `_ord` columns.", + "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], + "returns": { "type": "eql_v3.ore_block_256", "description": "the ORE term" }, + "throws": [], "notes": "", "warnings": "", "seeAlso": "", + "source": { "file": "src/ore_block_u64_8_256/functions.sql", "line": 8 } + } + ] +} diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts new file mode 100644 index 0000000..d442a07 --- /dev/null +++ b/scripts/generate-eql-api-docs.ts @@ -0,0 +1,177 @@ +#!/usr/bin/env tsx +/** + * EQL API reference generator + drift guard (docs V2). + * + * Consumes the structured `eql-manifest.json` that the EQL repo emits from its + * Doxygen'd SQL (cipherstash/encrypt-query-language#364) and: + * + * 1. Generates a version-stamped function catalog at + * content/docs/reference/eql/functions.mdx — the exhaustive, drift-proof + * low-level reference the hand-written pedagogical pages link to. + * 2. Drift-lints the hand-written pages: every `eql_v3.<fn>(...)` referenced + * in them should exist in the manifest for the shipped EQL version. This + * is what would have caught the fabricated function names / stale payload + * that slipped into the v2 docs. + * + * ── Manifest source ──────────────────────────────────────────────────────── + * TODAY: reads a committed illustrative sample (shape only), so the generated + * format and the lint are reviewable before the EQL side ships. + * TARGET: once #364 releases, `generate-eql-docs.ts` already downloads the + * `eql-docs-<tag>` asset — extend it to also extract `json/eql-manifest.json` + * and point MANIFEST_PATH at it. The renderer and lint stay identical. + * + * The drift-lint is REPORT-ONLY against the sample (which is tiny, so most real + * symbols read as "unknown"); it becomes a failing gate once wired to the real + * manifest. See STRICT below. + */ +import fs from "node:fs"; +import path from "node:path"; + +const MANIFEST_PATH = path.join( + process.cwd(), + "scripts/fixtures/eql-manifest.sample.json", +); +const EQL_DIR = path.join(process.cwd(), "content/docs/reference/eql"); +const OUT_FILE = path.join(EQL_DIR, "functions.mdx"); +// Flip to true once MANIFEST_PATH points at the real release manifest. +const STRICT = false; + +interface Param { + name: string; + type?: string; + description?: string; +} +interface Fn { + name: string; + signature: string; + visibility: "public" | "private"; + brief: string; + description?: string; + params: Param[]; + returns?: { type?: string; description?: string }; + source?: { file?: string; line?: number }; +} +interface Manifest { + version: string; + functions: Fn[]; +} + +function loadManifest(): Manifest { + return JSON.parse(fs.readFileSync(MANIFEST_PATH, "utf8")); +} + +// ── Render the generated catalog ───────────────────────────────────────────── +function paramsTable(params: Param[]): string { + if (!params.length) return ""; + const rows = params + .map( + (p) => + `| \`${p.name}\` | ${p.type ? `\`${p.type}\`` : ""} | ${(p.description ?? "").replace(/\|/g, "\\|")} |`, + ) + .join("\n"); + return `\n| Parameter | Type | Description |\n| --- | --- | --- |\n${rows}\n`; +} + +function renderFn(fn: Fn): string { + const parts = [`### \`${fn.signature}\``, "", fn.brief]; + if (fn.description && fn.description !== fn.brief) parts.push("", fn.description); + if (fn.params.length) parts.push(paramsTable(fn.params)); + if (fn.returns?.type || fn.returns?.description) { + const t = fn.returns.type ? `\`${fn.returns.type}\`` : ""; + parts.push("", `**Returns:** ${t}${fn.returns.description ? ` — ${fn.returns.description}` : ""}`); + } + return parts.join("\n"); +} + +function render(manifest: Manifest): string { + const version = manifest.version; + const publicFns = manifest.functions.filter((f) => f.visibility === "public"); + const privateFns = manifest.functions.filter((f) => f.visibility === "private"); + + const frontmatter = [ + "---", + "title: Functions", + `description: "Generated catalog of EQL SQL functions and operators (EQL ${version})."`, + "type: reference", + "components: [eql]", + "verifiedAgainst:", + ` eql: "${version}"`, + "---", + ].join("\n"); + + const body = [ + frontmatter, + "", + `{/* GENERATED — do not edit. Produced by scripts/generate-eql-api-docs.ts from the EQL manifest (v${version}). */}`, + "", + `<Callout type="info">`, + `Generated from the **EQL ${version}** manifest (the Doxygen'd SQL is the source of truth). For the model behind these — variants, terms, typed operands — see [Core concepts](/reference/eql/core-concepts).`, + `</Callout>`, + "", + "The complete function surface of the `eql_v3` schema. The type and query pages explain *when* to use these; this page is the exhaustive signature reference they link to.", + "", + "## Functions", + "", + ...publicFns.map(renderFn), + ]; + + if (privateFns.length) { + body.push("", "## Internal functions", "", ...privateFns.map(renderFn)); + } + + return `${body.join("\n").trimEnd()}\n`; +} + +// ── Drift guard ────────────────────────────────────────────────────────────── +function driftCheck(manifest: Manifest): string[] { + const known = new Set(manifest.functions.map((f) => f.name)); + const referenced = new Map<string, Set<string>>(); // symbol -> pages + + for (const file of fs.readdirSync(EQL_DIR)) { + if (!file.endsWith(".mdx") || file === "functions.mdx") continue; + const text = fs.readFileSync(path.join(EQL_DIR, file), "utf8"); + for (const m of text.matchAll(/eql_v3\.([a-z0-9_]+)\s*\(/g)) { + const sym = m[1]; + if (!referenced.has(sym)) referenced.set(sym, new Set()); + referenced.get(sym)!.add(file); + } + } + + const unknown: string[] = []; + for (const [sym, pages] of referenced) { + if (!known.has(sym)) unknown.push(`${sym} (in ${[...pages].join(", ")})`); + } + return unknown.sort(); +} + +// ── Main ───────────────────────────────────────────────────────────────────── +function main() { + const manifest = loadManifest(); + + fs.mkdirSync(EQL_DIR, { recursive: true }); + fs.writeFileSync(OUT_FILE, render(manifest)); + console.log( + `✓ Generated ${path.relative(process.cwd(), OUT_FILE)} from EQL ${manifest.version} (${manifest.functions.length} functions)`, + ); + + const unknown = driftCheck(manifest); + if (unknown.length) { + const header = `⚠ ${unknown.length} eql_v3.* function(s) referenced in hand-written pages are not in the manifest:`; + console.warn(`\n${header}`); + for (const u of unknown) console.warn(` - ${u}`); + if (STRICT) { + console.error( + "\nDrift check failed (STRICT). Fix the reference or update the pinned EQL version.", + ); + process.exit(1); + } else { + console.warn( + "\n(Report-only: using the illustrative sample manifest. Becomes a failing gate once wired to the real release manifest — set STRICT = true.)", + ); + } + } else { + console.log("✓ Drift check: all referenced eql_v3.* functions are in the manifest."); + } +} + +main(); From 1f08e4fea42c648a18bfc32fee145a2a6f5ee6d7 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Mon, 6 Jul 2026 09:59:26 +1000 Subject: [PATCH 30/38] docs(v2): render the EQL domain/variant matrix + extend the drift-lint to domains MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The generated functions.mdx now leads with the encrypted-domain variant matrix (domain × type × variant → capabilities → index terms), consumed from the manifest's new `domains` array (cipherstash/encrypt-query-language#364). The drift-lint now covers domain-type references too, not just function calls. Running it surfaced a real mismatch worth fixing: the hand-written v3 pages reference eql_v3.int4_*/int8_*/float4_*/float8_* domains, but the shipped SQL defines eql_v3.integer_*/bigint_*/real_*/double_* — so casts like $1::eql_v3.int8_ord would fail. Docs-vs-codegen naming needs reconciling; this is exactly the drift the lint exists to catch. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/eql/functions.mdx | 15 +++++++- scripts/fixtures/eql-manifest.sample.json | 34 ++++++++++++++--- scripts/generate-eql-api-docs.ts | 45 +++++++++++++++++++++-- 3 files changed, 83 insertions(+), 11 deletions(-) diff --git a/content/docs/reference/eql/functions.mdx b/content/docs/reference/eql/functions.mdx index 5f1eca2..20b83e3 100644 --- a/content/docs/reference/eql/functions.mdx +++ b/content/docs/reference/eql/functions.mdx @@ -13,7 +13,20 @@ verifiedAgainst: Generated from the **EQL 3.0.0-sample** manifest (the Doxygen'd SQL is the source of truth). For the model behind these — variants, terms, typed operands — see [Core concepts](/reference/eql/core-concepts). </Callout> -The complete function surface of the `eql_v3` schema. The type and query pages explain *when* to use these; this page is the exhaustive signature reference they link to. +The `eql_v3` schema surface — encrypted domains and the functions behind them. The type and query pages explain *when* to use these; this page is the exhaustive reference they link to. + +## Encrypted domains + +A column's capability is declared by its **domain variant**, enforced by a `CHECK` on the required index terms (`hm` equality · `ob`/`op` order · `bf` match · `sv` JSON). See [Core concepts](/reference/eql/core-concepts) for the model. + +| Domain | Type | Variant | Capabilities | Index terms | +| --- | --- | --- | --- | --- | +| `eql_v3.integer` | integer | _(storage only)_ | storage | — | +| `eql_v3.integer_eq` | integer | `_eq` | equality | `hm` | +| `eql_v3.integer_ord` | integer | `_ord` | order | `ob` | +| `eql_v3.integer_ord_ope` | integer | `_ord_ope` | order | `op` | +| `eql_v3.text_match` | text | `_match` | match | `bf` | +| `eql_v3.text_search` | text | `_search` | equality, order, match | `hm` `ob` `bf` | ## Functions diff --git a/scripts/fixtures/eql-manifest.sample.json b/scripts/fixtures/eql-manifest.sample.json index 99ce5d5..68ded54 100644 --- a/scripts/fixtures/eql-manifest.sample.json +++ b/scripts/fixtures/eql-manifest.sample.json @@ -1,10 +1,10 @@ { - "_note": "ILLUSTRATIVE SAMPLE — shape only. Replaced at build time by the real eql-manifest.json from the eql-docs release asset once cipherstash/encrypt-query-language#364 ships. Do not treat these signatures as authoritative.", + "_note": "ILLUSTRATIVE SAMPLE — shape only. Replaced at build time by the real eql-manifest.json from the eql-docs release asset once cipherstash/encrypt-query-language#364 ships. Do not treat these signatures/domains as authoritative.", "$schema": "https://schemas.cipherstash.com/eql/manifest/v1.json", "name": "eql", "version": "3.0.0-sample", - "generatedFrom": "doxygen-xml", - "counts": { "functions": 3, "public": 3, "private": 0 }, + "generatedFrom": "doxygen-xml + sql-domains", + "counts": { "functions": 3, "public": 3, "private": 0, "domains": 6 }, "functions": [ { "name": "hmac_256", @@ -15,7 +15,7 @@ "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], "returns": { "type": "text", "description": "the HMAC term" }, "throws": [], "notes": "", "warnings": "", "seeAlso": "", - "source": { "file": "src/hmac_256/functions.sql", "line": 12 } + "source": { "file": "src/v3/sem/hmac_256/functions.sql", "line": 12 } }, { "name": "bloom_filter", @@ -26,7 +26,7 @@ "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], "returns": { "type": "smallint[]", "description": "the set bit positions" }, "throws": [], "notes": "", "warnings": "", "seeAlso": "", - "source": { "file": "src/bloom_filter/functions.sql", "line": 20 } + "source": { "file": "src/v3/sem/bloom_filter/functions.sql", "line": 20 } }, { "name": "ore_block_256", @@ -37,7 +37,29 @@ "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], "returns": { "type": "eql_v3.ore_block_256", "description": "the ORE term" }, "throws": [], "notes": "", "warnings": "", "seeAlso": "", - "source": { "file": "src/ore_block_u64_8_256/functions.sql", "line": 8 } + "source": { "file": "src/v3/sem/ore_block_256/functions.sql", "line": 8 } } + ], + "domains": [ + { "name": "eql_v3.integer", "type": "integer", "variant": "", "base": "jsonb", + "brief": "Encrypted domain eql_v3.integer.", "terms": [], "capabilities": ["storage"], + "termFunctions": [], "source": { "file": "src/v3/scalars/integer/integer_types.sql", "line": 14 } }, + { "name": "eql_v3.integer_eq", "type": "integer", "variant": "eq", "base": "jsonb", + "brief": "Encrypted domain eql_v3.integer_eq.", "terms": ["hm"], "capabilities": ["equality"], + "termFunctions": ["eql_v3.hmac_256"], "source": { "file": "src/v3/scalars/integer/integer_types.sql", "line": 29 } }, + { "name": "eql_v3.integer_ord", "type": "integer", "variant": "ord", "base": "jsonb", + "brief": "Encrypted domain eql_v3.integer_ord.", "terms": ["ob"], "capabilities": ["order"], + "termFunctions": ["eql_v3.ore_block_256"], "source": { "file": "src/v3/scalars/integer/integer_types.sql", "line": 63 } }, + { "name": "eql_v3.integer_ord_ope", "type": "integer", "variant": "ord_ope", "base": "jsonb", + "brief": "Encrypted domain eql_v3.integer_ord_ope.", "terms": ["op"], "capabilities": ["order"], + "termFunctions": [], "source": { "file": "src/v3/scalars/integer/integer_types.sql", "line": 81 } }, + { "name": "eql_v3.text_match", "type": "text", "variant": "match", "base": "jsonb", + "brief": "Encrypted domain eql_v3.text_match.", "terms": ["bf"], "capabilities": ["match"], + "termFunctions": ["eql_v3.bloom_filter"], "source": { "file": "src/v3/scalars/text/text_types.sql", "line": 44 } }, + { "name": "eql_v3.text_search", "type": "text", "variant": "search", "base": "jsonb", + "brief": "Encrypted domain eql_v3.text_search.", "terms": ["hm", "ob", "bf"], + "capabilities": ["equality", "order", "match"], + "termFunctions": ["eql_v3.hmac_256", "eql_v3.ore_block_256", "eql_v3.bloom_filter"], + "source": { "file": "src/v3/scalars/text/text_types.sql", "line": 60 } } ] } diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index d442a07..e5054dc 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -51,9 +51,19 @@ interface Fn { returns?: { type?: string; description?: string }; source?: { file?: string; line?: number }; } +interface Domain { + name: string; + type: string; + variant: string; + base?: string; + terms: string[]; + capabilities: string[]; + termFunctions?: string[]; +} interface Manifest { version: string; functions: Fn[]; + domains?: Domain[]; } function loadManifest(): Manifest { @@ -83,6 +93,27 @@ function renderFn(fn: Fn): string { return parts.join("\n"); } +function renderDomains(domains: Domain[]): string { + if (!domains.length) return ""; + const rows = domains + .map((d) => { + const variant = d.variant ? `\`_${d.variant}\`` : "_(storage only)_"; + const terms = d.terms.length ? d.terms.map((t) => `\`${t}\``).join(" ") : "—"; + return `| \`${d.name}\` | ${d.type} | ${variant} | ${d.capabilities.join(", ")} | ${terms} |`; + }) + .join("\n"); + return [ + "## Encrypted domains", + "", + "A column's capability is declared by its **domain variant**, enforced by a `CHECK` on the required index terms (`hm` equality · `ob`/`op` order · `bf` match · `sv` JSON). See [Core concepts](/reference/eql/core-concepts) for the model.", + "", + "| Domain | Type | Variant | Capabilities | Index terms |", + "| --- | --- | --- | --- | --- |", + rows, + "", + ].join("\n"); +} + function render(manifest: Manifest): string { const version = manifest.version; const publicFns = manifest.functions.filter((f) => f.visibility === "public"); @@ -108,8 +139,9 @@ function render(manifest: Manifest): string { `Generated from the **EQL ${version}** manifest (the Doxygen'd SQL is the source of truth). For the model behind these — variants, terms, typed operands — see [Core concepts](/reference/eql/core-concepts).`, `</Callout>`, "", - "The complete function surface of the `eql_v3` schema. The type and query pages explain *when* to use these; this page is the exhaustive signature reference they link to.", + "The `eql_v3` schema surface — encrypted domains and the functions behind them. The type and query pages explain *when* to use these; this page is the exhaustive reference they link to.", "", + renderDomains(manifest.domains ?? []), "## Functions", "", ...publicFns.map(renderFn), @@ -124,13 +156,18 @@ function render(manifest: Manifest): string { // ── Drift guard ────────────────────────────────────────────────────────────── function driftCheck(manifest: Manifest): string[] { - const known = new Set(manifest.functions.map((f) => f.name)); + // Known = every function AND every domain (short) name. + const known = new Set<string>([ + ...manifest.functions.map((f) => f.name), + ...(manifest.domains ?? []).map((d) => d.name.replace(/^eql_v3\./, "")), + ]); const referenced = new Map<string, Set<string>>(); // symbol -> pages for (const file of fs.readdirSync(EQL_DIR)) { if (!file.endsWith(".mdx") || file === "functions.mdx") continue; const text = fs.readFileSync(path.join(EQL_DIR, file), "utf8"); - for (const m of text.matchAll(/eql_v3\.([a-z0-9_]+)\s*\(/g)) { + // Any eql_v3.<symbol> — function call, domain cast, or type reference. + for (const m of text.matchAll(/eql_v3\.([a-z0-9_]+)/g)) { const sym = m[1]; if (!referenced.has(sym)) referenced.set(sym, new Set()); referenced.get(sym)!.add(file); @@ -156,7 +193,7 @@ function main() { const unknown = driftCheck(manifest); if (unknown.length) { - const header = `⚠ ${unknown.length} eql_v3.* function(s) referenced in hand-written pages are not in the manifest:`; + const header = `⚠ ${unknown.length} eql_v3.* symbol(s) referenced in hand-written pages are not in the manifest (functions or domains):`; console.warn(`\n${header}`); for (const u of unknown) console.warn(` - ${u}`); if (STRICT) { From 200ee2e6ad4d998173f1ea2abc41ea3da70b022f Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Mon, 6 Jul 2026 10:45:22 +1000 Subject: [PATCH 31/38] docs(v2): switch the domain matrix to the catalog-driven shape Matches the manifest change in encrypt-query-language#364: domains now come from the Rust catalog (`eql-codegen dump-catalog`), carrying authoritative type names and the exact `supportedOperators` per domain (replacing the CHECK-key-derived `terms`/`termFunctions`). The generated variant matrix renders an Operators column; the sample fixture is updated to match. Note `_ord` correctly shows equality + order now (ORE collapses to equality). Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/eql/functions.mdx | 14 +++++++------- scripts/fixtures/eql-manifest.sample.json | 21 +++++++-------------- scripts/generate-eql-api-docs.ts | 13 +++++++------ 3 files changed, 21 insertions(+), 27 deletions(-) diff --git a/content/docs/reference/eql/functions.mdx b/content/docs/reference/eql/functions.mdx index 20b83e3..dce19cd 100644 --- a/content/docs/reference/eql/functions.mdx +++ b/content/docs/reference/eql/functions.mdx @@ -17,16 +17,16 @@ The `eql_v3` schema surface — encrypted domains and the functions behind them. ## Encrypted domains -A column's capability is declared by its **domain variant**, enforced by a `CHECK` on the required index terms (`hm` equality · `ob`/`op` order · `bf` match · `sv` JSON). See [Core concepts](/reference/eql/core-concepts) for the model. +A column's capability is declared by its **domain variant**. This matrix comes straight from the Rust catalog (`eql-codegen dump-catalog`) — the source of truth the SQL is generated from. See [Core concepts](/reference/eql/core-concepts) for the model. -| Domain | Type | Variant | Capabilities | Index terms | +| Domain | Type | Variant | Capabilities | Operators | | --- | --- | --- | --- | --- | | `eql_v3.integer` | integer | _(storage only)_ | storage | — | -| `eql_v3.integer_eq` | integer | `_eq` | equality | `hm` | -| `eql_v3.integer_ord` | integer | `_ord` | order | `ob` | -| `eql_v3.integer_ord_ope` | integer | `_ord_ope` | order | `op` | -| `eql_v3.text_match` | text | `_match` | match | `bf` | -| `eql_v3.text_search` | text | `_search` | equality, order, match | `hm` `ob` `bf` | +| `eql_v3.integer_eq` | integer | `_eq` | equality | `=` `<>` | +| `eql_v3.integer_ord` | integer | `_ord` | equality, order | `=` `<>` `<` `<=` `>` `>=` | +| `eql_v3.integer_ord_ope` | integer | `_ord_ope` | equality, order | `=` `<>` `<` `<=` `>` `>=` | +| `eql_v3.text_match` | text | `_match` | match | `@>` `<@` | +| `eql_v3.text_search` | text | `_search` | equality, order, match | `=` `<>` `<` `<=` `>` `>=` `@>` `<@` | ## Functions diff --git a/scripts/fixtures/eql-manifest.sample.json b/scripts/fixtures/eql-manifest.sample.json index 68ded54..dba9248 100644 --- a/scripts/fixtures/eql-manifest.sample.json +++ b/scripts/fixtures/eql-manifest.sample.json @@ -3,7 +3,7 @@ "$schema": "https://schemas.cipherstash.com/eql/manifest/v1.json", "name": "eql", "version": "3.0.0-sample", - "generatedFrom": "doxygen-xml + sql-domains", + "generatedFrom": "doxygen-xml + catalog", "counts": { "functions": 3, "public": 3, "private": 0, "domains": 6 }, "functions": [ { @@ -42,24 +42,17 @@ ], "domains": [ { "name": "eql_v3.integer", "type": "integer", "variant": "", "base": "jsonb", - "brief": "Encrypted domain eql_v3.integer.", "terms": [], "capabilities": ["storage"], - "termFunctions": [], "source": { "file": "src/v3/scalars/integer/integer_types.sql", "line": 14 } }, + "capabilities": ["storage"], "supportedOperators": [] }, { "name": "eql_v3.integer_eq", "type": "integer", "variant": "eq", "base": "jsonb", - "brief": "Encrypted domain eql_v3.integer_eq.", "terms": ["hm"], "capabilities": ["equality"], - "termFunctions": ["eql_v3.hmac_256"], "source": { "file": "src/v3/scalars/integer/integer_types.sql", "line": 29 } }, + "capabilities": ["equality"], "supportedOperators": ["=", "<>"] }, { "name": "eql_v3.integer_ord", "type": "integer", "variant": "ord", "base": "jsonb", - "brief": "Encrypted domain eql_v3.integer_ord.", "terms": ["ob"], "capabilities": ["order"], - "termFunctions": ["eql_v3.ore_block_256"], "source": { "file": "src/v3/scalars/integer/integer_types.sql", "line": 63 } }, + "capabilities": ["equality", "order"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">="] }, { "name": "eql_v3.integer_ord_ope", "type": "integer", "variant": "ord_ope", "base": "jsonb", - "brief": "Encrypted domain eql_v3.integer_ord_ope.", "terms": ["op"], "capabilities": ["order"], - "termFunctions": [], "source": { "file": "src/v3/scalars/integer/integer_types.sql", "line": 81 } }, + "capabilities": ["equality", "order"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">="] }, { "name": "eql_v3.text_match", "type": "text", "variant": "match", "base": "jsonb", - "brief": "Encrypted domain eql_v3.text_match.", "terms": ["bf"], "capabilities": ["match"], - "termFunctions": ["eql_v3.bloom_filter"], "source": { "file": "src/v3/scalars/text/text_types.sql", "line": 44 } }, + "capabilities": ["match"], "supportedOperators": ["@>", "<@"] }, { "name": "eql_v3.text_search", "type": "text", "variant": "search", "base": "jsonb", - "brief": "Encrypted domain eql_v3.text_search.", "terms": ["hm", "ob", "bf"], "capabilities": ["equality", "order", "match"], - "termFunctions": ["eql_v3.hmac_256", "eql_v3.ore_block_256", "eql_v3.bloom_filter"], - "source": { "file": "src/v3/scalars/text/text_types.sql", "line": 60 } } + "supportedOperators": ["=", "<>", "<", "<=", ">", ">=", "@>", "<@"] } ] } diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index e5054dc..cd5f72d 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -56,9 +56,8 @@ interface Domain { type: string; variant: string; base?: string; - terms: string[]; capabilities: string[]; - termFunctions?: string[]; + supportedOperators?: string[]; } interface Manifest { version: string; @@ -98,16 +97,18 @@ function renderDomains(domains: Domain[]): string { const rows = domains .map((d) => { const variant = d.variant ? `\`_${d.variant}\`` : "_(storage only)_"; - const terms = d.terms.length ? d.terms.map((t) => `\`${t}\``).join(" ") : "—"; - return `| \`${d.name}\` | ${d.type} | ${variant} | ${d.capabilities.join(", ")} | ${terms} |`; + const ops = d.supportedOperators?.length + ? d.supportedOperators.map((o) => `\`${o}\``).join(" ") + : "—"; + return `| \`${d.name}\` | ${d.type} | ${variant} | ${d.capabilities.join(", ")} | ${ops} |`; }) .join("\n"); return [ "## Encrypted domains", "", - "A column's capability is declared by its **domain variant**, enforced by a `CHECK` on the required index terms (`hm` equality · `ob`/`op` order · `bf` match · `sv` JSON). See [Core concepts](/reference/eql/core-concepts) for the model.", + "A column's capability is declared by its **domain variant**. This matrix comes straight from the Rust catalog (`eql-codegen dump-catalog`) — the source of truth the SQL is generated from. See [Core concepts](/reference/eql/core-concepts) for the model.", "", - "| Domain | Type | Variant | Capabilities | Index terms |", + "| Domain | Type | Variant | Capabilities | Operators |", "| --- | --- | --- | --- | --- |", rows, "", From 1e9f6f8eafee9bb1f19295146220272f430d4eab Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Mon, 6 Jul 2026 12:08:17 +1000 Subject: [PATCH 32/38] docs(v2): consume jsonb domains + extractor functions from the manifest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Matches encrypt-query-language#368 (stacked on #364): the manifest now carries the 3 jsonb (SteVec) domains and per-domain `termFunctions`. The variant matrix renders the jsonb domains (json capability), and the drift-lint adds each domain's extractor functions (eq_term / ord_term / match_term / ord_ope_term) to its known set — resolving those false-positives (61 → 57 unknowns on the sample). Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/eql/functions.mdx | 3 +++ scripts/fixtures/eql-manifest.sample.json | 25 ++++++++++++++++------- scripts/generate-eql-api-docs.ts | 8 +++++++- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/content/docs/reference/eql/functions.mdx b/content/docs/reference/eql/functions.mdx index dce19cd..a3b3dc9 100644 --- a/content/docs/reference/eql/functions.mdx +++ b/content/docs/reference/eql/functions.mdx @@ -27,6 +27,9 @@ A column's capability is declared by its **domain variant**. This matrix comes s | `eql_v3.integer_ord_ope` | integer | `_ord_ope` | equality, order | `=` `<>` `<` `<=` `>` `>=` | | `eql_v3.text_match` | text | `_match` | match | `@>` `<@` | | `eql_v3.text_search` | text | `_search` | equality, order, match | `=` `<>` `<` `<=` `>` `>=` `@>` `<@` | +| `eql_v3.json` | jsonb | _(storage only)_ | json | — | +| `eql_v3.jsonb_entry` | jsonb | _(storage only)_ | json | — | +| `eql_v3.jsonb_query` | jsonb | _(storage only)_ | json | — | ## Functions diff --git a/scripts/fixtures/eql-manifest.sample.json b/scripts/fixtures/eql-manifest.sample.json index dba9248..e08d50d 100644 --- a/scripts/fixtures/eql-manifest.sample.json +++ b/scripts/fixtures/eql-manifest.sample.json @@ -4,7 +4,7 @@ "name": "eql", "version": "3.0.0-sample", "generatedFrom": "doxygen-xml + catalog", - "counts": { "functions": 3, "public": 3, "private": 0, "domains": 6 }, + "counts": { "functions": 3, "public": 3, "private": 0, "domains": 9 }, "functions": [ { "name": "hmac_256", @@ -42,17 +42,28 @@ ], "domains": [ { "name": "eql_v3.integer", "type": "integer", "variant": "", "base": "jsonb", - "capabilities": ["storage"], "supportedOperators": [] }, + "capabilities": ["storage"], "supportedOperators": [], "termFunctions": [] }, { "name": "eql_v3.integer_eq", "type": "integer", "variant": "eq", "base": "jsonb", - "capabilities": ["equality"], "supportedOperators": ["=", "<>"] }, + "capabilities": ["equality"], "supportedOperators": ["=", "<>"], + "termFunctions": ["eql_v3.eq_term"] }, { "name": "eql_v3.integer_ord", "type": "integer", "variant": "ord", "base": "jsonb", - "capabilities": ["equality", "order"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">="] }, + "capabilities": ["equality", "order"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">="], + "termFunctions": ["eql_v3.ord_term"] }, { "name": "eql_v3.integer_ord_ope", "type": "integer", "variant": "ord_ope", "base": "jsonb", - "capabilities": ["equality", "order"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">="] }, + "capabilities": ["equality", "order"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">="], + "termFunctions": ["eql_v3.ord_ope_term"] }, { "name": "eql_v3.text_match", "type": "text", "variant": "match", "base": "jsonb", - "capabilities": ["match"], "supportedOperators": ["@>", "<@"] }, + "capabilities": ["match"], "supportedOperators": ["@>", "<@"], + "termFunctions": ["eql_v3.match_term"] }, { "name": "eql_v3.text_search", "type": "text", "variant": "search", "base": "jsonb", "capabilities": ["equality", "order", "match"], - "supportedOperators": ["=", "<>", "<", "<=", ">", ">=", "@>", "<@"] } + "supportedOperators": ["=", "<>", "<", "<=", ">", ">=", "@>", "<@"], + "termFunctions": ["eql_v3.eq_term", "eql_v3.ord_term", "eql_v3.match_term"] }, + { "name": "eql_v3.json", "type": "jsonb", "variant": "", "base": "jsonb", "shape": "stevec", + "capabilities": ["json"], "supportedOperators": [], "termFunctions": [] }, + { "name": "eql_v3.jsonb_entry", "type": "jsonb", "variant": "", "base": "jsonb", "shape": "stevec", + "capabilities": ["json"], "supportedOperators": [], "termFunctions": [] }, + { "name": "eql_v3.jsonb_query", "type": "jsonb", "variant": "", "base": "jsonb", "shape": "stevec", + "capabilities": ["json"], "supportedOperators": [], "termFunctions": [] } ] } diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index cd5f72d..665682d 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -58,6 +58,8 @@ interface Domain { base?: string; capabilities: string[]; supportedOperators?: string[]; + termFunctions?: string[]; + shape?: string; } interface Manifest { version: string; @@ -157,10 +159,14 @@ function render(manifest: Manifest): string { // ── Drift guard ────────────────────────────────────────────────────────────── function driftCheck(manifest: Manifest): string[] { - // Known = every function AND every domain (short) name. + // Known = every function, every domain (short) name, and every domain's + // extractor functions (eq_term / ord_term / …, authoritative from the catalog). const known = new Set<string>([ ...manifest.functions.map((f) => f.name), ...(manifest.domains ?? []).map((d) => d.name.replace(/^eql_v3\./, "")), + ...(manifest.domains ?? []).flatMap((d) => + (d.termFunctions ?? []).map((fn) => fn.replace(/^eql_v3\./, "")), + ), ]); const referenced = new Map<string, Set<string>>(); // symbol -> pages From 7e5f3019be3f9ee1e2b5f0215fd71e61b0e4c609 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Mon, 6 Jul 2026 17:48:36 +1000 Subject: [PATCH 33/38] docs(eql): schema-aware drift-lint + sweep pages to the shipped surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wires the v2 EQL reference to the corrected manifest (public. domains, eql_v3./eql_v3_internal. functions — cipherstash/encrypt-query-language#369) and turns the drift-lint into a real gate. - generate-eql-api-docs.ts: schema-aware drift-lint — matches public.<domain>, eql_v3.<fn>, eql_v3_internal.<fn> by schema AND name, so right-name/wrong-schema refs are caught, not just fabricated ones. Group public overloads by name (the raw 1680-function surface -> 39 functions); omit internal functions from the page (kept in the lint's known set). Manifest source: EQL_MANIFEST_PATH -> release cache -> sample, auto-strict against any real manifest. - generate-eql-docs.ts: extract json/eql-manifest.json from the release tarball to a gitignored cache the reference generator reads (best-effort; falls back to the sample until a release ships the manifest). - Swept content/docs/reference/eql/*.mdx (165 refs): domains -> public., internal ctors -> eql_v3_internal., PG aliases -> canonical catalog names (int4->integer, float8->double, bool->boolean, ste_vec_*->jsonb_*). - Updated the illustrative sample fixture to the corrected schema. Drift-lint verified green against a locally-generated corrected manifest; report-only against the committed sample until a release ships the manifest. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- .gitignore | 2 + content/docs/reference/eql/booleans.mdx | 22 +- content/docs/reference/eql/core-concepts.mdx | 32 +-- .../docs/reference/eql/dates-and-times.mdx | 22 +- content/docs/reference/eql/filtering.mdx | 38 ++-- content/docs/reference/eql/functions.mdx | 61 +++--- .../reference/eql/grouping-and-aggregates.mdx | 4 +- content/docs/reference/eql/indexes.mdx | 16 +- content/docs/reference/eql/joins.mdx | 10 +- content/docs/reference/eql/json.mdx | 38 ++-- content/docs/reference/eql/numbers.mdx | 28 +-- content/docs/reference/eql/sorting.mdx | 4 +- content/docs/reference/eql/text.mdx | 38 ++-- scripts/fixtures/eql-manifest.sample.json | 193 ++++++++++++++---- scripts/generate-eql-api-docs.ts | 172 +++++++++++----- scripts/generate-eql-docs.ts | 19 ++ 16 files changed, 456 insertions(+), 243 deletions(-) diff --git a/.gitignore b/.gitignore index 7d245b9..856a089 100644 --- a/.gitignore +++ b/.gitignore @@ -42,3 +42,5 @@ content/stack/reference/drizzle/v*/ # serena (local MCP tooling state) .serena/ +# EQL release manifest extracted by generate-eql-docs.ts (see generate-eql-api-docs.ts) +.eql-manifest.release.json diff --git a/content/docs/reference/eql/booleans.mdx b/content/docs/reference/eql/booleans.mdx index 5403fc9..6aad0f9 100644 --- a/content/docs/reference/eql/booleans.mdx +++ b/content/docs/reference/eql/booleans.mdx @@ -1,13 +1,13 @@ --- title: Booleans -description: "Encrypted booleans are storage-only by design: eql_v3.bool stores and decrypts, carries no index terms, and blocks every comparison." +description: "Encrypted booleans are storage-only by design: public.boolean stores and decrypts, carries no index terms, and blocks every comparison." type: reference components: [eql] verifiedAgainst: eql: "3.0.0" --- -Every scalar type has a storage-only variant — for `bool` it's the only one. EQL ships `eql_v3.bool` and nothing else: there is no `bool_eq` and no `bool_ord`. An encrypted boolean column can be stored, decrypted, and null-checked; it cannot be filtered, sorted, grouped, or joined on. +Every scalar type has a storage-only variant — for `bool` it's the only one. EQL ships `public.boolean` and nothing else: there is no `bool_eq` and no `bool_ord`. An encrypted boolean column can be stored, decrypted, and null-checked; it cannot be filtered, sorted, grouped, or joined on. ## Why there are no query variants @@ -15,11 +15,11 @@ A two-value column has too little cardinality for any searchable index to be saf ## What works, what raises -`eql_v3.bool` follows the bare-variant contract described in [Core concepts](/reference/eql/core-concepts#variants-declare-capability): it carries no index terms, so `IS NULL` / `IS NOT NULL` are the only predicates that work. Every comparison operator routes to a blocker and raises — the [fail-loud behavior](/reference/eql/core-concepts#unsupported-operations-fail-loudly) shared by all encrypted variants: +`public.boolean` follows the bare-variant contract described in [Core concepts](/reference/eql/core-concepts#variants-declare-capability): it carries no index terms, so `IS NULL` / `IS NOT NULL` are the only predicates that work. Every comparison operator routes to a blocker and raises — the [fail-loud behavior](/reference/eql/core-concepts#unsupported-operations-fail-loudly) shared by all encrypted variants: ```sql --- ❌ Raises: operator = is not supported for eql_v3.bool -SELECT * FROM users WHERE is_active = $1::eql_v3.bool; +-- ❌ Raises: operator = is not supported for public.boolean +SELECT * FROM users WHERE is_active = $1::public.boolean; -- ✅ Works: NULL columns are not encrypted SELECT * FROM users WHERE is_active IS NOT NULL; @@ -32,16 +32,16 @@ Query on other columns, decrypt the boolean in your application, and filter ther ```sql CREATE TABLE users ( id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v3.text_eq, -- exact lookup - created_at eql_v3.timestamp_ord, -- range queries, ORDER BY - is_active eql_v3.bool -- storage only (by design) + email public.text_eq, -- exact lookup + created_at public.timestamp_ord, -- range queries, ORDER BY + is_active public.boolean -- storage only (by design) ); ``` ```sql -- Narrow the result set with the columns that do carry index terms… SELECT id, email, is_active FROM users -WHERE created_at >= $1::eql_v3.timestamp_ord; +WHERE created_at >= $1::public.timestamp_ord; -- …then decrypt is_active in the client and filter on the plaintext. ``` @@ -51,12 +51,12 @@ If a boolean genuinely needs to be a server-side predicate, that is a data-model ## Storing without querying -`bool` is the forced case of a pattern available to every scalar type: the bare variant `eql_v3.<T>` (for example `eql_v3.int4`, `eql_v3.text`, `eql_v3.timestamp`) is storage-and-decryption only. It carries no index terms, and every comparison operator raises — use it for columns you only ever store and decrypt, so the database holds no searchable material for them at all. +`bool` is the forced case of a pattern available to every scalar type: the bare variant `eql_v3.<T>` (for example `public.integer`, `public.text`, `public.timestamp`) is storage-and-decryption only. It carries no index terms, and every comparison operator raises — use it for columns you only ever store and decrypt, so the database holds no searchable material for them at all. For every type other than `bool`, storage-only is a choice you can walk back. If you later need to query, retype the column as a query variant — or, if the payloads already carry the needed term (the client decides which terms travel in the payload), cast at the call site: ```sql -SELECT * FROM readings WHERE value::eql_v3.int4_ord > $1::eql_v3.int4_ord; +SELECT * FROM readings WHERE value::public.integer_ord > $1::public.integer_ord; ``` The variant families and what each one enables are covered in [Core concepts](/reference/eql/core-concepts); the per-type specifics live in [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), and [Text](/reference/eql/text). diff --git a/content/docs/reference/eql/core-concepts.mdx b/content/docs/reference/eql/core-concepts.mdx index 427bc7b..63ce2f1 100644 --- a/content/docs/reference/eql/core-concepts.mdx +++ b/content/docs/reference/eql/core-concepts.mdx @@ -23,8 +23,8 @@ For any scalar type `<T>`, the family looks like this: | `eql_v3.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | | `eql_v3.<T>_ord` | Comparisons (`<` … `>=`), `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | | `eql_v3.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | -| `eql_v3.text_match` (text only) | Free-text token containment: `@>` / `<@`. | -| `eql_v3.text_search` (text only) | Equality + ordering + token containment. | +| `public.text_match` (text only) | Free-text token containment: `@>` / `<@`. | +| `public.text_search` (text only) | Equality + ordering + token containment. | Two things worth calling out: @@ -45,13 +45,13 @@ Declaring a table is just typing each column as the variant it needs: ```sql CREATE TABLE users ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v3.text_eq, -- equality only - salary eql_v3.int4_ord, -- equality + range + ORDER BY - created_at eql_v3.timestamp_ord + email public.text_eq, -- equality only + salary public.integer_ord, -- equality + range + ORDER BY + created_at public.timestamp_ord ); ``` -Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.0 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use a separate domain, `eql_v3.json`, with its own operator surface — see [JSON](/reference/eql/json). +Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.0 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use a separate domain, `public.json`, with its own operator surface — see [JSON](/reference/eql/json). ## Anatomy of an encrypted value @@ -81,13 +81,13 @@ Alongside the envelope, a payload carries the index terms for its column's capab | Key | SEM type | Wire shape | Enables | Reveals | | --- | --- | --- | --- | --- | -| `hm` | `eql_v3.hmac_256` (domain over `text`) | Hex string (HMAC-SHA-256) | `=`, `<>` on `_eq` and `text_search` domains | Whether two values are equal — nothing else | -| `ob` | `eql_v3.ore_block_256` (composite: array of `bytea` block terms) | Array of hex-encoded ORE blocks (block count varies by scalar width) | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ore` domains — and `=` / `<>`, since ORE comparison collapses to equality | The relative order of two values | -| `bf` | `eql_v3.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit — large filters emit negative positions) | `@>` / `<@` token containment on `_match` domains | Probabilistic token overlap between values | +| `hm` | `eql_v3_internal.hmac_256` (domain over `text`) | Hex string (HMAC-SHA-256) | `=`, `<>` on `_eq` and `text_search` domains | Whether two values are equal — nothing else | +| `ob` | `eql_v3_internal.ore_block_256` (composite: array of `bytea` block terms) | Array of hex-encoded ORE blocks (block count varies by scalar width) | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ore` domains — and `=` / `<>`, since ORE comparison collapses to equality | The relative order of two values | +| `bf` | `eql_v3_internal.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit — large filters emit negative positions) | `@>` / `<@` token containment on `_match` domains | Probabilistic token overlap between values | -The capability is encoded as **required keys**: the payload for an `eql_v3.text_eq` column must carry `hm`; an `eql_v3.int4_ord` payload must carry `ob` (and only `ob`); a `text_match` payload must carry `bf`; a `text_search` payload carries all three. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. +The capability is encoded as **required keys**: the payload for an `public.text_eq` column must carry `hm`; an `public.integer_ord` payload must carry `ob` (and only `ob`); a `text_match` payload must carry `bf`; a `text_search` payload carries all three. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. -A scalar payload for an `eql_v3.text_search` column (lookup + ordering + free-text match, so all three terms are required): +A scalar payload for an `public.text_search` column (lookup + ordering + free-text match, so all three terms are required): ```json { @@ -124,10 +124,10 @@ The `eql_v3` domains are backed by `jsonb`. When an operand has no known type SELECT * FROM users WHERE email = $1; -- ✅ Right: typed operand — the encrypted `=` resolves. -SELECT * FROM users WHERE email = $1::eql_v3.text_eq; +SELECT * FROM users WHERE email = $1::public.text_eq; ``` -Always type the operand: a typed parameter (`$1::eql_v3.text_eq`) or an explicit cast (`'…'::eql_v3.int4_ord`). The [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) type bound parameters automatically — raw SQL must do it by hand. +Always type the operand: a typed parameter (`$1::public.text_eq`) or an explicit cast (`'…'::public.integer_ord`). The [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) type bound parameters automatically — raw SQL must do it by hand. This is the one place where a mistake is *silent*. Everything else fails loudly: @@ -136,9 +136,9 @@ This is the one place where a mistake is *silent*. Everything else fails loudly: Unsupported operators are not silent no-ops. Every operator that a variant doesn't support is still *defined* — it routes to a blocker function that raises an `operator … is not supported` exception. A mis-typed query fails loudly instead of silently returning wrong results: ```sql --- salary is eql_v3.int8_eq (equality only) -SELECT * FROM users WHERE salary > $1::eql_v3.int8_eq; --- ERROR: operator > is not supported for eql_v3.int8_eq +-- salary is public.bigint_eq (equality only) +SELECT * FROM users WHERE salary > $1::public.bigint_eq; +-- ERROR: operator > is not supported for public.bigint_eq ``` A `NULL` operand still raises — the blockers are deliberately not `STRICT`, so PostgreSQL can't skip the check. (A SQL `NULL` column value is not encrypted, so `IS NULL` / `IS NOT NULL` themselves always work, on every variant.) diff --git a/content/docs/reference/eql/dates-and-times.mdx b/content/docs/reference/eql/dates-and-times.mdx index 3af2b9c..0fa3d25 100644 --- a/content/docs/reference/eql/dates-and-times.mdx +++ b/content/docs/reference/eql/dates-and-times.mdx @@ -24,8 +24,8 @@ And every concrete domain this page covers: | Type | Variants | | --- | --- | -| `date` | `eql_v3.date` · `eql_v3.date_eq` · `eql_v3.date_ord` · `eql_v3.date_ord_ore` | -| `timestamp` | `eql_v3.timestamp` · `eql_v3.timestamp_eq` · `eql_v3.timestamp_ord` · `eql_v3.timestamp_ord_ore` | +| `date` | `public.date` · `public.date_eq` · `public.date_ord` · `public.date_ord_ore` | +| `timestamp` | `public.timestamp` · `public.timestamp_eq` · `public.timestamp_ord` · `public.timestamp_ord_ore` | Time columns are nearly always ranged and sorted, so `_ord` is the usual choice. Declare only the capability you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)), and the variant model itself is covered in [Core concepts](/reference/eql/core-concepts). @@ -36,9 +36,9 @@ An audit-events table where the timestamps drive time-window queries and sorting ```sql CREATE TABLE audit_events ( id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - occurred_at eql_v3.timestamp_ord, -- time windows, newest-first, MIN/MAX - review_due eql_v3.date_ord, -- range filters - sealed_on eql_v3.date -- store and decrypt only + occurred_at public.timestamp_ord, -- time windows, newest-first, MIN/MAX + review_due public.date_ord, -- range filters + sealed_on public.date -- store and decrypt only ); ``` @@ -55,7 +55,7 @@ The EQL v3 release adds an OPE specifier for every orderable type; unspecified ` ## Payload -A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus the `ob` ordering term. Here is a payload for the `eql_v3.timestamp_ord` `occurred_at` column: +A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus the `ob` ordering term. Here is a payload for the `public.timestamp_ord` `occurred_at` column: ```json { @@ -85,7 +85,7 @@ A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — | `ORDER BY` | ❌ | ❌ | ✅ | | `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | -Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::eql_v3.timestamp_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). +Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::public.timestamp_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). ## Functions @@ -103,17 +103,17 @@ Every operator has a function form, for managed platforms that disallow custom o ```sql SELECT * FROM audit_events -WHERE occurred_at BETWEEN $1::eql_v3.timestamp_ord AND $2::eql_v3.timestamp_ord; +WHERE occurred_at BETWEEN $1::public.timestamp_ord AND $2::public.timestamp_ord; SELECT * FROM audit_events -WHERE review_due BETWEEN $1::eql_v3.date_ord AND $2::eql_v3.date_ord; +WHERE review_due BETWEEN $1::public.date_ord AND $2::public.date_ord; ``` ### Retention cutoff ```sql SELECT id FROM audit_events -WHERE occurred_at < $1::eql_v3.timestamp_ord; +WHERE occurred_at < $1::public.timestamp_ord; ``` ### Newest-first listing @@ -122,7 +122,7 @@ Write the sort key in extractor form to stream rows out of the index already ord ```sql SELECT * FROM audit_events -WHERE occurred_at >= $1::eql_v3.timestamp_ord +WHERE occurred_at >= $1::public.timestamp_ord ORDER BY eql_v3.ord_term(occurred_at) DESC LIMIT 10; ``` diff --git a/content/docs/reference/eql/filtering.mdx b/content/docs/reference/eql/filtering.mdx index 9fe45a2..9dc4394 100644 --- a/content/docs/reference/eql/filtering.mdx +++ b/content/docs/reference/eql/filtering.mdx @@ -7,25 +7,25 @@ verifiedAgainst: eql: "3.0.0" --- -Every filter below is ordinary SQL — the encrypted operators resolve from the column's domain variant, and a functional index on the matching term extractor serves the predicate. One rule applies throughout: **operands must be typed** (`$1::eql_v3.text_eq`, not a bare literal), or PostgreSQL falls through to native `jsonb` semantics. See [Core concepts](/reference/eql/core-concepts) for the typed-operand rule and how unsupported operators fail loudly instead of returning wrong rows. +Every filter below is ordinary SQL — the encrypted operators resolve from the column's domain variant, and a functional index on the matching term extractor serves the predicate. One rule applies throughout: **operands must be typed** (`$1::public.text_eq`, not a bare literal), or PostgreSQL falls through to native `jsonb` semantics. See [Core concepts](/reference/eql/core-concepts) for the typed-operand rule and how unsupported operators fail loudly instead of returning wrong rows. ## Equality: `=` and `<>` Works on `_eq` and `_ord` / `_ord_ore` variants of every scalar, and on `text_search`: ```sql -SELECT * FROM users WHERE email = $1::eql_v3.text_eq; -SELECT * FROM users WHERE tax_id <> $1::eql_v3.text_eq; +SELECT * FROM users WHERE email = $1::public.text_eq; +SELECT * FROM users WHERE tax_id <> $1::public.text_eq; ``` On `_eq` and `text_search` columns equality compares the HMAC (`hm`) term. On `_ord` variants there is no `hm` — equality compares the ORE (`ob`) term, which collapses to equality, so `_ord` columns get `=` and `<>` for free. See [Core concepts](/reference/eql/core-concepts) for the mechanism. ```sql --- salary is eql_v3.int8_ord: equality works without an hm term -SELECT * FROM users WHERE salary = $1::eql_v3.int8_ord; +-- salary is public.bigint_ord: equality works without an hm term +SELECT * FROM users WHERE salary = $1::public.bigint_ord; ``` -Bare storage-only variants (`eql_v3.text`, `eql_v3.int4`, …) block every comparison — see the type pages for what each variant supports: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), [Booleans](/reference/eql/booleans). +Bare storage-only variants (`public.text`, `public.integer`, …) block every comparison — see the type pages for what each variant supports: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), [Booleans](/reference/eql/booleans). ## `IN` lists @@ -33,7 +33,7 @@ Bare storage-only variants (`eql_v3.text`, `eql_v3.int4`, …) block every compa ```sql SELECT * FROM users -WHERE email IN ($1::eql_v3.text_eq, $2::eql_v3.text_eq, $3::eql_v3.text_eq); +WHERE email IN ($1::public.text_eq, $2::public.text_eq, $3::public.text_eq); ``` There is no way to encrypt a list as one value — the client encrypts each element and binds it as its own parameter. `IN (subquery)` also works, subject to the same-keyset rule covered in [Joins](/reference/eql/joins). @@ -43,19 +43,19 @@ There is no way to encrypt a list as one value — the client encrypts each elem `<`, `<=`, `>`, `>=` work on `_ord` / `_ord_ore` variants and `text_search` — the variants carrying an ORE (`ob`) term: ```sql -SELECT * FROM users WHERE salary >= $1::eql_v3.int8_ord; +SELECT * FROM users WHERE salary >= $1::public.bigint_ord; -- BETWEEN desugars to >= and <= SELECT * FROM users -WHERE created_at BETWEEN $1::eql_v3.timestamp_ord AND $2::eql_v3.timestamp_ord; +WHERE created_at BETWEEN $1::public.timestamp_ord AND $2::public.timestamp_ord; ``` Half-open ranges compose the same way: ```sql SELECT * FROM events -WHERE occurred_at >= $1::eql_v3.timestamp_ord - AND occurred_at < $2::eql_v3.timestamp_ord; +WHERE occurred_at >= $1::public.timestamp_ord + AND occurred_at < $2::public.timestamp_ord; ``` ## Text token matching: `@>` @@ -63,25 +63,25 @@ WHERE occurred_at >= $1::eql_v3.timestamp_ord There is no `LIKE` on encrypted columns — encrypted free-text matching is bloom-filter token containment via `@>` on a `text_match` or `text_search` column: ```sql -SELECT * FROM users WHERE name @> $1::eql_v3.text_match; +SELECT * FROM users WHERE name @> $1::public.text_match; ``` The client encrypts the search term into a bloom-filter query value; matching is probabilistic (false positives possible, false negatives not). For the full no-`LIKE` story and match-term tuning, see [Text](/reference/eql/text). ## JSON containment and path filters -Encrypted JSON documents (`eql_v3.json`) filter by containment and path existence: +Encrypted JSON documents (`public.json`) filter by containment and path existence: ```sql -- Does the document contain this (encrypted) structure? -SELECT * FROM orders WHERE metadata @> $1::eql_v3.ste_vec_query; +SELECT * FROM orders WHERE metadata @> $1::public.jsonb_query; -- Does this path exist in the document? SELECT * FROM orders WHERE eql_v3.jsonb_path_exists(metadata, 'region_selector'); -- Equality on an extracted leaf SELECT * FROM orders -WHERE metadata -> 'email_selector'::text = $1::eql_v3.ste_vec_entry; +WHERE metadata -> 'email_selector'::text = $1::public.jsonb_entry; ``` Field access is by selector hash, not plaintext path. The full JSON surface — containment, field access, path queries, and range filters on extracted leaves — is in [JSON](/reference/eql/json). @@ -93,9 +93,9 @@ Encrypted predicates compose with `AND`, `OR`, `NOT`, and parentheses like any o ```sql SELECT * FROM users WHERE status = 'active' -- plaintext column, native operator - AND created_at >= $1::eql_v3.timestamp_ord -- encrypted range - AND (email = $2::eql_v3.text_eq -- encrypted equality - OR name @> $3::eql_v3.text_match); -- encrypted token match + AND created_at >= $1::public.timestamp_ord -- encrypted range + AND (email = $2::public.text_eq -- encrypted equality + OR name @> $3::public.text_match); -- encrypted token match ``` The planner treats each encrypted predicate independently, so it can combine an index on a plaintext column with a functional index on an encrypted one (bitmap-AND, or whichever plan is cheapest). @@ -118,7 +118,7 @@ Don't confuse this with a JSON `null` *inside* an encrypted document, which is a | Equality | `=` `<>` `IN` | `_eq`, `_ord` / `_ord_ore`, `text_search` | hash (or btree) on `eql_v3.eq_term` — btree on `eql_v3.ord_term` for `_ord` | | Range | `<` `<=` `>` `>=` `BETWEEN` | `_ord` / `_ord_ore`, `text_search` | btree on `eql_v3.ord_term` | | Text token match | `@>` `<@` | `text_match`, `text_search` | GIN on `eql_v3.match_term` | -| JSON containment | `@>` `<@` | `eql_v3.json` | GIN on `eql_v3.to_ste_vec_query(col)::jsonb` | +| JSON containment | `@>` `<@` | `public.json` | GIN on `eql_v3.to_ste_vec_query(col)::jsonb` | | Null check | `IS NULL` / `IS NOT NULL` | every variant | — | Every one of these has a full index recipe — which method, which extractor, and how to confirm the index engages with `EXPLAIN` — in [Indexes](/reference/eql/indexes). diff --git a/content/docs/reference/eql/functions.mdx b/content/docs/reference/eql/functions.mdx index a3b3dc9..e9f86a5 100644 --- a/content/docs/reference/eql/functions.mdx +++ b/content/docs/reference/eql/functions.mdx @@ -13,7 +13,7 @@ verifiedAgainst: Generated from the **EQL 3.0.0-sample** manifest (the Doxygen'd SQL is the source of truth). For the model behind these — variants, terms, typed operands — see [Core concepts](/reference/eql/core-concepts). </Callout> -The `eql_v3` schema surface — encrypted domains and the functions behind them. The type and query pages explain *when* to use these; this page is the exhaustive reference they link to. +The EQL SQL surface — encrypted domains (in the `public` schema) and the `eql_v3` functions behind them. The type and query pages explain *when* to use these; this page is the exhaustive reference they link to. ## Encrypted domains @@ -21,51 +21,50 @@ A column's capability is declared by its **domain variant**. This matrix comes s | Domain | Type | Variant | Capabilities | Operators | | --- | --- | --- | --- | --- | -| `eql_v3.integer` | integer | _(storage only)_ | storage | — | -| `eql_v3.integer_eq` | integer | `_eq` | equality | `=` `<>` | -| `eql_v3.integer_ord` | integer | `_ord` | equality, order | `=` `<>` `<` `<=` `>` `>=` | -| `eql_v3.integer_ord_ope` | integer | `_ord_ope` | equality, order | `=` `<>` `<` `<=` `>` `>=` | -| `eql_v3.text_match` | text | `_match` | match | `@>` `<@` | -| `eql_v3.text_search` | text | `_search` | equality, order, match | `=` `<>` `<` `<=` `>` `>=` `@>` `<@` | -| `eql_v3.json` | jsonb | _(storage only)_ | json | — | -| `eql_v3.jsonb_entry` | jsonb | _(storage only)_ | json | — | -| `eql_v3.jsonb_query` | jsonb | _(storage only)_ | json | — | +| `public.integer` | integer | _(storage only)_ | storage | — | +| `public.integer_eq` | integer | `_eq` | equality | `=` `<>` | +| `public.integer_ord` | integer | `_ord` | equality, order | `=` `<>` `<` `<=` `>` `>=` | +| `public.integer_ord_ope` | integer | `_ord_ope` | equality, order | `=` `<>` `<` `<=` `>` `>=` | +| `public.text_match` | text | `_match` | match | `@>` `<@` | +| `public.text_search` | text | `_search` | equality, order, match | `=` `<>` `<` `<=` `>` `>=` `@>` `<@` | +| `public.json` | jsonb | _(storage only)_ | json | — | +| `public.jsonb_entry` | jsonb | _(storage only)_ | json | — | +| `public.jsonb_query` | jsonb | _(storage only)_ | json | — | ## Functions -### `hmac_256(jsonb)` +The public `eql_v3` API — 2 functions (2 overloads). Internal `eql_v3_internal` functions (3) are implementation detail and omitted. -Extract the HMAC-SHA-256 equality term from an encrypted value. +### `jsonb_path_query` -Backs equality (`=`, `IN`, joins) on `_eq` and `text_search` columns. +Query encrypted JSONB for sv elements matching a selector. -| Parameter | Type | Description | -| --- | --- | --- | -| `val` | `jsonb` | the encrypted value | - - -**Returns:** `text` — the HMAC term -### `bloom_filter(jsonb)` +Returns one jsonb_entry row per matching encrypted element. -Extract the bloom-filter match term from an encrypted value. +**Signature:** -Backs token containment (`@>`) on `text_match` / `text_search` columns. +```sql +jsonb_path_query(jsonb, text) +``` | Parameter | Type | Description | | --- | --- | --- | -| `val` | `jsonb` | the encrypted value | +| `val` | `jsonb` | encrypted EQL payload with sv | +| `selector` | `text` | selector hash | -**Returns:** `smallint[]` — the set bit positions -### `ore_block_256(jsonb)` +**Returns:** `public.jsonb_entry` — matching encrypted entries -Extract the ORE ordering term from an encrypted value. +### `version` -Backs range and ordering (`<`, `>`, `ORDER BY`) on `_ord` columns. +Get the installed EQL version string. -| Parameter | Type | Description | -| --- | --- | --- | -| `val` | `jsonb` | the encrypted value | +Returns the version string for the installed EQL library. + +**Signature:** +```sql +version() +``` -**Returns:** `eql_v3.ore_block_256` — the ORE term +**Returns:** `text` — the version string diff --git a/content/docs/reference/eql/grouping-and-aggregates.mdx b/content/docs/reference/eql/grouping-and-aggregates.mdx index 544a91c..da735e7 100644 --- a/content/docs/reference/eql/grouping-and-aggregates.mdx +++ b/content/docs/reference/eql/grouping-and-aggregates.mdx @@ -39,7 +39,7 @@ Note the trade-off: grouping on `eq_term` returns the *term*, not the encrypted Plain `COUNT(col)` counts non-`NULL` rows — it never compares values, so it works on **any** variant, including storage-only ones: ```sql -SELECT COUNT(tax_id) FROM users; -- works even on bare eql_v3.text +SELECT COUNT(tax_id) FROM users; -- works even on bare public.text ``` `COUNT(DISTINCT col)` deduplicates, so it needs an equality-capable variant — and the same extractor advice applies: @@ -74,7 +74,7 @@ SELECT eql_v3.eq_term(department_code) AS dept, eql_v3.max(salary) If the column is generic `jsonb` rather than a domain, cast to the right variant at the call site so overload resolution can pick the aggregate: ```sql -SELECT eql_v3.min(salary_jsonb::eql_v3.int8_ord) FROM users; +SELECT eql_v3.min(salary_jsonb::public.bigint_ord) FROM users; ``` A btree on `eql_v3.ord_term(col)` serves `MIN` / `MAX` — the [Indexes](/reference/eql/indexes) page has the recipe. diff --git a/content/docs/reference/eql/indexes.mdx b/content/docs/reference/eql/indexes.mdx index ff4b1fe..71fd42a 100644 --- a/content/docs/reference/eql/indexes.mdx +++ b/content/docs/reference/eql/indexes.mdx @@ -18,7 +18,7 @@ EQL indexes are ordinary PostgreSQL functional indexes over **term-extractor fun The extractors are inlinable SQL functions, so the planner rewrites a bare-form predicate into the same expression the index was built on. You don't rewrite queries to use the index: ```sql -SELECT * FROM users WHERE email = $1::eql_v3.text_eq; +SELECT * FROM users WHERE email = $1::public.text_eq; -- planner inlines `=` to: eql_v3.eq_term(email) = eql_v3.eq_term($1) -- Index Cond on USING hash (eql_v3.eq_term(email)) ``` @@ -44,7 +44,7 @@ CREATE INDEX users_created_at_ord ON users USING btree (eql_v3.ord_term(created_at)); -- Text match: GIN index on match_term --- (columns typed eql_v3.text_match or text_search) +-- (columns typed public.text_match or text_search) CREATE INDEX users_name_match ON users USING gin (eql_v3.match_term(name)); @@ -65,7 +65,7 @@ All three must hold: ```sql -- ✓ resolves the encrypted operator → uses the index -WHERE email = $1::eql_v3.text_eq; +WHERE email = $1::public.text_eq; WHERE email = $1; -- only when the client (Stack SDK / Proxy) binds $1 typed -- ✗ falls through to native jsonb semantics @@ -77,7 +77,7 @@ WHERE email = '{"hm":"abc"}'::jsonb; ### Equality ```sql -SELECT * FROM users WHERE email = $1::eql_v3.text_eq; +SELECT * FROM users WHERE email = $1::public.text_eq; -- Index Scan using users_email_eq -- Index Cond: (eql_v3.eq_term(email) = eql_v3.eq_term($1)) ``` @@ -87,14 +87,14 @@ SELECT * FROM users WHERE email = $1::eql_v3.text_eq; The `<`, `<=`, `>`, `>=` operators inline to comparisons on `eql_v3.ord_term`, so natural-form range predicates match the btree: ```sql -SELECT * FROM users WHERE created_at < $1::eql_v3.timestamp_ord; +SELECT * FROM users WHERE created_at < $1::public.timestamp_ord; ``` `ORDER BY` needs care. The planner inlines operators in *predicates* but does not rewrite *sort keys*: `ORDER BY created_at` uses the index for the `WHERE` clause but still adds a `Sort` node, which scales linearly with the rows passing the filter. To stream rows out of the btree already ordered, write the sort key in extractor form: ```sql SELECT * FROM users - WHERE created_at < $1::eql_v3.timestamp_ord + WHERE created_at < $1::public.timestamp_ord ORDER BY eql_v3.ord_term(created_at) DESC LIMIT 10; ``` @@ -119,7 +119,7 @@ SELECT eql_v3.eq_term(email), count(*) ## Encrypted JSON -Containment (`@>` / `<@`) on `eql_v3.json` document columns uses a GIN index over `eql_v3.to_ste_vec_query(col)::jsonb`, and field-level equality and ordering have their own extractor recipes. See [JSON](/reference/eql/json). +Containment (`@>` / `<@`) on `public.json` document columns uses a GIN index over `eql_v3.to_ste_vec_query(col)::jsonb`, and field-level equality and ordering have their own extractor recipes. See [JSON](/reference/eql/json). ## Verify with EXPLAIN @@ -175,7 +175,7 @@ Everything above is a functional index over an `IMMUTABLE` SQL function — no o FROM users LIMIT 1; ``` -2. Verify the operand is typed (`$1::eql_v3.text_eq`, not `$1::jsonb`). +2. Verify the operand is typed (`$1::public.text_eq`, not `$1::jsonb`). 3. Recreate the index if the column's terms changed after it was built. 4. Run `ANALYZE`. Very small tables may still choose a sequential scan — that's correct. diff --git a/content/docs/reference/eql/joins.mdx b/content/docs/reference/eql/joins.mdx index fdcc0e4..b4e2850 100644 --- a/content/docs/reference/eql/joins.mdx +++ b/content/docs/reference/eql/joins.mdx @@ -21,7 +21,7 @@ Equijoins work on equality-capable variants (`_eq`, `_ord` / `_ord_ore`, `text_s SELECT u.*, o.total FROM users u JOIN orders o ON u.email = o.customer_email; --- both columns eql_v3.text_eq, encrypted with the same keyset +-- both columns public.text_eq, encrypted with the same keyset ``` No typed-operand cast is needed here — both operands are encrypted columns, so their domain types resolve the encrypted operator directly. All join types (`INNER`, `LEFT`, `RIGHT`, `FULL`) work; `LEFT JOIN` null-extension behaves normally because SQL `NULL`s are not encrypted. @@ -47,17 +47,17 @@ If the two columns are under different keysets, `IN (subquery)` matches nothing, ## Worked example -Two tables sharing an encrypted customer identifier, both columns typed `eql_v3.text_eq` and encrypted by the same client configuration (same keyset): +Two tables sharing an encrypted customer identifier, both columns typed `public.text_eq` and encrypted by the same client configuration (same keyset): ```sql CREATE TABLE users ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v3.text_eq + email public.text_eq ); CREATE TABLE orders ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - customer_email eql_v3.text_eq, + customer_email public.text_eq, total BIGINT NOT NULL ); @@ -72,7 +72,7 @@ Orders per user, filtered by an encrypted lookup on one side: SELECT u.id, COUNT(o.id) AS order_count FROM users u LEFT JOIN orders o ON u.email = o.customer_email -WHERE u.email = $1::eql_v3.text_eq +WHERE u.email = $1::public.text_eq GROUP BY u.id; ``` diff --git a/content/docs/reference/eql/json.mdx b/content/docs/reference/eql/json.mdx index 93eb920..96d8cce 100644 --- a/content/docs/reference/eql/json.mdx +++ b/content/docs/reference/eql/json.mdx @@ -1,15 +1,15 @@ --- title: JSON -description: "The complete reference for encrypted JSON documents with eql_v3.json — the ste_vec payload shape, containment, field access, and path queries over ciphertext, with the native jsonb operators that don't apply blocked outright." +description: "The complete reference for encrypted JSON documents with public.json — the ste_vec payload shape, containment, field access, and path queries over ciphertext, with the native jsonb operators that don't apply blocked outright." type: reference components: [eql] verifiedAgainst: eql: "3.0.0" --- -`eql_v3.json` is EQL's encrypted JSON document type, built on structured encryption (**ste_vec**). The document is encrypted as a vector of encrypted entries — one entry per path inside the document — and every path is queryable without decryption: containment, field and array access, and equality or range comparisons on extracted leaves. +`public.json` is EQL's encrypted JSON document type, built on structured encryption (**ste_vec**). The document is encrypted as a vector of encrypted entries — one entry per path inside the document — and every path is queryable without decryption: containment, field and array access, and equality or range comparisons on extracted leaves. -Like every EQL type, `eql_v3.json` holds ciphertext the database can't read. Encryption, decryption, and selector generation happen in the client — the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). See [Searchable encryption](/concepts/searchable-encryption) for how querying ciphertext works at all. +Like every EQL type, `public.json` holds ciphertext the database can't read. Encryption, decryption, and selector generation happen in the client — the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). See [Searchable encryption](/concepts/searchable-encryption) for how querying ciphertext works at all. ## The types @@ -17,9 +17,9 @@ Three `jsonb`-backed domains make up the encrypted JSON surface: | Type | What it is | | --- | --- | -| `eql_v3.json` | The column type. An encrypted document envelope carrying an `sv` array — one encrypted entry per path in the document. | -| `eql_v3.ste_vec_entry` | A single entry from the vector: a selector, a ciphertext, and exactly one index term. This is what `->` returns. | -| `eql_v3.ste_vec_query` | A containment needle: entries with selectors and index terms but **no ciphertext**. This is what you cast a `@>` operand to. | +| `public.json` | The column type. An encrypted document envelope carrying an `sv` array — one encrypted entry per path in the document. | +| `public.jsonb_entry` | A single entry from the vector: a selector, a ciphertext, and exactly one index term. This is what `->` returns. | +| `public.jsonb_query` | A containment needle: entries with selectors and index terms but **no ciphertext**. This is what you cast a `@>` operand to. | ## Payload shape @@ -34,7 +34,7 @@ An encrypted JSON document uses a different payload shape from the scalar types: The decoded `oc` value starts with a domain-tag byte (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext, so numeric and string values in one column keep a consistent total order. Earlier payload versions split this into two fields — `ocf` (fixed-width, numeric) and `ocv` (variable-width, string) — which consolidated into the single `oc` key; the tag byte now carries the distinction. -A document payload for an `eql_v3.json` column: +A document payload for an `public.json` column: ```json { @@ -53,7 +53,7 @@ A document payload for an `eql_v3.json` column: - Second entry: a string leaf — `oc` starting with tag `01` - Third entry: a numeric leaf — `oc` starting with tag `00` -A containment **query** payload (`eql_v3.ste_vec_query`) has the same `sv` shape but its entries carry no `c` — containment matches selectors and index terms, never ciphertexts. This is the needle the client builds for a `@>` query: +A containment **query** payload (`public.jsonb_query`) has the same `sv` shape but its entries carry no `c` — containment matches selectors and index terms, never ciphertexts. This is the needle the client builds for a `@>` query: ```json { @@ -65,16 +65,16 @@ A containment **query** payload (`eql_v3.ste_vec_query`) has the same `sv` shape ## Storing encrypted JSON -Type the column as `eql_v3.json`: +Type the column as `public.json`: ```sql CREATE TABLE orders ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - metadata eql_v3.json + metadata public.json ); ``` -There is no database-side configuration step. Which index terms a document carries is decided by the encryption client; typing the column as `eql_v3.json` is what makes the encrypted operators and functions resolve. The domain's `CHECK` constraint validates the payload shape on insert, so malformed values are rejected at write time. +There is no database-side configuration step. Which index terms a document carries is decided by the encryption client; typing the column as `public.json` is what makes the encrypted operators and functions resolve. The domain's `CHECK` constraint validates the payload shape on insert, so malformed values are rejected at write time. Insert and read through the Stack SDK or Proxy, which encrypt the document into the ste_vec payload on write and decrypt it on read. @@ -96,7 +96,7 @@ JSON `null` here means a `null` literal *inside* the document. A SQL `NULL` colu ## Blocked native jsonb operators -These native PostgreSQL `jsonb` operators are **blocked** on `eql_v3.json`. They raise an error rather than silently running plaintext-jsonb semantics against the encrypted payload: +These native PostgreSQL `jsonb` operators are **blocked** on `public.json`. They raise an error rather than silently running plaintext-jsonb semantics against the encrypted payload: - Key/path existence: `?`, `?|`, `?&`, `@?`, `@@` - Path extraction: `#>`, `#>>` @@ -111,11 +111,11 @@ Use containment (`@>` / `<@`), field access (`->` / `->>`), or the `eql_v3.jsonb ## Containment: `@>` and `<@` -`@>` tests whether the encrypted document contains a structure; `<@` is the reverse. Build the needle with the client and cast it to `eql_v3.ste_vec_query` (a typed `eql_v3.json` or `eql_v3.ste_vec_entry` operand also works): +`@>` tests whether the encrypted document contains a structure; `<@` is the reverse. Build the needle with the client and cast it to `public.jsonb_query` (a typed `public.json` or `public.jsonb_entry` operand also works): ```sql SELECT * FROM orders -WHERE metadata @> $1::eql_v3.ste_vec_query; +WHERE metadata @> $1::public.jsonb_query; ``` This is the encrypted equivalent of the plaintext `metadata @> '{"customer": {"tier": "premium"}}'`: containment checks that every encrypted term in the needle exists in the document's `sv` vector. `eql_v3.to_ste_vec_query(doc)` converts a stored document into the needle shape, and `eql_v3.ste_vec_contains(a, b)` is the function form backing `@>`. @@ -135,7 +135,7 @@ See [Indexes](/reference/eql/indexes) for the full recipes. Fields are addressed by **selector hash** — the deterministic identifier the client emits for a JSON path during encryption — not a plaintext path string like `$.customer.tier`. ```sql --- Field access by selector (returns eql_v3.ste_vec_entry) +-- Field access by selector (returns public.jsonb_entry) SELECT metadata -> 'selector_hash'::text FROM orders; -- The entry serialized as text (ciphertext JSON, not decrypted plaintext) @@ -145,7 +145,7 @@ SELECT metadata ->> 'selector_hash'::text FROM orders; SELECT metadata -> 0 FROM orders; ``` -The extracted `eql_v3.ste_vec_entry` is itself comparable: +The extracted `public.jsonb_entry` is itself comparable: - `=` / `<>` resolve via `eql_v3.eq_term` — works on every node type - `<` / `<=` / `>` / `>=` resolve via `eql_v3.ore_cllw` — String and Number leaves only @@ -154,7 +154,7 @@ The extracted `eql_v3.ste_vec_entry` is itself comparable: ```sql -- Equality on an extracted leaf SELECT * FROM orders -WHERE metadata -> 'email_selector'::text = $1::eql_v3.ste_vec_entry; +WHERE metadata -> 'email_selector'::text = $1::public.jsonb_entry; -- Group by an extracted leaf's equality term SELECT eql_v3.eq_term(metadata -> 'region_selector'::text) AS region, COUNT(*) @@ -213,7 +213,7 @@ The client encrypts this into a ste_vec payload with selectors for `$`, `$.custo ```sql CREATE TABLE orders ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - metadata eql_v3.json + metadata public.json ); INSERT INTO orders (metadata) VALUES ($1); @@ -229,7 +229,7 @@ Find premium orders. The client encrypts the needle `{"customer": {"tier": "prem ```sql SELECT id FROM orders -WHERE metadata @> $1::eql_v3.ste_vec_query; +WHERE metadata @> $1::public.jsonb_query; ``` Add the GIN index from above once the table grows. diff --git a/content/docs/reference/eql/numbers.mdx b/content/docs/reference/eql/numbers.mdx index 2d9a2a0..b30fab3 100644 --- a/content/docs/reference/eql/numbers.mdx +++ b/content/docs/reference/eql/numbers.mdx @@ -26,12 +26,12 @@ And every concrete domain this page covers: | Type | Variants | | --- | --- | -| `int2` | `eql_v3.int2` · `eql_v3.int2_eq` · `eql_v3.int2_ord` · `eql_v3.int2_ord_ore` | -| `int4` | `eql_v3.int4` · `eql_v3.int4_eq` · `eql_v3.int4_ord` · `eql_v3.int4_ord_ore` | -| `int8` | `eql_v3.int8` · `eql_v3.int8_eq` · `eql_v3.int8_ord` · `eql_v3.int8_ord_ore` | -| `float4` | `eql_v3.float4` · `eql_v3.float4_eq` · `eql_v3.float4_ord` · `eql_v3.float4_ord_ore` | -| `float8` | `eql_v3.float8` · `eql_v3.float8_eq` · `eql_v3.float8_ord` · `eql_v3.float8_ord_ore` | -| `numeric` | `eql_v3.numeric` · `eql_v3.numeric_eq` · `eql_v3.numeric_ord` · `eql_v3.numeric_ord_ore` | +| `int2` | `public.smallint` · `public.smallint_eq` · `public.smallint_ord` · `public.smallint_ord_ore` | +| `int4` | `public.integer` · `public.integer_eq` · `public.integer_ord` · `public.integer_ord_ore` | +| `int8` | `public.bigint` · `public.bigint_eq` · `public.bigint_ord` · `public.bigint_ord_ore` | +| `float4` | `public.real` · `public.real_eq` · `public.real_ord` · `public.real_ord_ore` | +| `float8` | `public.double` · `public.double_eq` · `public.double_ord` · `public.double_ord_ore` | +| `numeric` | `public.numeric` · `public.numeric_eq` · `public.numeric_ord` · `public.numeric_ord_ore` | Declare only the capability you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)), and the variant model itself is covered in [Core concepts](/reference/eql/core-concepts). @@ -42,9 +42,9 @@ A payroll table mixing the variants by how each column is queried: ```sql CREATE TABLE employees ( id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - salary eql_v3.int8_ord, -- range queries, ORDER BY, MIN/MAX - tax_rate eql_v3.numeric_eq, -- exact lookup only - net_worth eql_v3.numeric -- store and decrypt only, never queried + salary public.bigint_ord, -- range queries, ORDER BY, MIN/MAX + tax_rate public.numeric_eq, -- exact lookup only + net_worth public.numeric -- store and decrypt only, never queried ); ``` @@ -61,7 +61,7 @@ The EQL v3 release adds an OPE specifier for every orderable type; unspecified ` ## Payload -A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus the `ob` ordering term. Here is a payload for the `eql_v3.int8_ord` `salary` column: +A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — see [Core concepts](/reference/eql/core-concepts)) plus the `ob` ordering term. Here is a payload for the `public.bigint_ord` `salary` column: ```json { @@ -90,7 +90,7 @@ A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — | `ORDER BY` | ❌ | ❌ | ✅ | | `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | -Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::eql_v3.int8_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). +Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::public.bigint_ord`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). ## Functions @@ -110,10 +110,10 @@ Every operator has a function form, for managed platforms that disallow custom o ```sql SELECT * FROM employees -WHERE salary >= $1::eql_v3.int8_ord; +WHERE salary >= $1::public.bigint_ord; SELECT * FROM employees -WHERE salary BETWEEN $1::eql_v3.int8_ord AND $2::eql_v3.int8_ord; +WHERE salary BETWEEN $1::public.bigint_ord AND $2::public.bigint_ord; ``` ### MIN and MAX @@ -140,7 +140,7 @@ LIMIT 10; On a generic `jsonb` column whose payloads already carry the `ob` term, cast to the right domain in the query: ```sql -SELECT eql_v3.min(salary_jsonb::eql_v3.int8_ord) FROM employees; +SELECT eql_v3.min(salary_jsonb::public.bigint_ord) FROM employees; ``` ## Where to next diff --git a/content/docs/reference/eql/sorting.mdx b/content/docs/reference/eql/sorting.mdx index eb34f12..2e7ef2b 100644 --- a/content/docs/reference/eql/sorting.mdx +++ b/content/docs/reference/eql/sorting.mdx @@ -33,7 +33,7 @@ CREATE INDEX users_created_at_ord ANALYZE users; SELECT * FROM users - WHERE created_at < $1::eql_v3.timestamp_ord + WHERE created_at < $1::public.timestamp_ord ORDER BY eql_v3.ord_term(created_at) DESC LIMIT 10; -- Index Scan Backward using users_created_at_ord — no Sort node @@ -64,7 +64,7 @@ SELECT id, email, created_at FROM users -- Next page: pass the last row's created_at back, re-encrypted as the cursor SELECT id, email, created_at FROM users - WHERE created_at < $1::eql_v3.timestamp_ord + WHERE created_at < $1::public.timestamp_ord ORDER BY eql_v3.ord_term(created_at) DESC LIMIT 20; ``` diff --git a/content/docs/reference/eql/text.mdx b/content/docs/reference/eql/text.mdx index f08eafc..408ed97 100644 --- a/content/docs/reference/eql/text.mdx +++ b/content/docs/reference/eql/text.mdx @@ -15,12 +15,12 @@ All six are `jsonb`-backed domains. Which one you declare fixes the column's que | Domain variant | Capability | | --- | --- | -| `eql_v3.text` | Storage and decryption only. | -| `eql_v3.text_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | -| `eql_v3.text_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | -| `eql_v3.text_ord_ore` | As `text_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | -| `eql_v3.text_match` | Free-text token containment: `@>` / `<@`. | -| `eql_v3.text_search` | Equality + ordering + token containment. | +| `public.text` | Storage and decryption only. | +| `public.text_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | +| `public.text_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | +| `public.text_ord_ore` | As `text_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | +| `public.text_match` | Free-text token containment: `@>` / `<@`. | +| `public.text_search` | Equality + ordering + token containment. | Declare only the capabilities you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)). @@ -31,10 +31,10 @@ A users table mixing the variants by how each column is queried: ```sql CREATE TABLE users ( id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v3.text_search, -- lookup, sort, and free-text match - name eql_v3.text_match, -- free-text match only - tax_id eql_v3.text_eq, -- exact lookup only - notes eql_v3.text -- store and decrypt only + email public.text_search, -- lookup, sort, and free-text match + name public.text_match, -- free-text match only + tax_id public.text_eq, -- exact lookup only + notes public.text -- store and decrypt only ); ``` @@ -74,7 +74,7 @@ The narrower variants carry only their own term: a `text_eq` payload carries `hm ## Operators -| SQL operator | `eql_v3.text` | `text_eq` | `text_ord` / `text_ord_ore` | `text_match` | `text_search` | +| SQL operator | `public.text` | `text_eq` | `text_ord` / `text_ord_ore` | `text_match` | `text_search` | | --- | :---: | :---: | :---: | :---: | :---: | | `=` / `<>` | ❌ | ✅ | ✅ | ❌ | ✅ | | `<` `<=` `>` `>=` | ❌ | ❌ | ✅ | ❌ | ✅ | @@ -84,7 +84,7 @@ The narrower variants carry only their own term: a `text_eq` payload carries `hm | `ORDER BY` | ❌ | ❌ | ✅ | ❌ | ✅ | | `IS NULL` / `IS NOT NULL` | ✅ | ✅ | ✅ | ✅ | ✅ | -Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::eql_v3.text_eq`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). +Blocked *operator* cells raise an `operator … is not supported` exception — they never silently return wrong rows. `ORDER BY` is the one blocked cell that doesn't raise: it isn't an operator, so sorting a variant without an ordering term runs — but the order is meaningless (see [Sorting](/reference/eql/sorting)). Operands must be typed (`$1::public.text_eq`), or PostgreSQL resolves the native `jsonb` operator instead of the encrypted one. Both rules are covered in [Core concepts](/reference/eql/core-concepts). ## Functions @@ -108,7 +108,7 @@ There are no `like` / `ilike` function forms — encrypted text matching is `eql SELECT * FROM users WHERE email LIKE '%alice%'; -- ✅ Encrypted free-text match -SELECT * FROM users WHERE email @> $1::eql_v3.text_match; +SELECT * FROM users WHERE email @> $1::public.text_match; ``` `@>` / `<@` here is **probabilistic ngram-bloom containment** — it tests whether the encrypted text contains the (encrypted) search terms. It is not JSONB containment and not `LIKE`. The client encrypts the search term into a bloom-filter query value; false positives are possible, false negatives are not. There are no `like` / `ilike` function forms either — text matching is `eql_v3.contains` on a `text_match` value. @@ -120,10 +120,10 @@ SELECT * FROM users WHERE email @> $1::eql_v3.text_match; Equality on a `text_eq` column compares HMAC terms. `IN` desugars to `=`: ```sql -SELECT * FROM users WHERE tax_id = $1::eql_v3.text_eq; +SELECT * FROM users WHERE tax_id = $1::public.text_eq; SELECT * FROM users -WHERE tax_id IN ($1::eql_v3.text_eq, $2::eql_v3.text_eq); +WHERE tax_id IN ($1::public.text_eq, $2::public.text_eq); ``` ### Free-text match @@ -131,10 +131,10 @@ WHERE tax_id IN ($1::eql_v3.text_eq, $2::eql_v3.text_eq); The client encrypts the search term into the bloom-filter needle: ```sql -SELECT * FROM users WHERE name @> $1::eql_v3.text_match; +SELECT * FROM users WHERE name @> $1::public.text_match; -- Function form, for platforms without custom operators -SELECT * FROM users WHERE eql_v3.contains(name, $1::eql_v3.text_match); +SELECT * FROM users WHERE eql_v3.contains(name, $1::public.text_match); ``` ### The works: `text_search` @@ -143,8 +143,8 @@ A `text_search` column answers exact lookup, free-text match, and ordering — h ```sql SELECT id, email FROM users -WHERE email @> $1::eql_v3.text_match -- token containment on bf - AND email <> $2::eql_v3.text_eq -- exclude an exact value via hm +WHERE email @> $1::public.text_match -- token containment on bf + AND email <> $2::public.text_eq -- exclude an exact value via hm ORDER BY eql_v3.ord_term(email) -- sort on ob LIMIT 20; ``` diff --git a/scripts/fixtures/eql-manifest.sample.json b/scripts/fixtures/eql-manifest.sample.json index e08d50d..25a5246 100644 --- a/scripts/fixtures/eql-manifest.sample.json +++ b/scripts/fixtures/eql-manifest.sample.json @@ -1,69 +1,192 @@ { - "_note": "ILLUSTRATIVE SAMPLE — shape only. Replaced at build time by the real eql-manifest.json from the eql-docs release asset once cipherstash/encrypt-query-language#364 ships. Do not treat these signatures/domains as authoritative.", + "_note": "ILLUSTRATIVE SAMPLE — shape only. Replaced at build time by the real eql-manifest.json from the eql-docs release asset (cipherstash/encrypt-query-language). Do not treat these signatures/domains as authoritative. Mirrors the real schema: domains live in `public.`, public functions in `eql_v3.`, internal ctors in `eql_v3_internal.` (visibility: private).", "$schema": "https://schemas.cipherstash.com/eql/manifest/v1.json", "name": "eql", "version": "3.0.0-sample", "generatedFrom": "doxygen-xml + catalog", - "counts": { "functions": 3, "public": 3, "private": 0, "domains": 9 }, + "counts": { "functions": 5, "public": 2, "private": 3, "domains": 9 }, "functions": [ + { + "name": "version", + "signature": "version()", + "visibility": "public", + "brief": "Get the installed EQL version string.", + "description": "Returns the version string for the installed EQL library.", + "params": [], + "returns": { "type": "text", "description": "the version string" }, + "throws": [], + "notes": "", + "warnings": "", + "seeAlso": "", + "source": { "file": "src/v3/version.sql", "line": 28 } + }, + { + "name": "jsonb_path_query", + "signature": "jsonb_path_query(jsonb, text)", + "visibility": "public", + "brief": "Query encrypted JSONB for sv elements matching a selector.", + "description": "Returns one jsonb_entry row per matching encrypted element.", + "params": [ + { + "name": "val", + "type": "jsonb", + "description": "encrypted EQL payload with sv" + }, + { "name": "selector", "type": "text", "description": "selector hash" } + ], + "returns": { + "type": "public.jsonb_entry", + "description": "matching encrypted entries" + }, + "throws": [], + "notes": "", + "warnings": "", + "seeAlso": "", + "source": { "file": "src/v3/jsonb/functions.sql", "line": 358 } + }, { "name": "hmac_256", "signature": "hmac_256(jsonb)", - "visibility": "public", + "visibility": "private", "brief": "Extract the HMAC-SHA-256 equality term from an encrypted value.", "description": "Backs equality (`=`, `IN`, joins) on `_eq` and `text_search` columns.", - "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], + "params": [ + { "name": "val", "type": "jsonb", "description": "the encrypted value" } + ], "returns": { "type": "text", "description": "the HMAC term" }, - "throws": [], "notes": "", "warnings": "", "seeAlso": "", + "throws": [], + "notes": "", + "warnings": "", + "seeAlso": "", "source": { "file": "src/v3/sem/hmac_256/functions.sql", "line": 12 } }, { "name": "bloom_filter", "signature": "bloom_filter(jsonb)", - "visibility": "public", + "visibility": "private", "brief": "Extract the bloom-filter match term from an encrypted value.", "description": "Backs token containment (`@>`) on `text_match` / `text_search` columns.", - "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], - "returns": { "type": "smallint[]", "description": "the set bit positions" }, - "throws": [], "notes": "", "warnings": "", "seeAlso": "", + "params": [ + { "name": "val", "type": "jsonb", "description": "the encrypted value" } + ], + "returns": { + "type": "smallint[]", + "description": "the set bit positions" + }, + "throws": [], + "notes": "", + "warnings": "", + "seeAlso": "", "source": { "file": "src/v3/sem/bloom_filter/functions.sql", "line": 20 } }, { "name": "ore_block_256", "signature": "ore_block_256(jsonb)", - "visibility": "public", + "visibility": "private", "brief": "Extract the ORE ordering term from an encrypted value.", "description": "Backs range and ordering (`<`, `>`, `ORDER BY`) on `_ord` columns.", - "params": [{ "name": "val", "type": "jsonb", "description": "the encrypted value" }], - "returns": { "type": "eql_v3.ore_block_256", "description": "the ORE term" }, - "throws": [], "notes": "", "warnings": "", "seeAlso": "", + "params": [ + { "name": "val", "type": "jsonb", "description": "the encrypted value" } + ], + "returns": { + "type": "eql_v3_internal.ore_block_256", + "description": "the ORE term" + }, + "throws": [], + "notes": "", + "warnings": "", + "seeAlso": "", "source": { "file": "src/v3/sem/ore_block_256/functions.sql", "line": 8 } } ], "domains": [ - { "name": "eql_v3.integer", "type": "integer", "variant": "", "base": "jsonb", - "capabilities": ["storage"], "supportedOperators": [], "termFunctions": [] }, - { "name": "eql_v3.integer_eq", "type": "integer", "variant": "eq", "base": "jsonb", - "capabilities": ["equality"], "supportedOperators": ["=", "<>"], - "termFunctions": ["eql_v3.eq_term"] }, - { "name": "eql_v3.integer_ord", "type": "integer", "variant": "ord", "base": "jsonb", - "capabilities": ["equality", "order"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">="], - "termFunctions": ["eql_v3.ord_term"] }, - { "name": "eql_v3.integer_ord_ope", "type": "integer", "variant": "ord_ope", "base": "jsonb", - "capabilities": ["equality", "order"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">="], - "termFunctions": ["eql_v3.ord_ope_term"] }, - { "name": "eql_v3.text_match", "type": "text", "variant": "match", "base": "jsonb", - "capabilities": ["match"], "supportedOperators": ["@>", "<@"], - "termFunctions": ["eql_v3.match_term"] }, - { "name": "eql_v3.text_search", "type": "text", "variant": "search", "base": "jsonb", + { + "name": "public.integer", + "type": "integer", + "variant": "", + "base": "jsonb", + "capabilities": ["storage"], + "supportedOperators": [], + "termFunctions": [] + }, + { + "name": "public.integer_eq", + "type": "integer", + "variant": "eq", + "base": "jsonb", + "capabilities": ["equality"], + "supportedOperators": ["=", "<>"], + "termFunctions": ["eql_v3.eq_term"] + }, + { + "name": "public.integer_ord", + "type": "integer", + "variant": "ord", + "base": "jsonb", + "capabilities": ["equality", "order"], + "supportedOperators": ["=", "<>", "<", "<=", ">", ">="], + "termFunctions": ["eql_v3.ord_term"] + }, + { + "name": "public.integer_ord_ope", + "type": "integer", + "variant": "ord_ope", + "base": "jsonb", + "capabilities": ["equality", "order"], + "supportedOperators": ["=", "<>", "<", "<=", ">", ">="], + "termFunctions": ["eql_v3.ord_ope_term"] + }, + { + "name": "public.text_match", + "type": "text", + "variant": "match", + "base": "jsonb", + "capabilities": ["match"], + "supportedOperators": ["@>", "<@"], + "termFunctions": ["eql_v3.match_term"] + }, + { + "name": "public.text_search", + "type": "text", + "variant": "search", + "base": "jsonb", "capabilities": ["equality", "order", "match"], "supportedOperators": ["=", "<>", "<", "<=", ">", ">=", "@>", "<@"], - "termFunctions": ["eql_v3.eq_term", "eql_v3.ord_term", "eql_v3.match_term"] }, - { "name": "eql_v3.json", "type": "jsonb", "variant": "", "base": "jsonb", "shape": "stevec", - "capabilities": ["json"], "supportedOperators": [], "termFunctions": [] }, - { "name": "eql_v3.jsonb_entry", "type": "jsonb", "variant": "", "base": "jsonb", "shape": "stevec", - "capabilities": ["json"], "supportedOperators": [], "termFunctions": [] }, - { "name": "eql_v3.jsonb_query", "type": "jsonb", "variant": "", "base": "jsonb", "shape": "stevec", - "capabilities": ["json"], "supportedOperators": [], "termFunctions": [] } + "termFunctions": [ + "eql_v3.eq_term", + "eql_v3.ord_term", + "eql_v3.match_term" + ] + }, + { + "name": "public.json", + "type": "jsonb", + "variant": "", + "base": "jsonb", + "shape": "stevec", + "capabilities": ["json"], + "supportedOperators": [], + "termFunctions": [] + }, + { + "name": "public.jsonb_entry", + "type": "jsonb", + "variant": "", + "base": "jsonb", + "shape": "stevec", + "capabilities": ["json"], + "supportedOperators": [], + "termFunctions": ["eql_v3.eq_term", "eql_v3.ore_cllw"] + }, + { + "name": "public.jsonb_query", + "type": "jsonb", + "variant": "", + "base": "jsonb", + "shape": "stevec", + "capabilities": ["json"], + "supportedOperators": [], + "termFunctions": [] + } ] } diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index 665682d..c6cbfa6 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -8,33 +8,44 @@ * 1. Generates a version-stamped function catalog at * content/docs/reference/eql/functions.mdx — the exhaustive, drift-proof * low-level reference the hand-written pedagogical pages link to. - * 2. Drift-lints the hand-written pages: every `eql_v3.<fn>(...)` referenced - * in them should exist in the manifest for the shipped EQL version. This - * is what would have caught the fabricated function names / stale payload - * that slipped into the v2 docs. + * 2. Drift-lints the hand-written pages: every schema-qualified reference + * (`public.<domain>`, `eql_v3.<fn>`, `eql_v3_internal.<fn>`) must match the + * manifest by schema AND name. This catches fabricated names, stale + * payloads, and right-name/wrong-schema refs (e.g. `eql_v3.text_eq` for a + * domain that lives in `public`) — the classes of drift that slipped into + * the v2 docs. * * ── Manifest source ──────────────────────────────────────────────────────── - * TODAY: reads a committed illustrative sample (shape only), so the generated - * format and the lint are reviewable before the EQL side ships. - * TARGET: once #364 releases, `generate-eql-docs.ts` already downloads the - * `eql-docs-<tag>` asset — extend it to also extract `json/eql-manifest.json` - * and point MANIFEST_PATH at it. The renderer and lint stay identical. + * Reads the committed illustrative sample by default. Set EQL_MANIFEST_PATH to + * a real manifest (the `eql-docs-<tag>` release asset extracted by + * `generate-eql-docs.ts`, or a locally-generated one) — the renderer and lint + * are identical either way. * - * The drift-lint is REPORT-ONLY against the sample (which is tiny, so most real - * symbols read as "unknown"); it becomes a failing gate once wired to the real - * manifest. See STRICT below. + * The drift-lint is REPORT-ONLY against the sample (tiny, so most real symbols + * read as "unknown"); it becomes a failing gate against a real manifest + * (auto-strict when EQL_MANIFEST_PATH is set). See STRICT below. */ import fs from "node:fs"; import path from "node:path"; -const MANIFEST_PATH = path.join( +// Manifest source, in priority order: +// 1. EQL_MANIFEST_PATH — explicit override (local testing / CI). +// 2. .eql-manifest.release.json — extracted from the eql-docs release asset +// by generate-eql-docs.ts (present once a release ships the manifest). +// 3. the committed illustrative sample — offline fallback. +const SAMPLE_MANIFEST = path.join( process.cwd(), "scripts/fixtures/eql-manifest.sample.json", ); +const RELEASE_MANIFEST = path.join(process.cwd(), ".eql-manifest.release.json"); +const MANIFEST_PATH = + process.env.EQL_MANIFEST_PATH ?? + (fs.existsSync(RELEASE_MANIFEST) ? RELEASE_MANIFEST : SAMPLE_MANIFEST); const EQL_DIR = path.join(process.cwd(), "content/docs/reference/eql"); const OUT_FILE = path.join(EQL_DIR, "functions.mdx"); -// Flip to true once MANIFEST_PATH points at the real release manifest. -const STRICT = false; +// Report-only against the illustrative sample; a failing gate against any real +// manifest (the release asset or an explicit override). +const STRICT = MANIFEST_PATH !== SAMPLE_MANIFEST; interface Param { name: string; @@ -83,15 +94,49 @@ function paramsTable(params: Param[]): string { return `\n| Parameter | Type | Description |\n| --- | --- | --- |\n${rows}\n`; } -function renderFn(fn: Fn): string { - const parts = [`### \`${fn.signature}\``, "", fn.brief]; - if (fn.description && fn.description !== fn.brief) parts.push("", fn.description); - if (fn.params.length) parts.push(paramsTable(fn.params)); - if (fn.returns?.type || fn.returns?.description) { - const t = fn.returns.type ? `\`${fn.returns.type}\`` : ""; - parts.push("", `**Returns:** ${t}${fn.returns.description ? ` — ${fn.returns.description}` : ""}`); +// Public functions are heavily overloaded — one name per operation with a +// per-encrypted-type variant, all sharing the same doc. Group by name so the +// page is one entry per function (its distinct signatures listed) instead of +// ~hundreds of near-identical overload blocks. +function renderPublicFunctions(fns: Fn[]): string { + const byName = new Map<string, Fn[]>(); + for (const f of fns) { + const g = byName.get(f.name) ?? []; + g.push(f); + byName.set(f.name, g); } - return parts.join("\n"); + const sections: string[] = []; + const entries = [...byName.entries()].sort((a, b) => + a[0].localeCompare(b[0]), + ); + for (const [name, overloads] of entries) { + const rep = + overloads.find((o) => o.params.length || o.brief) ?? overloads[0]; + const sigs = [...new Set(overloads.map((o) => o.signature))].sort(); + const parts = [`### \`${name}\``, "", rep.brief]; + if (rep.description && rep.description !== rep.brief) + parts.push("", rep.description); + parts.push( + "", + sigs.length > 1 + ? `**${sigs.length} overloads** (one per encrypted type):` + : "**Signature:**", + "", + "```sql", + ...sigs, + "```", + ); + if (rep.params.length) parts.push(paramsTable(rep.params)); + if (rep.returns?.type || rep.returns?.description) { + const t = rep.returns.type ? `\`${rep.returns.type}\`` : ""; + parts.push( + "", + `**Returns:** ${t}${rep.returns.description ? ` — ${rep.returns.description}` : ""}`, + ); + } + sections.push(parts.join("\n")); + } + return sections.join("\n\n"); } function renderDomains(domains: Domain[]): string { @@ -119,8 +164,16 @@ function renderDomains(domains: Domain[]): string { function render(manifest: Manifest): string { const version = manifest.version; - const publicFns = manifest.functions.filter((f) => f.visibility === "public"); - const privateFns = manifest.functions.filter((f) => f.visibility === "private"); + // Operators (`->`, `>`) reach the manifest as pseudo-functions via Doxygen's + // operator-name remapping; they render poorly and are already covered by the + // domain matrix's Operators column, so keep only real named functions here. + const isNamed = (f: Fn) => /^[a-z_][a-z0-9_]*$/.test(f.name); + const publicFns = manifest.functions.filter( + (f) => f.visibility === "public" && isNamed(f), + ); + const privateFns = manifest.functions.filter( + (f) => f.visibility === "private", + ); const frontmatter = [ "---", @@ -142,48 +195,63 @@ function render(manifest: Manifest): string { `Generated from the **EQL ${version}** manifest (the Doxygen'd SQL is the source of truth). For the model behind these — variants, terms, typed operands — see [Core concepts](/reference/eql/core-concepts).`, `</Callout>`, "", - "The `eql_v3` schema surface — encrypted domains and the functions behind them. The type and query pages explain *when* to use these; this page is the exhaustive reference they link to.", + "The EQL SQL surface — encrypted domains (in the `public` schema) and the `eql_v3` functions behind them. The type and query pages explain *when* to use these; this page is the exhaustive reference they link to.", "", renderDomains(manifest.domains ?? []), "## Functions", "", - ...publicFns.map(renderFn), + `The public \`eql_v3\` API — ${new Set(publicFns.map((f) => f.name)).size} functions (${publicFns.length} overloads). Internal \`eql_v3_internal\` functions (${privateFns.length}) are implementation detail and omitted.`, + "", + renderPublicFunctions(publicFns), ]; - if (privateFns.length) { - body.push("", "## Internal functions", "", ...privateFns.map(renderFn)); - } - return `${body.join("\n").trimEnd()}\n`; } // ── Drift guard ────────────────────────────────────────────────────────────── +// The known surface is fully schema-qualified: domains live in `public.`, +// functions in `eql_v3.` (public) or `eql_v3_internal.` (private), and the +// per-domain extractor term-functions come pre-qualified from the catalog. +// Matching schema-AND-name means a right-name/wrong-schema reference — e.g. +// `eql_v3.text_eq`, whose domain is actually `public.text_eq` — is flagged too, +// not just fabricated names. +function knownSymbols(manifest: Manifest): Set<string> { + const known = new Set<string>(); + for (const d of manifest.domains ?? []) { + known.add(d.name); // public.<domain> + for (const fn of d.termFunctions ?? []) known.add(fn); // eql_v3.<extractor> + } + for (const f of manifest.functions) { + const schema = f.visibility === "private" ? "eql_v3_internal" : "eql_v3"; + known.add(`${schema}.${f.name}`); + } + return known; +} + function driftCheck(manifest: Manifest): string[] { - // Known = every function, every domain (short) name, and every domain's - // extractor functions (eq_term / ord_term / …, authoritative from the catalog). - const known = new Set<string>([ - ...manifest.functions.map((f) => f.name), - ...(manifest.domains ?? []).map((d) => d.name.replace(/^eql_v3\./, "")), - ...(manifest.domains ?? []).flatMap((d) => - (d.termFunctions ?? []).map((fn) => fn.replace(/^eql_v3\./, "")), - ), - ]); - const referenced = new Map<string, Set<string>>(); // symbol -> pages + const known = knownSymbols(manifest); + const referenced = new Map<string, Set<string>>(); // fqn -> pages for (const file of fs.readdirSync(EQL_DIR)) { if (!file.endsWith(".mdx") || file === "functions.mdx") continue; const text = fs.readFileSync(path.join(EQL_DIR, file), "utf8"); - // Any eql_v3.<symbol> — function call, domain cast, or type reference. - for (const m of text.matchAll(/eql_v3\.([a-z0-9_]+)/g)) { - const sym = m[1]; - if (!referenced.has(sym)) referenced.set(sym, new Set()); - referenced.get(sym)!.add(file); + // Any schema-qualified reference — function call, domain cast, or type. + // A trailing `*` marks a prose family (e.g. `eql_v3.jsonb_path_*`), which + // names a set rather than one symbol, so it's skipped. + for (const m of text.matchAll( + /\b(public|eql_v3_internal|eql_v3)\.([a-z0-9_]+)(\*?)/g, + )) { + if (m[3] === "*") continue; + const fqn = `${m[1]}.${m[2]}`; + const pages = referenced.get(fqn) ?? new Set<string>(); + pages.add(file); + referenced.set(fqn, pages); } } const unknown: string[] = []; - for (const [sym, pages] of referenced) { - if (!known.has(sym)) unknown.push(`${sym} (in ${[...pages].join(", ")})`); + for (const [fqn, pages] of referenced) { + if (!known.has(fqn)) unknown.push(`${fqn} (in ${[...pages].join(", ")})`); } return unknown.sort(); } @@ -200,7 +268,7 @@ function main() { const unknown = driftCheck(manifest); if (unknown.length) { - const header = `⚠ ${unknown.length} eql_v3.* symbol(s) referenced in hand-written pages are not in the manifest (functions or domains):`; + const header = `⚠ ${unknown.length} schema-qualified symbol(s) referenced in hand-written pages are not in the manifest (domains or functions):`; console.warn(`\n${header}`); for (const u of unknown) console.warn(` - ${u}`); if (STRICT) { @@ -210,11 +278,13 @@ function main() { process.exit(1); } else { console.warn( - "\n(Report-only: using the illustrative sample manifest. Becomes a failing gate once wired to the real release manifest — set STRICT = true.)", + "\n(Report-only: using the illustrative sample manifest, which covers only a few symbols. A real manifest — the release asset or EQL_MANIFEST_PATH — makes this a failing gate.)", ); } } else { - console.log("✓ Drift check: all referenced eql_v3.* functions are in the manifest."); + console.log( + "✓ Drift check: all schema-qualified symbols resolve against the manifest.", + ); } } diff --git a/scripts/generate-eql-docs.ts b/scripts/generate-eql-docs.ts index 975b9c1..cd21d6f 100644 --- a/scripts/generate-eql-docs.ts +++ b/scripts/generate-eql-docs.ts @@ -15,6 +15,9 @@ const GITHUB_API_URL = "https://api.github.com/repos/cipherstash/encrypt-query-language/releases"; const TEMP_DIR = ".tmp-eql"; const OUTPUT_DIR = path.join(process.cwd(), "content/stack/reference/eql"); +// Where the machine-readable manifest is surfaced for the v2 API-reference +// generator (scripts/generate-eql-api-docs.ts). Gitignored; best-effort. +const RELEASE_MANIFEST = path.join(process.cwd(), ".eql-manifest.release.json"); /** * Check if a tarball URL exists (returns HTTP 200) @@ -261,6 +264,22 @@ async function main() { "utf8", ); + // Surface the machine-readable manifest for the v2 API-reference generator. + // Only releases packaged with the manifest carry it; older ones don't, so + // this is best-effort and the generator falls back to the committed sample. + const manifestSrc = path.join(extractPath, "json", "eql-manifest.json"); + try { + await fs.copyFile(manifestSrc, RELEASE_MANIFEST); + console.log( + `✓ Extracted eql-manifest.json → ${path.basename(RELEASE_MANIFEST)}`, + ); + } catch { + await fs.rm(RELEASE_MANIFEST, { force: true }); // clear any stale cache + console.log( + "• No eql-manifest.json in this release; API reference uses the sample.", + ); + } + // Clean up console.log("Cleaning up..."); await fs.rm(extractPath, { recursive: true, force: true }); From d5e1910fd4275e4bb1f25bbbf5e7e419616dd9d3 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Mon, 6 Jul 2026 19:41:40 +1000 Subject: [PATCH 34/38] docs(eql): fix schema placeholders + add 'The three schemas' section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-ups from reviewing the rendered preview: - Variant tables used the eql_v3.<T> placeholder (booleans, core-concepts, dates-and-times, grouping-and-aggregates, indexes, numbers) — the concrete- token sweep missed it. Now public.<T>. - core-concepts: new 'The three schemas' section — public (encrypted domain types, kept in an unversioned schema so EQL upgrades don't rewrite your column types), eql_v3 (user-callable functions/operators), eql_v3_internal (implementation, don't call). Fixed prose that placed the domains and the SEM index-term types in eql_v3 (they're public / eql_v3_internal). - index: domains are public not eql_v3; corrected the uninstall Callout — DROP SCHEMA eql_v3, eql_v3_internal CASCADE removes operators/functions but leaves the public.* types + columns (data survives), where the old text claimed dropping eql_v3 drops the columns. - Grammar: 'an public.*' -> 'a public.*' left over from the schema sweep. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA --- content/docs/reference/eql/booleans.mdx | 2 +- content/docs/reference/eql/core-concepts.mdx | 38 +++++++++++++------ .../docs/reference/eql/dates-and-times.mdx | 10 ++--- .../reference/eql/grouping-and-aggregates.mdx | 8 ++-- content/docs/reference/eql/index.mdx | 6 +-- content/docs/reference/eql/indexes.mdx | 4 +- content/docs/reference/eql/json.mdx | 2 +- content/docs/reference/eql/numbers.mdx | 10 ++--- 8 files changed, 48 insertions(+), 32 deletions(-) diff --git a/content/docs/reference/eql/booleans.mdx b/content/docs/reference/eql/booleans.mdx index 6aad0f9..3cbbf73 100644 --- a/content/docs/reference/eql/booleans.mdx +++ b/content/docs/reference/eql/booleans.mdx @@ -51,7 +51,7 @@ If a boolean genuinely needs to be a server-side predicate, that is a data-model ## Storing without querying -`bool` is the forced case of a pattern available to every scalar type: the bare variant `eql_v3.<T>` (for example `public.integer`, `public.text`, `public.timestamp`) is storage-and-decryption only. It carries no index terms, and every comparison operator raises — use it for columns you only ever store and decrypt, so the database holds no searchable material for them at all. +`bool` is the forced case of a pattern available to every scalar type: the bare variant `public.<T>` (for example `public.integer`, `public.text`, `public.timestamp`) is storage-and-decryption only. It carries no index terms, and every comparison operator raises — use it for columns you only ever store and decrypt, so the database holds no searchable material for them at all. For every type other than `bool`, storage-only is a choice you can walk back. If you later need to query, retype the column as a query variant — or, if the payloads already carry the needed term (the client decides which terms travel in the payload), cast at the call site: diff --git a/content/docs/reference/eql/core-concepts.mdx b/content/docs/reference/eql/core-concepts.mdx index 63ce2f1..ffdd927 100644 --- a/content/docs/reference/eql/core-concepts.mdx +++ b/content/docs/reference/eql/core-concepts.mdx @@ -11,7 +11,7 @@ Everything in the EQL reference builds on four ideas: columns are typed as **dom ## Variants declare capability -EQL ships its searchable-encryption surface as PostgreSQL **domains in the `eql_v3` schema**, all backed by `jsonb`. Each scalar type generates a *family* of domain variants, and the variant you type a column as fixes its query capability. Each domain carries a `CHECK` constraint that validates the encrypted payload on insert, so a malformed or wrong-version value is rejected at write time rather than surfacing at query time. +EQL ships its searchable-encryption surface as PostgreSQL **domains in the `public` schema**, all backed by `jsonb` (the functions behind them live in `eql_v3` — [The three schemas](#the-three-schemas) explains the split). Each scalar type generates a *family* of domain variants, and the variant you type a column as fixes its query capability. Each domain carries a `CHECK` constraint that validates the encrypted payload on insert, so a malformed or wrong-version value is rejected at write time rather than surfacing at query time. There is no database-side configuration table. Earlier EQL versions tracked encryption config in the database (`config_add_table`, `config_add_column`, and friends) — those are gone in v3. The searchable surface of a column is fixed by the domain variant you type it as, and which index terms travel in a value's payload is decided by the encryption client (the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy)). The domain makes the matching operators resolve; the term in the payload is what makes them answer. @@ -19,24 +19,24 @@ For any scalar type `<T>`, the family looks like this: | Domain variant | Capability | | --- | --- | -| `eql_v3.<T>` | Storage and decryption only. | -| `eql_v3.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | -| `eql_v3.<T>_ord` | Comparisons (`<` … `>=`), `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | -| `eql_v3.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | +| `public.<T>` | Storage and decryption only. | +| `public.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | +| `public.<T>_ord` | Comparisons (`<` … `>=`), `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | +| `public.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | | `public.text_match` (text only) | Free-text token containment: `@>` / `<@`. | | `public.text_search` (text only) | Equality + ordering + token containment. | Two things worth calling out: -- **The bare variant blocks everything.** `eql_v3.<T>` carries no index term. Querying it with any comparison operator raises an "operator not supported" exception. Use it for columns you only ever store and decrypt — [Booleans](/reference/eql/booleans) covers this pattern in full. +- **The bare variant blocks everything.** `public.<T>` carries no index term. Querying it with any comparison operator raises an "operator not supported" exception. Use it for columns you only ever store and decrypt — [Booleans](/reference/eql/booleans) covers this pattern in full. - **Which index term backs each capability** is an implementation detail of the payload — covered in [Anatomy of an encrypted value](#anatomy-of-an-encrypted-value) below. ### SEM specifiers A trailing mechanism suffix — the `_ore` in `_ord_ore` — is a **SEM specifier**: it pins *which* searchable-encryption mechanism implements the capability, rather than just declaring the capability itself. -- `eql_v3.<T>_ord` declares *orderable* and leaves the mechanism to EQL's default — currently ORE (order-revealing encryption). -- `eql_v3.<T>_ord_ore` declares *orderable via ORE, explicitly*. Today the two are byte-identical surfaces backed by the same term. +- `public.<T>_ord` declares *orderable* and leaves the mechanism to EQL's default — currently ORE (order-revealing encryption). +- `public.<T>_ord_ore` declares *orderable via ORE, explicitly*. Today the two are byte-identical surfaces backed by the same term. The distinction earns its keep as mechanisms multiply: the EQL v3 release adds an **OPE** (order-preserving encryption) specifier for every orderable type — including `text` — at which point pinning a specifier documents and freezes a column's mechanism choice, while unspecified variants track the default. Each type page lists its available specifiers under an "SEM specifiers" heading. @@ -53,6 +53,22 @@ CREATE TABLE users ( Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.0 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use a separate domain, `public.json`, with its own operator surface — see [JSON](/reference/eql/json). +## The three schemas + +EQL spreads its surface across three PostgreSQL schemas, and the split is what makes EQL **upgradable in place**: + +| Schema | Holds | Do you call it? | +| --- | --- | --- | +| `public` | The encrypted **domain types** — every `public.<T>` variant you type a column as. | Referenced in your table DDL. | +| `eql_v3` | All **user-callable functions and operators** — the searchable-encryption API (`eql_v3.eq_term`, `eql_v3.jsonb_path_query`, the encrypted `=` / `<` / `@>` operators, `eql_v3.version()`). | Yes — this is the API. | +| `eql_v3_internal` | The **implementation functions** the domains and operators are built from. | No — never call these directly. | + +**Why the types live in `public`.** Your columns are typed as `public.integer_ord`, `public.text_eq`, and so on — never `eql_v3.*`. Keeping the types in the unversioned `public` schema means an EQL upgrade never rewrites your table definitions: the logic ships in a *versioned* schema (`eql_v3` today, a future `eql_v4` alongside it tomorrow) while the type names your schema depends on stay put. `eql_v2` was removed wholesale in 3.0.0 without any `public.*` type changing. + +**Why `eql_v3` is versioned.** The schema name encodes the major API version and is itself part of the public contract — a breaking change introduces a new `eql_vN` schema *beside* the old one rather than mutating it, so you migrate on your own timeline. Everything in `eql_v3` is fair game to call. + +**Why `eql_v3_internal` is off-limits.** These are the building blocks — term comparators, unsupported-operator blockers, cast helpers — that the operators and domain `CHECK` constraints wire together. They carry no stability guarantee and can change or disappear between releases. If you're reaching for `eql_v3_internal.*`, there's a `public` type or an `eql_v3` function that does what you want. + ## Anatomy of an encrypted value Every EQL encrypted value is a `jsonb` payload with a shared envelope plus the index terms that make it queryable. Earlier CipherStash docs called this format the **CipherCell** — this section is the current definition of the same structure. @@ -77,7 +93,7 @@ A `k` discriminator (`"ct"` for a scalar ciphertext, `"sv"` for a JSON document) ### Index-term keys -Alongside the envelope, a payload carries the index terms for its column's capability. Each key is backed by a SEM (searchable encrypted metadata) type in the `eql_v3` schema: +Alongside the envelope, a payload carries the index terms for its column's capability. Each key is backed by a SEM (searchable encrypted metadata) type in the `eql_v3_internal` schema: | Key | SEM type | Wire shape | Enables | Reveals | | --- | --- | --- | --- | --- | @@ -85,9 +101,9 @@ Alongside the envelope, a payload carries the index terms for its column's capab | `ob` | `eql_v3_internal.ore_block_256` (composite: array of `bytea` block terms) | Array of hex-encoded ORE blocks (block count varies by scalar width) | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ore` domains — and `=` / `<>`, since ORE comparison collapses to equality | The relative order of two values | | `bf` | `eql_v3_internal.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit — large filters emit negative positions) | `@>` / `<@` token containment on `_match` domains | Probabilistic token overlap between values | -The capability is encoded as **required keys**: the payload for an `public.text_eq` column must carry `hm`; an `public.integer_ord` payload must carry `ob` (and only `ob`); a `text_match` payload must carry `bf`; a `text_search` payload carries all three. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. +The capability is encoded as **required keys**: the payload for a `public.text_eq` column must carry `hm`; a `public.integer_ord` payload must carry `ob` (and only `ob`); a `text_match` payload must carry `bf`; a `text_search` payload carries all three. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. -A scalar payload for an `public.text_search` column (lookup + ordering + free-text match, so all three terms are required): +A scalar payload for a `public.text_search` column (lookup + ordering + free-text match, so all three terms are required): ```json { diff --git a/content/docs/reference/eql/dates-and-times.mdx b/content/docs/reference/eql/dates-and-times.mdx index 0fa3d25..1635a89 100644 --- a/content/docs/reference/eql/dates-and-times.mdx +++ b/content/docs/reference/eql/dates-and-times.mdx @@ -15,10 +15,10 @@ Both types generate the same `jsonb`-backed domain variants. The generic form: | Domain variant | Capability | | --- | --- | -| `eql_v3.<T>` | Storage and decryption only. | -| `eql_v3.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | -| `eql_v3.<T>_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | -| `eql_v3.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | +| `public.<T>` | Storage and decryption only. | +| `public.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | +| `public.<T>_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | +| `public.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | And every concrete domain this page covers: @@ -75,7 +75,7 @@ A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — ## Operators -| SQL operator | `eql_v3.<T>` | `<T>_eq` | `<T>_ord` / `<T>_ord_ore` | +| SQL operator | `public.<T>` | `<T>_eq` | `<T>_ord` / `<T>_ord_ore` | | --- | :---: | :---: | :---: | | `=` / `<>` | ❌ | ✅ | ✅ | | `<` `<=` `>` `>=` | ❌ | ❌ | ✅ | diff --git a/content/docs/reference/eql/grouping-and-aggregates.mdx b/content/docs/reference/eql/grouping-and-aggregates.mdx index da735e7..f4db3f3 100644 --- a/content/docs/reference/eql/grouping-and-aggregates.mdx +++ b/content/docs/reference/eql/grouping-and-aggregates.mdx @@ -53,10 +53,10 @@ SELECT COUNT(DISTINCT eql_v3.eq_term(email)) FROM logins; EQL ships `min` / `max` aggregates per ord-capable variant of every scalar type. The input type selects the aggregate, and the return type matches the input: ```sql -eql_v3.min(eql_v3.<T>_ord) RETURNS eql_v3.<T>_ord -eql_v3.max(eql_v3.<T>_ord) RETURNS eql_v3.<T>_ord -eql_v3.min(eql_v3.<T>_ord_ore) RETURNS eql_v3.<T>_ord_ore -eql_v3.max(eql_v3.<T>_ord_ore) RETURNS eql_v3.<T>_ord_ore +eql_v3.min(public.<T>_ord) RETURNS public.<T>_ord +eql_v3.max(public.<T>_ord) RETURNS public.<T>_ord +eql_v3.min(public.<T>_ord_ore) RETURNS public.<T>_ord_ore +eql_v3.max(public.<T>_ord_ore) RETURNS public.<T>_ord_ore ``` Comparison routes through the variant's `<` / `>` operator on the ORE term — no decryption happens in the database, and the result is an encrypted value the client decrypts. `NULL` inputs are skipped; an all-`NULL` input set returns `NULL`, matching native aggregate semantics. diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index 6b9ce9e..dc52454 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -11,7 +11,7 @@ Encrypt Query Language (EQL) is a set of types, operators, and functions for sto EQL itself never encrypts anything. Encryption and decryption happen in the client, using the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). EQL provides the database-side surface those clients query against: encrypted column types, the operators that compare them, and the term-extractor functions that make indexes work. -Every encrypted column is a `jsonb`-backed domain type in the `eql_v3` schema, and the domain variant you choose declares what the column can do — the full model is in [Core concepts](/reference/eql/core-concepts). +Every encrypted column is a `jsonb`-backed domain type in the `public` schema (the functions and operators behind them live in `eql_v3`), and the domain variant you choose declares what the column can do — the full model is in [Core concepts](/reference/eql/core-concepts). ## Install @@ -51,7 +51,7 @@ SELECT eql_v3.version(); </Steps> <Callout type="warn"> -`DROP SCHEMA eql_v3 CASCADE` drops every column typed as an `eql_v3` domain. The domain types live in the schema, and your columns depend on them. +To uninstall, drop both EQL schemas: `DROP SCHEMA eql_v3, eql_v3_internal CASCADE`. This removes the encrypted operators and functions, so queries against encrypted columns stop resolving — but your data survives: the `public.*` domain types (and the columns typed as them) live in the `public` schema and are left untouched. Remove those separately only if you intend to drop or re-type the columns. </Callout> ### dbdev @@ -81,7 +81,7 @@ GRANT CREATE ON SCHEMA public TO your_migration_user; GRANT ALTER ON ALL TABLES IN SCHEMA public TO your_migration_user; ``` -`CREATE ON DATABASE` creates the `eql_v3` schema and its types; `CREATE ON SCHEMA` and `ALTER` are needed to add encrypted columns (typed as `eql_v3` domains, with their `CHECK` constraints) to your tables. +`CREATE ON DATABASE` lets EQL create its schemas (`eql_v3`, `eql_v3_internal`) and the `public.*` domain types; `CREATE ON SCHEMA` and `ALTER` are needed to add encrypted columns (typed as `public.*` domains, with their `CHECK` constraints) to your tables. **Runtime user** — the application's day-to-day access: diff --git a/content/docs/reference/eql/indexes.mdx b/content/docs/reference/eql/indexes.mdx index 71fd42a..553a10b 100644 --- a/content/docs/reference/eql/indexes.mdx +++ b/content/docs/reference/eql/indexes.mdx @@ -33,13 +33,13 @@ Type the column as the domain variant that carries the term (see [Core concepts] ```sql -- Equality: hash index on eq_term --- (columns typed eql_v3.<T>_eq or text_search; equality on _ord columns +-- (columns typed public.<T>_eq or text_search; equality on _ord columns -- compares ORE terms, so the btree on ord_term below serves it) CREATE INDEX users_email_eq ON users USING hash (eql_v3.eq_term(email)); -- Range / ordering: btree index on ord_term --- (columns typed eql_v3.<T>_ord or _ord_ore) +-- (columns typed public.<T>_ord or _ord_ore) CREATE INDEX users_created_at_ord ON users USING btree (eql_v3.ord_term(created_at)); diff --git a/content/docs/reference/eql/json.mdx b/content/docs/reference/eql/json.mdx index 96d8cce..44df57e 100644 --- a/content/docs/reference/eql/json.mdx +++ b/content/docs/reference/eql/json.mdx @@ -34,7 +34,7 @@ An encrypted JSON document uses a different payload shape from the scalar types: The decoded `oc` value starts with a domain-tag byte (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext, so numeric and string values in one column keep a consistent total order. Earlier payload versions split this into two fields — `ocf` (fixed-width, numeric) and `ocv` (variable-width, string) — which consolidated into the single `oc` key; the tag byte now carries the distinction. -A document payload for an `public.json` column: +A document payload for a `public.json` column: ```json { diff --git a/content/docs/reference/eql/numbers.mdx b/content/docs/reference/eql/numbers.mdx index b30fab3..aeb3a63 100644 --- a/content/docs/reference/eql/numbers.mdx +++ b/content/docs/reference/eql/numbers.mdx @@ -17,10 +17,10 @@ Each numeric type generates the same `jsonb`-backed domain variants. The generic | Domain variant | Capability | | --- | --- | -| `eql_v3.<T>` | Storage and decryption only. | -| `eql_v3.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | -| `eql_v3.<T>_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | -| `eql_v3.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | +| `public.<T>` | Storage and decryption only. | +| `public.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | +| `public.<T>_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | +| `public.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | And every concrete domain this page covers: @@ -80,7 +80,7 @@ A value for an `_ord` column carries the shared envelope keys (`v`, `i`, `c` — ## Operators -| SQL operator | `eql_v3.<T>` | `<T>_eq` | `<T>_ord` / `<T>_ord_ore` | +| SQL operator | `public.<T>` | `<T>_eq` | `<T>_ord` / `<T>_ord_ore` | | --- | :---: | :---: | :---: | | `=` / `<>` | ❌ | ✅ | ✅ | | `<` `<=` `>` `>=` | ❌ | ❌ | ✅ | From a0e92041e9fa578660fc106a41241c40a0033826 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 17:39:30 +1000 Subject: [PATCH 35/38] docs(eql): rename public.jsonb_query -> eql_v3.query_jsonb EQL v3 moved the query-operand domains into the eql_v3 schema and renamed them to the query_<name> prefix, shipped in eql-3.0.0-alpha.3. The jsonb containment needle is now `eql_v3.query_jsonb`; `public.jsonb_query` no longer exists. Update the two hand-written pages (filtering.mdx, json.mdx) that cast `@>` operands to it. This clears the STRICT drift-lint, which (now that alpha.3 ships a real json/eql-manifest.json) correctly flagged public.jsonb_query as absent from the manifest. Verified: drift check passes and the build compiles all pages. functions.mdx regenerates from the manifest at build. --- content/docs/reference/eql/filtering.mdx | 2 +- content/docs/reference/eql/json.mdx | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/content/docs/reference/eql/filtering.mdx b/content/docs/reference/eql/filtering.mdx index 9dc4394..272b127 100644 --- a/content/docs/reference/eql/filtering.mdx +++ b/content/docs/reference/eql/filtering.mdx @@ -74,7 +74,7 @@ Encrypted JSON documents (`public.json`) filter by containment and path existenc ```sql -- Does the document contain this (encrypted) structure? -SELECT * FROM orders WHERE metadata @> $1::public.jsonb_query; +SELECT * FROM orders WHERE metadata @> $1::eql_v3.query_jsonb; -- Does this path exist in the document? SELECT * FROM orders WHERE eql_v3.jsonb_path_exists(metadata, 'region_selector'); diff --git a/content/docs/reference/eql/json.mdx b/content/docs/reference/eql/json.mdx index 44df57e..bf0156f 100644 --- a/content/docs/reference/eql/json.mdx +++ b/content/docs/reference/eql/json.mdx @@ -19,7 +19,7 @@ Three `jsonb`-backed domains make up the encrypted JSON surface: | --- | --- | | `public.json` | The column type. An encrypted document envelope carrying an `sv` array — one encrypted entry per path in the document. | | `public.jsonb_entry` | A single entry from the vector: a selector, a ciphertext, and exactly one index term. This is what `->` returns. | -| `public.jsonb_query` | A containment needle: entries with selectors and index terms but **no ciphertext**. This is what you cast a `@>` operand to. | +| `eql_v3.query_jsonb` | A containment needle: entries with selectors and index terms but **no ciphertext**. This is what you cast a `@>` operand to. | ## Payload shape @@ -53,7 +53,7 @@ A document payload for a `public.json` column: - Second entry: a string leaf — `oc` starting with tag `01` - Third entry: a numeric leaf — `oc` starting with tag `00` -A containment **query** payload (`public.jsonb_query`) has the same `sv` shape but its entries carry no `c` — containment matches selectors and index terms, never ciphertexts. This is the needle the client builds for a `@>` query: +A containment **query** payload (`eql_v3.query_jsonb`) has the same `sv` shape but its entries carry no `c` — containment matches selectors and index terms, never ciphertexts. This is the needle the client builds for a `@>` query: ```json { @@ -111,11 +111,11 @@ Use containment (`@>` / `<@`), field access (`->` / `->>`), or the `eql_v3.jsonb ## Containment: `@>` and `<@` -`@>` tests whether the encrypted document contains a structure; `<@` is the reverse. Build the needle with the client and cast it to `public.jsonb_query` (a typed `public.json` or `public.jsonb_entry` operand also works): +`@>` tests whether the encrypted document contains a structure; `<@` is the reverse. Build the needle with the client and cast it to `eql_v3.query_jsonb` (a typed `public.json` or `public.jsonb_entry` operand also works): ```sql SELECT * FROM orders -WHERE metadata @> $1::public.jsonb_query; +WHERE metadata @> $1::eql_v3.query_jsonb; ``` This is the encrypted equivalent of the plaintext `metadata @> '{"customer": {"tier": "premium"}}'`: containment checks that every encrypted term in the needle exists in the document's `sv` vector. `eql_v3.to_ste_vec_query(doc)` converts a stored document into the needle shape, and `eql_v3.ste_vec_contains(a, b)` is the function form backing `@>`. @@ -229,7 +229,7 @@ Find premium orders. The client encrypts the needle `{"customer": {"tier": "prem ```sql SELECT id FROM orders -WHERE metadata @> $1::public.jsonb_query; +WHERE metadata @> $1::eql_v3.query_jsonb; ``` Add the GIN index from above once the table grows. From 79b88330d0d274e1173c1fbb244506cb3a5f16d2 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 18:40:23 +1000 Subject: [PATCH 36/38] docs(eql): add an EQL-version banner linking to the v2 reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every EQL v3 reference page carries a shared <EqlVersion> callout stating the EQL release it's generated/validated against, with a link to the retained EQL v2 reference (/docs/reference/eql/v2) for readers still on the older generation. The version is single-sourced from the EQL release manifest: generate-eql-api-docs.ts writes the manifest's own `version` to src/lib/eql-version.ts, which the component reads. No hardcoded version — whatever the built release reports (e.g. a bare "DEV" today) is what shows, on every page including functions.mdx. --- content/docs/reference/eql/booleans.mdx | 2 ++ content/docs/reference/eql/core-concepts.mdx | 2 ++ .../docs/reference/eql/dates-and-times.mdx | 2 ++ content/docs/reference/eql/filtering.mdx | 2 ++ content/docs/reference/eql/functions.mdx | 2 ++ .../reference/eql/grouping-and-aggregates.mdx | 2 ++ content/docs/reference/eql/index.mdx | 2 ++ content/docs/reference/eql/indexes.mdx | 2 ++ content/docs/reference/eql/joins.mdx | 2 ++ content/docs/reference/eql/json.mdx | 2 ++ content/docs/reference/eql/numbers.mdx | 2 ++ content/docs/reference/eql/sorting.mdx | 2 ++ content/docs/reference/eql/text.mdx | 2 ++ scripts/generate-eql-api-docs.ts | 14 +++++++++++++ src/components/eql-version.tsx | 20 +++++++++++++++++++ src/lib/eql-version.ts | 4 ++++ src/mdx-components.tsx | 2 ++ 17 files changed, 66 insertions(+) create mode 100644 src/components/eql-version.tsx create mode 100644 src/lib/eql-version.ts diff --git a/content/docs/reference/eql/booleans.mdx b/content/docs/reference/eql/booleans.mdx index 3cbbf73..294a846 100644 --- a/content/docs/reference/eql/booleans.mdx +++ b/content/docs/reference/eql/booleans.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + Every scalar type has a storage-only variant — for `bool` it's the only one. EQL ships `public.boolean` and nothing else: there is no `bool_eq` and no `bool_ord`. An encrypted boolean column can be stored, decrypted, and null-checked; it cannot be filtered, sorted, grouped, or joined on. ## Why there are no query variants diff --git a/content/docs/reference/eql/core-concepts.mdx b/content/docs/reference/eql/core-concepts.mdx index ffdd927..ea9884b 100644 --- a/content/docs/reference/eql/core-concepts.mdx +++ b/content/docs/reference/eql/core-concepts.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + Everything in the EQL reference builds on four ideas: columns are typed as **domain variants** that declare what they can do, every value is a **`jsonb` payload** carrying encrypted index terms, **operands must be typed** for the encrypted operators to resolve, and anything a column can't do **fails loudly** instead of returning wrong rows. This page is the canonical home for all four — the per-type and per-query pages link back here rather than restating them. ## Variants declare capability diff --git a/content/docs/reference/eql/dates-and-times.mdx b/content/docs/reference/eql/dates-and-times.mdx index 1635a89..ea8f0a9 100644 --- a/content/docs/reference/eql/dates-and-times.mdx +++ b/content/docs/reference/eql/dates-and-times.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + `date` and `timestamp` columns carry the same capabilities as [encrypted numbers](/reference/eql/numbers) — equality, ranges, ordering, `MIN` / `MAX` — but the queries they serve are temporal: time windows, newest-first listings, retention cutoffs, "when did this last happen". ## Variants diff --git a/content/docs/reference/eql/filtering.mdx b/content/docs/reference/eql/filtering.mdx index 272b127..6210731 100644 --- a/content/docs/reference/eql/filtering.mdx +++ b/content/docs/reference/eql/filtering.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + Every filter below is ordinary SQL — the encrypted operators resolve from the column's domain variant, and a functional index on the matching term extractor serves the predicate. One rule applies throughout: **operands must be typed** (`$1::public.text_eq`, not a bare literal), or PostgreSQL falls through to native `jsonb` semantics. See [Core concepts](/reference/eql/core-concepts) for the typed-operand rule and how unsupported operators fail loudly instead of returning wrong rows. ## Equality: `=` and `<>` diff --git a/content/docs/reference/eql/functions.mdx b/content/docs/reference/eql/functions.mdx index e9f86a5..22cf3c0 100644 --- a/content/docs/reference/eql/functions.mdx +++ b/content/docs/reference/eql/functions.mdx @@ -9,6 +9,8 @@ verifiedAgainst: {/* GENERATED — do not edit. Produced by scripts/generate-eql-api-docs.ts from the EQL manifest (v3.0.0-sample). */} +<EqlVersion /> + <Callout type="info"> Generated from the **EQL 3.0.0-sample** manifest (the Doxygen'd SQL is the source of truth). For the model behind these — variants, terms, typed operands — see [Core concepts](/reference/eql/core-concepts). </Callout> diff --git a/content/docs/reference/eql/grouping-and-aggregates.mdx b/content/docs/reference/eql/grouping-and-aggregates.mdx index f4db3f3..615d6b2 100644 --- a/content/docs/reference/eql/grouping-and-aggregates.mdx +++ b/content/docs/reference/eql/grouping-and-aggregates.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + Grouping and deduplication need an equality term, so they work on the same variants as `=`: `_eq`, `_ord` / `_ord_ore`, and `text_search`. `MIN` / `MAX` need an ordering term (`_ord` / `_ord_ore`, `text_search`). Arithmetic aggregates don't work at all — that's the last section. As everywhere, operands and call-site casts must be typed; see [Core concepts](/reference/eql/core-concepts). ## `GROUP BY` and `DISTINCT` diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index dc52454..205916c 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + Encrypt Query Language (EQL) is a set of types, operators, and functions for storing and querying encrypted data in PostgreSQL. It installs as a single plain-SQL script — no extension packaging, no superuser, no operator classes — so it runs on Supabase, RDS, Cloud SQL, and self-hosted Postgres alike. EQL itself never encrypts anything. Encryption and decryption happen in the client, using the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). EQL provides the database-side surface those clients query against: encrypted column types, the operators that compare them, and the term-extractor functions that make indexes work. diff --git a/content/docs/reference/eql/indexes.mdx b/content/docs/reference/eql/indexes.mdx index 553a10b..df87fb4 100644 --- a/content/docs/reference/eql/indexes.mdx +++ b/content/docs/reference/eql/indexes.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + EQL indexes are ordinary PostgreSQL functional indexes over **term-extractor functions** — never an index or operator class on the column itself. Each extractor returns a small per-row index term whose return type already carries a default operator class: | Extractor | Index method | Term | Capability | diff --git a/content/docs/reference/eql/joins.mdx b/content/docs/reference/eql/joins.mdx index b4e2850..2b31347 100644 --- a/content/docs/reference/eql/joins.mdx +++ b/content/docs/reference/eql/joins.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + Equijoins work on equality-capable variants (`_eq`, `_ord` / `_ord_ore`, `text_search`) — the join condition is just encrypted equality. But there is one constraint that has no plaintext equivalent, and it is the single thing to internalize on this page: <Callout type="warn"> diff --git a/content/docs/reference/eql/json.mdx b/content/docs/reference/eql/json.mdx index bf0156f..273fc8c 100644 --- a/content/docs/reference/eql/json.mdx +++ b/content/docs/reference/eql/json.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + `public.json` is EQL's encrypted JSON document type, built on structured encryption (**ste_vec**). The document is encrypted as a vector of encrypted entries — one entry per path inside the document — and every path is queryable without decryption: containment, field and array access, and equality or range comparisons on extracted leaves. Like every EQL type, `public.json` holds ciphertext the database can't read. Encryption, decryption, and selector generation happen in the client — the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). See [Searchable encryption](/concepts/searchable-encryption) for how querying ciphertext works at all. diff --git a/content/docs/reference/eql/numbers.mdx b/content/docs/reference/eql/numbers.mdx index aeb3a63..7c74fd2 100644 --- a/content/docs/reference/eql/numbers.mdx +++ b/content/docs/reference/eql/numbers.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + Six numeric types share one identical query surface: `int2`, `int4`, `int8`, `float4`, `float8`, and `numeric`. These are the columns you filter by range, sort, and take a `MIN` / `MAX` over — salaries, totals, rates, quantities. Date and time columns have the same capabilities but their own semantics — see [Dates & times](/reference/eql/dates-and-times). There is no free-text matching for numeric types — `_match` and `_search` are [text-only variants](/reference/eql/text). diff --git a/content/docs/reference/eql/sorting.mdx b/content/docs/reference/eql/sorting.mdx index 2e7ef2b..c847404 100644 --- a/content/docs/reference/eql/sorting.mdx +++ b/content/docs/reference/eql/sorting.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + `ORDER BY` on an encrypted column needs an ORE ordering term: it works on `_ord` / `_ord_ore` variants of every scalar and on `text_search`. ORE terms are order-preserving, so the database sorts ciphertext in exactly the order the plaintext would sort — without decrypting anything. Which variants carry the term is covered in [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), and [Text](/reference/eql/text); the variant model itself is in [Core concepts](/reference/eql/core-concepts). Sorting a variant *without* an ORE term (`_eq`, `text_match`, bare storage variants) won't raise — but the order is meaningless. Type the column as an `_ord` variant when ordering matters. diff --git a/content/docs/reference/eql/text.mdx b/content/docs/reference/eql/text.mdx index 408ed97..7865e8c 100644 --- a/content/docs/reference/eql/text.mdx +++ b/content/docs/reference/eql/text.mdx @@ -7,6 +7,8 @@ verifiedAgainst: eql: "3.0.0" --- +<EqlVersion /> + Text is the richest encrypted scalar. Beyond the four variants every scalar type gets, `text` adds two of its own: `text_match` for encrypted free-text matching, and `text_search` for columns you need to look up, sort, *and* search. Emails, names, tax IDs, addresses — this page is the full surface for all of them. ## Variants diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index c6cbfa6..0967d92 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -43,6 +43,10 @@ const MANIFEST_PATH = (fs.existsSync(RELEASE_MANIFEST) ? RELEASE_MANIFEST : SAMPLE_MANIFEST); const EQL_DIR = path.join(process.cwd(), "content/docs/reference/eql"); const OUT_FILE = path.join(EQL_DIR, "functions.mdx"); +// Single source for the EQL version the whole reference is built against: the +// release manifest's own `version`. Written here so the <EqlVersion> banner on +// every EQL page reads the same release-derived value (no hardcoded constant). +const VERSION_FILE = path.join(process.cwd(), "src/lib/eql-version.ts"); // Report-only against the illustrative sample; a failing gate against any real // manifest (the release asset or an explicit override). const STRICT = MANIFEST_PATH !== SAMPLE_MANIFEST; @@ -191,6 +195,8 @@ function render(manifest: Manifest): string { "", `{/* GENERATED — do not edit. Produced by scripts/generate-eql-api-docs.ts from the EQL manifest (v${version}). */}`, "", + "<EqlVersion />", + "", `<Callout type="info">`, `Generated from the **EQL ${version}** manifest (the Doxygen'd SQL is the source of truth). For the model behind these — variants, terms, typed operands — see [Core concepts](/reference/eql/core-concepts).`, `</Callout>`, @@ -262,6 +268,14 @@ function main() { fs.mkdirSync(EQL_DIR, { recursive: true }); fs.writeFileSync(OUT_FILE, render(manifest)); + + // Emit the release version for the <EqlVersion> banner (shared by every EQL + // reference page, hand-written and generated alike). + fs.mkdirSync(path.dirname(VERSION_FILE), { recursive: true }); + fs.writeFileSync( + VERSION_FILE, + `// GENERATED by scripts/generate-eql-api-docs.ts from the EQL release manifest.\n// Do not edit; the prebuild step overwrites it with the version of the EQL\n// release the docs are built against.\nexport const EQL_VERSION = ${JSON.stringify(manifest.version)};\n`, + ); console.log( `✓ Generated ${path.relative(process.cwd(), OUT_FILE)} from EQL ${manifest.version} (${manifest.functions.length} functions)`, ); diff --git a/src/components/eql-version.tsx b/src/components/eql-version.tsx new file mode 100644 index 0000000..d3c43e0 --- /dev/null +++ b/src/components/eql-version.tsx @@ -0,0 +1,20 @@ +import { Callout } from "fumadocs-ui/components/callout"; +import Link from "next/link"; +import { EQL_VERSION } from "@/lib/eql-version"; + +/** + * Version banner for the EQL v3 reference. Shows the EQL release the docs were + * generated/validated against — sourced from the release manifest's own version + * (written to `@/lib/eql-version` by generate-eql-api-docs.ts at build time) — + * and links to the retained EQL v2 reference for readers on the older + * generation. + */ +export function EqlVersion() { + return ( + <Callout title="EQL version" type="info"> + This reference is generated and validated against{" "} + <strong>EQL {EQL_VERSION}</strong>. Running EQL 2.x? See the{" "} + <Link href="/reference/eql/v2">EQL v2 reference</Link>. + </Callout> + ); +} diff --git a/src/lib/eql-version.ts b/src/lib/eql-version.ts new file mode 100644 index 0000000..58f5f8a --- /dev/null +++ b/src/lib/eql-version.ts @@ -0,0 +1,4 @@ +// GENERATED by scripts/generate-eql-api-docs.ts from the EQL release manifest. +// Do not edit; the prebuild step overwrites it with the version of the EQL +// release the docs are built against. +export const EQL_VERSION = "3.0.0-sample"; diff --git a/src/mdx-components.tsx b/src/mdx-components.tsx index ba31fd2..b684d9c 100644 --- a/src/mdx-components.tsx +++ b/src/mdx-components.tsx @@ -3,6 +3,7 @@ import { Step, Steps } from "fumadocs-ui/components/steps"; import defaultMdxComponents from "fumadocs-ui/mdx"; import type { MDXComponents } from "mdx/types"; import { TrackedCodeBlock } from "@/components/code-block"; +import { EqlVersion } from "@/components/eql-version"; export function getMDXComponents(components?: MDXComponents): MDXComponents { return { @@ -13,6 +14,7 @@ export function getMDXComponents(components?: MDXComponents): MDXComponents { Callout, Steps, Step, + EqlVersion, ...components, }; } From 78a4a2257e6078039cfe1199de54aa7753c46931 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 14:53:08 +1000 Subject: [PATCH 37/38] docs(eql): install via `stash eql install` CLI Replace the curl + psql manual install with the Stash CLI (`npx stash eql install --eql-version 3`), validated against the cipherstash/stack main branch. Covers DATABASE_URL / --database-url, idempotent re-runs, `eql status`/`eql upgrade`, and the CLI's no-superuser auto-fallback on managed Postgres. --- content/docs/reference/eql/index.mdx | 39 ++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index 205916c..52cb643 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -9,7 +9,7 @@ verifiedAgainst: <EqlVersion /> -Encrypt Query Language (EQL) is a set of types, operators, and functions for storing and querying encrypted data in PostgreSQL. It installs as a single plain-SQL script — no extension packaging, no superuser, no operator classes — so it runs on Supabase, RDS, Cloud SQL, and self-hosted Postgres alike. +Encrypt Query Language (EQL) is a set of types, operators, and functions for storing and querying encrypted data in PostgreSQL. The [Stash CLI](/reference/cli) installs it over an ordinary database connection — no extension packaging, no superuser, no operator classes — so it runs on Supabase, RDS, Cloud SQL, and self-hosted Postgres alike. EQL itself never encrypts anything. Encryption and decryption happen in the client, using the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). EQL provides the database-side surface those clients query against: encrypted column types, the operators that compare them, and the term-extractor functions that make indexes work. @@ -20,30 +20,45 @@ Every encrypted column is a `jsonb`-backed domain type in the `public` schema (t <Steps> <Step> -### Download the install script +### Run the installer -Each [GitHub release](https://github.com/cipherstash/encrypt-query-language/releases) publishes a versioned `cipherstash-encrypt.sql`: +Point the [Stash CLI](/reference/cli) at your database and install the `eql_v3` schema: -```sh -curl -sLo cipherstash-encrypt.sql https://github.com/cipherstash/encrypt-query-language/releases/latest/download/cipherstash-encrypt.sql +```bash +npx stash eql install --eql-version 3 ``` +`stash eql install` scaffolds a `stash.config.ts` if your project doesn't have one, then installs the `eql_v3` schema — all domain types, operators, functions, and aggregates — along with the `cs_migrations` tracking schema that the `stash encrypt` commands depend on. It is idempotent: re-running reports that EQL is already installed and changes nothing. Pass `--force` to reinstall in place. + +<Callout> +`--eql-version 3` selects the native `eql_v3` domain schema this reference documents. Without it the CLI installs EQL v2. v3 installs directly against the database, so the `--drizzle`, `--migration`, and `--latest` flags — all v2-only — don't apply. +</Callout> + </Step> <Step> -### Run it against each database +### Point it at each database -```sh -psql -f cipherstash-encrypt.sql +The CLI reads the connection string from `DATABASE_URL` (your environment or a `.env` file) and prompts if it can't find one. Override it for a single run — handy when installing across several databases — with `--database-url`, which is used for that run only and never written to disk: + +```bash +npx stash eql install --eql-version 3 \ + --database-url postgres://user:pass@host:5432/mydb ``` -The script installs the `eql_v3` schema with all domain types, operators, functions, and aggregates. It is idempotent: re-running it upgrades the `eql_v3` surface in place and won't remove anything you've built on top of it. To upgrade, download the latest script and run it again. +Run the installer against every database that stores encrypted columns. </Step> <Step> ### Verify +```bash +npx stash eql status +``` + +Or check directly from SQL: + ```sql SELECT eql_v3.version(); -- '3.0.0' @@ -52,13 +67,15 @@ SELECT eql_v3.version(); </Step> </Steps> +To pull in a newer EQL release later, run `npx stash eql upgrade --eql-version 3`. + <Callout type="warn"> To uninstall, drop both EQL schemas: `DROP SCHEMA eql_v3, eql_v3_internal CASCADE`. This removes the encrypted operators and functions, so queries against encrypted columns stop resolving — but your data survives: the `public.*` domain types (and the columns typed as them) live in the `public` schema and are left untouched. Remove those separately only if you intend to drop or re-type the columns. </Callout> ### dbdev -EQL is also published to [dbdev](https://database.dev/cipherstash/eql). The dbdev release can lag behind GitHub releases, so prefer the install script when you need the latest version. +EQL is also published to [dbdev](https://database.dev/cipherstash/eql). The dbdev release can lag behind GitHub releases, so prefer `stash eql install` when you need the latest version. ### Docker for local development @@ -100,7 +117,7 @@ Schema changes — adding or removing encrypted columns — always go through th ## Managed Postgres and Supabase -EQL v3 is designed to install without superuser. There are no custom operator classes (which managed platforms typically block), no `postgresql.conf` changes, and no separate Supabase build — the single install script is the same artefact everywhere. Indexing works through ordinary functional indexes over EQL's term-extractor functions, which any user who can `CREATE INDEX` can build. See the [Supabase integration](/integrations/supabase) for platform-specific setup. +EQL v3 is designed to install without superuser. There are no custom operator classes (which managed platforms typically block), no `postgresql.conf` changes, and no separate Supabase build — the same schema installs everywhere. When the connected role lacks superuser, `stash eql install` detects it and automatically falls back to the no-operator-family install variant, so it works on Supabase, Neon, and RDS without extra flags. Indexing works through ordinary functional indexes over EQL's term-extractor functions, which any user who can `CREATE INDEX` can build. See the [Supabase integration](/integrations/supabase) for platform-specific setup. ## Understand From e19f7d0a6f9413bc0738f1233c52504a6ca0c560 Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Wed, 8 Jul 2026 15:01:08 +1000 Subject: [PATCH 38/38] docs(eql): document stash.config.ts Add a Configuration subsection covering the two defineConfig fields (databaseUrl / client), the resolveDatabaseUrl resolution chain, and the scaffolded template. Link the inline stash.config.ts mention to it. --- content/docs/reference/eql/index.mdx | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index 52cb643..693645a 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -28,7 +28,7 @@ Point the [Stash CLI](/reference/cli) at your database and install the `eql_v3` npx stash eql install --eql-version 3 ``` -`stash eql install` scaffolds a `stash.config.ts` if your project doesn't have one, then installs the `eql_v3` schema — all domain types, operators, functions, and aggregates — along with the `cs_migrations` tracking schema that the `stash encrypt` commands depend on. It is idempotent: re-running reports that EQL is already installed and changes nothing. Pass `--force` to reinstall in place. +`stash eql install` scaffolds a [`stash.config.ts`](#configuration) if your project doesn't have one, then installs the `eql_v3` schema — all domain types, operators, functions, and aggregates — along with the `cs_migrations` tracking schema that the `stash encrypt` commands depend on. It is idempotent: re-running reports that EQL is already installed and changes nothing. Pass `--force` to reinstall in place. <Callout> `--eql-version 3` selects the native `eql_v3` domain schema this reference documents. Without it the CLI installs EQL v2. v3 installs directly against the database, so the `--drizzle`, `--migration`, and `--latest` flags — all v2-only — don't apply. @@ -73,6 +73,24 @@ To pull in a newer EQL release later, run `npx stash eql upgrade --eql-version 3 To uninstall, drop both EQL schemas: `DROP SCHEMA eql_v3, eql_v3_internal CASCADE`. This removes the encrypted operators and functions, so queries against encrypted columns stop resolving — but your data survives: the `public.*` domain types (and the columns typed as them) live in the `public` schema and are left untouched. Remove those separately only if you intend to drop or re-type the columns. </Callout> +### Configuration + +`stash eql install` reads a `stash.config.ts` at your project root, scaffolding one on first run if it's missing. The same file is used by the `stash db` and `stash encrypt` commands. It exports a `defineConfig` object with two fields: + +| Field | Required | Description | +| --- | --- | --- | +| `databaseUrl` | Yes | PostgreSQL connection string. The scaffold sets it to `await resolveDatabaseUrl()`, which resolves from the `--database-url` flag, then `DATABASE_URL` (environment or `.env`), then `supabase status`, then an interactive prompt. The resolved connection string is never written to disk — only the declarative call is. | +| `client` | No | Path to your encryption client file. Defaults to `./src/encryption/index.ts`. | + +```ts filename="stash.config.ts" +import { defineConfig, resolveDatabaseUrl } from "stash"; + +export default defineConfig({ + databaseUrl: await resolveDatabaseUrl(), + client: "./src/encryption/index.ts", +}); +``` + ### dbdev EQL is also published to [dbdev](https://database.dev/cipherstash/eql). The dbdev release can lag behind GitHub releases, so prefer `stash eql install` when you need the latest version.