Update EQL v3 concrete type names via FFI 0.28#575
Open
tobyhede wants to merge 5 commits into
Open
Conversation
🦋 Changeset detectedLatest commit: 6e6e932 The changes in this PR will be included in the next version bump. This PR includes changesets to release 6 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Squashes the concrete-type domain work and the protect-ffi 0.24 -> 0.28 bump into a single commit, removing the intermediate 0.27 -> 0.28 rename and re-bump churn (package.json, pnpm-lock.yaml, changeset).
tobyhede
force-pushed
the
chore/eql-v3-ffi-0-28
branch
from
fcbe7da to
89b903f
Compare
Adopt the latest EQL v3 SQL bundle and realign the v3 DSL so the SDK's emitted domain names byte-match the installed bundle (a mismatch fails CREATE TABLE/cast resolution at runtime, only caught by live-PG tests). - Re-vendor packages/stack/__tests__/fixtures/eql-v3/cipherstash-encrypt-v3.sql - Type domains eql_v3.* -> public.* (user-column domains now live in public so app tables survive EQL uninstall); eql_v3.bool -> public.boolean - Index-term constructors hmac_256/ore_block_256/bloom_filter moved to eql_v3_internal in live-test SQL; operator extractors (eq_term/ord_term/ match_term) stay in eql_v3 - Remap eqlType strings + catalog keys + type-level/unit assertions Verified: test:types 60/60, schema-v3 unit 162/162, no regressions. Live-PG contract suite runs in CI.
The live suites gate themselves on CS_WORKSPACE_CRN / CS_CLIENT_ID / CS_CLIENT_KEY / CS_CLIENT_ACCESS_KEY (vitest `describe.skip`, Deno `test.ignore`). If any secret is unset, rotated, or absent (e.g. a fork PR), those suites silently skip and CI goes green with ZERO live encryption coverage — no signal that the whole v3 live matrix never ran. Add a `require-cs-secrets` local composite action that asserts all four vars are present and fails the job otherwise, and wire it into every credential-dependent job: - tests.yml: run-tests, e2e-tests, wasm-e2e-tests - prisma-next-e2e.yml - prisma-example-readme-e2e.yml The wasm-e2e-tests job already had an inline version of this guard; it's now folded into the shared action so the check lives in one place. Note: run-tests-bun is intentionally left unguarded — it is `continue-on-error: true` (and runs vitest with `|| true`), so a guard there could not fail CI regardless.
…to operand query API
Root cause of the red v3 live suites: the SDK never passed `eqlVersion` to
protect-ffi's `newClient`, so v3 clients defaulted to `eqlVersion: 2`. A v2-mode
client cannot resolve v3 concrete-type columns and the native addon throws
"Cannot convert undefined or null to object" for every encrypt.
Source fix (thread the version through the existing config seam):
- types.ts: add `eqlVersion?: 2 | 3` to `ClientConfig`.
- encryption/index.ts: `init()` forwards `eqlVersion` to `newClient`.
- encryption/v3.ts: `EncryptionV3` forces `eqlVersion: 3` (v3-only invariant).
v2 clients leave it unset -> protect-ffi default 2, so no v2 change.
Test fixes:
- matrix-live{,-pg}.test.ts: `slug` stripped the stale `eql_v3.` prefix, but the
domains were renamed to `public.*`, producing invalid dotted column names
(`public.boolean`) that also fail FFI identifier resolution. Strip `public.`.
- matrix-live-pg + schema-v3-pg: protect-ffi 0.28 has no v3 scalar query wire
shape (`encryptQuery` throws EQL_V3_QUERY_UNSUPPORTED). Upgrade both suites to
the supported API: encrypt a FULL operand with `client.encrypt` and compare
with the public two-arg functions `eql_v3.eq/gte/lte/contains(col, $::jsonb)`.
- text_ord/text_ord_ore carry both `hm` and `ob`; `eql_v3.eq` would compare
`hm`, so the ORE proof uses a degenerate `gte AND lte` range. Pure-ORE
numeric/date domains keep `eql_v3.eq` (resolves to `ord_term`).
- schema-v3-pg used the base `Encryption` factory with v3 tables; pass
`config: { eqlVersion: 3 }` so its storage encrypts succeed.
- Rebuild the query-only-rejection test without `encryptQuery` (strip the
ciphertext from a full operand).
- Range test asserts membership via `ORDER BY label`: `eql_v3.gte/lte` order
correctly (selects grace+zora, excludes ada/alan), but `ORDER BY
eql_v3.ord_term` sorts by the raw term, not ORE order, for text_search.
- Both pg suites: `DROP TABLE IF EXISTS` before create, so a stale local table
from before the `public.*` rename can't mask the run (no-op in CI).
Verified live: matrix-live 176/176, matrix-live-pg 40/40, schema-v3-pg 8/8,
type tests 60/60.
…defect
Two follow-ups surfaced while making the v3 suites green.
1. Live-test timeout flakiness. vitest.config.ts set no global testTimeout, so
it used Vitest's 5000ms default. 300+ live tests already pass an explicit
`, 30000)`; the few without one inherited 5000ms and intermittently timed out
under concurrent ZeroKMS load (they pass in isolation). Set testTimeout and
hookTimeout to 30000 to match the dominant explicit value — high enough for
real network latency, low enough to still surface a genuine hang. Full live
suite now 1036 passed / 0 timeouts.
2. `ORDER BY eql_v3.ord_term(col)` does not sort in ORE order. Root cause: the
ORE-correct btree opclass for the composite `ore_block_256` type is created
in a superuser-gated DO block in the vendored bundle and is skipped on
managed/non-superuser Postgres, so ORDER BY silently falls back to record
(raw-byte) comparison. The `eql_v3.gte/lte/lt/gt` predicates are ORE-correct
regardless (they don't use sort/index machinery). This is an upstream bundle
limitation, so the vendored .sql is NOT patched; instead:
- docs/eql-v3-ord-term-ordering-defect.md records the defect, live evidence,
a minimal repro, and the recommended upstream fix, and warns that a future
v3 orderBy must not lower to a bare `ORDER BY ord_term(...)`.
- schema-v3-pg.test.ts corrects the range-test comment and adds a positive
ORE-total-order proof reconstructed from pairwise `eql_v3.lt` (the only
ORE-correct path), independent of the superuser-gated opclass.
Verified live: schema-v3-pg 9/9; full package suite 1036 passed / 3 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@cipherstash/protect-ffito0.27.0.Integer,Smallint,Boolean,Real,Double, andBigint.Validation
pnpm --filter @cipherstash/stack buildpnpm exec vitest run __tests__/schema-v3.test.ts __tests__/v3-matrix/matrix.test.tspnpm exec vitest --run --typecheck.only __tests__/schema-v3.test-d.ts __tests__/typed-client-v3.test-d.ts __tests__/v3-matrix/matrix.test-d.tsNotes
Summary by CodeRabbit
New Features
Datevalues in v3 encryption and query encryption flows.Bug Fixes