Skip to content

Fix indexing and queries for numeric values with the high bit set#36

Closed
SewerynKras wants to merge 1 commit into
mainfrom
fix/numeric-high-bit-values
Closed

Fix indexing and queries for numeric values with the high bit set#36
SewerynKras wants to merge 1 commit into
mainfrom
fix/numeric-high-bit-values

Conversation

@SewerynKras

Copy link
Copy Markdown

What was broken

A numeric annotation value of 2^63 or more (the user hit 2^64−1, the documented max) could not be bound to SQLite: Go's database/sql rejects uint64 arguments with the high bit set:

sql: converting argument $2 type: uint64 values with high bit set are not supported

Since event-following is deterministic, one such entity was a poison message: every node failed the same block, rolled back, and retried forever. This is what took down all Braga RPC nodes on 2026-07-03. The same limit applied to the read path — an arkiv_query with such a literal failed the same way.

The fix

Numeric values are now stored with their top bit flipped (value XOR 2^63), mapping the full uint64 range [0, 2^64) one-to-one onto SQLite's signed INTEGER range [−2^63, 2^63) while preserving order. Unsigned ordering matches SQLite's signed ordering, so equality, IN, and range comparisons (<, <=, >, >=) are all correct for every possible value — including across the 2^63 boundary.

This supersedes #35, which used the raw int64 bit-pattern cast: that unblocks indexing but leaves range queries silently wrong for the high band (big > 5 misses 2^64−1; big < 10 matches it), and once high-bit values exist in databases under that encoding, converting to an order-preserving one requires a table rebuild instead of a single UPDATE. Doing it in one shot keeps the migration trivial.

Changes

  • store/numeric_value.go (new): NumericValue — the encoding in one place, as a driver.Valuer/sql.Scanner
  • store/schema/000002_rebias_numeric_values.up.sql (new): re-encodes existing rows (value - 9223372036854775807 - 1; the literal is split because SQLite parses 9223372036854775808 as a REAL, which would corrupt values via float precision loss)
  • store/sqlc.yaml + regenerated code (sqlc v1.30.0, type-only diff): the value column binds as NumericValue
  • bitmap_cache.go, query/evaluate.go: wrap values at every bind site; also fix %q applied to uint64 in error messages (this is why the incident log showed value '�')
  • sqlitestore_test.go: regression tests — the production failure replayed through FollowEvents with 7, 2^63−1, 2^63, 2^64−1; range operators asserted across the 2^63 boundary; the full RPC path (parse → evaluate) with a 18446744073709551615 literal; and a migration test that builds a schema-v1 database with raw-encoded rows and verifies they are found after re-encoding

Rollout

The migration runs automatically on startup and is a no-op on fresh databases. Wedged nodes apply it, resume from last_block, and index the previously-fatal entity. All nodes must upgrade — un-upgraded nodes stay stuck on the poison block regardless.

🤖 Generated with Claude Code

Numeric attribute values of 2^63 or more could not be bound to SQLite
(Go's database/sql rejects uint64 with the high bit set), so a single
entity with such an annotation permanently wedged the event follower on
every node.

Store values with the top bit flipped (value XOR 2^63): the full uint64
range maps one-to-one onto SQLite's signed INTEGER with order preserved,
so equality, IN and range comparisons all stay correct. Existing rows
are re-encoded by migration 000002; wedged nodes heal on upgrade and
resume from last_block.

Also fix %q being applied to uint64 in bitmap cache error messages,
which rendered values as unicode garbage in logs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@SewerynKras SewerynKras closed this Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant