Fix indexing and queries for numeric values with the high bit set#36
Closed
SewerynKras wants to merge 1 commit into
Closed
Fix indexing and queries for numeric values with the high bit set#36SewerynKras wants to merge 1 commit into
SewerynKras wants to merge 1 commit into
Conversation
Numeric attribute values of 2^63 or more could not be bound to SQLite (Go's database/sql rejects uint64 with the high bit set), so a single entity with such an annotation permanently wedged the event follower on every node. Store values with the top bit flipped (value XOR 2^63): the full uint64 range maps one-to-one onto SQLite's signed INTEGER with order preserved, so equality, IN and range comparisons all stay correct. Existing rows are re-encoded by migration 000002; wedged nodes heal on upgrade and resume from last_block. Also fix %q being applied to uint64 in bitmap cache error messages, which rendered values as unicode garbage in logs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken
A numeric annotation value of 2^63 or more (the user hit 2^64−1, the documented max) could not be bound to SQLite: Go's
database/sqlrejectsuint64arguments with the high bit set:Since event-following is deterministic, one such entity was a poison message: every node failed the same block, rolled back, and retried forever. This is what took down all Braga RPC nodes on 2026-07-03. The same limit applied to the read path — an
arkiv_querywith such a literal failed the same way.The fix
Numeric values are now stored with their top bit flipped (
value XOR 2^63), mapping the full uint64 range[0, 2^64)one-to-one onto SQLite's signed INTEGER range[−2^63, 2^63)while preserving order. Unsigned ordering matches SQLite's signed ordering, so equality,IN, and range comparisons (<,<=,>,>=) are all correct for every possible value — including across the 2^63 boundary.This supersedes #35, which used the raw
int64bit-pattern cast: that unblocks indexing but leaves range queries silently wrong for the high band (big > 5misses2^64−1;big < 10matches it), and once high-bit values exist in databases under that encoding, converting to an order-preserving one requires a table rebuild instead of a single UPDATE. Doing it in one shot keeps the migration trivial.Changes
store/numeric_value.go(new):NumericValue— the encoding in one place, as adriver.Valuer/sql.Scannerstore/schema/000002_rebias_numeric_values.up.sql(new): re-encodes existing rows (value - 9223372036854775807 - 1; the literal is split because SQLite parses9223372036854775808as a REAL, which would corrupt values via float precision loss)store/sqlc.yaml+ regenerated code (sqlc v1.30.0, type-only diff): the value column binds asNumericValuebitmap_cache.go,query/evaluate.go: wrap values at every bind site; also fix%qapplied to uint64 in error messages (this is why the incident log showedvalue '�')sqlitestore_test.go: regression tests — the production failure replayed throughFollowEventswith 7, 2^63−1, 2^63, 2^64−1; range operators asserted across the 2^63 boundary; the full RPC path (parse → evaluate) with a18446744073709551615literal; and a migration test that builds a schema-v1 database with raw-encoded rows and verifies they are found after re-encodingRollout
The migration runs automatically on startup and is a no-op on fresh databases. Wedged nodes apply it, resume from
last_block, and index the previously-fatal entity. All nodes must upgrade — un-upgraded nodes stay stuck on the poison block regardless.🤖 Generated with Claude Code