fix(backend): detect ledger gaps via a contiguous-cursor checkpoint table - #1572
Merged
ogazboiz merged 1 commit intoJul 30, 2026
Merged
Conversation
…able
The event indexer advanced its cursor (indexer_state.last_ledger) as long
as it was monotonic, with no record of which ledger ranges had actually
been confirmed scanned. A restart that resumes from a stale or clamped
starting point — or any other source of a skipped range — could silently
drop events (e.g. a LoanDefaulted) with no error raised, since empty
ledgers legitimately produce no events and the old code couldn't tell an
empty range from a skipped one.
- Add ledger_checkpoints (contract, range_start, range_end, status) and
record a checkpoint after every poll iteration. A range is 'verified'
when it immediately follows the previously-recorded range; otherwise the
gap between them is flagged 'suspect' and logged.
- Expose EventIndexer.getSuspectRanges() and a standalone
hasUnresolvedLedgerGaps(contract) so consumers can check before trusting
indexed events.
- Gate defaultChecker.checkOverdueLoans on the loan manager contract having
no unresolved gap, since fetchOverdueLoanIds/fetchOverdueStats derive
loan state entirely from indexed contract_events — a missed
LoanRepaid/LoanDefaulted there risks a false default. The gate fails open
on a query error so a checkpoint-table hiccup alone doesn't block runs.
- Fix migrations/1801..._create-pause-state-table.js: it used CommonJS
`exports.up = async (db) => ...` in an ESM package ("type": "module"),
which throws `ReferenceError: exports is not defined` when node-pg-migrate
loads it — this runs before the new 1802 migration in file order, so it
had to be fixed to verify migrate:up/down at all. Converted to the same
ESM pgm-style used by every other migration in this directory.
Out of scope (see issue LabsCrypt#1376 for the full spec): reorg detection via a
content digest, automatic backfill of a detected gap, the contract-level
event_seq identifier, the frontend SSE resync flow, and the CI/docker-compose
fixture RPC wiring — each is its own multi-hour, multi-file change; this PR
covers the backend detection primitive and the one known consumer that
needed gating.
Verified backend/migrations/1802..._create-ledger-checkpoints.js up/down/
up-again against a local Postgres 16 with no residual objects. Pre-existing
main already fails `npm run build`/`npm run typecheck` (transactionController.ts,
pagination.ts, pauseGuard.ts, etc.) and the migration-check job's full
down-everything loop breaks on migration 1800's down() referencing an index
name that doesn't match what its own up() created — both unrelated to this
change and left as-is.
Closes LabsCrypt#1376
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
eventIndexer.tsadvanced its cursor (indexer_state.last_ledger) whenever it was monotonic, with no record of which ledger ranges had actually been confirmed scanned by the RPC. A restart that resumes from a stale or clamped starting point could silently skip a range of ledgers with no error — empty ledgers legitimately produce no events, so the old code had no way to distinguish "empty" from "skipped."This adds a checkpoint table so the indexer can detect that kind of gap and flag it, and gates the one consumer (
defaultChecker.ts) that draws conclusions from indexed events on there being no unresolved gap for the loan manager contract.ledger_checkpointstable (contract,range_start,range_end,status). After every poll iteration,EventIndexer.recordCheckpointrecords the range that was just requested from the RPC:'verified'whenrange_startimmediately follows the previously-recorded range'srange_end.'suspect'for the gap in between when it doesn't — logged and persisted for follow-up.EventIndexer.getSuspectRanges()and a standalonehasUnresolvedLedgerGaps(contract)(in the newledgerCheckpoints.ts, kept separate fromeventIndexer.tsso consumers don't pull in the indexer's full dependency graph) expose this to callers.defaultChecker.checkOverdueLoans()now checkshasUnresolvedLedgerGapsforLOAN_MANAGER_CONTRACT_IDbefore evaluating overdue loans, sincefetchOverdueLoanIds/fetchOverdueStatsderive loan state entirely from indexedcontract_events— a missedLoanRepaidorLoanDefaultedin a gap risks a false default. The run is skipped (not failed) when a gap is unresolved, returning{ skipped: true, skippedReason: 'unresolved_ledger_gap' }. The gate itself fails open on a query error so a checkpoint-table hiccup alone doesn't block default checking.migrations/1801..._create-pause-state-table.js: it used CommonJSexports.up = async (db) => ...in an ESM package ("type": "module"), which throwsReferenceError: exports is not definedwhen node-pg-migrate loads it. This migration runs immediately before the new1802..._create-ledger-checkpoints.jsin file order, so it had to be fixed to verifymigrate:up/migrate:downat all. Converted to the same ESMpgm-style used by every other migration in the directory — no behavior change.Out of scope
Issue #1376's full spec covers five phases across contracts, backend, frontend, and CI. This PR is the backend detection primitive plus the one known consumer that needed gating. Left out, each being its own multi-hour, multi-file change:
range_digest) — requires the contract-levelevent_seqidentifier added toloan_manager/lending_pool/multisig_governance/remittance_nftfirst.backfillRange).resyncframe and TanStack Query cache invalidation.docker-compose.ymlfixture RPC and the corresponding CI job wiring.Also unrelated and pre-existing on
main(verified against a clean checkout before making any changes, not introduced by this PR):npm run build/npm run typecheckinbackend/already fail ontransactionController.ts,pagination.ts,pauseGuard.ts,sorobanService.ts, and others.1800..._keyset-pagination-seq-columns.js: itsdown()drops an index name that doesn't match what its ownup()created.Test plan
npm test -- --testPathPatterns="eventIndexer|defaultChecker"— 32/32 passing (7 new: gap/contiguous/first-checkpoint scenarios,getSuspectRanges,hasUnresolvedLedgerGapshappy/empty/error paths; 2 new:defaultCheckerskips when a gap exists, proceeds when it doesn't).npm test— same 30 pre-existing failing suites / 15 pre-existing failing tests as a clean checkout ofmain(confirmed viagit stash); no new failures, 9 more passing tests than baseline.npx eslinton every changed/added file — clean.migrate:up→migrate:down(removesledger_checkpointsandpause_statewith no residual objects) →migrate:upagain succeeds.Closes #1376