Feed maintenance: inline dedup for small feeds + identity in the light pass, named full/light#204
Closed
jozef2svrcek wants to merge 2 commits into
Closed
Feed maintenance: inline dedup for small feeds + identity in the light pass, named full/light#204jozef2svrcek wants to merge 2 commits into
jozef2svrcek wants to merge 2 commits into
Conversation
… light pass Follow-up to the feed-maintenance discussion. - Dedup by size, not source kind. `sources_sync` now sets skip_dedup = bulk (the size-based `source_import_is_bulk`): a small incremental feed update dedups games INLINE so cross-feed/repeat duplicates never accumulate on the daily path, while a bulk run (Ajedrez, or any large first-run/catch-up) inserts everything and defers to the single batch dedup_games — inline-fingerprinting millions of games would be far too slow. The maintenance level is now size-based too (request_maintenance( bulk) instead of kind==Bulk), so anything that skipped inline dedup gets the full batch pass — closing the gap where a large-ish feed increment (a >12 MB Lichess month) would have skipped both inline and batch dedup. - Light pass gains the player-bounded identity steps. It now runs resolve_fide + dedup_players + normalise + index (was normalise + index), so feed players get FIDE IDs matched by name and duplicate rows merge on the daily path rather than waiting for a full pass. The O(all-games) dedup_games stays full-only. This also makes the shared "Prepare database — resolve · dedup · normalise · index" activity placeholder accurate for the light pass, which decomposes into these jobs as before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2
The coalesced "Prepare database" pending row read the same whether a full
or light pass was owed. Surface the level: the /jobs synthetic row now
carries `{full}` (from maintenance_pending_level), and the activity panel
labels it "Full maintenance — resolve FIDE · dedup players · normalise ·
dedup games · index" or "Light maintenance — resolve FIDE · dedup players ·
normalise · index". It still decomposes into those step jobs when it
reaches the top of the queue.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2
Contributor
Author
|
Superseded by the incremental-dedup approach (option 2): |
jozef2svrcek
added a commit
that referenced
this pull request
Jul 25, 2026
`dedup_games` runs after every daily sync, but its self-join is O(N) over the whole games table — re-vetting millions of already-deduplicated games each time. This adds a `deduped` boolean marker so a pass only examines pairs where at least one side is still unvetted, then flips survivors to TRUE once it completes. A daily run now re-checks only the games imported since the last pass. Because the incremental pass is cheap, the former light-vs-full maintenance split collapses into one identity-first pass (resolve-fide → dedup_players → normalise → dedup_games → index) that every source — feed or bulk — requests. This drops the inline per-import dedup entirely: imports always `skip_dedup`, and the single background pass does the deduplication. Supersedes PR #204's light/full naming. Details: - schema: nullable `games.deduped`; one-time backfill marks pre-existing rows TRUE (NULL sentinel keeps it idempotent — new games are written FALSE, never NULL, so a re-init never re-marks them). - importer: games appender/INSERT write `deduped = FALSE` (awaiting dedup). - dedup_games: `(g1.deduped IS NOT TRUE OR g2.deduped IS NOT TRUE)` candidate filter — robust to id reuse; a new game (FALSE) paired with an old vetted game (TRUE) is still a candidate. Marks vetted after a *complete* pass only (cancel and dry-run skip it), including the no-candidates case so unique games aren't rescanned forever. - jobs/serve: `request_maintenance()` loses its `full` arg; a single pass. - tests: incremental second-pass, dry-run no-op, prefix-dup removal, and the backfill idempotency migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2
jozef2svrcek
added a commit
that referenced
this pull request
Jul 25, 2026
…jedrez, per-card metrics + incremental dedup Lands the tested batch behind the 0.9.9 draft: - #194 (PR #200): configurable daily update-check time + next-run visibility. - #195 (PR #198): enabling a feed triggers an immediate sync. - #196 (PR #199): Ajedrez one-shot "Download & import" action instead of a toggle. - #197 (PR #201): per-source metrics moved into each card; aggregated blocks dropped. - PR #203: crisp high-contrast toggle icon. Plus follow-up work validated during testing: - Incremental dedup_games via a `deduped` marker; unified single maintenance pass (supersedes the closed PR #204). - Sync init shows the indeterminate bar instead of a stuck 100%. - Enable-flag write persists before the sync is dispatched; toggle state derived from the job pipeline + a module-level intent store, so it survives navigation. - FIDE card reflects background (scheduler / post-sync) refreshes. - Coalesced maintenance is pinned to the queue tail the moment a feed is enabled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the feed-maintenance discussion — makes the post-sync maintenance correct for feeds and clearly labelled.
Dedup by size, not source kind
sources_syncnow setsskip_dedup = bulk(the size-basedsource_import_is_bulk):dedup_games— inline-fingerprinting millions of games would be far too slow. (Your requirement: bulk never dedups inline.)The maintenance level is size-based too now —
request_maintenance(bulk)instead ofkind == Bulk. That closes a gap: a large-ish feed increment (a >12 MB Lichess month) is size-bulk → skips inline dedup, and would previously have taken the light pass → no batch dedup either. Now it takes the full pass, so anything that skipped inline dedup gets batch-deduped.Light pass gains player-bounded identity
The light pass now runs
resolve_fide+dedup_players+normalise+index(was normalise + index). These are bounded by player count (cheap), so feed players get FIDE IDs matched by name and duplicate rows merge on the daily path. The O(all-games)dedup_gamesstays full-only.Named full vs light in the activity panel
The coalesced "Prepare database" placeholder read the same for both. It now carries
{full}(maintenance_pending_level) and the panel shows:Same "single row that decomposes into its step jobs at the top of the queue" concept as before, just labelled.
clippy (
-D warnings) + 45 tests green; frontend builds.🤖 Generated with Claude Code
https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2