Skip to content

fix(backfill): replay every MV feeding the target, not just the first (NUM-7480)#185

Merged
alvarogar4 merged 1 commit into
mainfrom
alvaro/num-7480-chkit-backfill-mv_replay-only-replays-the-first-mv-when-a
Jul 7, 2026
Merged

fix(backfill): replay every MV feeding the target, not just the first (NUM-7480)#185
alvarogar4 merged 1 commit into
mainfrom
alvaro/num-7480-chkit-backfill-mv_replay-only-replays-the-first-mv-when-a

Conversation

@alvarogar4

@alvarogar4 alvarogar4 commented Jul 7, 2026

Copy link
Copy Markdown
Member

Summary

Fixes NUM-7480. backfill mv_replay rebuilt a target table from only the first materialized view whose to matched it. ClickHouse allows several MVs to share one destination table, so the rest were silently dropped and the backfill came out incomplete.

  • detect.ts: findMvForTargetfindMvsForTarget — returns every MV feeding the target.
  • planner.ts / types.ts: the plan now carries mvReplayQueries: string[] (renamed from mvAsQuery).
  • chunking/sql.ts: buildChunkExecutionSql replays all MV queries in a single INSERT INTO target SELECT … UNION ALL SELECT … per chunk, so one deterministic query_id and one insert_deduplication_token still cover the chunk (separate statements would break query-id tracking and dedup-drop every insert after the first). Single-MV plans emit byte-identical SQL.
  • Threaded through both execution venues: local run (plugin.ts) and managed submit (plugin-obsessiondb).
  • Docs: clarified the mv_replay strategy description for multi-MV targets.

Clean rename with no back-compat shim — @chkit/plugin-backfill is 0.1.2-beta, plans are ephemeral/regenerated and read via plain JSON.parse.

Test plan

  • New unit coverage: findMvsForTarget returns all matching MVs; planner produces a 2-element mvReplayQueries + UNION ALL SQL; managed submit path does the same.
  • typecheck, lint, build pass; @chkit/plugin-backfill and @chkit/plugin-obsessiondb test tasks (incl. live e2e) pass under bun verify.
  • Eyeballed generated multi-MV SQL: valid ClickHouse — correct per-branch WHERE injection, preserved GROUP BYs, single trailing SETTINGS/dedup token.
  • Optional: live multi-MV backfill against a real ObsessionDB service.

Note: the only bun verify failures are pre-existing live-cluster auth errors in packages/cli (untouched by this PR) — no code-path connection to this change.

🤖 Generated with Claude Code

… (NUM-7480)

mv_replay rebuilt a target table from only the first materialized view
whose `to` matched it. ClickHouse allows several MVs to share one
destination table, so the rest were silently dropped and the backfill
came out incomplete.

Collect all MVs targeting the table and replay them in a single
`INSERT INTO target SELECT … UNION ALL SELECT …` per chunk, so one
query id and one idempotency token still cover the chunk. Single-MV
plans emit byte-identical SQL. Covers the local `run` and managed
`submit` paths (both build chunk SQL through the shared builder).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgxmdeV7WhPGGxDfv5H2Jp
@alvarogar4 alvarogar4 merged commit 3f9a246 into main Jul 7, 2026
6 checks passed
@alvarogar4 alvarogar4 deleted the alvaro/num-7480-chkit-backfill-mv_replay-only-replays-the-first-mv-when-a branch July 7, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant