Rearchitect the warehouse executor: named backends, one route resolver, structural ingest routing

[Makisuo]

Why

The query-execution stack had grown hard to trace across the three backends (managed Tinybird, Tinybird's ClickHouse gateway, BYO self-hosted ClickHouse):

• The backends were encoded as combinations of two independent flags (_tag × provider), branched on in ~6 places across the package executor and the host wiring. Nothing in the code named the three deployments.
• Which of four host config resolvers ran was decided inside the package executor from a per-call-site pinToIngestConfig boolean — forgetting it on a new read of an ingest-only datasource silently returned empty rows for BYO-CH orgs.
• Three near-identical interface/option types (ExecutorQueryOptions had already drifted), plus a duplicate WarehouseExecutor.* span nested around every canonical executeSql span.
• Observability routes flattened every warehouse error (quota 429, validation 400, transient 503) into a bare 500; the v2 alerts mapper flattened all 8 tags to upstreamError.
• CLI local mode hand-copied ~190 lines of executor plumbing (spans, truncation, error wrapping, decoding).

What changed

One backend enum + one behavior table — WarehouseBackendKind = "tinybird" | "tinybird-gateway" | "clickhouse" | "chdb" with a single BackendDialect table (execution/backend.ts): driver, Tinybird settings-strip policy, SQL normalization, peer-service label. All scattered branches became table lookups. The gateway keeps peerService: "clickhouse" deliberately so service-map DB nodes don't churn.

Route-as-data — the executor computes a purpose (read / raw / ingest) and calls a single injected resolveRoute, implemented as one ordered decision function in WarehouseQueryService.ts. Raw-SQL isolation is preserved verbatim (textual OrgId filter in the executor, org-scoped JWT for shared Tinybird, per-org creds for BYO, self_hosted-only gate for shared vanilla CH), and clientCacheKey strings are byte-identical so warm clients aren't churned.

Structural ingest routing — control-plane queries declare .routing("ingest") at their definition (new chain on the CH builder, carried on CompiledQuery); alert_checks + the cross-org activity scans are tagged. Hand-written SQL uses SqlQueryOptions.route: "ingest" (the only runtime-conditional site is the Cloudflare usage read, gated on write-readiness). A safety net warns when a BYO route reads an ingest-pinned table. pinToIngestConfig is deleted.

One option type, one span — SqlQueryOptions lives next to the profiles and is shared by the executor, the observability facade, and the runtime port; the ExecutorQuery* duplicates are deleted. asExecutor no longer wraps duplicate spans — WarehouseQueryService.executeSql is the single client span, now also carrying warehouse.backend, warehouse.route, and warehouse.config_source so a trace answers "which path and why" in one place. Legacy query.routing/clientSource/db.client are dual-emitted for one release.

Canonical error statuses — observability endpoints declare warehouseHttpErrors, so quota surfaces as 429, validation as 400, transient as 503 instead of a blanket 500 (the mapError flatten is gone). The v2 alerts mapper sends quota → rateLimited() and validation → invalidRequest. ingest failures are classified with mapWarehouseError.

CLI local mode runs the real executor — a chdb dialect row + constant local route replace the hand-rolled shape (deleted). /local/query returns 400 (not 500) for failing SQL so the shared executor doesn't classify it as transient and retry. The raw-SQL escape hatch (maple query) posts directly to /local/query: arbitrary local SQL has no OrgId, and the executor's scoping guard must not weaken.

Net −59 lines of production code despite ~130 lines of new tests.

Reviewer notes

• Client-visible change: the observability endpoints' error union is widened (domain + api + web deploy together; no web code referenced ObservabilityApiError by name, so no web edits were needed). Dashboards filtering on the legacy span attributes should migrate to warehouse.* before the dual-emit is removed.
• lib/clickhouse-builder gained the .routing("ingest") chain — its dist is rebuilt as part of this change.
• Local span names change from warehouse.sqlQuery → WarehouseQueryService.executeSql (local mode only).
• Verified: full monorepo typecheck (29 packages) plus test suites — query-engine 782, apps/api 913, CLI 95 (3 new wiring tests), clickhouse-builder 73. Phase-0 characterization tests pin the gateway strip+normalize behavior, ingest routing, raw response limits, cache keying, and timeout semantics.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

[pullfrog]

Your Pullfrog Router balance is exhausted.

You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

  ^{｜ ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot ｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}