fix: resolve top production error sources from 24h error sweep#235
Merged
Conversation
Three fixes from an error sweep of the last 24h (~120k events, almost all from the alerting worker): - alerting: bind TINYBIRD_SIGNING_KEY + TINYBIRD_WORKSPACE_ID (same as the api worker) — without them every scheduler tick fails alert-rule evaluation for Tinybird-scoped orgs with WarehouseConfigError (~23k/day) - query-engine: appendSettings now inserts SETTINGS before a trailing FORMAT clause; Tinybird rejects `FORMAT JSON SETTINGS …` with a syntax error (caused an ~85k-error burst on Jul 19) - cli: chunk chDB inserts on line boundaries under a 200KB escaped-payload budget so large OTLP batches no longer trip chDB's ~256KB max_query_size (Code: 62, ~700/day in local mode) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Your Pullfrog Router balance is exhausted. You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Makisuo
added a commit
that referenced
this pull request
Jul 20, 2026
…gate non-prod alerting crons (#239) Follow-up to the Jul 20 error sweep (#235/#237). Those fixes verifiably worked (prod errors ~110k/day -> ~220/10h); this resolves the steady error sources that remained: - query-engine: classify Tinybird's "invalid authentication token. Workspace not found" as WarehouseAuthError (it previously fell through to the generic WarehouseQueryError and looked retryable). - alerting ticks: new warehouse-org-quarantine (edge-cache bucket, 6h TTL) parks an org whose warehouse fails with an auth/config-class error and skips it in the errors/anomaly/digest ticks — three prod orgs with dead Tinybird workspace tokens were failing every 1-5min tick forever (~75 errors/10h prod, ~3k/10h across stages, plus secondary *PersistenceError cascades). TTL expiry retries automatically once the org's config is repaired. DigestService gains an EdgeCacheService dep. - oauth: new oauth_connections.revoked_at (migration 0016), stamped in the shared refresh path and at the Cloudflare analytics poller's revoked sites (a mid-poll 401 never reaches the refresh path). pollAllOrgs filters revoked connections, so an org whose grant was revoked stops re-erroring every 5 minutes; reconnect (upsertConnection) and resetOrgState clear the stamp. - alerting worker: skip all crons on non-prod stages unless MAPLE_ALERTING_ALLOW_NONPROD=1 — stg shares the prod DB, so its crons iterated real orgs with stage-local Tinybird/Clerk credentials (~10k errors/10h; same class of leak as the #237 email gating). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three fixes from a sweep of the last 24h of production errors (~120k events, almost all from the
alertingworker):Alerting worker: bind
TINYBIRD_SIGNING_KEY+TINYBIRD_WORKSPACE_ID(apps/alerting/alchemy.run.ts).TinybirdOrgTokenServicerequires both for Tinybird-scoped raw SQL; the api worker binds them but the alerting worker never did, so every scheduler tick failed alert-rule evaluation for Tinybird-scoped orgs withWarehouseConfigError: TINYBIRD_SIGNING_KEY is required for Tinybird-scoped raw SQL(~23k errors/day, ongoing until this deploys).query-engine: emit
SETTINGSbefore a trailingFORMATclause (packages/query-engine/src/profiles/query-profile.ts).appendSettingsappended the clause to the end of DSL SQL that already terminates inFORMAT JSON, producing… FORMAT JSON SETTINGS …, which Tinybird rejects withDB::Exception: Syntax error … (FORMAT). This caused an ~85k-error burst on Jul 19. TheWarehouseQueryServiceFORMAT-detection regex already accepts both orders; its comment is updated and both orders are now covered by tests.CLI local mode: chunk chDB inserts (
apps/cli/src/server/inserts.ts,serve.ts). The whole NDJSON batch was inlined into oneINSERT … FROM format(JSONEachRow, …)statement; large log batches exceeded chDB's default ~256KBmax_query_size(Code: 62, ~700 errors/day). NewbuildInsertStatementssplits batches on line boundaries under a 200KB escaped-payload budget; a single oversized line still goes out intact as its own statement.Reviewer notes
17797887319973565170/14262904916114617522should stop afterwards.FORMAT JSON SETTINGS …order inWarehouseQueryServiceso nothing regresses mid-rollout.permission deniedonalert_rules/issue_escalations/error_issue_states/oauth_connections(ops-side grants, started Jul 19 16:20 UTC) and quarantining orgs with permanently broken configs (dead Tinybird workspace org, deleted Clerk org).Testing
packages/query-engine: 808 tests pass (newappendSettingsFORMAT-ordering cases included)apps/api:WarehouseQueryService.test.ts34 tests pass (new canonical-order case)apps/cli: 108 tests pass (newtest/inserts.test.tscovers splitting, round-trip, escaped-byte budgeting, oversized lines)bun typecheckgreen across all 29 packages🤖 Generated with Claude Code
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.