fix: quarantine broken-warehouse orgs, skip revoked Cloudflare integrations, gate non-prod alerting crons#239
Merged
Conversation
…gate non-prod alerting crons Follow-up to the Jul 20 error sweep (#235/#237). Those fixes verifiably worked (prod errors ~110k/day -> ~220/10h); this resolves the steady error sources that remained: - query-engine: classify Tinybird's "invalid authentication token. Workspace not found" as WarehouseAuthError (it previously fell through to the generic WarehouseQueryError and looked retryable). - alerting ticks: new warehouse-org-quarantine (edge-cache bucket, 6h TTL) parks an org whose warehouse fails with an auth/config-class error and skips it in the errors/anomaly/digest ticks — three prod orgs with dead Tinybird workspace tokens were failing every 1-5min tick forever (~75 errors/10h prod, ~3k/10h across stages, plus secondary *PersistenceError cascades). TTL expiry retries automatically once the org's config is repaired. DigestService gains an EdgeCacheService dep. - oauth: new oauth_connections.revoked_at (migration 0016), stamped in the shared refresh path and at the Cloudflare analytics poller's revoked sites (a mid-poll 401 never reaches the refresh path). pollAllOrgs filters revoked connections, so an org whose grant was revoked stops re-erroring every 5 minutes; reconnect (upsertConnection) and resetOrgState clear the stamp. - alerting worker: skip all crons on non-prod stages unless MAPLE_ALERTING_ALLOW_NONPROD=1 — stg shares the prod DB, so its crons iterated real orgs with stage-local Tinybird/Clerk credentials (~10k errors/10h; same class of leak as the #237 email gating). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Your Pullfrog Router balance is exhausted. You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
Ingest Rust Test + Benchmark ResultsCommit: Load Benchmark —
|
| Metric | main (median) | PR (median) | Delta |
|---|---|---|---|
| Requests/sec | 7755.68 | 12314.89 | +58.8% better |
| Rows/sec | 77556.81 | 123148.89 | +58.8% better |
| p50 latency | 7.92 ms | 4.88 ms | -38.4% better |
| p95 latency | 15.95 ms | 9.58 ms | -40.0% better |
| p99 latency | 22.73 ms | 11.38 ms | -49.9% better |
| Export catch-up | 0.026 s | 0.026 s | +2.0% worse |
| Max RSS | 14.52 MiB | 16.01 MiB | +10.3% worse |
| Failures | 0 | 0 | same |
Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.
PR load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 9,
"duration_seconds": 0.162470532,
"export_catchup_seconds": 0.026181473,
"request_rps": 12309.924608359133,
"row_rps": 123099.24608359134,
"p50_ms": 4.88,
"p95_ms": 9.576,
"p99_ms": 13.668,
"max_rss_mb": 16.07421875,
"max_cpu_percent": 0.0,
"avg_cpu_percent": 0.0
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 11,
"duration_seconds": 0.153215118,
"export_catchup_seconds": 0.025782954,
"request_rps": 13053.542144581319,
"row_rps": 130535.42144581319,
"p50_ms": 4.784,
"p95_ms": 9.268,
"p99_ms": 10.623,
"max_rss_mb": 15.43359375,
"max_cpu_percent": 0.0,
"avg_cpu_percent": 0.0
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 11,
"duration_seconds": 0.162405038,
"export_catchup_seconds": 0.026569382,
"request_rps": 12314.888901414499,
"row_rps": 123148.889014145,
"p50_ms": 4.959,
"p95_ms": 9.813,
"p99_ms": 11.385,
"max_rss_mb": 16.01171875,
"max_cpu_percent": 0.0,
"avg_cpu_percent": 0.0
}
]main load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 15,
"duration_seconds": 0.262720703,
"export_catchup_seconds": 0.026687848,
"request_rps": 7612.64710836283,
"row_rps": 76126.4710836283,
"p50_ms": 7.997,
"p95_ms": 15.951,
"p99_ms": 23.132,
"max_rss_mb": 14.515625,
"max_cpu_percent": 16.6,
"avg_cpu_percent": 16.6
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 13,
"duration_seconds": 0.250551491,
"export_catchup_seconds": 0.025679296,
"request_rps": 7982.391132527724,
"row_rps": 79823.91132527724,
"p50_ms": 7.752,
"p95_ms": 15.512,
"p99_ms": 22.731,
"max_rss_mb": 14.2109375,
"max_cpu_percent": 0.0,
"avg_cpu_percent": 0.0
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 13,
"duration_seconds": 0.257875492,
"export_catchup_seconds": 0.025447242,
"request_rps": 7755.680791876105,
"row_rps": 77556.80791876104,
"p50_ms": 7.922,
"p95_ms": 16.631,
"p99_ms": 22.631,
"max_rss_mb": 14.8671875,
"max_cpu_percent": 20.0,
"avg_cpu_percent": 20.0
}
]WAL-acked microbench (cargo bench --bench ingest_bench)
Compiling maple-ingest v0.1.0 (/home/runner/_work/maple/maple/apps/ingest)
Finished `bench` profile [optimized] target(s) in 13.28s
Running benches/ingest_bench.rs (target/release/deps/ingest_bench-581d2100de893627)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench: 81405 ns/iter (+/- 4507)
thread 'main' (5090) panicked at benches/ingest_bench.rs:73:22:
accept traces: QueueUnavailable("Telemetry WAL lane is full")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test ingest_accept/traces_10_spans_wal_ack ...
error: bench failed, to rerun pass `--bench ingest_bench`
cargo test
test telemetry::tests::metrics_summary_data_points_are_dropped ... ok
test telemetry::tests::clickhouse_breaker_trips_sheds_and_recovers ... ok
test telemetry::tests::pipeline_can_start_for_clickhouse_only_without_tinybird_credentials ... ok
test telemetry::tests::migrate_legacy_shard_relocates_frames_into_lanes ... ok
test telemetry::tests::clickhouse_export_drops_passworded_non_https_endpoint_without_sending ... ok
test telemetry::tests::sampling_keeps_errors_even_when_ratio_low ... ok
test telemetry::tests::scraper_contract::scraper_otlp_json_decodes_with_gateway_serde_and_encodes_to_rows ... ok
test telemetry::tests::signal_tag_round_trips_all_variants ... ok
test telemetry::tests::pipeline_e2e_exports_metrics_to_fake_tinybird ... ok
test telemetry::tests::telemetry_signal_as_str_is_canonical_lowercase ... ok
test telemetry::tests::timestamp_has_nano_precision ... ok
test telemetry::tests::timestamps_match_clickhouse_datetime64_nine_format ... ok
test telemetry::tests::trace_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::traces_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::pipeline_e2e_exports_gzip_ndjson_to_fake_tinybird ... ok
test telemetry::tests::wal_round_trips_frame ... ok
test telemetry::tests::wal_partial_drain_advances_cursor_without_truncating ... ok
test telemetry::tests::wal_truncates_after_full_drain_allowing_further_appends ... ok
test telemetry::tests::pipeline_e2e_exports_traces_to_fake_tinybird ... ok
test telemetry::tests::pipeline_exports_ready_org_to_clickhouse_without_tinybird_calls ... ok
test telemetry::tests::slow_clickhouse_lane_does_not_block_cosharded_tinybird_org ... ok
test telemetry::tests::clickhouse_breaker_sheds_after_threshold_failures ... ok
test result: ok. 36 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.74s
Running unittests src/bin/load_test.rs (target/debug/deps/load_test-661a0aa1eb3f6d6d)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/main.rs (target/debug/deps/maple_ingest-c33bf80c577edb95)
running 38 tests
test autumn::tests::flat_sub_one_gb_remaining_still_allows ... ok
test autumn::tests::flat_hardcap_depleted_blocks ... ok
test autumn::tests::flat_hardcap_with_remaining_allows ... ok
test autumn::tests::allowed_only_no_balance_field ... ok
test autumn::tests::flat_overage_allows ... ok
test autumn::tests::flat_unlimited_allows ... ok
test autumn::tests::nested_balance_object_depleted_blocks ... ok
test autumn::tests::nested_overage_allows ... ok
test autumn::tests::null_balance_no_subscription_blocks ... ok
test autumn::tests::nested_balance_object_with_remaining_allows ... ok
test autumn::tests::unrecognized_shape_returns_none ... ok
test tests::api_error_from_pipeline_maps_variants_to_status ... ok
test tests::api_error_kind_maps_status_to_stable_label ... ok
test tests::clickhouse_destination_is_terminal_in_dual_mode ... ok
test tests::clickhouse_destination_uses_native_pipeline_even_in_forward_mode ... ok
test tests::clickhouse_target_resolver_requires_current_schema ... ok
test tests::clickhouse_target_resolver_decrypts_current_schema_password ... ok
test tests::clickhouse_target_resolver_rejects_password_over_http ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::decrypt_aes256_gcm_matches_node_crypto_fixture ... ok
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::rejection_span_status_is_error_only_for_5xx ... ok
test tests::hash_is_deterministic ... ok
test tests::resolve_ingest_key_keeps_stale_schema_on_managed_native_path ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::tinybird_destination_keeps_forward_mode_on_forward_path ... ok
test tests::resolve_connector_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_refreshes_routing_before_auth_cache_expires ... ok
test autumn::tests::fails_open_on_transport_error ... ok
test tests::resolve_ingest_key_serves_last_known_routing_when_refresh_fails ... ok
test tests::forward_mode_switches_ready_org_to_clickhouse_without_forwarding_again ... ok
test result: ok. 38 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.19s
Doc-tests maple_ingest
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+31
to
+46
| const hasQuarantineTag = (value: unknown, depth = 0): boolean => { | ||
| if (depth > 8 || typeof value !== "object" || value === null) return false | ||
| const candidate = value as { readonly _tag?: unknown; readonly cause?: unknown } | ||
| if (typeof candidate._tag === "string" && QUARANTINE_ERROR_TAGS.has(candidate._tag)) return true | ||
| // Services wrap warehouse failures in their own tagged errors (e.g. | ||
| // AnomalyPersistenceError) with the original error as `cause` — walk it. | ||
| return hasQuarantineTag(candidate.cause, depth + 1) | ||
| } | ||
|
|
||
| /** Does this cause (or anything wrapped inside it) carry an auth/config-class warehouse error? */ | ||
| export const causeHasWarehouseConfigClassError = <E>(cause: Cause.Cause<E>): boolean => | ||
| cause.reasons.some((reason) => | ||
| Cause.isFailReason(reason) | ||
| ? hasQuarantineTag(reason.error) | ||
| : Cause.isDieReason(reason) && hasQuarantineTag(reason.defect), | ||
| ) |
There was a problem hiding this comment.
🔴 Broken-warehouse organizations are never actually parked, so the error floods this change targets keep happening
The check that decides whether to park a broken organization (causeHasWarehouseConfigClassError at apps/api/src/lib/warehouse-org-quarantine.ts:41-46) looks for the warehouse auth/config marker by walking an error's cause, but the ticks first repackage the warehouse failure into a persistence error whose cause is flattened to plain text, so the marker is gone and the organization keeps being retried every tick.
Impact: The three dead-token organizations this PR aims to silence keep failing on every errors/anomaly tick and keep flooding the error dashboards, exactly the behavior the change was meant to stop.
How the tag is destroyed before the quarantine matcher sees it
In AnomalyDetectionService, every warehouse call is wrapped: warehouse.compiledQuery(...).pipe(Effect.mapError(makePersistenceError)) (e.g. apps/api/src/services/AnomalyDetectionService.ts:858,884). makePersistenceError (apps/api/src/services/AnomalyDetectionService.ts:116-125) builds an AnomalyPersistenceError whose cause is describeCause(error.cause), and describeCause (apps/api/src/services/AnomalyDetectionService.ts:105-114) returns a string (stack/JSON.stringify/String). So the failure reaching runTick's catchCause is an AnomalyPersistenceError (_tag = @maple/http/anomalies/AnomalyPersistenceError, not in QUARANTINE_ERROR_TAGS) whose .cause is a plain string.
hasQuarantineTag (apps/api/src/lib/warehouse-org-quarantine.ts:31-38) returns false for the wrapper tag, then recurses into .cause; because that value is a string (typeof value !== "object") it immediately returns false. The original WarehouseAuthError/WarehouseConfigError object — the only thing carrying @maple/http/errors/WarehouseAuthError — was discarded during wrapping, so quarantineOnConfigClassCause returns false and the org is never parked.
The same wrapping happens in ErrorsService (apps/api/src/services/ErrorsService.ts:2357-2359 maps the per-org warehouse scan through makePersistenceError, which also stringifies the cause at :178-190).
The unit test warehouse-org-quarantine.test.ts:27-30 masks this by hand-constructing { _tag: ..., cause: authError } with the real error object as cause, which never occurs at runtime.
Prompt for agents
The quarantine matcher causeHasWarehouseConfigClassError / hasQuarantineTag in apps/api/src/lib/warehouse-org-quarantine.ts detects a broken-warehouse org by finding a WarehouseAuthError/WarehouseConfigError _tag either directly on a Cause reason or by walking the error's .cause chain. However, AnomalyDetectionService and ErrorsService wrap warehouse failures via makePersistenceError before the error reaches the tick's catchCause, and those wrappers store cause via describeCause(...) which flattens the original error into a plain STRING (stack / JSON.stringify / String). As a result the WarehouseAuthError/WarehouseConfigError object (the only carrier of the quarantine _tag) is discarded, .cause is a string, and hasQuarantineTag's recursion bails at `typeof value !== 'object'`. The quarantine therefore never fires for the errors/anomaly ticks — the primary fix in this PR is inert. Consider one of: (a) preserve the original tagged error object as the wrapper's cause instead of stringifying it (add a structured cause field on the persistence errors, or keep the tagged error and don't run describeCause on it); (b) have the matcher also inspect the wrapper's message string for the warehouse-auth/config signatures; or (c) classify the warehouse failure BEFORE it is wrapped (e.g. quarantine at the warehouse-call seam rather than at the tick catchCause). Also fix warehouse-org-quarantine.test.ts so the wrapped-cause test reflects the real runtime shape (stringified cause) rather than an object cause, so it actually guards the behavior.
Was this helpful? React with 👍 or 👎 to provide feedback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Follow-up to the Jul 20 error sweep (#235 / #237). Verified against live telemetry first: those fixes worked — production errors dropped from ~110k/day to ~220 in the 10 hours after deploy, with the
WarehouseConfigError/TinybirdOrgTokenErrorfloods stopping at the exact deploy minute. This PR resolves the steady error sources that remained.What changed
1. Org warehouse quarantine (main fix)
Three prod orgs whose Tinybird auth is dead ("invalid authentication token. Workspace not found") failed every digest/error/anomaly tick forever (~75 errors/10h prod, ~3k/10h across stages, plus secondary
*PersistenceErrorcascades).packages/query-engine/src/execution/errors.ts: that Tinybird message now classifies asWarehouseAuthErrorinstead of falling through to the genericWarehouseQueryError.apps/api/src/lib/warehouse-org-quarantine.ts: when a tick's per-org failure carries an auth/config-class warehouse error (walks wrappedcausechains), the org is parked in the edge cache for 6h and skipped by the errors/anomaly/digest ticks; TTL expiry retries automatically once the org's config is repaired. Transient upstream failures keep retrying as before.DigestServicegains anEdgeCacheServicedependency (layers wired inapp.ts+ alerting worker); the digest skip happens before the daily-attempt claim so a parked org isn't marked attempted.2. Revoked Cloudflare integrations stop being polled
pollAllOrgsselected orgs purely fromoauth_connectionsexistence, so revoked grants re-errored every 5 minutes forever (steady ~6/h "Unauthorized" for 2+ days).oauth_connections.revoked_atcolumn (migration0016, additive).connection-helpers.ts) and at the analytics poller's three revoked sites — the mid-poll 401 (unexpired token revoked upstream, the live failure mode) never reaches the refresh path, hence the explicit stamps.pollAllOrgsfiltersrevoked_at IS NULL; reconnecting (upsertConnection) andresetOrgStateclear the stamp. PlanetScale's poller uses its ownplanetscale_connectionstable and is unaffected.3. Non-prod alerting crons gated off
The stg alerting worker shares the prod DB, so its crons iterated real orgs with stage-local Tinybird/Clerk credentials (~10k errors/10h: per-org auth failures,
permission denied for table issue_escalations, Clerk member-list failures) — the same class of leak the #237 email gating fixed.scheduled()exits early unlessMAPLE_ENVIRONMENT === "production";MAPLE_ALERTING_ALLOW_NONPROD=1(new alchemy passthrough) re-enables crons on a stage deliberately.Reviewer notes
db:generate.pollAllOrgsskipping a revoked connection /resetOrgStateclearing the stamp. Full suite green (944 api / 724 web) + repo typecheck clean.org_clickhouse_settingsfor the three affected orgs (org_39XiWVrKBo3Yr90WSjklOpc7EP8,org_3Fe7Jyx5UQc4E6iJ69ZpOmWxXWk,org_3Fe2H9c0s67N9o2JoyszvoEuUdg) to clean up whatever stale config points at the deleted workspace. The quarantine mutes them meanwhile.512677025343832041and12523048052930950221should drop to ~0 within a few hours.🤖 Generated with Claude Code
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.