fix: stop duplicate notification emails#237
Merged
Merged
Conversation
Users reported receiving the same email repeatedly. Production telemetry traced it to four compounding causes, fixed here: - Gate the EMAIL binding to the prd stage only (api + alerting alchemy configs): PR-preview and stg workers ran the same onboarding/digest/ alert crons against branched real data with a live binding, so every live stage sent its own copy of each email to real users. EmailService also gains a runtime guard (MAPLE_ENVIRONMENT must be "production", override via MAPLE_EMAIL_ALLOW_NONPROD) for bindings that reach non-prod workers anyway. - Flap suppression (AlertsService): incidents reopening within the renotify interval of the previously notified incident open silently and inherit its notify anchor; their resolves are silent too. A rule oscillating around its threshold no longer emails an identical trigger/resolve pair every few minutes. - Renotify gate closes at queue time, not delivery success: a failing destination used to re-queue a fresh renotify chain every minute (deliveryKey embeds scheduledAt, so the unique index never collapsed them) and the whole backlog fired on recovery. - Email fan-out counts partial member success as success (failed members are reported, not retried): retrying the event re-emailed members who already received it. - Error-issue incident open/resolve transitions are CAS-guarded (setWhere + returning), so overlapping error ticks can no longer double-dispatch the immediate-path notifications. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Your Pullfrog Router balance is exhausted. You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Makisuo
added a commit
that referenced
this pull request
Jul 20, 2026
…gate non-prod alerting crons (#239) Follow-up to the Jul 20 error sweep (#235/#237). Those fixes verifiably worked (prod errors ~110k/day -> ~220/10h); this resolves the steady error sources that remained: - query-engine: classify Tinybird's "invalid authentication token. Workspace not found" as WarehouseAuthError (it previously fell through to the generic WarehouseQueryError and looked retryable). - alerting ticks: new warehouse-org-quarantine (edge-cache bucket, 6h TTL) parks an org whose warehouse fails with an auth/config-class error and skips it in the errors/anomaly/digest ticks — three prod orgs with dead Tinybird workspace tokens were failing every 1-5min tick forever (~75 errors/10h prod, ~3k/10h across stages, plus secondary *PersistenceError cascades). TTL expiry retries automatically once the org's config is repaired. DigestService gains an EdgeCacheService dep. - oauth: new oauth_connections.revoked_at (migration 0016), stamped in the shared refresh path and at the Cloudflare analytics poller's revoked sites (a mid-poll 401 never reaches the refresh path). pollAllOrgs filters revoked connections, so an org whose grant was revoked stops re-erroring every 5 minutes; reconnect (upsertConnection) and resetOrgState clear the stamp. - alerting worker: skip all crons on non-prod stages unless MAPLE_ALERTING_ALLOW_NONPROD=1 — stg shares the prod DB, so its crons iterated real orgs with stage-local Tinybird/Clerk credentials (~10k errors/10h; same class of leak as the #237 email gating). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Users reported being spammed by the same email over and over. This traces and fixes four compounding causes, verified against production telemetry (internal-org alerting logs via the Maple MCP):
apps/api/alchemy.run.ts,apps/alerting/alchemy.run.ts). PR-preview and stg workers run the same onboarding/digest/alert crons against branched real data and had a live binding — telemetry showed five PR previews each sending "Welcome to Maple" within 250 ms of each other, so a new user got one copy from prod plus one per live preview.EmailServicealso gains a runtime guard: sends requireMAPLE_ENVIRONMENT=production(escape hatch:MAPLE_EMAIL_ALLOW_NONPROD=true), covering any binding that still reaches a non-prod worker (wrangler dev, manual deploys).AlertsService). Confirmed live on 07-16: a rule oscillating around its threshold opened/resolved short-lived incidents all day, emailing identical trigger/resolve pairs every few minutes. Incidents reopening within the renotify interval of the previously notified incident now open silently, inherit its notify anchor (so the renotify gate enforces one email per interval while flapping persists), and resolve silently.scheduledAt, so the(deliveryKey, attemptNumber)unique index never collapsed cross-tick re-queues) and the entire backlog fired once the destination recovered. The per-event retry chain (5 attempts, backoff) already covers transient failures.AlertDeliveryDispatch). One member failing used to fail the whole event; the retry re-emailed members who already received it. Partial success now succeeds with the failures annotated inproviderMessage; only all-members-failed is retried (first member error preserved verbatim so timeout classification survives). This also flows throughNotificationDispatcher, fixing the EscalationService re-page on partially-delivered email destinations.ErrorsService). Incident open/resolve dispatch immediately (no outbox) and were guarded only by a read-then-act check; overlapping ticks — including stg's crons, which currently share the prod DB — could double-dispatch. Both transitions are now CAS-guarded (onConflictDoUpdate+setWhere+.returning()for open; conditionalstatus='open'update for resolve); only the winner notifies.Reviewer notes
alert_incidents.lastNotifiedAtnow means "last notification queued" until a delivery success overwrites it with delivery time. All other readers are display-only (audited: web collections, incident-timeline MCP tool, v2 serializer).lastDeliveredEventType == null && lastNotifiedAt != null— an incident that renotified successfully mid-flap still gets its resolve email.ErrorsServicedeliberately skips occurrence bumps (the winner already counted the same scan window).issue_escalationsgrant gap — those are preview-only hygiene; prod was healthy.Testing
bun typecheck— 29/29 cleanbun run test— full monorepo green (api 935, web 719, ingest 38)🤖 Generated with Claude Code
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.