fix(queue): enforce max 5 retries and make queues healthy

[riderx]

Summary (AI generated)

• Hard-cap every queue at MAX_QUEUE_READS = 5 (removed VERSION_QUEUE_MAX_READS = 30)
• Cap webhook delivery attempts at WEBHOOK_MAX_ATTEMPTS = 5 (DB default + handler clamp) and shorten the retry delay ladder
• Raise org_stats_queue to batch 100 on every 10s tick so cron_stat_org can drain
• /queue_health ignores delayed messages (vt > now()) for never-read stale checks
• Migration purges existing poison rows with read_ct > 5
• Document the hard rule in AGENTS.md (Production Scale Guardrails + dedicated Queue Retry Budget section)

Motivation (AI generated)

Production /queue_health was unhealthy for three distinct reasons:

1. cron_stat_org — org_stats_queue was batch 10 / every 5 minutes (~100 msgs/5m) and could not keep up with ingress
2. on_version_update — special-case 30 retries kept poison/partial cleanup messages looping and masked them from health (stuck > 5 while consumer still retried)
3. webhook_delivery — delayed retries (pgmq.send with future vt) looked like 12h never-read backlog; app-level retries also went up to 10 / 24h

Retries above 5 are not an acceptable progress mechanism. Leftover work must be re-enqueued by sweepers/crons.

Business Impact (AI generated)

Restores a trustworthy queue monitoring signal, stops unbounded retry storms on version cleanup, drains org plan-stat backlog faster, and fails bad webhook endpoints sooner instead of parking delayed messages for up to a day.

Test Plan (AI generated)

• bunx vitest run tests/queue-consumer-message-shape.unit.test.ts tests/queue-health.unit.test.ts tests/webhook-delivery-security.unit.test.ts tests/webhook-delivery-handler-security.unit.test.ts
• Apply migration queue_health_retry_budget_5 in prod (also purges read_ct > 5)
• Deploy API/functions so consumer + webhook caps take effect
• Confirm GET /queue_health returns status: ok after org_stats backlog drains (a few minutes post-deploy)
• Confirm no queue has max_read_ct > 5 and Discord alerts fire at the shared budget

Generated with AI

  

Summary by CodeRabbit

• Enhancements

  • Standardized queue and webhook retries to a maximum of five attempts.
  • Improved queue health reporting with clearer retry and message-age details.
  • Automatically cleans up messages that exceed the retry limit.
  • Tightened webhook retry delays, including a maximum two-hour server-requested delay.

• Maintenance

  • Updated scheduled queue processing to run more frequently and handle larger batches.
  • Clarified retry behavior documentation and updated automated coverage.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 31 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: e56f1b6f-7d7f-43e4-bdb0-001ed00e31ae

📥 Commits

Reviewing files that changed from the base of the PR and between c19312b and d3598b7.

📒 Files selected for processing (5)

• supabase/functions/_backend/public/queue_health.ts
• supabase/functions/_backend/utils/webhook.ts
• supabase/migrations/20260726134739_queue_health_retry_budget_5.sql
• tests/queue-health.unit.test.ts
• tests/webhook-delivery-security.unit.test.ts

📝 Walkthrough

Walkthrough

The PR standardizes queue and webhook retries at a hard maximum of five, updates queue-health calculations and messaging, purges over-budget PGMQ messages during migration, and documents the resulting retry and cleanup behavior.

Changes

Retry budget enforcement

Layer / File(s)	Summary
Queue consumer budget supabase/functions/_backend/triggers/queue_consumer.ts, tests/queue-consumer-message-shape.unit.test.ts	All queues now use MAX_QUEUE_READS = 5, with failure-action tests updated for the shared budget.
Webhook delivery budget supabase/functions/_backend/utils/webhook.ts, supabase/functions/_backend/triggers/webhook_delivery.ts, tests/webhook-delivery-security.unit.test.ts	Webhook attempts and retry delays are capped at five, retry-after parsing is capped at two hours, and delivery tests reflect the shortened schedule.
Health and poison-message handling supabase/functions/_backend/public/queue_health.ts, supabase/migrations/20260726134739_queue_health_retry_budget_5.sql, tests/queue-health.unit.test.ts	Queue-health metrics account for visible messages, while the migration clamps webhook attempts and deletes PGMQ messages with read_ct > 5 in bounded batches.
Retry semantics and guidance AGENTS.md, supabase/functions/_backend/triggers/on_version_update.ts	Operational guidance and manifest deletion comments describe the five-attempt budget, poison cleanup, retry safety, and idempotency.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant QueueConsumer
  participant PGMQ
  participant QueueHealth
  participant CleanupMigration
  QueueConsumer->>PGMQ: read messages with max read count 5
  PGMQ-->>QueueConsumer: message and read count
  QueueHealth->>PGMQ: evaluate visible and stuck messages
  CleanupMigration->>PGMQ: delete messages with read_ct > 5

Loading

Possibly related PRs

• Cap-go/capgo.app#2721: Aligns poison-message cleanup with the read_ct > 5 retry threshold.
• Cap-go/capgo.app#2724: Directly relates to version-queue retry-budget behavior.
• Cap-go/capgo.app#2756: Introduces the queue-health endpoint whose metrics this PR updates.

Suggested labels: codex

Suggested reviewers: dalanir, wcaleniewolny

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: capping retries and improving queue health.
Description check	✅ Passed	The description covers summary, motivation, impact, and a concrete test plan; screenshots are reasonably omitted for this backend-only change.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 43 untouched benchmarks
⏩ 2 skipped benchmarks¹

---

_{Comparing cursor/queue-health-all-388e (d3598b7) with main (57e6571)}

Footnotes

1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_195508c1-2806-46dd-9c0d-f563a9aa71a0)

[cursor]

Stale comment

Risk: high. Cursor Bugbot did not complete (skipped due to usage limit), so the configured automated review signal is unavailable. This PR changes production queue retry budgets, webhook delivery limits, and includes a migration that deletes stuck queue messages — human review is required.

  

_{Sent by Cursor Approval Agent: Pull Request Approver External}

[cursor]

Stale comment

Risk: medium. Not approving: Cursor Bugbot completed with status skipping (usage limit reached) and did not produce a clean review signal. Human review is needed for this queue retry-budget and migration change. Assigned WcaleNieWolny and victorleduc as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@supabase/functions/_backend/public/queue_health.ts`:
- Around line 376-389: Restore never_read_count to count all rows with read_ct =
0, preserving its existing public API meaning. Keep the vt <= now() visibility
filter on never_read_stale_count, and if the visible-only metric is required
elsewhere, expose it through a distinct field such as never_read_visible_count
and update only those consumers to use it.

In `@supabase/functions/_backend/utils/webhook.ts`:
- Around line 99-110: The retry schedule and cap in getWebhookRetryDelaySeconds
are inconsistent: the fifth delay exceeds WEBHOOK_MAX_RETRY_AFTER_SECONDS.
Reduce the final value in WEBHOOK_RETRY_DELAYS_SECONDS so every retry remains
within the existing 2-hour cap, preserving the current cap logic and attempt
ordering.

In `@supabase/migrations/20260726134739_queue_health_retry_budget_5.sql`:
- Around line 26-76: Add production-scale support for the stuck-row purge loop
in the migration’s anonymous DO block: ensure the DELETE subquery over each pgmq
queue table can efficiently locate rows matching read_ct > 5, preferably by
creating or reusing an appropriate partial index before batching, or validate
the approach with production-like EXPLAIN (ANALYZE, BUFFERS) results and adjust
it if it exceeds the migration time budget. Preserve the existing per-queue
batch and total-work bounds.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 024dacb6-33d5-4200-a86c-5d00c84724ab

📥 Commits

Reviewing files that changed from the base of the PR and between 57e6571 and c19312b.

📒 Files selected for processing (10)

• AGENTS.md
• supabase/functions/_backend/public/queue_health.ts
• supabase/functions/_backend/triggers/on_version_update.ts
• supabase/functions/_backend/triggers/queue_consumer.ts
• supabase/functions/_backend/triggers/webhook_delivery.ts
• supabase/functions/_backend/utils/webhook.ts
• supabase/migrations/20260726134739_queue_health_retry_budget_5.sql
• tests/queue-consumer-message-shape.unit.test.ts
• tests/queue-health.unit.test.ts
• tests/webhook-delivery-security.unit.test.ts

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• Cap-go/capacitor-updater (manual)

[cubic-dev-ai]

All reported issues were addressed across 10 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_17460284-86fe-41f1-8eb6-1452585cca7a)

[cursor]

Risk: high. Not approving: Cursor Bugbot completed with status skipping (usage limit reached) and did not produce a clean review signal. Human review is required for this queue retry-budget and migration change; Dalanir and WcaleNieWolny are already assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver External}

[cursor]

Risk: medium. Not approving: Cursor Bugbot completed with status skipping (usage limit reached) and did not produce a clean review signal. Human review is still needed for this queue retry-budget and migration change; Dalanir and WcaleNieWolny are already assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud