fix: SendGrid webhook batching, idempotency-key stability, TTS circuit breaker, and cross-tenant job isolation

[Mystery-CLI]

Summary

Fixes four issues surfaced by the automated multi-agent codebase audit, spanning the email service (Rust/services/api) and the TTS service (TypeScript/services/tts). Each fix is isolated to its own commit.

1. Sequential SendGrid webhook batch processing (#1128)

handle_sendgrid_webhook looped over events one at a time, awaiting process_event sequentially. Each process_event performs up to 4 sequential DB/Redis round trips, so a batch of ~300 events could hold a DB pool connection through 600–1200 sequential round trips before responding — risking SendGrid webhook timeouts and pool contention.

Fix: Process events concurrently with bounded parallelism (futures::stream::buffer_unordered, capped at 16 in-flight) instead of a plain sequential loop. DB/Redis round trips overlap instead of stacking serially, so wall-clock time no longer scales linearly with batch size.

• services/api/src/email/webhook.rs

2. Idempotency key rotation can outlive retry backoff (#1129)

idempotency_key derived its hourly rotation bucket from SystemTime::now() on every call. mark_failed's exponential backoff (2/4/8/16/32/~64min...) crosses that 1-hour boundary by the 6th retry. If a send actually succeeded at SendGrid but the worker crashed before mark_completed, a retry scheduled more than an hour later would compute a different key and skip deduplication — resulting in a duplicate email to the recipient.

Fix: The hour bucket is now derived from the job's original created_at timestamp instead of wall-clock "now", so the key stays stable across a job's entire retry lifetime regardless of how long the backoff delay runs. Added a test simulating a retry 90 minutes after creation (crossing the hour boundary) asserting the key is unchanged.

• services/api/src/email/service.rs
• services/api/src/email/queue.rs

3. Retry logic wraps the circuit breaker (#1131)

withRetry() called _callProvider() — which itself called breaker.fire() — on every retry attempt. A single failed user request could generate up to maxRetries + 1 breaker failures instead of one, tripping the volume/error thresholds far earlier than intended. Worse, once the breaker was OPEN, _callProvider re-wrapped the open-circuit error as a 503 TTSProviderError, which isRetryable() treats as retryable — so withRetry kept retrying an open breaker with full exponential backoff, the opposite of the breaker's purpose.

Fix: Moved the retry/backoff loop inside each circuit breaker's wrapped action, so breaker.fire() is invoked exactly once per external request. When the circuit is open, fire() now rejects immediately without ever invoking the action (and therefore without any retry/backoff), giving true fast-fail behavior.

• services/tts/src/TTSService.ts

4. TTS job endpoints leak data across tenants (#1130)

jobStore was a single global Map with no owner/tenant field. GET /tts/jobs and GET /tts/job/:id returned every job in the process — including other tenants' narration text, voice, provider, and output paths — to any caller that passed the (optional) auth check, since neither endpoint filtered by the requesting credential. Additionally, /tts/enqueue and /tts/generate never passed the authenticated request's credential down to the service layer at all.

Fix: Each job is now tagged with an owner identity resolved from its creating credential (the API key itself, or a JWT's sub claim; "anonymous" when no auth is configured). getJob/listJobs require the owner to match, and getJob returns the same 404 for "not found" and "belongs to another tenant" so a credential can't distinguish the two by probing IDs. Internal operational code (health checks, queue-depth metrics) that needs a cross-tenant view now uses a new listAllJobsUnscoped(). Wired the missing credential through /tts/enqueue and /tts/generate. Added tests creating jobs under two different API keys and asserting cross-tenant access is denied.

• services/tts/src/TTSService.ts
• services/tts/src/server.ts
• services/tts/src/HealthCheck.ts
• services/tts/src/__tests__/auth.test.ts

Test plan

• cd services/api && cargo clippy --all-targets --all-features -- -D warnings
• cd services/api && cargo test webhook
• cd services/api && cargo test service
• cd services/api && cargo build --release
• cd services/tts && npx tsc --noEmit --project tsconfig.json
• cd services/tts && npm test -- TTSService
• cd services/tts && npm test -- server
• cd services/tts && npm run build

Note: this environment had no Rust toolchain available, so the Rust-side changes (#1128, #1129) were not compiled or test-run locally — please verify in CI before merging.

Closes #1128
Closes #1129
Closes #1130
Closes #1131

🤖 Generated with Claude Code

[drips-wave]

@Mystery-CLI Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits