[HARD][PAYMENTS] Add durable webhook inbox, retry, dead-letter, and replay system

[Obiajulu-gif]

Overview

Harden Paystack webhook processing by introducing a durable webhook inbox that records verified events before business processing and supports safe retries, dead-letter handling, and controlled replay.

Problem

The current webhook route verifies the signature and immediately performs database work. If parsing, MongoDB, a downstream service, or deployment fails after Paystack sends the event, retry visibility and recovery are limited. Returning errors can cause repeated delivery, while returning success too early can lose financial events.

Scope

• Persist the raw verified event envelope before domain processing.
• Store provider event ID/reference, event type, payload hash, received time, processing status, attempts, next retry, and last error.
• Deduplicate identical deliveries and detect conflicting payloads for the same reference.
• Separate HTTP acknowledgement from idempotent domain processing.
• Add retry policy with exponential backoff and maximum attempts.
• Move exhausted or structurally invalid events to a dead-letter state.
• Add an admin-only replay command/API with reason and audit log.
• Redact or encrypt sensitive payer/bank metadata in stored payloads.
• Add retention/cleanup policy for webhook bodies.
• Support mock events for contributor development.

Reliability requirements

• A verified event must not be lost after acknowledgement.
• Replaying an already completed event must not create value twice.
• One failing event must not block unrelated events.
• Conflicting payloads sharing a reference must be quarantined.
• Processing must expose correlation IDs in logs and audit records.

Test requirements

• duplicate delivery before and after successful processing
• database/service failure followed by retry
• malformed JSON and valid signature edge case
• conflicting amount/reference payload
• dead-letter transition
• authorized and unauthorized replay
• concurrent workers claiming the same event
• retention cleanup

Acceptance criteria

• Verified webhooks are durably stored before processing.
• Processing is asynchronous or otherwise retryable without provider redelivery.
• Duplicate/replayed events remain idempotent.
• Dead-letter events can be inspected and safely replayed by admins.
• Sensitive payload data is protected and retention is documented.
• Metrics/logs expose pending, failed, retried, and dead-letter counts.
• Lint, typecheck, tests, and build pass.

Difficulty

Hard