Structured JSON logging with correlation IDs across request lifecycle

[Armolas]

Summary

Closes #925. Pino was already wired up for structured JSON logging, and a requestId middleware already existed — but correlation IDs never made it past the access-log line into jobs, RPC calls, or business-logic logs, and there was no secret redaction at all. This PR closes those gaps.

What's new

Correlation context

• New AsyncLocalStorage-based correlation context (backend/src/lib/requestContext.js) so any code in a request or job's call chain can read the current correlation ID without threading it through every function signature.
• Wired into pino via a mixin — every log.*() call automatically picks up requestId from the active context, no call site needs to remember to pass it.
• X-Request-Id added to CORS allowed/exposed headers so cross-origin clients can read and supply it.

Propagation to jobs + RPC

• Both jobRunner (in-memory) and durableJobQueue (SQLite-backed) capture the enqueuing request's correlation ID and re-establish it as context when the job actually runs, so job logs — including retries — tie back to the request that triggered them.
• The durable queue persists the ID on the job row (migration 035) so it survives a process restart.
• rpcPool.js and sorobanRpc.js had zero logging before this PR. Added structured log lines for circuit-breaker transitions, pool saturation, endpoint health changes, and RPC success/failure/latency — all correlated via the outbound x-request-id header and log context.

No secret leakage

• Added a pino redact config stripping API keys, auth headers, passwords, and tokens by convention (previously none existed).

Cleanup (blocking "all logs are structured")

• Fixed ~26 call sites where log.X('message', {fields}) was silently dropping the fields object — pino's real argument order is (fields, message), verified empirically. Affected status.js, webhooks.js, faucet.js, pruningJob.js.
• Migrated routes/variants.js and routes/audit.js off raw console.error onto the structured logger.

Bugs found along the way

Tightening checkSorobanRpcHealth's typing (needed for correct test coverage) surfaced two compounding pre-existing bugs in routes/status.js: it called the health check with no rpcUrl (always fetching undefined), and treated the returned result object — always truthy, even on status: 'error' — as a boolean. Net effect: the public status page's RPC component permanently reported "operational" regardless of actual health. Fixed both.

Testing

~60 new/extended tests across lib/requestContext, middleware/requestContext, middleware/requestId (previously untested), middleware/logger (mixin + redact verified against a real pino instance, not mocked), jobRunner, durableJobQueue, rpcPool, sorobanRpc, and index.test.js integration tests.

Live end-to-end smoke test: booted the real backend and confirmed a client-supplied X-Request-Id header shows up correlated across the access log, RPC log, and response headers; confirmed job retries share a consistent correlation ID across attempts; confirmed an Authorization header never leaks into logs.

eslint/tsc --noEmit clean on all touched files (pre-existing, unrelated warnings/errors elsewhere untouched).

Acceptance criteria

• All logs are structured + correlated.
• IDs propagate to jobs + RPC.
• No secret leakage in logs.

[drips-wave]

@Armolas Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits