feat(init): Handle::drain_logs — flush the log pipeline before a fatal exit

[nicolasauler]

Summary

A fatal error logged right before process exit — e.g. a missing required env var at boot (env::load failing before the server can serve) — was frequently lost. main logs the error, then the trail::Handle drops, and Drop → shutdown() is best-effort: its result is only eprintln!'d, and prod ingests logs only via trail's OTLP exporter (no pod-stdout scraping). So a boot crash was invisible in Grafana/Sentry — you only saw CrashLoopBackOff.

This adds a way for the exit path to flush the log pipeline and learn whether the record landed.

The change (purely additive)

Handle::drain_logs(&self) -> OTelSdkResult:

pub fn drain_logs(&self) -> opentelemetry_sdk::error::OTelSdkResult {
    let flushed = self.logger_provider.force_flush();
    let _ = self.logger_provider.shutdown_with_timeout(Duration::from_secs(12));
    flushed
}

Why both calls — the two SDK primitives each miss one half, verified against opentelemetry_sdk 0.32.1:

• force_flush() propagates the true export result (recv_timeout(..)?), but the SDK hard-codes its wait to 5s.
• shutdown_with_timeout(t) waits t (can exceed the 10s export RPC), but discards the result (recv_timeout(t).map(|_| Ok(()))) — it returns Ok even when the export failed (e.g. connection-refused).

So: force_flush first for a real acknowledgement on the fast path (the common case — a healthy in-cluster collector ACKs in ms), then shutdown_with_timeout(12s) to give a slower export time to finish before the process exits. Contract: Ok = exported and acked within 5s; Err = unconfirmed in that window (may still land via the drain) — not proof of loss.

Also suppresses the benign AlreadyShutdown that Drop's shutdown() would otherwise eprintln! after drain_logs — an error-shaped line in the exact stderr an operator reads to triage the crash.

No behavior change to steady-state export, Drop, or SIGTERM drain — the 10s OTLP default and the existing shutdown() are untouched. Only a new opt-in method.

Consumer

The bipa server will call drain_logs() on its fatal boot-error branch (separate PR). It pairs with the env two-phase change (bipa#16485) that moves telemetry init ahead of the bulk env parse so the error reaches the pipeline in the first place.

Review

Two rounds of adversarial (Fable-5) review folded in: the first version set .with_timeout(3s) on the exporter — which was backwards (it cancels the export RPC at 3s and regresses delivery fleet-wide, since the batch worker exports on a detached thread the old 5s wait never actually cut); the second flagged that shutdown_with_timeout discards the result (false Ok). Both are fixed here. Mechanics verified directly against the SDK source.

Test plan

• cargo check --features init + --all-features clean; cargo fmt --check clean
• CI green
• Behavioral: exercised via the bipa boot-QA (missing required env var → confirm the ERROR reaches Loki, and drain_logs returns Ok)