Skip to content

fix: prevent OpenTelemetry scope leaks masking original exceptions#681

Merged
dgafka merged 1 commit into
mainfrom
scope-detach-tracking
Jul 2, 2026
Merged

fix: prevent OpenTelemetry scope leaks masking original exceptions#681
dgafka merged 1 commit into
mainfrom
scope-detach-tracking

Conversation

@dgafka

@dgafka dgafka commented Jul 2, 2026

Copy link
Copy Markdown
Member

Why is this change proposed?

Fixes #419. When a message send failed inside a traced flow (e.g. a broker or database outage while running ecotone:run), the OpenTelemetry scope activated in TracingChannelInterceptor::preSend was never detached. Every subsequent span closing then triggered Scope: unexpected call to Scope::detach() for scope #XXXX, scope successfully detached but another scope should have been detached first. Under Symfony's error handler this notice becomes an ErrorException that replaces the original exception, so users saw a confusing tracing error instead of the real cause of the failure.

Description of Changes

  • SendingInterceptorAdapter now guarantees the channel-interceptor lifecycle: every interceptor whose preSend completed receives exactly one afterSendCompletion on every path — send success, send failure, failure during another interceptor's preSend, and message drop (preSend returning null, used in production by the collector module)
  • Cleanup hooks (afterSendCompletion/postSend) are individually guarded, so one interceptor failing cannot prevent the others from cleaning up — and the original send exception always takes priority over cleanup failures
  • TracingChannelInterceptor tracks its open span in a per-channel LIFO stack (new ChannelSendSpan with idempotent close) instead of storing the scope object in a temporary message header, so a preSend that rebuilds the message can no longer break scope cleanup
  • TracerInterceptor::traceAsynchronousEndpoint uses try/finally for scope detach; traceDistributedBus now records the exception and closes its span on failure instead of leaking the scope
  • Added TracingScopeCleanupTest covering 8 scenarios: channel send failure, interceptor failing before send, message filtered out mid-send, cleanup failure of another interceptor, postSend failure, message rebuilt without framework headers, error-channel send failure, and Distributed Bus send failure

What users experience

Before — real failure hidden behind a tracing notice:

ErrorException: User Notice: Scope: unexpected call to Scope::detach() for scope #11818,
scope successfully detached but another scope should have been detached first

After — the original exception propagates and the failed send is recorded on the span:

RuntimeException: SQLSTATE[08006] Connection refused

No configuration changes are needed; existing tracing setups keep working as-is:

#[Asynchronous('orders')]
#[CommandHandler('order.place', endpointId: 'placeOrderEndpoint')]
public function placeOrder(PlaceOrder $command, EventBus $eventBus): void
{
    $eventBus->publish(new OrderWasPlaced($command->orderId));
}

Use case scenarios

  1. Broker outage during event publishing — an async handler publishes to RabbitMQ/Kafka while the broker is down: the connection exception now reaches the error channel / dead letter intact instead of being replaced by a Scope::detach() notice
  2. Error channel on a failing transport — the consumer routes a failed message to an error channel whose transport is also unavailable: both spans close correctly, no notices
  3. Message filtering/deduplication combined with tracing — an interceptor drops a message after the tracing span was started: the span is closed instead of leaking into all subsequent traces of the worker process
sequenceDiagram
    participant Consumer as Polling Consumer
    participant Tracer as TracerInterceptor
    participant Adapter as SendingInterceptorAdapter
    participant Tracing as TracingChannelInterceptor
    participant Channel as Message Channel

    Consumer->>Tracer: consume message (activate consumer scope)
    Tracer->>Adapter: handler publishes event
    Adapter->>Tracing: preSend (activate send scope)
    Adapter->>Channel: send
    Channel-->>Adapter: ❌ broker failure
    rect rgb(220, 245, 220)
        Note over Adapter,Tracing: NEW: afterSendCompletion always invoked
        Adapter->>Tracing: afterSendCompletion(exception)
        Tracing->>Tracing: close span, detach scope
    end
    Adapter-->>Tracer: original exception rethrown
    Tracer->>Tracer: finally: detach consumer scope (balanced)
    Tracer-->>Consumer: original exception, no Scope notices
Loading

Pull Request Contribution Terms

  • I have read and agree to the contribution terms outlined in CONTRIBUTING.

When a message send failed inside a traced flow, the scope activated in
TracingChannelInterceptor::preSend was never detached, causing
'unexpected call to Scope::detach()' notices that replaced the original
exception under Symfony's error handler.

- SendingInterceptorAdapter guarantees exactly one afterSendCompletion
  per executed preSend on every path (success, failure, preSend failure,
  message drop), with per-interceptor guarded cleanup
- TracingChannelInterceptor tracks open spans in a per-channel LIFO
  stack instead of a temporary message header
- TracerInterceptor uses try/finally for the consumer scope and closes
  the Distributed Bus span on failure

Fixes #419
@dgafka
dgafka merged commit 226aa8f into main Jul 2, 2026
9 checks passed
@dgafka
dgafka deleted the scope-detach-tracking branch July 2, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unexpected call to Scope::detach() for scope #XXXX, scope successfully detached but another scope should have been detached first

1 participant