Skip to content

fix: executionTimeLimit no longer blocks graceful shutdown#680

Merged
dgafka merged 1 commit into
mainfrom
signal-handling-consumers
Jul 2, 2026
Merged

fix: executionTimeLimit no longer blocks graceful shutdown#680
dgafka merged 1 commit into
mainfrom
signal-handling-consumers

Conversation

@dgafka

@dgafka dgafka commented Jul 2, 2026

Copy link
Copy Markdown
Member

Why is this change proposed?

When a consumer configured with executionTimeLimit (e.g. ecotone:run internal --executionTimeLimit 3600000) has no messages to process, it fails to respond to SIGTERM/SIGINT/SIGQUIT or even its own time limit in a timely manner. The root cause: DBAL/AMQP/SQS/Redis/Kafka inbound channel adapters used executionTimeLimit as the timeout for a single blocking broker receive() call, fully overriding the channel's normal, short receiveTimeoutInMilliseconds. In production this means a consumer with a 1-hour execution time limit sits blocked inside one broker call for up to an hour before ever checking for termination — breaking graceful shutdown in containerized/orchestrated environments (Kubernetes SIGTERM during pod termination, docker stop, etc). Fixes #440.

Description of Changes

  • Cap the per-poll receive timeout with min(executionTimeLimit, receiveTimeoutInMilliseconds) instead of letting executionTimeLimit fully override it, in EnqueueInboundChannelAdapter (covers DBAL, SQS, Redis), AmqpStreamInboundChannelAdapter, and KafkaInboundChannelAdapter.
  • This preserves the existing "make tests fast" behavior (PollingMetadata::withTestingSetup() intentionally sets a small executionTimeLimit to shorten polling in tests) while fixing production runs where executionTimeLimit is large.
  • Added ExecutionTimeLimitReceiveTimeoutTest — a fast, low-level reproduction proving the receive call now respects the channel's configured timeout instead of the execution time limit.
  • Added GracefulShutdownOnSignalTest — an end-to-end reproduction that sends a real SIGTERM to a running consumer (mirroring the issue's exact repro steps) and asserts it shuts down promptly instead of ignoring the signal.
  • Bumped maxExecutionTimeInMilliseconds in several Kafka tests that consume multiple messages within a single time-bounded run() call, from the pre-existing consumer-group rebalancing floor (10000ms) up to 20000ms — those tests had no margin for a second poll cycle and were prone to flaking under load (unrelated to the fix itself, but surfaced while validating it).

No code changes needed for end users — consumers configured with executionTimeLimit now behave correctly out of the box:

// php bin/console ecotone:run async --executionTimeLimit 3600000
// Before: SIGTERM ignored for up to 1 hour while the queue is idle.
// After: consumer keeps polling at its normal cadence and responds to
//        SIGTERM/SIGINT/SIGQUIT promptly, regardless of executionTimeLimit.

Use case scenarios:

  1. Running an Ecotone consumer under Kubernetes/Docker with a rolling deploy — the orchestrator sends SIGTERM and expects the process to exit within its grace period; previously a consumer with a large executionTimeLimit and no traffic could ignore that signal for the full duration.
  2. Long-running batch consumers using executionTimeLimit purely as a safety net (e.g., process restarted hourly by a supervisor) — these silently failed to self-terminate on schedule when idle.
  3. Local development where Ctrl+C (SIGINT) is sent to a running consumer, expecting immediate shutdown.
sequenceDiagram
    participant OS
    participant InterceptedConsumer
    participant SignalInterceptor
    participant Adapter as Inbound Channel Adapter
    participant Broker

    Note over InterceptedConsumer,Broker: Before fix
    InterceptedConsumer->>Adapter: poll (timeout = executionTimeLimit, e.g. 3600000ms)
    Adapter->>Broker: receive(3600000ms)
    OS--)InterceptedConsumer: SIGTERM
    Note right of InterceptedConsumer: signal flagged but not checked -<br/>still blocked inside receive()
    Broker--)Adapter: (nothing, up to 1 hour later)

    Note over InterceptedConsumer,Broker: After fix
    InterceptedConsumer->>Adapter: poll (timeout = min(3600000, 1000)ms)
    Adapter->>Broker: receive(1000ms)
    OS--)InterceptedConsumer: SIGTERM
    Broker--)Adapter: null (timeout)
    Adapter--)InterceptedConsumer: control returns
    InterceptedConsumer->>SignalInterceptor: shouldBeStopped()
    SignalInterceptor--)InterceptedConsumer: true
    InterceptedConsumer->>InterceptedConsumer: graceful shutdown
Loading

Pull Request Contribution Terms

  • I have read and agree to the contribution terms outlined in CONTRIBUTING.

Cap the per-poll receive timeout with min(executionTimeLimit, receiveTimeoutInMilliseconds)
instead of letting executionTimeLimit fully override the channel's configured receive
timeout in EnqueueInboundChannelAdapter (DBAL/SQS/Redis), AmqpStreamInboundChannelAdapter,
and KafkaInboundChannelAdapter. Previously a consumer started with a large executionTimeLimit
(e.g. --executionTimeLimit 3600000) could block inside a single broker receive() call for
the full duration while idle, ignoring SIGTERM/SIGINT/SIGQUIT and its own time limit until
that call returned.

Fixes #440
@dgafka
dgafka merged commit d90d5c2 into main Jul 2, 2026
9 checks passed
@dgafka
dgafka deleted the signal-handling-consumers branch July 2, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Signal handling in consumers fails with executionTimeLimit option

1 participant