Skip to content

feat: asynchronous message publishing with batching and pre-commit delivery guarantees (Enterprise)#684

Open
dgafka wants to merge 32 commits into
mainfrom
async-publishing
Open

feat: asynchronous message publishing with batching and pre-commit delivery guarantees (Enterprise)#684
dgafka wants to merge 32 commits into
mainfrom
async-publishing

Conversation

@dgafka

@dgafka dgafka commented Jul 18, 2026

Copy link
Copy Markdown
Member

Why is this change proposed?

Publishing to a message broker blocks the handler on one full round-trip per message — a
handler emitting many messages pays N sequential broker round-trips, which caps publishing
throughput as systems scale.

Ecotone now provides an Enterprise feature that makes publishing scale, while still keeping
all the delivery guarantees end users rely on: every message is still confirmed by the
broker, a failed delivery still fails the business operation or is routed to the error
channel, and every failure points to the exact message that failed — nothing is ever lost
silently.

This is achieved by firing messages to the broker immediately, batching them natively per
provider, and deferring confirmation awaiting to the last responsible moment — messages
travel to the broker while the handler keeps working, and all confirmations are collected
before the operation completes.

Benchmarks

Publishing 1,000 messages, relative to each provider's previous message-by-message
synchronous publishing (= 100%). Mode of 10 iterations, all delivery confirmations awaited:

Provider Feature used Time vs previous Speed-up
Kafka Batching + async sending 9% ~11×
SQS Batching + async sending 9% ~11×
DBAL Batching 10% ~10×
AMQP Batching + async sending 32% ~3×
Redis Batching 36% ~3×
Kafka Async sending (single messages) 36% ~3×
SQS Async sending (single messages) 65% ~1.5×

Batching + async sending publishes the workload as provider-native batches with
confirmation awaiting deferred; async sending fires individual messages without waiting.
DBAL and Redis write synchronously, so batching is their applicable feature.

Resulting flow

graph TD
    A[Handler emits messages] --> B[Collector gathers them]
    B -->|one BatchMessage| C[Adapter: provider-native batch publish, no wait]
    C --> D[Registry: PendingDelivery per publish]
    D --> E[Waiter interceptor: await all confirmations]
    E -->|all confirmed| F[Operation completes]
    E -->|some failed| G[Only failed messages to error channel, or operation fails]
Loading

Enabling

ServiceConfiguration::createWithDefaults()
    ->withExtensionObjects([
        // Message Channel: messages published from handlers are gathered,
        // sent as one batch and confirmed before the operation completes
        AmqpBackedMessageChannelBuilder::create('orders')
            ->withAsyncPublishing(),
        KafkaMessageChannelBuilder::create('orders')
            ->withAsyncPublishing(timeoutInMilliseconds: 12000),

        // Message Publisher: enables MessagePublisher::asyncPublish()
        AmqpMessagePublisherConfiguration::create()
            ->withAsyncPublishing(),
    ]);

Example

#[CommandHandler]
public function place(PlaceOrder $command, EventBus $eventBus): void
{
    // events are batched to the broker and their confirmations
    // awaited automatically before the operation completes
    $eventBus->publish(new OrderWasPlaced($command->orderId));
}

// explicit async publishing with a Future
$future = $messagePublisher->asyncPublish(
    BatchMessage::constructEmpty()
        ->append($firstOrder)
        ->append($secondOrder)
);

$future->resolve(); // throws AsyncPublishingFailedException listing only the messages that failed

Pull Request Contribution Terms

  • I have read and agree to the contribution terms outlined in CONTRIBUTING.

dgafka added 30 commits July 3, 2026 07:00
…en by command handler batch publishing tests with delay and ttl across providers
…cycle

- catch \RdKafka\Exception instead of the undefined KafkaException so
  producer queue-full backpressure actually polls and retries rather than
  failing the publish on the first full-queue signal
- require AsyncPublishingRegistry in the SQS and Enqueue outbound adapters,
  matching DBAL and Redis, removing the nullable dereference
- drop the register_shutdown_function auto-flush; unresolved deliveries are
  now flushed via explicit flushUnawaitedDeliveries and the backlog limit
dgafka added 2 commits July 18, 2026 08:00
…nd error channel routing, shutdown flush of unawaited deliveries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant