Skip to content

tech debt: notify Jira when workflow execution exhausts queue retries #140

Description

@eshulman2

Context

PR #58 centralizes Jira notifications for handled workflow failures represented by last_error. A separate visibility gap remains when compiled_workflow.ainvoke() raises instead of returning workflow state: the worker logs and re-raises the exception, and the queue retries the event, but Jira is never notified if all retries fail.

This is intentionally out of scope for PR #58 so handled workflow errors and queue-level execution failures remain separate concerns.

Proposed behavior

  • Do not notify Jira for each retryable queue failure.
  • When the existing retry mechanism exhausts its attempts and moves the event to the dead-letter queue, emit one terminal Jira notification.
  • Keep the existing queue retry behavior unchanged.
  • Include the ticket key, sanitized error summary, event/correlation ID, and recovery instructions.
  • Use the shared error formatting/redaction path where practical.
  • Add idempotency protection keyed by the event ID so terminal processing cannot create duplicate comments.
  • A Jira notification failure must not prevent the message from reaching the dead-letter queue.

Suggested design

Add an optional terminal-failure callback or event to QueueConsumer. Invoke it when RetryQueue.enqueue_for_retry() reports that the message was moved to the dead-letter queue, in both the initial consumer failure path and the retry poller. The orchestrator can provide the Jira-specific handler without coupling the generic retry queue directly to Jira.

Acceptance criteria

  • Retryable escaped workflow exceptions do not post Jira comments.
  • Exhausting queue retries posts exactly one terminal Jira comment.
  • The error text is secret-redacted.
  • Duplicate terminal handling for the same event does not create another comment.
  • Notification failure does not interfere with dead-lettering.
  • Both the initial processing path and retry-poller path are covered by tests.

Related: #58

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:integrationsJira and other external service integrationsarea:workflowWorkflow graphs, orchestration, state, routing, and gatesenhancementNew feature or requesttech-debtInternal maintainability or architectural debt

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions