fix(ai-sre-relay): dedup alert tickets and sanitize llm analysis before filing

[jdwillmsen]

Fixes the three AI-SRE relay bugs observed in tickets it filed (duplicate pairs JDWLABS-151/153 vs 150/152; conversational filler in 151; raw tool-call dump in 153).

Root causes

1. Duplicate tickets (JDWLABS-151 dup of 150, 153 dup of 152)

Dedup was keyed only on the Alertmanager fingerprint label (amfp-<fp>). Fingerprints hash the full labelset, so the same alert re-firing under a different labelset (e.g. TargetDown for a different job, KubeAggregatedAPIDown for a different apiservice set during the same node incident) gets a brand-new fingerprint — the label search legitimately finds nothing and a sibling duplicate is filed. The observed pairs confirm this: each pair shares an alertname but has different amfp-* labels. A second latent hazard: Jira's JQL index is eventually consistent, so a search issued shortly after a create can miss the fresh issue and duplicate it too.

2. Conversational prose in ticket body (JDWLABS-151)

Holmes /api/chat returns the backing LLM's final message verbatim in analysis, and the relay pasted it straight into the Jira description. Weaker models prepend chat narration — 151 opened with "Now I have a complete picture. Let me summarize the investigation findings." — and nothing stripped it.

3. Raw tool-call dump as ticket body (JDWLABS-153)

When the model emits an un-executed tool call as its final text (typical when the agent loop hits its iteration cap), Holmes still returns it in analysis. 153's entire description was a hermes-style <tool_call><function=execute_prometheus_instant_query>... block with zero analysis. The relay had no validation between the LLM response and the ticket create.

Fix

• jira.go — layered dedup. Upserts are serialized and remembered in an in-process label -> issue key registry (survives JQL index lag; verified against live status via a strongly-consistent issue GET, not search). Search order: (1) fingerprint label — comments, reopening Done matches as before; (2) new amalert-<alertname> label restricted to open issues — a re-fire of the same alert under a new labelset is grouped into the open ticket as a comment instead of a sibling duplicate (Done issues excluded: a refire after close under a new labelset is new work). New issues carry both labels.
• dispatcher.go — in-flight coalescing. A repeat notification for a fingerprint still queued/under investigation is acknowledged but not re-investigated (investigations run for minutes; repeats doubled LLM spend and raced the upsert).
• sanitize.go (new) + holmes.go. All Holmes output passes sanitizeAnalysis before any output path (Jira, Discord, patch gen) sees it: strips <tool_call>/<function=...> markup (terminated or not), rejects JSON-shaped tool calls and empty/all-filler responses, and trims leading/trailing conversational paragraphs (conservative: only short plain prose with first-person markers; markdown structure always survives). Investigate re-asks once on unusable output, then fails the investigation — the pipeline then sends the existing failure notice instead of filing a garbage ticket.

Test evidence

New tests (all following existing httptest fake-server patterns):

• TestJiraUpsertGroupsSameAlertnameOpenTicket — replays the 152/153 pair: new fingerprint, open same-alertname ticket → comment, no create.
• TestJiraUpsertDedupSurvivesSearchIndexLag — create then two refires (same fingerprint, then new fingerprint of same alertname) while search returns empty: exactly 1 create, 2 comments.
• TestDispatcherCoalescesInFlightFingerprint — repeat while in flight is coalesced; same fingerprint investigable again after completion.
• sanitize_test.go (9 tests) — preamble strip (151's exact opening line), trailing chat-offer strip, clean-analysis passthrough, tool-call rejection (153's exact markup, unterminated, and JSON forms), empty/all-filler rejection, embedded-tool-call strip keeping prose.
• TestHolmesInvestigateRetriesOnToolCallResponse, ...ErrorsWhenAnalysisStaysUnusable, ...StripsConversationalFiller — retry-then-fail behavior at the client boundary.

Validation (worktree, Windows):

go build ./apps/backend/ai-sre-relay/   ok
go vet   ./apps/backend/ai-sre-relay/   ok
go test  -count=1 ./apps/backend/ai-sre-relay/          46 passed
go test  -count=1 -race ./apps/backend/ai-sre-relay/    46 passed
pnpm exec nx run ai-sre-relay:lint      ok
pnpm exec nx run ai-sre-relay:test      ok
pnpm exec nx format:check               ok

One existing test updated: TestDispatcherShutdownDrains used one fingerprint for 4 alerts, which the new coalescing intentionally collapses — it now uses distinct fingerprints.

Notes / limitations

• The in-process registry is per-replica; the deployment runs a single replica, and cross-restart dedup is covered by the Jira labels once the JQL index catches up (the observed windows were minutes, well past index lag).
• The filler heuristic is deliberately conservative (short, plain, first-person paragraphs only) to avoid ever deleting substantive analysis.
• Live relay currently runs jdwlabs/ai-sre-relay:0.2.0 with LITELLM_MODEL=sre-investigator; no cluster changes made (read-only inspection only).

🤖 Generated with Claude Code