Add human handoff, CSAT capture, and analytics dashboard - #64
Merged
Conversation
Closes the P0 competitive gaps vs. Intercom Fin/Zendesk AI Agents: sessions can now escalate to a human agent (server-side tool + max-tool-rounds fallback), operators can reply/resolve escalated sessions from the dashboard, users can submit a 1-5 CSAT rating after a resolved turn, and the dashboard exposes a resolution-rate/escalation-rate/CSAT analytics page. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reused-session-history endpoint filtered messages to user/assistant only, so a human_agent reply sent while a client was offline would never appear on reconnect (only via the live human_message SSE event). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sessions previously only had reply/resolve as raw API endpoints with no way to use them from the dashboard. Adds an escalation banner with escalation reason, a "Mark resolved" button, and a reply composer on escalated sessions in the Sessions page. Also surfaces escalated_at/ escalation_reason/resolved_by from sessionOut, which the serializer was previously dropping. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The CI pipeline builds one dashboard image and bakes NEXT_PUBLIC_API_BASE_URL to production's domain for every deploy target. Since `dashboard` and `api` are the same Next.js app/image in every topology this repo supports, route all client requests same-origin (matching how the cookie-bound auth routes already behaved) instead of trusting that build-time constant. Fixes staging (and any other non-prod deploy) silently calling production's API and getting 401s / "failed to load" for data that only exists in that environment's own database. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two gaps found while testing the human-handoff flow end to end:
- There was no way for an operator to join a conversation the AI hadn't
already escalated. New POST .../escalate route + "Take over" button on
active sessions in the dashboard manually flips a session to escalated,
which unlocks the existing reply/resolve UI.
- CSAT was only ever requested after an AI-resolved turn, never after a
human manually resolves an escalated session. New internal endpoint
(POST /internal/sessions/{id}/feedback-requested) + dashboard resolve
route now fires feedback_requested when an operator marks a session
resolved, on top of the existing AI-turn trigger.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
feedback_requested now carries an "immediate" flag: false for the existing post-AI-turn trigger (SDKs debounce it), true for the human-resolve trigger (SDKs should show it right away, since the conversation is definitively over). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root-caused a real bug hit while testing the human-handoff flow: a brief Redis stall (e.g. a scheduled BGSAVE) during a tool-call round caused a cascade of failures that left a session in a broken, duplicated-turn state. Three contributing bugs, all in Redis-touching code paths: 1. resolve_pending_tool_result (called when the SDK submits a tool result) let a transient Redis publish/lock-check error propagate, which skipped the same-process local future resolve entirely -- silently dropping an already-successful tool result. 2. run_agent_loop's tool-wait only caught asyncio.TimeoutError. Any other failure (e.g. that same Redis error, or its knock-on effects) crashed the whole turn unhandled. _run_turn's `finally: clear_turn()` then released the per-session turn lock anyway, letting a second message start a *concurrent* second turn for the same session while the first call sat forever unresolved -- exactly the "duplicate tool request, stuck approval card" state observed. 3. The SSE /events endpoint's `except TimeoutError` doesn't catch redis.exceptions.TimeoutError (it isn't a subclass of the builtin), so a Redis I/O hiccup crashed the whole stream instead of degrading to a keep-alive-and-retry, producing the misleading "something went wrong" bubble on the client. All three now degrade gracefully: a transient error resolves locally where possible, is recorded as a normal (recoverable) tool_result error and the loop continues, or falls back to a keep-alive -- instead of corrupting turn-lock exclusivity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The internal service-to-service JWT settings (added earlier for the
dashboard->agent human-reply endpoint) were only ever added to
agent/config.py's Python defaults and .env.example -- never actually
passed through to the dashboard/api or backend containers in any of
the three docker-compose files. The dashboard's token signer throws at
runtime if this key is unset/left as the insecure placeholder, so every
reply attempt failed silently server-side ("cannot send the message to
the user"). Wires it into all three compose files the same way
RK_KNOWLEDGE_BASES_SIGNING_KEY already is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The manual "Take over" route updated the session's status in the DB
but never told the connected SDK client -- so isEscalated never became
true on-device, the escalated banner never showed, and critically the
CSAT-suppression-on-escalation logic never fired. A CSAT prompt from
the AI's last turn could keep showing (or later reappear) mid human
handoff, on an unresolved session.
New internal endpoint (POST /internal/sessions/{id}/session-escalated)
mirrors the existing feedback-requested one: the dashboard route now
pushes a real session_escalated SSE event after taking over, exactly
like the AI-triggered escalation path already does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the P0 competitive gaps identified vs. Intercom Fin / Zendesk AI Agents:
Test plan
uv run python -m pytest tests/— 230 passed, 2 pre-existing unrelated failures (confirmed viagit stashagainst main)uv run python scripts/check_openapi_sync.py