feat(voice-relay): framework agents join the external TTS relay as bot clients - #68
Open
nickmahdavi wants to merge 4 commits into
Open
feat(voice-relay): framework agents join the external TTS relay as bot clients#68nickmahdavi wants to merge 4 commits into
nickmahdavi wants to merge 4 commits into
Conversation
Two framework-core changes that let a channel-scoped consumer (e.g. a TTS
relay bridge) follow an agent's turn without reaching into private
framework state.
1. Optional channelId on the inference:* trace family, stamped from the
turn's pinned locus by a read-only helper at the existing trace-emit
sites, so a consumer can key per-channel streams. With a channelRegistry
the pin is the only source read: a mid-turn channel_open moves the
trigger bookkeeping without moving actual routing, so falling back to it
would stamp a channel the turn's speech never lands in. Registry-less
hosts fall back to the triggering channel. Turns with no channel
(heartbeats, timers) leave the field unset.
2. abortInference(name, { reason, keepText }). When a user-initiated abort
supplies keepText — the text a voice client delivered aloud before the
user interrupted — its not-yet-committed part is saved as the assistant's turn, and the
channel receives only the part the turn's live speak-while-acting posts
have not already delivered (undeliveredSuffix matches the spoken text
against live-routed prose; re-routing it verbatim would double-post).
Silencing is honored as at completion, and a keepText racing a
framework-internal cancel is discarded with its stream. With no keepText
the partial turn is discarded as before, and framework-internal cancels
are unchanged. The existing string form, abortInference(name, reason),
still works.
prose-segments also gains isWhitespaceInsensitivePrefix, the same
whitespace-tolerant walk answering only yes/no. The relay client uses it to
judge whether a voice client's reported spoken text belongs to the utterance
currently streaming in a channel — a report that does not match is stale and
must not cut off a newer turn.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y test helpers Adds the pieces shared by the relay-client work: - The v2 TTS-relay wire-protocol types, ported from melodeus-tts-relay to match it exactly on the wire. - InferenceTraceBridge, which translates the framework's inference:* traces into the relay's streaming messages (activation_start, block_start, chunk, block_complete, activation_end), keyed by the new channelId. - sha256 token helpers. - Test helpers: a recording socket, a simulated v2 voice client, and a helper that spawns the reference melodeus-tts-relay used by the end-to-end tests. The reference-repo lookup no longer false-positives on the framework repo itself when HOME is unset. Tests cover the bridge's translation table: identity resolution, the visible flag (text chunks are voiced, thinking is not), block-content accumulation, terminal-trace mapping to activation_end reasons, channel-less drops, and unsubscribe on stop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nal TTS relay
Connects framework agents to an external melodeus TTS relay the same way the
relay's own bots do: the module opens one WebSocket to the relay's /bot
endpoint, authenticates as a single bot identity, and streams the agent's
turn up the socket as it is produced. The stream is the relay's own
messages (activation_start, block_start, chunk, block_complete,
activation_end), translated from the agent's inference:* traces and scoped
to a channel by the trace channelId.
Interruptions coming back from the relay are mapped to
abortInference(agentName, { keepText }), so an interrupted agent's context
keeps the words the client verifiably reported spoken. Addressing is strict: an
interruption naming a channel the module never streamed to is dropped
(guessing could abort an unrelated agent), and only a channel-less
interruption may fall back to the single tracked agent. Before aborting,
the reported spoken text must prefix-match the current utterance's streamed text
(per-block or whole-activation, per client) — voice lags text, so a report that does not
match the current utterance describes an earlier one and must not cut off
the new turn. Channel tracking is bounded (256, oldest evicted) so a
long-lived process cannot grow it without limit.
Connection handling: the client treats relay heartbeats as liveness only,
reconnects with exponential backoff and re-authenticates, and — matching the
relay — does not queue while disconnected, so messages produced while the
socket is down are dropped. A close carrying the relay's "Replaced by new
connection" is fatal rather than retried: two clients sharing a bot
identity would otherwise evict each other forever. The backoff resets only
after a connection has stayed authenticated for a stability window, so an
auth-then-drop loop cannot hammer the relay from the floor delay.
Tested against an in-process mock /bot server (auth, bot-identity stamping,
reconnect and the replaced-connection stop, backoff stability, interruption
addressing and staleness, channel-tracking bounds) and end to end against
the real melodeus-tts-relay: a simulated voice client hears a framework
agent's streamed turn and interrupts it mid-sentence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing, keepText hygiene, connection hardening
Fixes from two review rounds: a six-dimension adversarially-verified
workflow, then a ten-agent sweep of the full PR.
Block emission (the load-bearing fix): the agent's production stream
requested emitBlocks: false, so inference:content_block never fired
outside tests and block_start/block_complete never reached the wire —
melodeus keys its utterance state machine (and its ability to interrupt)
on block_start, and iOS flushes the trailing sentence on block_complete.
Masked because the mock membrane ignored the flag; it now honors it, so
every block-dependent test doubles as a regression pin on the agent
actually requesting blocks.
Interruption correctness:
- Interruptions are dropped when the target agent is mid-turn on a
DIFFERENT channel (activeChannelByAgent, set on inference:started,
cleared on terminals): a late report for a finished turn must not abort
the unrelated turn the agent is running now.
- The staleness guard matches spokenText against two windows — since the
channel's last block_start AND since its activation_start — because the
reference clients differ (melodeus resets per block, iOS accumulates the
whole activation). Previously a legitimate iOS report on a multi-block
turn was dropped as stale.
- Both comparison walks ignore characters clients normalize away:
whitespace and narrator-markup asterisks. iOS voices *action* spans via
a narrator voice and strips the asterisks from its report, so any
narrated turn's interruption used to fail verification and the agent
kept talking.
- A non-empty report while the current utterance has voiced nothing yet
(activation_start seen, no visible text) is dropped as stale: real
clients never report speech for an utterance that voiced nothing, so it
can only describe the previous turn — previously it aborted the fresh
turn. Only a report with NO accumulator entry at all (connected
mid-turn) is treated as unverifiable: the abort goes through, but
unverifiable text is never forwarded as keepText — text we cannot match
against what we streamed must not be committed as words the agent said.
keepText lifecycle (framework core):
- keepText is stored only after a SUCCESSFUL abort. Storing before and
deleting on failure let a duplicate interruption report (same streamId —
cancelStream does not bump it) wipe the first abort's still-pending
keepText, discarding the spoken words entirely.
- abortKeepTexts is keyed `${agentName}:${streamId}` with a driveStream
finally backstop, so an abort racing turn completion goes inert instead
of attaching an old turn's spoken text to a later abort of the agent.
- The keepText branch snapshots the turn's locus BEFORE awaiting the
speech chain (a successor turn re-pins it) and re-checks agent.streamId
after its awaits: a stream started during that window is not
reset/settled against the wrong turn, and driveStream's finally no
longer deletes the successor's name-keyed per-turn state (which would
strand its tool round).
- A user abort is booked as a deliberate cancel (inference:exhausted with
errorType 'abort'): no consecutive-failure streak, no failures.log
entry, no false "[inference-failed] nothing was sent" chronicle marker —
three routine voice barge-ins must not page an operator as hard-down.
- The context commit dedupes keepText against turnCommittedProse (the
rounds already flushed with their tool calls), so a whole-activation
client's replayed prose is not committed twice.
Connection hardening:
- Heartbeat watchdog: the relay heartbeats every ~2s; when nothing arrives
for heartbeatTimeoutMs (default 8s) the link is presumed half-open and
terminated so the normal reconnect path recovers it. handshakeTimeout
covers the dial phase the watchdog cannot (a host that accepts TCP but
stalls the upgrade).
- Inbound frames are shape-checked (a bare JSON null previously crashed
the process), capped at 1 MiB, and handled inside a catch so a throw in
the abort path cannot escape the socket listener; the interruption
reason is coerced to the protocol enum.
- Outbound sends drop when the socket buffer exceeds 4 MiB (delivery is
best-effort by design) and reconnects carry ±20% jitter.
- The fatal replaced-connection stop tears down the trace subscriptions
with the connection — a permanently-down module must not keep
translating every turn into sends that go nowhere. auth_error drops the
authed flag, duplicate auth_ok cannot re-arm the stability timer, and a
restarted module begins at the backoff floor.
- The default logger is console-backed for info and above (matching
RelayLogger's documented contract) so auth rejection and the fatal
replaced-connection stop are visible without configuration.
Scope and typing cleanup:
- \`agents\` config filter scopes the bridge and interruption addressing to
the listed agents, making the multi-instance recipe in the header true;
the header documents the required construct → bind(framework) →
addModule wiring, and the iOS deployment constraint (unconfigured-bot
announcements prefix "<name> says:" to reports, which fails
verification — give the bot a voice entry).
- Outbound messages are typed BotStreamMessage (the bot-side union) so the
compiler rejects anything a /bot client must never send. NoopTraceBridge
(no consumers) and tokens.ts (relay-server account machinery, no
callers) are removed.
- inference:stream_restarted carries channelId and the bridge closes the
abandoned activation with activation_end('abort') — verified against
both clients ('abort' cancels iOS's queued audio for the abandoned
stream; the re-stream then delivers fresh), so a budget restart no
longer leaves an unpaired activation_start on the wire.
- trace.ts documents exactly which inference:* members carry channelId;
types.ts records the relay commit it mirrors (ec8f0f1) and glosses "v2"
and "ChapterX".
Tests: 438 → 454. New coverage: block emission (mock honors emitBlocks),
watchdog (dead link + kept-alive), malformed frames, the
unverifiable-vs-stale report split, narrator-asterisk reports, iOS-style
whole-activation reports, cross-turn late interruptions, duplicate-abort
keepText survival, multi-round keepText dedupe, stream_restarted closure,
the agents filter, deliberate-cancel booking, fatal-stop teardown, and a
CI-runnable closed loop (real framework + mock /bot server: interruption
→ abort → activation_end('abort') on the wire, spoken prefix in context)
so the round trip no longer rides only on the locally-run e2e. All unit
tests tear down via t.after so a failed assertion cannot leak
sockets/timers and hang the runner; the e2e cleans up its spawned relay
on any failure; the backoff-stability window is widened against
CPU-starved CI runners.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Framework agents join the external melodeus TTS relay as first-class bot clients — the same
/botconnection the relay's existing (ChapterX) bots hold. The relay itself is untouched and stays external: one module instance dials{url}/bot, authenticates as one bot identity, streams the agent's turns as the relay's own wire messages, and maps interruptions coming back intoabortInference. Only the client connection each agent-framework process needs; nothing relay-side moves into connectome.Commits / how to review
channelIdon the turn-scopedinference:*traces (stamped read-only from the turn's pinned locus), andabortInference(name, { reason, keepText })so an interrupted agent's context keeps what was actually spoken. ~200 src lines; the only commit touching shared code — review this one hardest.types.tsis a near-verbatim mirror ofmelodeus-tts-relay/src/types.ts@ ec8f0f1 (fastest review: diff it against the relay). The bridge translatesinference:*traces into the five streaming messages.emitBlocks: true— block boundaries never reached the wire before (melodeus can't track or interrupt an utterance without them), masked by mocks that ignored the flag.Behavior notes for existing deployments
TraceEventadditions are optional fields;abortInferencestill accepts the plain string form; hosts without the module are unaffected.inference:exhaustedwitherrorType: 'abort') — no failure streak, nofailures.logentry, no[inference-failed]chronicle marker. Voice barge-ins are routine; three in a row must not page an operator as hard-down. Monitoring that counted aborts as failures will see fewer entries.bind(framework)→addModule. Withoutbind, outbound streaming works but interruptions are dropped (warned). One instance = one bot identity; scope withagents: [...]when running several instances in one process.Testing
/botserver — a turn streams to the wire, an interruption comes back,activation_end(reason: "abort")goes out, and the agent's context keeps the spoken prefix./ttsvoice client hears a framework agent through the real relay and interrupts it mid-sentence.