Skip to content

[agency] Create hybrid CLJS TTS planner with host-owned audio effects #12

Description

@meekmachine

Goal

Create the Polyester TTS agency as a hybrid ClojureScript planner plus host adapter: CLJS owns speech/timeline planning, while browser/backend/audio effects stay on the main thread or backend.

Current state

Polyester root still inherits the TypeScript TTS service from Latticework. There is no src-cljs/latticework/tts.cljs agency yet. The TypeScript src/tts/ttsService.ts currently owns Web Speech, SAPI/Azure backend calls, AudioContext playback, display-media playback reference, timeouts, and a VocalService instance.

This mirrors the Latticework research issue: meekmachine/Latticework#37.

Clojure-way direction

  • CLJS owns text token parsing, emoji/emotion event extraction, provider response normalization, timeline planning, utterance state, cancellation tokens, and commands to Vocal/Prosodic/EmotionExpression.
  • Main thread owns Web Speech, SpeechSynthesisUtterance, AudioContext, display-media capture, HTMLAudioElement, permission prompts, fetch calls, and playback start/end events.
  • Backend owns Azure credentials and synthesis for the buffered Azure path. Polyester workers should consume provider timing data; they should not own secrets.
  • Preserve current JS imports (createTTSService, TTSService) through an adapter during migration.

Azure/audio contract to support

  • Buffered Azure: host calls LoomLarge POST /api/azure-tts/synthesize, receives audio_base64, visemes, word_boundaries, duration, decodes audio, starts playback, then sends a clock-anchored provider timeline into CLJS/Vocal.
  • LiveKit drawer TTS: host calls POST /api/tts/synthesize, receives audio_base64 WAV and estimated word boundaries, plays audio, then sends text-predicted timeline plus word-boundary ticks.
  • Web Speech: host listens for onboundary and start/end events, while CLJS plans fallback text visemes and prosodic/emotion events.

Acceptance criteria

  • Existing TTS public API remains source-compatible.
  • No CLJS worker code references Web Speech, navigator.mediaDevices, AudioContext, DOM, backend fetch APIs, LiveKit objects, or Loom3.
  • TTS emits data commands for Vocal, Prosodic, and EmotionExpression instead of directly scheduling animation snippets.
  • Stop/interruption reliably cancels pending timelines and active speech state.
  • Current TTS tests continue to pass through the adapter.
  • Add CLJS tests for token parsing, emoji timing, provider response normalization, utterance cancellation, and speech lifecycle state.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions