[agency] Rework CLJS lipsync/viseme timing around provider timelines

[meekmachine]

Goal

Rework the LipSync agency as a ClojureScript provider/phoneme planner that feeds the active Vocal timeline model, with special care for Azure/LiveKit viseme timing and canonical Loom3 viseme mapping.

Current state

Polyester has both:

• Legacy TypeScript lip-sync: src/lipsync/lipSyncService.ts, lipSyncScheduler.ts, PhonemeExtractor.ts, VisemeMapper.ts, Azure mapping helpers, coarticulation/performance helpers.
• Newer CLJS work: src-cljs/latticework/lipsync.cljs plus the active sentence-level src-cljs/latticework/vocal.cljs path.

The stabilization discussion says the visible mouth should use the newer Vocal timeline path, not split ownership between legacy LipSyncScheduler and VocalService: meekmachine/LoomLarge#521.

This mirrors Latticework issue: meekmachine/Latticework#35.

Azure and audio transport findings

Buffered Azure drawer path:

• Frontend TTS calls LoomLarge backend POST /api/azure-tts/synthesize from the TTS service.
• Backend backend/src/api.py creates Azure Speech SDK synthesis, captures viseme_received and synthesis_word_boundary, and returns JSON: audio_base64, visemes[{viseme_id,audio_offset}], word_boundaries[{word,start_time,end_time}], and duration.
• Frontend decodes audio_base64 into an AudioBuffer, starts Web Audio playback locally, converts Azure visemes through azureVisemesToTimeline, and schedules a Vocal timeline. This is backend-generated buffered audio, not a streamed audio response.
• Azure credentials are normally backend env vars. The current service still supports optional X-Azure-Speech-Key / X-Azure-Speech-Region headers for override, but Polyester should not make worker agencies own secrets.

LiveKit conversation path:

• Backend backend/src/agent_with_visemes.py uses AzureTTSLiveKit when Azure env vars exist.
• backend/src/azure_tts_livekit.py captures Azure visemes/words during synthesis, sends text streams on lk.visemes and lk.words, and emits PCM audio frames to the LiveKit audio track.
• Frontend frontend/src/services/livekitConversationService.ts receives lk.visemes and lk.words, anchors word callbacks to the actual attached audio element play time, and the LiveKit module starts a Vocal timeline at audio start.
• This path streams audio through LiveKit; viseme/word timing arrives as LiveKit text streams.

LiveKit drawer TTS path:

• POST /api/tts/synthesize returns audio_base64 WAV and estimated word_boundaries, but no provider visemes.
• The frontend plays an HTMLAudioElement, calls startExternalSentence(text), and ticks word boundaries against audio.currentTime; this is text-predicted lip sync, not Azure provider-viseme lip sync.

Clojure-way direction

• Define one transport-neutral speech timing contract: utterance id, source, audio clock anchor, duration, provider visemes, word timings, cancellation token, and optional visual lead.
• Normalize Azure schemas from both paths: {viseme_id,audio_offset} and {id,time}; {start_time,end_time} and {start,end}.
• Convert provider events to canonical 15-slot Loom3/CC4 visemes using word-context heuristics for lossy Azure groups.
• Keep visual lead separate from audio/word clocks so mouth anticipation does not move callbacks or word timings.
• Emit data commands to Vocal/Animation; do not call engine, Web Audio, LiveKit, DOM, or backend APIs from worker code.

Acceptance criteria

• Correct visemes activate at the correct audible timing for buffered Azure and LiveKit Azure paths.
• Interruption/cancellation stops pending viseme/word events and removes active snippets without leaving the mouth stuck.
• Azure lossy groups have tests and documented behavior, especially EY/EH/UH, Y/IY/IH/IX, W/UW, D/T/N/TH, and diphthongs.
• Add a small phrase corpus covering B/M/P, F/V, TH, S/Z, CH/J, rounded vowels, long-E, short-I, R/ER, and diphthongs.
• Add parity tests comparing TypeScript and CLJS provider timelines for representative Azure event streams.
• Expose debug data for provider id, canonical viseme, word, timestamp, duration, active morph target key, jaw value, and total lip activation.
• Document that legacy per-word LipSync is compatibility/research; production mouth playback should route through Vocal sentence timelines.