Skip to content

[agency] Harden CLJS vocal agency as the active lip-sync timeline runtime #10

Description

@meekmachine

Goal

Finish Vocal as the active ClojureScript sentence-level lip-sync runtime for text and provider-timed speech timelines.

Current state

Polyester already has src-cljs/latticework/vocal.cljs, in-process and worker client APIs, text-to-viseme planning, sentence timelines, word-boundary drift detection, snippet building, jaw curves, coarticulation constraints, and cleanup planning. The TypeScript VocalService is currently the active lip-sync runtime used by TTS and LiveKit paths.

This mirrors the Latticework research issue: meekmachine/Latticework#34 and the stabilization discussion: meekmachine/LoomLarge#521.

Clojure-way direction

  • Keep sentence-level timeline planning in pure CLJS data: viseme events, word timings, snippet curves, active sentence state, drift decisions.
  • Keep all animation side effects at the host edge: schedule/remove/seek/pause/resume.
  • Provider timelines should be pre-timed; snippet playback rate should remain neutral to avoid double-applying speech rate.
  • Word boundaries should verify and correct drift by emitting seek requests instead of rebuilding clips on every word.

Acceptance criteria

  • createVocalAgency and worker client can replace the TypeScript VocalService for text and provider timelines behind the existing JS API.
  • Worker returns deterministic serializable snippets for text and provider viseme timelines.
  • Word-boundary drift still emits seek requests when threshold is exceeded.
  • Active sentence cleanup cannot remove unrelated snippets and does not leak timers.
  • Add TS-vs-CLJS parity tests for representative phrases, jaw curves, coarticulation caps, sentence cleanup, pause/resume, and drift correction.
  • Document that Vocal is the visible mouth runtime; legacy LipSync remains compatibility/research until folded into this path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions