Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions skills-manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
"files": 7
},
"hyperframes-core": {
"hash": "690ebb3ba5420b90",
"hash": "c3fdc0fb238d5a56",
"files": 14
},
"hyperframes-creative": {
Expand All @@ -46,7 +46,7 @@
"files": 10
},
"media-use": {
"hash": "49aa5adae0800403",
"hash": "0a0665d20e7ca91d",
"files": 122
},
"motion-graphics": {
Expand Down
2 changes: 1 addition & 1 deletion skills/hyperframes-core/references/script-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,4 @@ A header block, then one section per spoken line.

## To TTS

Feed each line's spoken text to `npx hyperframes tts` (pin `--voice` / `--provider` from the header; capture word timestamps for captions). Real per-word timing replaces the `**Time:**` guides. CLI contract → `media-use/audio/references/tts.md`.
Feed each line's spoken text to the TTS path selected in the header. For local Kokoro, run `npx hyperframes tts` and pin `--voice`; for cloud providers and word timestamps, use the provider-specific scripts. Real per-word timing replaces the `**Time:**` guides. Contract → `media-use/audio/references/tts.md`.
24 changes: 12 additions & 12 deletions skills/media-use/audio/references/transcribe.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ Create normalized word-level timestamps. **Always specify `--model` explicitly**

```bash
npx hyperframes transcribe audio.mp3 --model small.en # known English
npx hyperframes transcribe video.mp4 --model small --language es # known Spanish
npx hyperframes transcribe audio.mp3 --model small # unknown language (auto-detect)
npx hyperframes transcribe video.mp4 --model large-v3 --language es # known Spanish
npx hyperframes transcribe audio.mp3 --model large-v3 # unknown language (auto-detect)
npx hyperframes transcribe subtitles.srt # import existing
npx hyperframes transcribe subtitles.vtt
npx hyperframes transcribe openai-response.json
Expand All @@ -16,27 +16,27 @@ npx hyperframes transcribe openai-response.json
`.en` models (`tiny.en` / `base.en` / `small.en` / `medium.en`) **translate** non-English audio into English. This silently destroys the original language.

1. **Known English** → `--model small.en` (or `medium.en` for music / noisy audio)
2. **Known non-English** → `--model small --language <iso-code>` (no `.en` suffix)
3. **Unknown language** → `--model small` (whisper auto-detects)
2. **Known non-English** → `--model large-v3 --language <iso-code>`
3. **Unknown language** → `--model large-v3` (whisper auto-detects)

**CLI default is `small.en`** — do not rely on it; always pass `--model` to make the choice explicit. `--language` also filters out non-target-language segments from mixed-language audio.

## Model Sizes

| Model | Size | Speed | When |
| ---------- | ------ | -------- | ------------------------------------- |
| `tiny` | 75 MB | Fastest | Quick previews, smoke tests |
| `base` | 142 MB | Fast | Short clips, clear audio |
| `small` | 466 MB | Moderate | Default for most multilingual content |
| `medium` | 1.5 GB | Slow | Music with vocals, noisy audio |
| `large-v3` | 3.1 GB | Slowest | Production quality |
| Model | Size | Speed | When |
| ----------- | ------ | -------- | -------------------------------- |
| `tiny.en` | 75 MB | Fastest | English previews and smoke tests |
| `base.en` | 142 MB | Fast | Short, clear English speech |
| `small.en` | 466 MB | Moderate | Default for English speech |
| `medium.en` | 1.5 GB | Slow | Noisy English audio and vocals |
| `large-v3` | 3.1 GB | Slowest | Production quality |

### Picking a model by content type

1. Speech over silence / light background → `small.en`
2. Speech over music, or music with vocals → start with `medium.en`
3. Produced music track (vocals + full instrumentation) → start with `medium.en`; expect to need manual lyrics or an external API ([`captions/transcript-handling.md`](captions/transcript-handling.md) → "Using External Transcription APIs")
4. Multilingual → `medium` or `large-v3` (no `.en` suffix), pair with `--language`
4. Multilingual → `large-v3`, pair with `--language` when known

## Output Shape

Expand Down
5 changes: 3 additions & 2 deletions skills/media-use/audio/references/tts-to-captions.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ When no recorded voiceover exists, generate one and obtain word-level caption ti
HeyGen returns word timestamps in the same response as the audio. Pass `--words` and you're done:

```bash
npx hyperframes tts script.txt --provider heygen --output narration.wav --words narration.words.json
node <SKILL_DIR>/audio/scripts/heygen-tts.mjs \
script.txt --output narration.wav --words narration.words.json
```

`narration.words.json` is already in the `[{ id, text, start, end }]` shape the captions pipeline consumes — no separate transcribe pass.
Expand All @@ -21,4 +22,4 @@ npx hyperframes tts script.txt --voice af_heart --output narration.wav
npx hyperframes transcribe narration.wav --model small.en # voice af_heart is American English
```

Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match `--model` to the voice's language (use `small.en` for `a`/`b` prefixes, `small --language <code>` otherwise). Then consume `transcript.json` via the caption references in `captions/`.
Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match `--model` to the voice's language (use `small.en` for `a`/`b` prefixes, `large-v3 --language <code>` otherwise). Then consume `transcript.json` via the caption references in `captions/`.
30 changes: 15 additions & 15 deletions skills/media-use/audio/references/tts.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
# Text To Speech

`npx hyperframes tts` auto-detects a provider from env vars; explicit override via `--provider`.
`npx hyperframes tts` is the local Kokoro-82M command. It accepts `--voice`,
`--lang`, `--speed`, and `--output`; it does **not** accept `--provider` or
`--words`. Use the provider-specific scripts below for cloud voices and word
timestamps.

> **Run the Preflight first — no credential is not a green light to silently use the local voice.** Before generating a voiceover, complete the sign-in **Preflight** (see `../SKILL.md` → Preflight): run `npx hyperframes auth status`, recommend signing in, and **STOP for the user's choice** (sign in for HeyGen voices, or continue offline with local Kokoro). This applies to a one-off "generate a voiceover" request just as much as inside a full workflow.

## Provider chain
## Media workflow provider chain

| Order | Provider | Env trigger | Voice IDs | Word timestamps | Audio format |
| ----- | ----------------- | ------------------------------------------- | ------------------------------------------- | ----------------------------------------- | -------------------- |
Expand All @@ -13,16 +16,8 @@
| 3 | Kokoro-82M | always (local fallback) | `am_michael`, `af_heart`, … (54 voices) | No | wav direct |

```bash
# Auto-detect (HeyGen if key set, else ElevenLabs, else Kokoro)
# Local Kokoro
npx hyperframes tts "Welcome to HyperFrames" -o narration.wav

# Pin the provider explicitly
npx hyperframes tts "Hello" --provider kokoro
npx hyperframes tts "Hello" --provider heygen --voice <heygen-uuid>
npx hyperframes tts "Hello" --provider elevenlabs --voice 21m00Tcm4TlvDq8ikWAM

# HeyGen path: capture word timestamps in one call (skips a Whisper pass)
npx hyperframes tts "Hi there" --words narration.words.json
```

## Self-contained HeyGen (no CLI) — `scripts/heygen-tts.mjs`
Expand Down Expand Up @@ -70,7 +65,11 @@ node skills/media-use/audio/scripts/heygen-tts.mjs --list # public starfish vo

## ffmpeg requirement

HeyGen + ElevenLabs return mp3. The CLI transcodes to wav when `--output` ends in `.wav` (the default and what downstream `ffprobe` + Whisper expect). If you'd rather skip the transcode, pass `-o file.mp3`. Without `ffmpeg` on PATH, `.wav` output from the cloud providers fails — install ffmpeg or use `.mp3`.
HeyGen + ElevenLabs return mp3. Their provider-specific scripts transcode to wav
when `--output` ends in `.wav` (the default and what downstream `ffprobe` +
Whisper expect). If you'd rather skip the transcode, pass `-o file.mp3`.
Without `ffmpeg` on PATH, `.wav` output from cloud providers fails — install
ffmpeg or use `.mp3`. Local `hyperframes tts` writes Kokoro wav directly.

## Voice selection (Kokoro)

Expand Down Expand Up @@ -103,8 +102,8 @@ The first letter of a Kokoro voice ID picks the phonemizer language; `--lang` ov
| `z` | Mandarin |

```bash
npx hyperframes tts "La reunión empieza a las nueve" --voice ef_dora --provider kokoro
npx hyperframes tts "Today is a nice day" --voice af_heart --provider kokoro
npx hyperframes tts "La reunión empieza a las nueve" --voice ef_dora
npx hyperframes tts "Today is a nice day" --voice af_heart
```

Valid `--lang` codes (only needed to override the voice's auto-detected language): `en-us`, `en-gb`, `es`, `fr-fr`, `hi`, `it`, `pt-br`, `ja`, `zh`.
Expand All @@ -118,7 +117,8 @@ Non-English phonemization requires `espeak-ng` system-wide (`brew install espeak
- `1.1-1.2` — intros, transitions, upbeat content
- `1.5+` — rarely appropriate, test carefully

Honored by Kokoro + HeyGen; ElevenLabs ignores `--speed` (use voice settings on their dashboard).
`hyperframes tts --speed` controls local Kokoro. Cloud-provider speed support is
documented by each provider-specific script; do not pass CLI-only flags to them.

## Long scripts

Expand Down
3 changes: 3 additions & 0 deletions skills/media-use/scripts/lib/registry.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,9 @@ const REGISTRY = {
// Local design spec, not heygen — reads frame.md / design.md tokens.
A("design_spec", { search: brandProvider.search }),
],
// Local/direct-URL ingest only. Declaring video here lets `resolve --from`
// freeze and ledger prepared clips without implying a generative provider.
video: [A("ingest.only", {})],
grade: [
// Local deterministic cascade handled by resolve.mjs so grade records can
// carry an inline block as well as an optional frozen .cube file.
Expand Down
15 changes: 13 additions & 2 deletions skills/media-use/scripts/lib/registry.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,18 @@ import { getProviders, getProvider, listTypes, runProviders, runCapability } fro

test("listTypes exposes the v2 media types", () => {
const types = listTypes();
for (const t of ["bgm", "sfx", "image", "icon", "logo", "voice", "brand", "grade", "lut"]) {
for (const t of [
"bgm",
"sfx",
"image",
"icon",
"logo",
"voice",
"brand",
"video",
"grade",
"lut",
]) {
assert.ok(types.includes(t), `missing type: ${t}`);
}
});
Expand All @@ -21,7 +32,7 @@ test("heygen provider is first for every type it serves", () => {

test("sanctioned providers only: heygen, local mflux/kokoro, codex, design spec, logo tiers", () => {
const allowed =
/^heygen|^mflux\.local$|^kokoro\.local$|^codex\.image_gen$|^design_spec$|^svgl$|^simple-icons$|^github\.avatar$|^favicon\.ddg$|^color_grade\.local$|^cube_lut\.local$/;
/^heygen|^mflux\.local$|^kokoro\.local$|^codex\.image_gen$|^design_spec$|^svgl$|^simple-icons$|^github\.avatar$|^favicon\.ddg$|^ingest\.only$|^color_grade\.local$|^cube_lut\.local$/;
for (const t of listTypes()) {
for (const p of getProviders(t)) {
assert.ok(allowed.test(p.name), `${t} lists unsanctioned provider: ${p.name}`);
Expand Down
14 changes: 14 additions & 0 deletions skills/media-use/scripts/resolve.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -640,6 +640,20 @@ test("--from rejects invalid lut cube without registering or leaving a frozen fi
cleanup();
});

test("--from ingests a local video when type is video", () => {
setup();
const video = join(tmp, "demo.mp4");
writeFileSync(video, "video fixture");

const out = runResolve(["--from", video, "--type", "video", "--project", tmp, "--json"]);
const parsed = JSON.parse(out);

assert.equal(parsed.type, "video");
assert.ok(parsed.path.endsWith(".mp4"));
assert.ok(existsSync(join(tmp, parsed.path)));
cleanup();
});

test("grade miss exits explicitly with no partial file", () => {
setup();
const missIntent = `zqxv imaginary neutron ${process.pid}`;
Expand Down
Loading