diff --git a/README.md b/README.md index 6d098d97c..433ac1055 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,19 @@ The goal of this continuation is to keep OpenScreen alive as a fully open-source - Export to MP4 or GIF in multiple aspect ratios and resolutions. - Languages supported: Arabic, English, Spanish, French, Italian, Japanese, Korean, Portuguese (Brazil), Russian, Turkish, Vietnamese, Simplified Chinese, and Traditional Chinese. +## Command-line interface (headless) + +OpenScreen ships a CLI for scripts, CI, and AI coding agents: record the screen +headlessly, edit the `.openscreen` project JSON programmatically (zooms, +annotations, trims), and render MP4/GIF with the full export pipeline — no +visible windows, NDJSON output with `--json`. + +```bash +openscreen record --duration 20 --project demo.openscreen --json +openscreen export demo.openscreen -o demo.mp4 --json +``` + +See [docs/cli.md](./docs/cli.md). ## Installation diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 000000000..f546442d4 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,179 @@ +# OpenScreen CLI + +Headless command-line interface for recording the screen and exporting `.openscreen` +projects — no visible windows, machine-readable output. Designed so scripts, CI +pipelines, and AI coding agents can produce polished product demos automatically: + +```text +record → edit the project JSON programmatically → export → MP4/GIF +``` + +## Running + +Development (after `npm run build-vite` and, for recording, the native helper build): + +```bash +npm run cli -- [options] +# or directly: +./node_modules/.bin/electron . [options] +``` + +Packaged app (the CLI ships inside the normal binary): + +```bash +# macOS +/Applications/Openscreen.app/Contents/MacOS/Openscreen export demo.openscreen -o demo.mp4 +# Windows +"C:\Program Files\Openscreen\Openscreen.exe" export demo.openscreen -o demo.mp4 +``` + +CLI runs skip the single-instance lock, so they work while the GUI app is open. + +## Commands + +### `openscreen record` + +Records headlessly through the same pipeline as the GUI: the native +ScreenCaptureKit helper on macOS, the WGC helper on Windows (browser capture as +fallback). Recordings land in the app's recordings directory +(`/recordings/`), exactly like GUI recordings — including the +`.cursor.json` cursor-telemetry sidecar used for editable cursors and auto-zoom. + +```bash +openscreen record --duration 30 --project demo.openscreen --json +openscreen record --window "My App" --mic --system-audio +openscreen record --display 1 --cursor system +``` + +| Option | Meaning | +|---|---| +| `--display ` | Screen index to record (default 0) | +| `--window ` | Record the first window whose title contains `<title>` | +| `--mic` / `--mic-device <name>` | Capture microphone (optionally by device-label substring) | +| `--system-audio` | Capture system audio | +| `--cursor <editable-overlay\|system>` | Hide the system cursor and record telemetry (default), or bake it into the video | +| `--duration <seconds>` | Stop automatically | +| `--project <out.openscreen>` | Write a ready-to-export project file when done | +| `--json` | NDJSON events on stdout | + +Stopping without `--duration`: send SIGINT/SIGTERM to the process, or type +`stop` + Enter on its stdin. + +Platform notes: + +- **macOS**: requires the Swift helper (`npm run build:native:mac`, needs Xcode) + and the Screen Recording permission for whatever binary hosts Electron + (your terminal during development). Webcam capture is not available in CLI + recording on macOS (same limitation as the native helper). +- **Windows**: requires the WGC helper (`npm run build:native:win`). SIGTERM + does not exist on Windows — stop recordings with Ctrl+C, stdin `stop`, or + `--duration` (a hard `taskkill` loses the recording). Microphone access is + gated by Settings → Privacy → Microphone; there is no programmatic prompt. +- **Linux**: uses the browser capture pipeline; cursor options are limited, + matching the GUI. On Wayland, capture goes through the PipeWire portal, + which may show a system picker dialog and requires a desktop session + (headless/SSH sessions without a portal cannot record). + +### `openscreen export` + +Renders a project to MP4 or GIF using the app's real export pipeline (WebCodecs + +PixiJS, faster than realtime) in a hidden window. Falls back to SwiftShader when +no GPU is available (CI), and applies everything the editor would: zooms, trims, +speed regions, wallpaper/padding, annotations, cursor rendering, webcam layouts. + +```bash +openscreen export demo.openscreen # format/quality from the project +openscreen export demo.openscreen -o out.mp4 --quality source +openscreen export demo.openscreen -o out.gif --gif-fps 20 --gif-size large +openscreen export demo.openscreen --json | while read line; do ...; done +``` + +| Option | Meaning | +|---|---| +| `-o, --out <path>` | Output file; extension picks the format. Default: next to the project | +| `--format <mp4\|gif>` | Override the project's stored format | +| `--quality <medium\|good\|source>` | MP4 quality | +| `--gif-fps <15\|20\|25\|30>`, `--gif-size <medium\|large\|original>` | GIF settings | +| `--preview-size <WxH>` | Reference preview box (default `1280x720`), see below | +| `--audio <file>` | Mix a voiceover file into the MP4 (mp3/wav/m4a — anything Chromium can decode; AIFF is not supported) | +| `--audio-mode <mix\|replace>` | Layer the voiceover over the recording's audio (default `mix`) or replace it | +| `--audio-offset <seconds>` | Delay before the voiceover starts (default 0) | +| `--json` | NDJSON progress + result on stdout | + +`--audio` re-muxes after the render: video packets are copied untouched, and the +audio is mixed offline (OfflineAudioContext) and re-encoded to AAC. MP4 only. +In `mix` mode the original audio is ducked to 40% under the voiceover so the +sum cannot clip; use `replace` to drop the original entirely. + +**Media path rule**: for safety, a project's referenced media is only auto-approved +when it lives in the app's recordings directory or **next to the project file**. +Keep `.openscreen` files beside their media (or record via the CLI, which uses +the recordings directory). + +**`--preview-size`**: annotation font sizes and border radii are stored in +preview-pixel space and scaled by `export size / preview size` — in the GUI the +"preview" is the editor window, so results depend on window size. The CLI uses a +deterministic reference box instead (the composition fitted into 1280×720). +Authoring tip for scripts: treat annotation `fontSize` as "pixels in a +1280-wide preview". + +### `openscreen info` + +Prints a project summary (referenced media and whether it exists, format, +region counts). Exits non-zero if the referenced video is missing. + +```bash +openscreen info demo.openscreen --json +``` + +## Machine-readable output (`--json`) + +One JSON object per line on stdout (NDJSON). stderr carries diagnostics only. + +```jsonl +{"event":"started","command":"export"} +{"event":"progress","percentage":42,"currentFrame":50,"totalFrames":120,"estimatedTimeRemaining":3} +{"event":"done","success":true,"outputPath":"/path/out.mp4","format":"mp4","width":1920,"height":1080} +``` + +Record emits `log` events (`Recording started`, …), `stopping`, and a final +`done` carrying `screenVideoPath`, `cursorDataPath`, `durationMs`, and +`projectPath` when `--project` was used. Exit code is 0 on success, 1 on +failure, 2 on bad arguments. + +## Example: automated product demo (for scripts/agents) + +```bash +# 1. Record 20 seconds of the running app +openscreen record --window "MyProduct" --duration 20 --project demo.openscreen --json + +# 2. Edit the project: add a zoom and a caption (plain JSON) +node -e ' + const fs = require("fs"); + const p = JSON.parse(fs.readFileSync("demo.openscreen", "utf8")); + p.editor.zoomRegions.push({ id: "z1", startMs: 2000, endMs: 6000, depth: 3, + focus: { cx: 0.5, cy: 0.4 }, focusMode: "manual", source: "manual" }); + p.editor.annotationRegions.push({ id: "a1", startMs: 500, endMs: 4000, + type: "text", content: "One-click setup", textContent: "One-click setup", + position: { x: 8, y: 6 }, size: { width: 40, height: 12 }, + style: { fontSize: 24, color: "#fff" }, zIndex: 1 }); + fs.writeFileSync("demo.openscreen", JSON.stringify(p, null, 2)); +' + +# 3. Narrate with any TTS (macOS `say` shown; any engine producing mp3/wav/m4a works) +say -o voice.m4a --file-format=m4af "Welcome to my product. Here's a quick tour." + +# 4. Render with the voiceover mixed in +openscreen export demo.openscreen -o demo.mp4 --audio voice.m4a --audio-mode replace --json +``` + +## Architecture + +- `electron/cli/args.ts` — pure argv parser (unit-tested in `args.test.ts`). +- `electron/cli/cliMain.ts` — headless boot: no HUD/tray/menu/dock, stdio + protocol, signal handling, exit codes. Registers the same IPC surface as the + GUI (`registerIpcHandlers`) with inert window callbacks. +- `src/cli/CliExportRunner.tsx` / `src/cli/CliRecordRunner.tsx` — hidden-window + runners (`?windowType=cli-export|cli-record`) that drive the existing + exporter classes and `useScreenRecorder` hook. +- Contracts shared between main and renderer: `src/lib/cliContracts.ts`. diff --git a/electron/cli/args.test.ts b/electron/cli/args.test.ts new file mode 100644 index 000000000..f8886bf7e --- /dev/null +++ b/electron/cli/args.test.ts @@ -0,0 +1,155 @@ +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { parseCliArgs } from "./args"; + +const CWD = path.resolve("/work"); +const inCwd = (name: string) => path.resolve(CWD, name); + +function parse(args: string[]) { + return parseCliArgs(["electron", "app-path", ...args], 2, CWD); +} + +describe("parseCliArgs", () => { + it("returns null when no subcommand is present (GUI launch)", () => { + expect(parse([])).toBeNull(); + expect(parse(["--some-chromium-flag"])).toBeNull(); + }); + + it("skips leading Chromium switches before the subcommand (AppImage --no-sandbox)", () => { + expect(parse(["--no-sandbox", "export", "demo.openscreen"])).toMatchObject({ + kind: "export", + projectPath: inCwd("demo.openscreen"), + }); + expect(parse(["--no-sandbox", "--enable-unsafe-swiftshader", "record"])).toMatchObject({ + kind: "record", + }); + expect(parse(["--no-sandbox", "--help"])).toMatchObject({ kind: "help" }); + expect(parse(["--no-sandbox"])).toBeNull(); + }); + + it("parses a minimal export command and resolves relative paths", () => { + const cmd = parse(["export", "demo.openscreen"]); + expect(cmd).toMatchObject({ + kind: "export", + projectPath: inCwd("demo.openscreen"), + outPath: null, + format: null, + }); + }); + + it("parses export options and infers format from --out extension", () => { + const cmd = parse([ + "export", + "/p/demo.openscreen", + "-o", + "out.gif", + "--gif-fps", + "20", + "--preview-size", + "1600x900", + "--json", + ]); + expect(cmd).toMatchObject({ + kind: "export", + outPath: inCwd("out.gif"), + format: "gif", + gifFrameRate: 20, + previewWidth: 1600, + previewHeight: 900, + json: true, + }); + }); + + it("rejects a --format that conflicts with the --out extension", () => { + const cmd = parse(["export", "a.openscreen", "-o", "x.mp4", "--format", "gif"]); + expect(cmd).toMatchObject({ kind: "error" }); + }); + + it("rejects export without a project path", () => { + expect(parse(["export"])).toMatchObject({ kind: "error" }); + }); + + it("parses voiceover audio options", () => { + const cmd = parse([ + "export", + "a.openscreen", + "--audio", + "voice.mp3", + "--audio-mode", + "replace", + "--audio-offset", + "1.5", + ]); + expect(cmd).toMatchObject({ + kind: "export", + audioPath: inCwd("voice.mp3"), + audioMode: "replace", + audioOffsetSec: 1.5, + }); + }); + + it("defaults audio mode to mix and rejects --audio with gif", () => { + expect(parse(["export", "a.openscreen", "--audio", "v.mp3"])).toMatchObject({ + audioMode: "mix", + audioOffsetSec: 0, + }); + expect(parse(["export", "a.openscreen", "--audio", "v.mp3", "--format", "gif"])).toMatchObject({ + kind: "error", + }); + expect(parse(["export", "a.openscreen", "--audio-offset", "-1"])).toMatchObject({ + kind: "error", + }); + }); + + it("parses record defaults", () => { + expect(parse(["record"])).toMatchObject({ + kind: "record", + displayIndex: 0, + windowTitle: null, + mic: false, + systemAudio: false, + cursorMode: "editable-overlay", + durationMs: null, + }); + }); + + it("parses record options", () => { + const cmd = parse([ + "record", + "--display", + "1", + "--mic-device", + "MacBook", + "--system-audio", + "--duration", + "12.5", + "--project", + "demo.openscreen", + ]); + expect(cmd).toMatchObject({ + kind: "record", + displayIndex: 1, + mic: true, + micDevice: "MacBook", + systemAudio: true, + durationMs: 12500, + projectOut: inCwd("demo.openscreen"), + }); + }); + + it("rejects invalid record values", () => { + expect(parse(["record", "--duration", "0"])).toMatchObject({ kind: "error" }); + expect(parse(["record", "--cursor", "off"])).toMatchObject({ kind: "error" }); + expect(parse(["record", "--project", "demo.json"])).toMatchObject({ kind: "error" }); + }); + + it("parses info and help", () => { + expect(parse(["info", "demo.openscreen", "--json"])).toMatchObject({ + kind: "info", + projectPath: inCwd("demo.openscreen"), + json: true, + }); + expect(parse(["help"])).toMatchObject({ kind: "help" }); + expect(parse(["--help"])).toMatchObject({ kind: "help" }); + }); +}); diff --git a/electron/cli/args.ts b/electron/cli/args.ts new file mode 100644 index 000000000..69578198b --- /dev/null +++ b/electron/cli/args.ts @@ -0,0 +1,351 @@ +// Pure argv parser for the OpenScreen CLI. No Electron imports so it can be +// unit-tested under plain vitest. + +import path from "node:path"; +import type { CliExportRequest, CliRecordRequest, CliRequest } from "../../src/lib/cliContracts"; + +export interface CliInfoCommand { + kind: "info"; + projectPath: string; +} + +export interface CliHelpCommand { + kind: "help"; +} + +export interface CliErrorCommand { + kind: "error"; + message: string; +} + +export type CliCommand = (CliRequest | CliInfoCommand | CliHelpCommand | CliErrorCommand) & { + /** Machine-readable NDJSON output on stdout instead of human progress. */ + json?: boolean; +}; + +const SUBCOMMANDS = new Set(["export", "record", "info", "help", "--help", "-h"]); + +export const CLI_USAGE = `OpenScreen CLI + +Usage: + openscreen export <project.openscreen> [options] Render a project to MP4/GIF + openscreen record [options] Record the screen headlessly + openscreen info <project.openscreen> [--json] Inspect a project file + openscreen help Show this help + +Export options: + -o, --out <path> Output file (.mp4 or .gif). Default: next to the project file + --format <mp4|gif> Override the format stored in the project + --quality <medium|good|source> + MP4 quality (default: from project) + --gif-fps <15|20|25|30> GIF frame rate (default: from project) + --gif-size <medium|large|original> + GIF size preset (default: from project) + --preview-size <WxH> Reference preview box for annotation scaling (default 1280x720) + --audio <file> Mix a voiceover audio file into the MP4 (mp3/wav/m4a) + --audio-mode <mix|replace> + Layer over the recording's audio (default) or replace it + --audio-offset <seconds> Delay before the voiceover starts (default 0) + --json NDJSON progress/result on stdout + +Record options (recording is saved into the app's recordings directory): + --display <n> Screen index to record (default 0) + --window <title> Record the first window whose title contains <title> + --mic Capture the default microphone + --mic-device <name> Microphone device name (implies --mic) + --system-audio Capture system audio + --cursor <editable-overlay|system> + Cursor capture mode (default editable-overlay) + --duration <seconds> Stop automatically after this long + --project <out.openscreen> + Write a ready-to-export project file when done + --json NDJSON events on stdout + +Stopping a recording: send SIGINT/SIGTERM, type "stop" + Enter on stdin, +or pass --duration. +`; + +function takeValue(argv: string[], i: number, flag: string): [string, number] { + const next = argv[i + 1]; + if (next === undefined || next.startsWith("--")) { + throw new Error(`${flag} requires a value`); + } + return [next, i + 1]; +} + +function resolvePath(p: string, cwd: string): string { + return path.isAbsolute(p) ? p : path.resolve(cwd, p); +} + +/** + * Extracts CLI arguments from process.argv. Returns null when no subcommand is + * present (normal GUI launch). `firstArgIndex` should be 1 for packaged builds + * and 2 when running via `electron <app-path> ...`. + */ +export function parseCliArgs( + argv: string[], + firstArgIndex: number, + cwd: string = process.cwd(), +): CliCommand | null { + const rawArgs = argv.slice(firstArgIndex).filter((a) => !a.startsWith("--inspect")); + + // Skip *leading* Chromium/Electron switches (e.g. the AppImage's required + // `--no-sandbox`) so `Openscreen --no-sandbox export demo.openscreen` still + // enters CLI mode. Only leading dash-tokens are skipped — everything after + // the subcommand belongs to the subcommand parser. `--help`/`-h` are ours. + let subIndex = 0; + while ( + subIndex < rawArgs.length && + rawArgs[subIndex].startsWith("-") && + rawArgs[subIndex] !== "--help" && + rawArgs[subIndex] !== "-h" + ) { + subIndex++; + } + + const args = rawArgs.slice(subIndex); + const sub = args[0]; + if (!sub || !SUBCOMMANDS.has(sub)) return null; + if (sub === "help" || sub === "--help" || sub === "-h") return { kind: "help" }; + + try { + if (sub === "export") return parseExport(args.slice(1), cwd); + if (sub === "record") return parseRecord(args.slice(1), cwd); + return parseInfo(args.slice(1), cwd); + } catch (error) { + return { kind: "error", message: error instanceof Error ? error.message : String(error) }; + } +} + +function parseExport(args: string[], cwd: string): CliCommand { + const request: CliExportRequest & { json?: boolean } = { + kind: "export", + projectPath: "", + outPath: null, + format: null, + quality: null, + gifFrameRate: null, + gifSizePreset: null, + previewWidth: null, + previewHeight: null, + audioPath: null, + audioMode: "mix", + audioOffsetSec: 0, + }; + + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + switch (arg) { + case "-o": + case "--out": { + const [value, next] = takeValue(args, i, arg); + request.outPath = resolvePath(value, cwd); + i = next; + break; + } + case "--format": { + const [value, next] = takeValue(args, i, arg); + if (value !== "mp4" && value !== "gif") { + throw new Error(`--format must be mp4 or gif, got "${value}"`); + } + request.format = value; + i = next; + break; + } + case "--quality": { + const [value, next] = takeValue(args, i, arg); + if (value !== "medium" && value !== "good" && value !== "source") { + throw new Error(`--quality must be medium, good or source, got "${value}"`); + } + request.quality = value; + i = next; + break; + } + case "--gif-fps": { + const [value, next] = takeValue(args, i, arg); + const fps = Number(value); + if (fps !== 15 && fps !== 20 && fps !== 25 && fps !== 30) { + throw new Error(`--gif-fps must be 15, 20, 25 or 30, got "${value}"`); + } + request.gifFrameRate = fps; + i = next; + break; + } + case "--gif-size": { + const [value, next] = takeValue(args, i, arg); + if (value !== "medium" && value !== "large" && value !== "original") { + throw new Error(`--gif-size must be medium, large or original, got "${value}"`); + } + request.gifSizePreset = value; + i = next; + break; + } + case "--preview-size": { + const [value, next] = takeValue(args, i, arg); + const match = /^(\d+)x(\d+)$/.exec(value); + if (!match) throw new Error(`--preview-size must look like 1280x720, got "${value}"`); + const previewWidth = Number(match[1]); + const previewHeight = Number(match[2]); + if (previewWidth <= 0 || previewHeight <= 0) { + throw new Error(`--preview-size dimensions must be positive, got "${value}"`); + } + request.previewWidth = previewWidth; + request.previewHeight = previewHeight; + i = next; + break; + } + case "--audio": { + const [value, next] = takeValue(args, i, arg); + request.audioPath = resolvePath(value, cwd); + i = next; + break; + } + case "--audio-mode": { + const [value, next] = takeValue(args, i, arg); + if (value !== "mix" && value !== "replace") { + throw new Error(`--audio-mode must be mix or replace, got "${value}"`); + } + request.audioMode = value; + i = next; + break; + } + case "--audio-offset": { + const [value, next] = takeValue(args, i, arg); + const seconds = Number(value); + if (!Number.isFinite(seconds) || seconds < 0) { + throw new Error( + `--audio-offset must be a non-negative number of seconds, got "${value}"`, + ); + } + request.audioOffsetSec = seconds; + i = next; + break; + } + case "--json": + request.json = true; + break; + default: + if (arg.startsWith("-")) throw new Error(`Unknown export option: ${arg}`); + if (request.projectPath) throw new Error(`Unexpected extra argument: ${arg}`); + request.projectPath = resolvePath(arg, cwd); + } + } + + if (!request.projectPath) throw new Error("export requires a <project.openscreen> path"); + if (request.audioPath && request.format === "gif") { + throw new Error("--audio is only supported for MP4 exports"); + } + if (request.outPath) { + const ext = path.extname(request.outPath).toLowerCase(); + if (ext !== ".mp4" && ext !== ".gif") { + throw new Error(`--out must end in .mp4 or .gif, got "${request.outPath}"`); + } + const extFormat = ext === ".gif" ? "gif" : "mp4"; + if (request.format && request.format !== extFormat) { + throw new Error(`--format ${request.format} conflicts with --out extension ${ext}`); + } + request.format = extFormat; + } + return request; +} + +function parseRecord(args: string[], cwd: string): CliCommand { + const request: CliRecordRequest & { json?: boolean } = { + kind: "record", + displayIndex: 0, + windowTitle: null, + mic: false, + micDevice: null, + systemAudio: false, + cursorMode: "editable-overlay", + durationMs: null, + projectOut: null, + }; + + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + switch (arg) { + case "--display": { + const [value, next] = takeValue(args, i, arg); + const index = Number(value); + if (!Number.isInteger(index) || index < 0) { + throw new Error(`--display must be a non-negative integer, got "${value}"`); + } + request.displayIndex = index; + i = next; + break; + } + case "--window": { + const [value, next] = takeValue(args, i, arg); + request.windowTitle = value; + i = next; + break; + } + case "--mic": + request.mic = true; + break; + case "--mic-device": { + const [value, next] = takeValue(args, i, arg); + request.mic = true; + request.micDevice = value; + i = next; + break; + } + case "--system-audio": + request.systemAudio = true; + break; + case "--cursor": { + const [value, next] = takeValue(args, i, arg); + if (value !== "editable-overlay" && value !== "system") { + throw new Error(`--cursor must be editable-overlay or system, got "${value}"`); + } + request.cursorMode = value; + i = next; + break; + } + case "--duration": { + const [value, next] = takeValue(args, i, arg); + const seconds = Number(value); + if (!Number.isFinite(seconds) || seconds <= 0) { + throw new Error(`--duration must be a positive number of seconds, got "${value}"`); + } + request.durationMs = Math.round(seconds * 1000); + i = next; + break; + } + case "--project": { + const [value, next] = takeValue(args, i, arg); + if (!value.endsWith(".openscreen")) { + throw new Error(`--project must end in .openscreen, got "${value}"`); + } + request.projectOut = resolvePath(value, cwd); + i = next; + break; + } + case "--json": + request.json = true; + break; + default: + throw new Error(`Unknown record option: ${arg}`); + } + } + return request; +} + +function parseInfo(args: string[], cwd: string): CliCommand { + let projectPath = ""; + let json = false; + for (const arg of args) { + if (arg === "--json") { + json = true; + } else if (arg.startsWith("-")) { + throw new Error(`Unknown info option: ${arg}`); + } else if (projectPath) { + throw new Error(`Unexpected extra argument: ${arg}`); + } else { + projectPath = resolvePath(arg, cwd); + } + } + if (!projectPath) throw new Error("info requires a <project.openscreen> path"); + return { kind: "info", projectPath, json }; +} diff --git a/electron/cli/cliMain.ts b/electron/cli/cliMain.ts new file mode 100644 index 000000000..02fe23585 --- /dev/null +++ b/electron/cli/cliMain.ts @@ -0,0 +1,453 @@ +// Headless CLI mode: boots Electron without HUD/tray/menu, drives a hidden +// renderer window (windowType=cli-export | cli-record) that reuses the app's +// existing export and recording pipelines, and reports progress on stdio. + +import fs from "node:fs/promises"; +import path from "node:path"; +import readline from "node:readline"; +import { fileURLToPath } from "node:url"; +import { app, BrowserWindow, ipcMain, session, systemPreferences } from "electron"; +import type { CliDoneResult, CliProgressEvent, CliRequest } from "../../src/lib/cliContracts"; +import { getSelectedDesktopSource, registerIpcHandlers } from "../ipc/handlers"; +import { ASSET_BASE_URL_ARG } from "../windows"; +import { CLI_USAGE, type CliCommand } from "./args"; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const VITE_DEV_SERVER_URL = process.env["VITE_DEV_SERVER_URL"]; +const RENDERER_DIST = path.join(__dirname, "..", "dist"); + +interface CliOutput { + json: boolean; + event(event: string, data?: Record<string, unknown>): void; + info(message: string): void; + error(message: string): void; + progress(p: CliProgressEvent): void; +} + +// stdout/stderr may be a closed pipe (`openscreen export | head`); writes must +// never take the process down — Electron would show a GUI error dialog. +function safeWrite(stream: NodeJS.WriteStream, text: string): void { + try { + stream.write(text); + } catch { + // EPIPE or closed stream; drop the output. + } +} + +function createOutput(json: boolean): CliOutput { + const isTty = process.stdout.isTTY === true; + let progressLineActive = false; + let lastProgressText = ""; + const clearProgressLine = () => { + if (progressLineActive) { + safeWrite(process.stdout, "\n"); + progressLineActive = false; + } + }; + return { + json, + event(event, data = {}) { + if (json) { + safeWrite(process.stdout, `${JSON.stringify({ event, ...data })}\n`); + } + }, + info(message) { + if (json) return; + clearProgressLine(); + safeWrite(process.stdout, `${message}\n`); + }, + error(message) { + clearProgressLine(); + if (json) { + safeWrite(process.stdout, `${JSON.stringify({ event: "error", message })}\n`); + } else { + safeWrite(process.stderr, `Error: ${message}\n`); + } + }, + progress(p) { + if (json) { + safeWrite(process.stdout, `${JSON.stringify({ event: "progress", ...p })}\n`); + return; + } + const frames = + p.currentFrame !== undefined && p.totalFrames + ? ` frame ${p.currentFrame}/${p.totalFrames}` + : ""; + const eta = + p.estimatedTimeRemaining !== undefined && Number.isFinite(p.estimatedTimeRemaining) + ? ` ETA ${Math.max(0, Math.round(p.estimatedTimeRemaining))}s` + : ""; + const phase = p.phase ? ` [${p.phase}]` : ""; + const text = `Exporting ${Math.round(p.percentage)}%${frames}${eta}${phase}`; + if (isTty) { + if (text === lastProgressText) return; + lastProgressText = text; + safeWrite(process.stdout, `\r\x1b[2K${text}`); + progressLineActive = true; + } else { + // Piped/non-TTY consumers get one line per whole-percent (or phase) change. + const coarse = `${Math.round(p.percentage)}%${phase}`; + if (coarse === lastProgressText) return; + lastProgressText = coarse; + safeWrite(process.stdout, `${text}\n`); + } + }, + }; +} + +function loadRunnerWindow(windowType: string): BrowserWindow { + const win = new BrowserWindow({ + width: 1280, + height: 720, + show: false, + webPreferences: { + preload: path.join(__dirname, "preload.mjs"), + additionalArguments: [ASSET_BASE_URL_ARG], + nodeIntegration: false, + contextIsolation: true, + // Same relaxation as the editor window: exporters load recording media + // via file:// URLs. + webSecurity: false, + backgroundThrottling: false, + }, + }); + + if (VITE_DEV_SERVER_URL) { + win.loadURL(`${VITE_DEV_SERVER_URL}?windowType=${windowType}`); + } else { + win.loadFile(path.join(RENDERER_DIST, "index.html"), { query: { windowType } }); + } + return win; +} + +function registerAppHandlersForCli(cliWindowRef: () => BrowserWindow | null) { + // The recording/export pipelines are driven through the same IPC surface the + // GUI uses. Window-management callbacks become no-ops; "switch-to-editor" + // (fired by the recorder hook after it stores a finished session) is handled + // by the runner itself, so an inert factory is enough. + const noop = () => { + // Intentionally empty: CLI mode has no HUD/tray/editor windows to manage. + }; + const notAvailable = () => { + throw new Error("Window not available in CLI mode"); + }; + registerIpcHandlers( + noop, // createEditorWindow: recording finished; runner drives completion + notAvailable, // createSourceSelectorWindow + notAvailable, // createCountdownOverlayWindow + notAvailable, // createNotesWindow + cliWindowRef, + () => null, + () => null, + () => null, + noop, // onRecordingStateChange: no tray to update + noop, // switchToHud + ); +} + +async function writeProjectFile(projectOut: string, projectData: unknown): Promise<void> { + await fs.mkdir(path.dirname(projectOut), { recursive: true }); + await fs.writeFile(projectOut, JSON.stringify(projectData, null, 2), "utf8"); +} + +function setupRecordStopSignals(stop: (reason: string) => void): void { + // SIGINT covers Ctrl+C everywhere; SIGTERM never fires on Windows, where + // stdin "stop" or --duration are the graceful alternatives (see docs/cli.md). + process.on("SIGINT", () => stop("SIGINT")); + process.on("SIGTERM", () => stop("SIGTERM")); + try { + // Touching process.stdin can throw on Windows GUI-subsystem builds when + // spawned with stdio "ignore"; signals and --duration still work then. + const rl = readline.createInterface({ input: process.stdin }); + rl.on("line", (line) => { + const trimmed = line.trim().toLowerCase(); + if (trimmed === "stop" || trimmed === "q" || trimmed === "quit") { + stop("stdin"); + } + }); + rl.on("close", () => { + // stdin EOF is not a stop signal: agents may spawn the CLI with + // stdin closed and stop it via SIGINT/--duration instead. + }); + } catch { + // stdin unavailable; signals and --duration still work. + } +} + +async function runInfoCommand(projectPath: string, json: boolean): Promise<number> { + const raw = await fs.readFile(projectPath, "utf8"); + const data = JSON.parse(raw) as { + version?: number; + media?: { screenVideoPath?: string; webcamVideoPath?: string; cursorCaptureMode?: string }; + videoPath?: string; + editor?: Record<string, unknown>; + }; + const editor = data.editor ?? {}; + const count = (key: string) => + Array.isArray(editor[key]) ? (editor[key] as unknown[]).length : 0; + const screenVideoPath = data.media?.screenVideoPath ?? data.videoPath ?? null; + const mediaExists = screenVideoPath + ? await fs + .access(screenVideoPath) + .then(() => true) + .catch(() => false) + : false; + + const summary = { + projectPath, + version: data.version ?? null, + screenVideoPath, + screenVideoExists: mediaExists, + webcamVideoPath: data.media?.webcamVideoPath ?? null, + cursorCaptureMode: data.media?.cursorCaptureMode ?? null, + exportFormat: (editor.exportFormat as string) ?? null, + exportQuality: (editor.exportQuality as string) ?? null, + aspectRatio: (editor.aspectRatio as string) ?? null, + zoomRegions: count("zoomRegions"), + trimRegions: count("trimRegions"), + speedRegions: count("speedRegions"), + annotationRegions: count("annotationRegions"), + }; + + if (json) { + safeWrite(process.stdout, `${JSON.stringify(summary)}\n`); + } else { + safeWrite( + process.stdout, + [ + `Project: ${summary.projectPath} (version ${summary.version ?? "?"})`, + `Video: ${summary.screenVideoPath ?? "(none)"}${mediaExists ? "" : " [MISSING]"}`, + `Webcam: ${summary.webcamVideoPath ?? "(none)"}`, + `Cursor: ${summary.cursorCaptureMode ?? "(unknown)"}`, + `Export: ${summary.exportFormat ?? "?"} / ${summary.exportQuality ?? "?"} / ${summary.aspectRatio ?? "?"}`, + `Timeline: ${summary.zoomRegions} zooms, ${summary.trimRegions} trims, ${summary.speedRegions} speed regions, ${summary.annotationRegions} annotations`, + ].join("\n") + "\n", + ); + } + return summary.screenVideoPath && !mediaExists ? 1 : 0; +} + +export function runCli(command: CliCommand): void { + if (command.kind === "help") { + process.stdout.write(CLI_USAGE); + app.exit(0); + return; + } + if (command.kind === "error") { + process.stderr.write(`Error: ${command.message}\n\n${CLI_USAGE}`); + app.exit(2); + return; + } + + const output = createOutput(command.json === true); + + // stdout belongs to the CLI protocol (NDJSON / progress); reroute the app's + // own console chatter (e.g. "[native-sck] starting…") to stderr. + const stringifyArg = (value: unknown): string => { + if (typeof value === "string") return value; + try { + return JSON.stringify(value) ?? String(value); + } catch { + return String(value); + } + }; + for (const level of ["log", "info", "warn", "error", "debug"] as const) { + console[level] = (...args: unknown[]) => { + safeWrite(process.stderr, `${args.map(stringifyArg).join(" ")}\n`); + }; + } + + // A consumer closing the pipe (`openscreen export | head`) must not crash the + // process, and main-process exceptions must never surface as Electron's GUI + // error dialog — report on stderr and exit non-zero instead. + const ignoreStreamError = () => { + // Intentionally empty: EPIPE from a closed consumer is not an error here. + }; + process.stdout.on("error", ignoreStreamError); + process.stderr.on("error", ignoreStreamError); + process.on("uncaughtException", (error) => { + safeWrite(process.stderr, `Fatal: ${error?.stack ?? String(error)}\n`); + app.exit(1); + }); + process.on("unhandledRejection", (reason) => { + safeWrite( + process.stderr, + `Fatal (unhandled rejection): ${reason instanceof Error ? (reason.stack ?? reason.message) : String(reason)}\n`, + ); + app.exit(1); + }); + + // Set once cli-done has been received; suppresses the window-all-closed + // failure path during the normal teardown race after a successful run. + let finished = false; + + // GPU may be unavailable in CI/servers; let Chromium fall back to SwiftShader + // so the WebGL-based export renderer still works. + app.commandLine.appendSwitch("enable-unsafe-swiftshader"); + + // Never show the dock icon for CLI runs. + if (process.platform === "darwin") { + app.dock?.hide(); + } + + app.on("window-all-closed", () => { + // Completion is signalled via cli-done; a vanished window is a failure + // only while the run is still in flight. + if (finished) return; + output.error("Renderer window closed unexpectedly"); + app.exit(1); + }); + + void app + .whenReady() + .then(async () => { + if (command.kind === "info") { + const code = await runInfoCommand(command.projectPath, command.json === true); + app.exit(code); + return; + } + + await fs.mkdir(path.join(app.getPath("userData"), "recordings"), { recursive: true }); + + // Media/screen permissions for the renderer (mic metering, future browser + // capture paths). Mirrors the GUI allowlist. + const allowed = [ + "media", + "audioCapture", + "microphone", + "videoCapture", + "camera", + "screen", + "display-capture", + ]; + session.defaultSession.setPermissionCheckHandler((_wc, permission) => + allowed.includes(permission), + ); + session.defaultSession.setPermissionRequestHandler((_wc, permission, callback) => + callback(allowed.includes(permission)), + ); + + // Browser-pipeline recording fallback (e.g. Linux, missing native helper) + // resolves the pre-selected source exactly like the GUI does. + session.defaultSession.setDisplayMediaRequestHandler( + (request, callback) => { + const source = getSelectedDesktopSource(); + if (!request.videoRequested || !source) { + callback({}); + return; + } + callback({ + video: source, + ...(request.audioRequested && process.platform === "win32" + ? { audio: "loopback" as const } + : {}), + }); + }, + { useSystemPicker: false }, + ); + + if (command.kind === "record" && command.mic && process.platform === "darwin") { + const micStatus = systemPreferences.getMediaAccessStatus("microphone"); + if (micStatus !== "granted") { + await systemPreferences.askForMediaAccess("microphone"); + } + } + + let cliWindow: BrowserWindow | null = null; + registerAppHandlersForCli(() => cliWindow); + + // Registered by the GUI boot path (main.ts) rather than registerIpcHandlers; + // the renderer's i18n init invokes it unconditionally. + ipcMain.handle("set-locale", () => { + // Locale only affects GUI menus/tray, which do not exist in CLI mode. + }); + ipcMain.handle("update-global-shortcut", () => ({ success: false })); + + const request: CliRequest = command; + ipcMain.handle("cli-get-request", () => request); + ipcMain.on("cli-log", (_event, level: string, message: string) => { + if (level === "error") { + output.error(message); + } else { + output.info(message); + output.event("log", { message }); + } + }); + ipcMain.on("cli-progress", (_event, progress: CliProgressEvent) => { + output.progress(progress); + }); + + ipcMain.handle("cli-done", async (_event, result: CliDoneResult) => { + if (finished) return; + finished = true; + + try { + if (result.success && command.kind === "record" && command.projectOut) { + if (result.projectData !== undefined) { + await writeProjectFile(command.projectOut, result.projectData); + result.projectPath = command.projectOut; + } + } + } catch (error) { + result.success = false; + result.error = `Recording succeeded but writing the project file failed: ${String(error)}`; + } + + if (result.success) { + for (const warning of result.warnings ?? []) { + output.info(`Warning: ${warning}`); + output.event("warning", { message: warning }); + } + if (command.kind === "export") { + output.info(`Exported ${result.format ?? ""} → ${result.outputPath}`); + } else { + output.info(`Recording saved → ${result.screenVideoPath}`); + if (result.cursorDataPath) output.info(`Cursor data → ${result.cursorDataPath}`); + if (result.projectPath) output.info(`Project → ${result.projectPath}`); + } + output.event("done", { ...result }); + } else { + output.error(result.error ?? "Unknown failure"); + output.event("done", { success: false, error: result.error }); + } + + // Give the renderer a beat to resolve the invoke before exiting. + setTimeout(() => app.exit(result.success ? 0 : 1), 50); + }); + + if (command.kind === "record") { + const stop = (reason: string) => { + output.info(`Stopping recording (${reason})…`); + output.event("stopping", { reason }); + cliWindow?.webContents.send("cli-stop-recording"); + }; + setupRecordStopSignals(stop); + } + + const windowType = command.kind === "export" ? "cli-export" : "cli-record"; + cliWindow = loadRunnerWindow(windowType); + + // Surface renderer console errors/warnings on stderr — the hidden window + // has no other way to show what went wrong (toasts are invisible). + cliWindow.webContents.on("console-message", (details) => { + if (details.level === "error" || details.level === "warning") { + safeWrite(process.stderr, `[renderer] ${details.message}\n`); + } + }); + + cliWindow.webContents.on("did-fail-load", (_e, code, description) => { + output.error(`Failed to load runner window: ${description} (${code})`); + app.exit(1); + }); + cliWindow.webContents.on("render-process-gone", (_e, details) => { + output.error(`Renderer crashed: ${details.reason}`); + app.exit(1); + }); + + output.event("started", { command: command.kind }); + }) + .catch((error) => { + output.error(error instanceof Error ? (error.stack ?? error.message) : String(error)); + app.exit(1); + }); +} diff --git a/electron/electron-env.d.ts b/electron/electron-env.d.ts index f954fc99e..9f20fae44 100644 --- a/electron/electron-env.d.ts +++ b/electron/electron-env.d.ts @@ -311,6 +311,12 @@ interface Window { onRequestSaveBeforeClose: (callback: () => Promise<boolean> | boolean) => () => void; onRequestCloseConfirm: (callback: () => void) => () => void; sendCloseConfirmResponse: (choice: "save" | "discard" | "cancel") => void; + // CLI mode (hidden runner windows; see electron/cli/) + cliGetRequest: () => Promise<import("../src/lib/cliContracts").CliRequest>; + cliProgress: (progress: import("../src/lib/cliContracts").CliProgressEvent) => void; + cliLog: (level: "info" | "error", message: string) => void; + cliDone: (result: import("../src/lib/cliContracts").CliDoneResult) => Promise<void>; + onCliStopRecording: (callback: () => void) => () => void; setLocale: (locale: string) => Promise<void>; saveDiagnostic: (payload: { error: string; diff --git a/electron/main.ts b/electron/main.ts index 12ce817fc..1c4b7d71d 100644 --- a/electron/main.ts +++ b/electron/main.ts @@ -12,6 +12,8 @@ import { Tray, } from "electron"; import { ShortcutBinding } from "../src/lib/shortcuts"; +import { parseCliArgs } from "./cli/args"; +import { runCli } from "./cli/cliMain"; import { isDiagnosticModeEnabled, mainLogBuffer } from "./diagnostics/main-log-buffer"; import { loadAndRegisterGlobalShortcut, @@ -31,6 +33,10 @@ import { const __dirname = path.dirname(fileURLToPath(import.meta.url)); +// CLI mode: `openscreen export|record|info|help ...` runs headless without +// HUD/tray/menu. Parsed before any GUI side effects; see electron/cli/. +const cliCommand = parseCliArgs(process.argv, app.isPackaged ? 1 : 2); + // Use Screen & System Audio Recording permissions instead of the CoreAudio Tap API on macOS. // Tap needs NSAudioCaptureUsageDescription in the parent app's Info.plist, which breaks when // running from a terminal/IDE during dev. @@ -122,11 +128,15 @@ function showMainWindow() { createWindow(); } -const stableInstanceLock = acquireStableInstanceLock(); -const hasElectronSingleInstanceLock = app.requestSingleInstanceLock(); +// CLI runs skip the single-instance lock so `openscreen export/record` works +// while the GUI app is open (they share nothing but the recordings directory). +const stableInstanceLock = cliCommand ? null : acquireStableInstanceLock(); +const hasElectronSingleInstanceLock = cliCommand ? false : app.requestSingleInstanceLock(); const hasSingleInstanceLock = Boolean(stableInstanceLock && hasElectronSingleInstanceLock); -if (hasSingleInstanceLock) { +if (cliCommand) { + runCli(cliCommand); +} else if (hasSingleInstanceLock) { app.on("second-instance", () => { showMainWindow(); }); @@ -478,11 +488,15 @@ function createCountdownOverlayWindowWrapper() { // Closing every window quits the app (tray goes too). The in-app "Return to Recorder" // button covers the editor-to-HUD round-trip, so closing the last window means "I'm done". -app.on("window-all-closed", () => { - app.quit(); -}); +// CLI mode owns its own lifecycle (see electron/cli/cliMain.ts). +if (!cliCommand) { + app.on("window-all-closed", () => { + app.quit(); + }); +} app.on("activate", () => { + if (cliCommand) return; // On macOS, re-open a window when the dock icon is clicked and none are open. const hasVisibleWindow = BrowserWindow.getAllWindows().some((window) => { if (window.isDestroyed() || !window.isVisible()) { @@ -503,7 +517,7 @@ app.on("will-quit", () => { stableInstanceLock?.release(); }); -const appReady = hasSingleInstanceLock ? app.whenReady() : null; +const appReady = !cliCommand && hasSingleInstanceLock ? app.whenReady() : null; appReady?.then(async () => { if (isDiagnosticModeEnabled()) { diff --git a/electron/preload.ts b/electron/preload.ts index f02a2ccd2..94c243681 100644 --- a/electron/preload.ts +++ b/electron/preload.ts @@ -302,4 +302,22 @@ contextBridge.exposeInMainWorld("electronAPI", { sendCloseConfirmResponse: (choice: "save" | "discard" | "cancel") => { ipcRenderer.send("close-confirm-response", choice); }, + // --- CLI mode (hidden runner windows; see electron/cli/) --- + cliGetRequest: (): Promise<import("../src/lib/cliContracts").CliRequest> => { + return ipcRenderer.invoke("cli-get-request"); + }, + cliProgress: (progress: import("../src/lib/cliContracts").CliProgressEvent) => { + ipcRenderer.send("cli-progress", progress); + }, + cliLog: (level: "info" | "error", message: string) => { + ipcRenderer.send("cli-log", level, message); + }, + cliDone: (result: import("../src/lib/cliContracts").CliDoneResult) => { + return ipcRenderer.invoke("cli-done", result); + }, + onCliStopRecording: (callback: () => void) => { + const listener = () => callback(); + ipcRenderer.on("cli-stop-recording", listener); + return () => ipcRenderer.removeListener("cli-stop-recording", listener); + }, }); diff --git a/electron/windows.ts b/electron/windows.ts index 60dbc9df4..b330e514f 100644 --- a/electron/windows.ts +++ b/electron/windows.ts @@ -14,7 +14,7 @@ const HEADLESS = process.env["HEADLESS"] === "true"; const ASSET_BASE_DIR = process.defaultApp ? path.join(__dirname, "..", "public") : process.resourcesPath; -const ASSET_BASE_URL_ARG = `--asset-base-url=${pathToFileURL(`${ASSET_BASE_DIR}${path.sep}`).toString()}`; +export const ASSET_BASE_URL_ARG = `--asset-base-url=${pathToFileURL(`${ASSET_BASE_DIR}${path.sep}`).toString()}`; let hudOverlayWindow: BrowserWindow | null = null; diff --git a/package.json b/package.json index 02f778efc..00e9af3b5 100644 --- a/package.json +++ b/package.json @@ -47,6 +47,7 @@ "diagnostic:run": "node scripts/diagnostic-tool/diagnostic.mjs", "diagnostic:smoke:win": "node scripts/diagnostic-tool/diagnostic.mjs --duration 3", "build-vite": "tsc && vite build", + "cli": "electron .", "test:browser": "vitest --config vitest.browser.config.ts --run", "test:browser:install": "playwright install --with-deps chromium-headless-shell", "test:e2e": "playwright test", diff --git a/src/App.tsx b/src/App.tsx index 35749408c..7c72e4bdc 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -10,6 +10,8 @@ import { ShortcutsProvider } from "./contexts/ShortcutsContext"; import { loadAllCustomFonts } from "./lib/customFonts"; const VideoEditor = lazy(() => import("./components/video-editor/VideoEditor")); +const CliExportRunner = lazy(() => import("./cli/CliExportRunner")); +const CliRecordRunner = lazy(() => import("./cli/CliRecordRunner")); const ShortcutsConfigDialog = lazy(() => import("./components/video-editor/ShortcutsConfigDialog").then((module) => ({ default: module.ShortcutsConfigDialog, @@ -66,6 +68,18 @@ export default function App() { return <SourceSelector />; case "countdown-overlay": return <CountdownOverlay />; + case "cli-export": + return ( + <Suspense fallback={null}> + <CliExportRunner /> + </Suspense> + ); + case "cli-record": + return ( + <Suspense fallback={null}> + <CliRecordRunner /> + </Suspense> + ); case "editor": return ( <ShortcutsProvider> diff --git a/src/cli/CliExportRunner.tsx b/src/cli/CliExportRunner.tsx new file mode 100644 index 000000000..56f60581e --- /dev/null +++ b/src/cli/CliExportRunner.tsx @@ -0,0 +1,335 @@ +// Hidden-window runner for `openscreen export`. Loads an .openscreen project, +// rebuilds the same exporter configuration the editor's export dialog would, +// and streams progress back to the CLI controller in the main process. + +import { useEffect, useRef, useState } from "react"; +import { + DEFAULT_CURSOR_SETTINGS, + DEFAULT_SOURCE_DIMENSIONS, +} from "@/components/video-editor/editorDefaults"; +import { + normalizeProjectEditor, + resolveProjectMedia, + toFileUrl, + validateProjectData, +} from "@/components/video-editor/projectPersistence"; +import type { CursorTelemetryPoint } from "@/components/video-editor/types"; +import type { CliDoneResult, CliExportRequest } from "@/lib/cliContracts"; +import { hasNativeCursorRecordingData } from "@/lib/cursor/nativeCursor"; +import { calculateOutputDimensions, GifExporter } from "@/lib/exporter/gifExporter"; +import { + calculateEffectiveSourceDimensions, + calculateMp4ExportSettings, +} from "@/lib/exporter/mp4ExportSettings"; +import type { ExportProgress } from "@/lib/exporter/types"; +import { GIF_SIZE_PRESETS } from "@/lib/exporter/types"; +import { VideoExporter } from "@/lib/exporter/videoExporter"; +import { mixVoiceoverIntoVideo } from "@/lib/exporter/voiceoverMix"; +import { nativeBridgeClient } from "@/native"; +import type { CursorRecordingData, NativePlatform } from "@/native/contracts"; +import { getAspectRatioValue, getNativeAspectRatioValue } from "@/utils/aspectRatioUtils"; + +// Mirrors the private helper in VideoEditor.tsx. +function isClickInteractionType(interactionType: string | null | undefined) { + return ( + interactionType === "click" || + interactionType === "double-click" || + interactionType === "right-click" || + interactionType === "middle-click" + ); +} + +function probeVideoDimensions(url: string): Promise<{ width: number; height: number }> { + return new Promise((resolve, reject) => { + const video = document.createElement("video"); + video.preload = "metadata"; + video.muted = true; + const cleanup = () => { + clearTimeout(timer); + video.removeAttribute("src"); + video.load(); + }; + // A stalled load fires neither event; without a deadline the CLI hangs. + const timer = setTimeout(() => { + cleanup(); + reject(new Error(`Timed out reading video metadata: ${url}`)); + }, 30_000); + video.onloadedmetadata = () => { + const width = video.videoWidth; + const height = video.videoHeight; + cleanup(); + resolve({ width, height }); + }; + video.onerror = () => { + cleanup(); + reject(new Error(`Failed to load video metadata: ${url}`)); + }; + video.src = url; + }); +} + +/** Fit the composition aspect ratio into the reference preview box, mirroring + * how the editor sizes its on-screen preview container. */ +function fitPreviewBox(aspectRatioValue: number, boxWidth: number, boxHeight: number) { + let width = boxWidth; + let height = boxWidth / aspectRatioValue; + if (height > boxHeight) { + height = boxHeight; + width = boxHeight * aspectRatioValue; + } + return { width: Math.round(width), height: Math.round(height) }; +} + +function replaceExtension(filePath: string, newExtension: string): string { + return filePath.replace(/\.(openscreen|json)$/i, "") + newExtension; +} + +async function runExport(request: CliExportRequest): Promise<CliDoneResult> { + const loaded = await nativeBridgeClient.project.loadProjectFileFromPath(request.projectPath); + if (!loaded.success || loaded.project === undefined) { + throw new Error(loaded.error ?? loaded.message ?? "Failed to load project file"); + } + if (!validateProjectData(loaded.project)) { + throw new Error("Project file is not a valid .openscreen project"); + } + const project = loaded.project; + const media = resolveProjectMedia(project); + if (!media) { + throw new Error("Project file does not reference any recorded media"); + } + const editor = normalizeProjectEditor(project.editor ?? {}); + + const format = request.format ?? editor.exportFormat; + if (request.audioPath && format === "gif") { + throw new Error( + "--audio is only supported for MP4 exports (this project's stored format is gif; pass --format mp4)", + ); + } + const quality = request.quality ?? editor.exportQuality; + const gifFrameRate = request.gifFrameRate ?? editor.gifFrameRate; + const gifSizePreset = request.gifSizePreset ?? editor.gifSizePreset; + const outPath = + request.outPath ?? replaceExtension(request.projectPath, format === "gif" ? ".gif" : ".mp4"); + + const videoUrl = toFileUrl(media.screenVideoPath); + const webcamVideoUrl = media.webcamVideoPath ? toFileUrl(media.webcamVideoPath) : undefined; + + // Cursor sidecar data (native recordings). Both lookups tolerate missing files. + let cursorTelemetry: CursorTelemetryPoint[] = []; + let cursorRecordingData: CursorRecordingData | null = null; + try { + cursorTelemetry = await nativeBridgeClient.cursor.getTelemetry(media.screenVideoPath); + } catch { + cursorTelemetry = []; + } + try { + cursorRecordingData = await nativeBridgeClient.cursor.getRecordingData(media.screenVideoPath); + } catch { + cursorRecordingData = null; + } + + const recordingClicks = + cursorRecordingData?.samples + .filter((sample) => isClickInteractionType(sample.interactionType)) + .map((sample) => sample.timeMs) ?? []; + const cursorClickTimestamps = + recordingClicks.length > 0 + ? recordingClicks + : cursorTelemetry + .filter((sample) => isClickInteractionType(sample.interactionType)) + .map((sample) => sample.timeMs); + + let platform: NativePlatform | null = null; + try { + platform = await nativeBridgeClient.system.getPlatform(); + } catch { + platform = null; + } + const hasEditableCursorRecording = + (media.cursorCaptureMode ?? "editable-overlay") === "editable-overlay" && + (platform === "win32" || platform === "darwin") && + hasNativeCursorRecordingData(cursorRecordingData); + const effectiveShowCursor = DEFAULT_CURSOR_SETTINGS.show && hasEditableCursorRecording; + + const probed = await probeVideoDimensions(videoUrl); + const sourceWidth = probed.width || DEFAULT_SOURCE_DIMENSIONS.width; + const sourceHeight = probed.height || DEFAULT_SOURCE_DIMENSIONS.height; + const effectiveSourceDimensions = calculateEffectiveSourceDimensions( + sourceWidth, + sourceHeight, + editor.cropRegion, + ); + const aspectRatioValue = + editor.aspectRatio === "native" + ? getNativeAspectRatioValue(sourceWidth, sourceHeight, editor.cropRegion) + : getAspectRatioValue(editor.aspectRatio); + + const preview = fitPreviewBox( + aspectRatioValue, + request.previewWidth ?? 1280, + request.previewHeight ?? 720, + ); + + const onProgress = (progress: ExportProgress) => { + window.electronAPI.cliProgress({ + percentage: progress.percentage, + currentFrame: progress.currentFrame, + totalFrames: progress.totalFrames, + estimatedTimeRemaining: progress.estimatedTimeRemaining, + phase: progress.phase, + }); + }; + + const sharedConfig = { + videoUrl, + webcamVideoUrl, + wallpaper: editor.wallpaper, + zoomRegions: editor.zoomRegions, + cameraFullscreenRegions: editor.cameraFullscreenRegions, + trimRegions: editor.trimRegions, + speedRegions: editor.speedRegions, + showShadow: editor.shadowIntensity > 0, + shadowIntensity: editor.shadowIntensity, + showBlur: editor.showBlur, + motionBlurAmount: editor.motionBlurAmount, + borderRadius: editor.borderRadius, + padding: editor.padding, + cropRegion: editor.cropRegion, + cursorRecordingData, + cursorScale: effectiveShowCursor ? DEFAULT_CURSOR_SETTINGS.size : 0, + cursorSmoothing: DEFAULT_CURSOR_SETTINGS.smoothing, + cursorMotionBlur: DEFAULT_CURSOR_SETTINGS.motionBlur, + cursorClickBounce: DEFAULT_CURSOR_SETTINGS.clickBounce, + cursorClipToBounds: DEFAULT_CURSOR_SETTINGS.clipToBounds, + cursorTheme: editor.cursorTheme, + annotationRegions: editor.annotationRegions, + webcamLayoutPreset: editor.webcamLayoutPreset, + webcamMaskShape: editor.webcamMaskShape, + webcamMirrored: editor.webcamMirrored, + webcamReactiveZoom: editor.webcamReactiveZoom, + webcamSizePreset: editor.webcamSizePreset, + webcamPosition: editor.webcamPosition, + previewWidth: preview.width, + previewHeight: preview.height, + cursorTelemetry, + cursorClickTimestamps, + onProgress, + }; + + let blob: Blob; + let warnings: string[] | undefined; + let outWidth: number; + let outHeight: number; + + if (format === "gif") { + const gifDimensions = calculateOutputDimensions( + effectiveSourceDimensions.width, + effectiveSourceDimensions.height, + gifSizePreset, + GIF_SIZE_PRESETS, + aspectRatioValue, + ); + outWidth = gifDimensions.width; + outHeight = gifDimensions.height; + const gifExporter = new GifExporter({ + ...sharedConfig, + width: gifDimensions.width, + height: gifDimensions.height, + frameRate: gifFrameRate, + loop: editor.gifLoop, + sizePreset: gifSizePreset, + videoPadding: editor.padding, + }); + const result = await gifExporter.export(); + if (!result.success || !result.blob) { + throw new Error(result.error ?? "GIF export failed"); + } + blob = result.blob; + warnings = result.warnings; + } else { + const mp4Settings = calculateMp4ExportSettings({ + quality, + sourceWidth: effectiveSourceDimensions.width, + sourceHeight: effectiveSourceDimensions.height, + aspectRatioValue, + }); + outWidth = mp4Settings.width; + outHeight = mp4Settings.height; + const exporter = new VideoExporter({ + ...sharedConfig, + width: mp4Settings.width, + height: mp4Settings.height, + frameRate: 60, + bitrate: mp4Settings.bitrate, + codec: "avc1.640033", + }); + const result = await exporter.export(); + if (!result.success || !result.blob) { + throw new Error(result.error ?? "MP4 export failed"); + } + blob = result.blob; + warnings = result.warnings; + } + + if (request.audioPath && format === "mp4") { + window.electronAPI.cliProgress({ percentage: 100, phase: "mixing-voiceover" }); + const audioResponse = await fetch(toFileUrl(request.audioPath)); + if (!audioResponse.ok) { + throw new Error(`Failed to read voiceover file: ${request.audioPath}`); + } + const voiceoverData = await audioResponse.arrayBuffer(); + blob = await mixVoiceoverIntoVideo(blob, { + voiceoverData, + mode: request.audioMode, + offsetSec: request.audioOffsetSec, + }); + } + + const arrayBuffer = await blob.arrayBuffer(); + const saveResult = await window.electronAPI.writeExportToPath(arrayBuffer, outPath); + if (!saveResult.success || !saveResult.path) { + throw new Error(saveResult.message ?? `Failed to write output to ${outPath}`); + } + + return { + success: true, + outputPath: saveResult.path, + format, + width: outWidth, + height: outHeight, + warnings, + }; +} + +export function CliExportRunner() { + const startedRef = useRef(false); + const [status, setStatus] = useState("Starting export…"); + + useEffect(() => { + if (startedRef.current) return; + startedRef.current = true; + + void (async () => { + try { + const request = (await window.electronAPI.cliGetRequest()) as CliExportRequest; + if (request.kind !== "export") { + throw new Error(`cli-export window received a ${request.kind} request`); + } + setStatus(`Exporting ${request.projectPath}…`); + const result = await runExport(request); + await window.electronAPI.cliDone(result); + } catch (error) { + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); + await window.electronAPI.cliDone({ success: false, error: message }); + } + })(); + }, []); + + return ( + <div className="flex h-screen items-center justify-center bg-[#09090b] text-white/60 text-sm"> + {status} + </div> + ); +} + +export default CliExportRunner; diff --git a/src/cli/CliRecordRunner.tsx b/src/cli/CliRecordRunner.tsx new file mode 100644 index 000000000..e19c451f9 --- /dev/null +++ b/src/cli/CliRecordRunner.tsx @@ -0,0 +1,301 @@ +// Hidden-window runner for `openscreen record`. Reuses the full recording +// pipeline via useScreenRecorder (native macOS/Windows helpers with browser +// fallback), driven by the CLI controller instead of the HUD. + +import { useEffect, useRef, useState } from "react"; +import { + normalizeProjectEditor, + PROJECT_VERSION, +} from "@/components/video-editor/projectPersistence"; +import { useScreenRecorder } from "@/hooks/useScreenRecorder"; +import type { CliRecordRequest } from "@/lib/cliContracts"; + +type Phase = "init" | "recording" | "stopping" | "done"; + +async function pickSource(request: CliRecordRequest): Promise<ProcessedDesktopSource> { + const sources = await window.electronAPI.getSources({ + types: ["screen", "window"], + thumbnailSize: { width: 32, height: 18 }, + }); + + if (request.windowTitle) { + const needle = request.windowTitle.toLowerCase(); + const match = sources.find( + (source) => source.id.startsWith("window:") && source.name.toLowerCase().includes(needle), + ); + if (!match) { + const windows = sources + .filter((s) => s.id.startsWith("window:")) + .map((s) => ` - ${s.name}`) + .join("\n"); + throw new Error( + `No window title contains "${request.windowTitle}". Open windows:\n${windows}`, + ); + } + return match; + } + + const screens = sources.filter((source) => source.id.startsWith("screen:")); + const screen = screens[request.displayIndex]; + if (!screen) { + throw new Error( + `Display index ${request.displayIndex} not found (${screens.length} screen(s) available)`, + ); + } + return screen; +} + +async function resolveMicDeviceId(deviceNameFilter: string | null): Promise<{ + deviceId: string | undefined; + deviceName: string | undefined; +}> { + if (!deviceNameFilter) return { deviceId: undefined, deviceName: undefined }; + + // Labels require an active permission grant; a short-lived stream unlocks them. + let probeStream: MediaStream | null = null; + try { + probeStream = await navigator.mediaDevices.getUserMedia({ audio: true }); + } catch { + // Enumeration below may still work with empty labels. + } + try { + const devices = await navigator.mediaDevices.enumerateDevices(); + const needle = deviceNameFilter.toLowerCase(); + const match = devices.find( + (device) => device.kind === "audioinput" && device.label.toLowerCase().includes(needle), + ); + if (!match) { + const labels = devices + .filter((d) => d.kind === "audioinput" && d.label) + .map((d) => ` - ${d.label}`) + .join("\n"); + throw new Error(`No microphone label contains "${deviceNameFilter}". Devices:\n${labels}`); + } + return { deviceId: match.deviceId, deviceName: match.label }; + } finally { + probeStream?.getTracks().forEach((track) => track.stop()); + } +} + +/** A minimal .openscreen project referencing the finished recording, with all + * editor settings at their defaults — ready for `openscreen export` or the GUI. */ +function buildDefaultProject(session: { + screenVideoPath: string; + webcamVideoPath?: string; + cursorCaptureMode?: string; +}) { + return { + version: PROJECT_VERSION, + media: { + screenVideoPath: session.screenVideoPath, + ...(session.webcamVideoPath ? { webcamVideoPath: session.webcamVideoPath } : {}), + ...(session.cursorCaptureMode ? { cursorCaptureMode: session.cursorCaptureMode } : {}), + }, + editor: normalizeProjectEditor({}), + }; +} + +export function CliRecordRunner() { + const recorder = useScreenRecorder(); + const startedRef = useRef(false); + const requestRef = useRef<CliRecordRequest | null>(null); + // Re-render trigger once the bootstrap has applied all recorder settings; + // the start effect below cannot rely on recorder state deps alone because + // default-valued settings (no mic, no system audio) never change. + const [requestReady, setRequestReady] = useState<CliRecordRequest | null>(null); + const phaseRef = useRef<Phase>("init"); + const recordingStartedAtRef = useRef<number | null>(null); + // A stop (SIGINT/stdin) can land while the capture helper is still starting; + // remember it and apply as soon as recording flips on. + const stopRequestedRef = useRef(false); + const [status, setStatus] = useState("Preparing recording…"); + + const { + recording, + saving, + startRecordingImmediately, + toggleRecording, + setMicrophoneEnabled, + setMicrophoneDeviceId, + setMicrophoneDeviceName, + setSystemAudioEnabled, + setCursorCaptureMode, + } = recorder; + + // Keep latest values in refs for the stop/finish effects. + const toggleRecordingRef = useRef(toggleRecording); + const recordingRef = useRef(recording); + useEffect(() => { + toggleRecordingRef.current = toggleRecording; + recordingRef.current = recording; + }); + + const fail = async (error: unknown) => { + phaseRef.current = "done"; + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); + await window.electronAPI.cliDone({ success: false, error: message }); + }; + + // Bootstrap: pick source, configure recorder, start. + // biome-ignore lint/correctness/useExhaustiveDependencies: intentional run-once bootstrap; startedRef guards re-entry + useEffect(() => { + if (startedRef.current) return; + startedRef.current = true; + + void (async () => { + try { + // The recorder hook surfaces some failures via blocking alert(); a + // hidden window must never show (or hang on) a modal. + window.alert = (message?: unknown) => { + window.electronAPI.cliLog("error", `Recorder: ${String(message)}`); + }; + + const request = (await window.electronAPI.cliGetRequest()) as CliRecordRequest; + if (request.kind !== "record") { + throw new Error(`cli-record window received a ${request.kind} request`); + } + requestRef.current = request; + + const source = await pickSource(request); + await window.electronAPI.selectSource(source); + window.electronAPI.cliLog("info", `Recording source: ${source.name}`); + + if (request.mic) { + const mic = await resolveMicDeviceId(request.micDevice); + setMicrophoneEnabled(true); + setMicrophoneDeviceId(mic.deviceId); + setMicrophoneDeviceName(mic.deviceName); + if (mic.deviceName) { + window.electronAPI.cliLog("info", `Microphone: ${mic.deviceName}`); + } + } + setSystemAudioEnabled(request.systemAudio); + setCursorCaptureMode(request.cursorMode); + setStatus("Starting recording…"); + setRequestReady(request); + } catch (error) { + await fail(error); + } + })(); + }, []); + + // The setters above land on the *next* render; start only once they have. + const configuredRef = useRef(false); + // biome-ignore lint/correctness/useExhaustiveDependencies: fires when recorder settings match the request; other referenced values are stable refs/callbacks + useEffect(() => { + const request = requestReady; + if (!request || configuredRef.current || phaseRef.current !== "init") return; + const micReady = !request.mic || recorder.microphoneEnabled; + const systemAudioReady = recorder.systemAudioEnabled === request.systemAudio; + const cursorReady = recorder.cursorCaptureMode === request.cursorMode; + if (!micReady || !systemAudioReady || !cursorReady) return; + + configuredRef.current = true; + phaseRef.current = "recording"; + void (async () => { + try { + await startRecordingImmediately(); + // The hook reports start failures via toast/console, not by + // rejecting — without a deadline a failed start would hang the + // CLI forever. + setTimeout(() => { + if (recordingStartedAtRef.current === null && phaseRef.current === "recording") { + void fail( + new Error( + "Recording did not start within 30s — see stderr for the underlying capture error", + ), + ); + } + }, 30_000); + } catch (error) { + await fail(error); + } + })(); + }, [ + requestReady, + recorder.microphoneEnabled, + recorder.systemAudioEnabled, + recorder.cursorCaptureMode, + ]); + + // Recording state transitions: report start and arm the duration timer. + useEffect(() => { + const request = requestRef.current; + if (recording && recordingStartedAtRef.current === null) { + recordingStartedAtRef.current = Date.now(); + setStatus("Recording…"); + window.electronAPI.cliLog("info", "Recording started"); + + if (stopRequestedRef.current) { + phaseRef.current = "stopping"; + setStatus("Stopping…"); + toggleRecordingRef.current(); + return; + } + + if (request?.durationMs) { + const timer = setTimeout(() => { + if (recordingRef.current && phaseRef.current === "recording") { + phaseRef.current = "stopping"; + window.electronAPI.cliLog("info", `Duration reached (${request.durationMs}ms)`); + toggleRecordingRef.current(); + } + }, request.durationMs); + return () => clearTimeout(timer); + } + } + }, [recording]); + + // External stop (SIGINT / stdin via main process). + useEffect(() => { + return window.electronAPI.onCliStopRecording(() => { + if (recordingRef.current && phaseRef.current === "recording") { + phaseRef.current = "stopping"; + setStatus("Stopping…"); + toggleRecordingRef.current(); + } else { + // Capture is still starting; stop as soon as it comes up. + stopRequestedRef.current = true; + } + }); + }, []); + + // Completion: recording flipped off and the session finished saving. + // biome-ignore lint/correctness/useExhaustiveDependencies: completion is keyed on recording/saving; fail is stable + useEffect(() => { + if (recordingStartedAtRef.current === null) return; + if (recording || saving) return; + if (phaseRef.current === "done") return; + phaseRef.current = "done"; + + void (async () => { + try { + const sessionResult = await window.electronAPI.getCurrentRecordingSession(); + const session = sessionResult?.session; + if (!session?.screenVideoPath) { + throw new Error("Recording finished but no session manifest was stored"); + } + const durationMs = Date.now() - (recordingStartedAtRef.current ?? Date.now()); + const request = requestRef.current; + await window.electronAPI.cliDone({ + success: true, + screenVideoPath: session.screenVideoPath, + webcamVideoPath: session.webcamVideoPath, + cursorDataPath: `${session.screenVideoPath}.cursor.json`, + durationMs, + ...(request?.projectOut ? { projectData: buildDefaultProject(session) } : {}), + }); + } catch (error) { + await fail(error); + } + })(); + }, [recording, saving]); + + return ( + <div className="flex h-screen items-center justify-center bg-[#09090b] text-white/60 text-sm"> + {status} + </div> + ); +} + +export default CliRecordRunner; diff --git a/src/hooks/useScreenRecorder.ts b/src/hooks/useScreenRecorder.ts index af95b4722..42458f1e6 100644 --- a/src/hooks/useScreenRecorder.ts +++ b/src/hooks/useScreenRecorder.ts @@ -54,6 +54,8 @@ type UseScreenRecorderReturn = { saving: boolean; elapsedSeconds: number; toggleRecording: () => void; + /** Starts recording with no countdown overlay. Used by the headless CLI runner. */ + startRecordingImmediately: () => Promise<void>; togglePaused: () => void; canPauseRecording: boolean; restartRecording: () => void; @@ -1728,6 +1730,7 @@ export function useScreenRecorder(): UseScreenRecorderReturn { saving, elapsedSeconds, toggleRecording, + startRecordingImmediately: () => startRecording(), togglePaused, canPauseRecording, restartRecording, diff --git a/src/lib/cliContracts.ts b/src/lib/cliContracts.ts new file mode 100644 index 000000000..cdd7a4506 --- /dev/null +++ b/src/lib/cliContracts.ts @@ -0,0 +1,83 @@ +// Shared request/response contracts between the CLI entry in the Electron main +// process (electron/cli/) and the hidden renderer runners (src/cli/). Keep this +// file dependency-free so both build targets can import it. + +import type { ExportQuality, GifFrameRate, GifSizePreset } from "./exporter/types"; + +export type CliCursorCaptureMode = "editable-overlay" | "system"; + +export interface CliExportRequest { + kind: "export"; + /** Absolute path to the .openscreen project file. */ + projectPath: string; + /** Absolute output path; null = derive from projectPath + format. */ + outPath: string | null; + /** null = use the format stored in the project. */ + format: "mp4" | "gif" | null; + /** null = use the quality stored in the project. */ + quality: ExportQuality | null; + gifFrameRate: GifFrameRate | null; + gifSizePreset: GifSizePreset | null; + /** + * Reference preview box used to scale annotation text and border radii the + * same way the editor's on-screen preview does. The composition is fitted + * into this box, mirroring the editor layout. Defaults to 1280x720. + */ + previewWidth: number | null; + previewHeight: number | null; + /** Absolute path to a voiceover audio file to mix into the export (MP4 only). */ + audioPath: string | null; + /** "mix" layers the voiceover over the recording's audio; "replace" drops the original. */ + audioMode: "mix" | "replace"; + /** Delay before the voiceover starts, in seconds. */ + audioOffsetSec: number; +} + +export interface CliRecordRequest { + kind: "record"; + /** Index into the available screen sources (0 = primary display). */ + displayIndex: number; + /** Case-insensitive substring match against window titles; overrides displayIndex. */ + windowTitle: string | null; + mic: boolean; + /** Microphone device label substring; null = system default. */ + micDevice: string | null; + systemAudio: boolean; + cursorMode: CliCursorCaptureMode; + /** Auto-stop after this many milliseconds; null = stop via signal/stdin. */ + durationMs: number | null; + /** When set, write a ready-to-export .openscreen project here after recording. */ + projectOut: string | null; +} + +export type CliRequest = CliExportRequest | CliRecordRequest; + +export interface CliProgressEvent { + percentage: number; + currentFrame?: number; + totalFrames?: number; + estimatedTimeRemaining?: number; + phase?: string; +} + +export interface CliDoneResult { + success: boolean; + error?: string; + warnings?: string[]; + /** Export: the written output file. */ + outputPath?: string; + format?: string; + width?: number; + height?: number; + /** Record: produced artifacts. */ + screenVideoPath?: string; + webcamVideoPath?: string; + cursorDataPath?: string; + projectPath?: string; + durationMs?: number; + /** + * Record: a ready-to-save .openscreen project object built by the runner. + * The main process writes it to the --project path (renderer has no fs). + */ + projectData?: unknown; +} diff --git a/src/lib/exporter/voiceoverMix.ts b/src/lib/exporter/voiceoverMix.ts new file mode 100644 index 000000000..0c6305627 --- /dev/null +++ b/src/lib/exporter/voiceoverMix.ts @@ -0,0 +1,147 @@ +// Post-export voiceover mixing for the CLI (`openscreen export --audio`). +// +// Takes the finished MP4 blob, copies its video packets untouched (no +// re-encode), renders a new audio track with OfflineAudioContext — the +// original audio and the voiceover mixed, or the voiceover alone — and +// re-muxes both into a new MP4 via mediabunny. + +import { + ALL_FORMATS, + AudioBufferSource, + BlobSource, + BufferTarget, + EncodedPacketSink, + EncodedVideoPacketSource, + Input, + Mp4OutputFormat, + Output, +} from "mediabunny"; + +export type VoiceoverMixMode = "mix" | "replace"; + +export interface VoiceoverMixOptions { + /** Encoded audio file bytes (mp3/wav/m4a — anything decodeAudioData accepts). */ + voiceoverData: ArrayBuffer; + mode: VoiceoverMixMode; + /** Delay before the voiceover starts, in seconds. */ + offsetSec: number; + /** Gain applied to the original track in "mix" mode (0..1). */ + originalGain?: number; +} + +// Duck the original bed under the voiceover by default so the unity-gain sum +// of two loud sources doesn't hard-clip. +const DEFAULT_ORIGINAL_GAIN = 0.4; + +const OUTPUT_SAMPLE_RATE = 48_000; +const OUTPUT_CHANNELS = 2; +const VOICEOVER_AUDIO_BITRATE = 192_000; + +async function decodeToBuffer( + context: OfflineAudioContext, + data: ArrayBuffer, +): Promise<AudioBuffer> { + // decodeAudioData detaches the buffer, so hand it a copy. + return context.decodeAudioData(data.slice(0)); +} + +/** Renders the final audio track: original bed (optional) + offset voiceover. */ +async function renderMixedAudio( + videoData: ArrayBuffer | null, + durationSec: number, + options: VoiceoverMixOptions, +): Promise<AudioBuffer> { + const frameCount = Math.max(1, Math.ceil(durationSec * OUTPUT_SAMPLE_RATE)); + const context = new OfflineAudioContext(OUTPUT_CHANNELS, frameCount, OUTPUT_SAMPLE_RATE); + + const voiceover = await decodeToBuffer(context, options.voiceoverData); + const voiceoverNode = context.createBufferSource(); + voiceoverNode.buffer = voiceover; + voiceoverNode.connect(context.destination); + voiceoverNode.start(Math.max(0, options.offsetSec)); + + if (options.mode === "mix" && videoData) { + try { + const original = await decodeToBuffer(context, videoData); + const originalNode = context.createBufferSource(); + originalNode.buffer = original; + const gainNode = context.createGain(); + gainNode.gain.value = options.originalGain ?? DEFAULT_ORIGINAL_GAIN; + originalNode.connect(gainNode); + gainNode.connect(context.destination); + originalNode.start(0); + } catch { + // The exported video has no decodable audio track; the voiceover + // becomes the only audio, same as "replace". + } + } + + return context.startRendering(); +} + +/** + * Returns a new MP4 blob with the same video stream and the mixed audio track. + * The video packets are copied without re-encoding. + */ +export async function mixVoiceoverIntoVideo( + videoBlob: Blob, + options: VoiceoverMixOptions, +): Promise<Blob> { + const input = new Input({ source: new BlobSource(videoBlob), formats: ALL_FORMATS }); + try { + const videoTrack = await input.getPrimaryVideoTrack(); + if (!videoTrack) { + throw new Error("Exported file has no video track to remux"); + } + const codec = videoTrack.codec; + if (!codec) { + throw new Error("Exported file's video codec was not recognized"); + } + const decoderConfig = await videoTrack.getDecoderConfig(); + if (!decoderConfig) { + throw new Error("Exported file's video decoder config could not be read"); + } + const durationSec = await input.computeDuration(); + + // The full-file bytes are only needed to decode the original bed in + // "mix" mode; "replace" skips the copy entirely. + const videoData = options.mode === "mix" ? await videoBlob.arrayBuffer() : null; + const mixedAudio = await renderMixedAudio(videoData, durationSec, options); + + const target = new BufferTarget(); + const output = new Output({ + format: new Mp4OutputFormat({ fastStart: "in-memory" }), + target, + }); + try { + const videoSource = new EncodedVideoPacketSource(codec); + output.addVideoTrack(videoSource); + const audioSource = new AudioBufferSource({ + codec: "aac", + bitrate: VOICEOVER_AUDIO_BITRATE, + }); + output.addAudioTrack(audioSource); + await output.start(); + + const sink = new EncodedPacketSink(videoTrack); + let isFirstPacket = true; + for await (const packet of sink.packets()) { + await videoSource.add(packet, isFirstPacket ? { decoderConfig } : undefined); + isFirstPacket = false; + } + await audioSource.add(mixedAudio); + + await output.finalize(); + } catch (error) { + await output.cancel().catch(() => undefined); + throw error; + } + const buffer = target.buffer; + if (!buffer) { + throw new Error("Voiceover remux produced no output"); + } + return new Blob([buffer], { type: "video/mp4" }); + } finally { + input.dispose(); + } +}