feat: prototype editable cursor choreography for AI Edition#116
Draft
YoneRai12 wants to merge 105 commits into
Draft
feat: prototype editable cursor choreography for AI Edition#116YoneRai12 wants to merge 105 commits into
YoneRai12 wants to merge 105 commits into
Conversation
…; make it the only editor
…ate, region drag, Edit Clip dialog
…ent tool-selection rules
… (LCS -> timeline)
…sset camera/cursor links
…t, then drop it for Silero VAD
…nt, build+perf work)
…W word-timestamps
…ness Pure-logic per-stage timing accumulator (record / start-stop / time / timeAsync / snapshot / formatSummary) with an injectable clock, for measuring where export time goes (decode/render/readback/encode) before the D4 perf rewrite. No DOM/IPC; 13 table-driven vitest tests; tsc + biome clean. Delegated (MiniMax-M3), reviewed and verified by the supervisor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase-0 instrumentation showed the export is encode-bound: encodeWait is ~90% of wall, while render+readback are ~2.3%. Two changes follow from it. 1. Hardware-first encoding. getEncoderPreferences() pinned Windows to software-first (2a2d7e7 "Stabilize video export on Windows"); software always succeeds, so hardware was never tried. export()'s retry loop already falls back to software on error, so preferring hardware only trades up when it actually works. Measured 94.6s -> 72.7s on os_parity, output verified frame-identical (ffprobe + extracted frames). 2. GPU-resident composite. The compositor rendered the video in Pixi/WebGL but flattened every other layer onto 2D canvases, then handed the encoder new VideoFrame(compositeCanvas /* 2D */) - a full GPU->CPU->GPU round-trip per frame. Now wallpaper, shadowed recording, webcam, cursor and annotations are all Pixi nodes rendered in one pass, and getCanvas() returns the WebGL canvas so the frame never leaves the GPU. The 2D path stays as the fallback for Linux/Wayland (where drawImage(webglCanvas) is unreliable) and for 3D-rotation frames, and GIF export opts out via disableGpuComposite since gif.js reads pixels back anyway. The webcam/cursor geometry both paths need is extracted into computeWebcamDraw()/computeNativeCursorDraw() so the math has one source of truth. Also adds the StageTimings harness wiring and a flag-gated encoder throughput probe: encodeWait stays ~90% even with GPU frames + hardware, so the encoder itself is capped ~10fps @1080p and the probe isolates whether that is the platform encoder or our feeding path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… encoder probe Tells SwiftShader (CPU fallback) apart from a real GPU-backed ANGLE context, which decides whether GPU-resident compositing can pay off at all here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-segment time-stretch ran serially after the video loop, adding its full CPU cost (~4s on os_parity) to export wall time. It depends only on the decoded PCM and the plan's speed regions - never on segmentFrameCounts, which is the one thing the video loop produces - so it can start as soon as the audio pre-pass is done. Kick it off before the video loop and await it in the audio phase. The loop spends ~90% of its wall time blocked in encodeWait while the encoder works on another thread, so the stretch runs in that idle time for free. It yields between segments so the loop can interleave, and the 120-frame encode queue is deep enough that a segment-sized stretch never starves the encoder. A new audioStretch stage measures the leftover await; audioEncode no longer includes the stretch. Implemented by the cc-delegate worker (MiniMax-M3); reviewed, empty-catch lint fixed, and verified here (tsc 0, biome clean, 31 exporter tests green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured on os_parity: it removed the readback (0.2ms/frame) but encodeWait did not move (90.6% of wall, ~8fps) - because the wall was never the frame feeding. An isolated encoder probe settled it: the bare WebCodecs encoder caps at ~23-36fps on 1080p synthetic frames, and GPU canvas frames are SLOWER into it than CPU RGBA ones (22.8 vs 35.6 fps). The premise that GPU-backed frames unblock the encoder is false here. So this bought ~0 while adding a second render path to maintain, and it was never visually verified. Reverting frameRenderer/gifExporter to the proven 2D composite. It only ever touched the export path - the preview (PreviewCompositor.tsx) is unaffected either way. Kept from the same work: hardware-first encoding, the StageTimings harness, and the encoder probe that produced the verdict. For the record, the real ceiling is the WebCodecs API itself: native ffmpeg encodes the same 546 frames at 165fps (h264_amf) / 198fps (x264) through a Node pipe, vs ~8fps here - a 20-25x gap that no compositing work can close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ncode Phase-0 measurement disproved this spec's central perf premise, so the perf half is rewritten around what the numbers actually say. What was wrong: - S2.3 claimed WebCodecs 'is the OS hardware codec - the same path native ffmpeg uses'. It reaches the same silicon, but measures ~8fps @1080p where native ffmpeg h264_amf does 165fps on the same GPU. - S3/P2 called the per-frame readback the 'dominant time sink'. Measured: 0.2ms/frame = 0.1% of wall. encodeWait is 90%. - S7 concluded 'a native ffmpeg swap would not beat it'. Wrong by ~20x. Evidence added in S3.1: per-stage timings across software/hardware/ GPU-resident runs, an isolated encoder probe (no decode/render/composite), and native ffmpeg benchmarks on the same 546 frames. Plus the two traps that produced false readings - getGPUFeatureStatus() from a windowless Electron script, and pipe benchmarks through MSYS's emulated pipe. What changes: - Goal 2 and S4.4 rewritten: keep the Canvas2D composite, replace the encoder. GPU-resident compositing cancelled (built, measured ~0, reverted). - S7 inverted: native LGPL ffmpeg primary with per-platform hardware encoder selection; WebCodecs demoted to fallback. S7.2 documents why an LGPL build keeps the app MIT (we build it, no --enable-gpl) and records the rejected alternatives (@napi-rs/webcodecs lacks AMF; node-av/beamcoder ship GPL). - Phasing redone: Phase 3 is native encode (>=5x gate); parallel encode and Worker/OffscreenCanvas demoted behind it. - D4 resolved, and not as written. New risks R6-R9 (bundle size, LGPL compliance drift, vendor encoder variance, subprocess lifecycle). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fine) Etienne's call. It was never written down, so it leaked into decisions as an unexamined assumption - Phase 3 told us to strip the ffmpeg build to ~30MB, and two rejected options were partly argued on weight. Stating it explicitly and fixing what it contaminated: - Phase 3: don't strip. Enable HEVC/AV1/VP9/ProRes now so a future format needs no rebuild + re-qualification. Clarify that --enable-gpl/--enable-nonfree are the only levers that decide LGPL vs GPL - stripping was never a licence control. - R6: size drops out; the real risks are the 3 build matrices and macOS notarisation, and the app already ships native binaries so that path is solved. - S7 rejected table: GStreamer's 'heavier' no longer counts against it (it still loses on the missing Node binding); the N-API addon's 'ships nothing extra' upside is moot, leaving it with no advantage at all. Decision unchanged - native ffmpeg subprocess still wins. Only the reasoning was resting on a constraint we don't have. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…not the encoder
The 165 fps native-ffmpeg figure was node->ffmpeg in ONE process. The real
export adds a renderer->main crossing per frame, which I had never measured.
Measuring it moves the bottleneck and cuts the expected win from ~20x to ~6x.
New in S4.4.1, all measured on the reference machine:
- Extraction: new VideoFrame(canvas2D).copyTo() = 1.43 ms/frame (698 fps),
5x cheaper than getImageData (7.00). Reading back a WebGL canvas is 11.23 ms
- 8x slower than a 2D one, which retroactively explains why the GPU-resident
composite never paid off. That revert was right for a reason we only learned now.
- Format: VideoFrame(canvas).format is BGRA and copyTo({format:'NV12'|'I420'})
throws NotSupportedError. Chromium will not convert, so we ship 7.9 MB BGRA
and let swscale convert - the 3.0 MB NV12 frame is not obtainable.
- Transport: a credit window (8 frames in flight) is worth +56% over
stop-and-wait (130 vs 84 fps). Zero-copy is impossible: MessagePortMain's
transfer list takes MessagePortMain[] only, and transferring an ArrayBuffer
renderer->main silently drops the whole message (electron#34905). Every frame
is copied => ~390 MB/s => ~49 fps at 7.9 MB/frame.
So: target ~95s -> ~16s (~6x), IPC-bound. Phase 3 exit gate relaxed to >=4x and
told to use copyTo + window=8. New Phase 4 = GPU BGRA->NV12 packing to halve the
crossing (~13x), gated on not reintroducing the slow GL readback. Parallel encode
demoted to Phase 5: the encoder now has 3x headroom over the crossing.
Also records the third measurement trap of this exercise: new VideoFrame(canvas)
is LAZY, so timing it measures nothing - the old 'readback is 0.1% of wall' claim
came from that. Added the rule the spec keeps re-learning: benchmark the stage in
the process topology it will really run in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s NV12 The previous numbers (49/131 fps) were extrapolated from a 3.0MB-frame run by dividing throughput. Measured directly, both sizes, with and without ffmpeg: IPC alone NV12 3.0MB 130 fps 387 MB/s IPC alone BGRA 7.9MB 53 fps 418 MB/s IPC+ffmpeg NV12 3.0MB 81 fps 240 MB/s IPC+ffmpeg BGRA 7.9MB 34 fps 268 MB/s Two findings the extrapolation could not have given us: - The crossing is BANDWIDTH-bound, not per-message-bound (387 vs 418 MB/s at wildly different frame sizes). This is what actually justifies Phase 4: halving the bytes really does double the rate. - Main is a serialisation point. Attaching ffmpeg costs ~40% of the crossing (418 -> 268 MB/s) because one process both receives the IPC message and writes stdin. A utilityProcess would NOT fix this - it only moves which process pays both halves. Also records that Buffer.from(typedArray) copies: wrapping the memory instead (Buffer.from(buf.buffer, byteOffset, byteLength)) is worth +31% (26 -> 34 fps). At 7.9MB x 546 frames a stray copy is 4.3GB of waste. So the honest headline drops from ~6x to 4.3x for Phase 3, and 13x to 10x for Phase 4. Purged the extrapolated figures from S1, S4.4.1, Phase 3/4 and D4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… LGPL gate) First piece of the native-encode path: pure, I/O-free capability detection so the streaming layer can stay focused on process handling and the tests never spawn anything. Mirrors electron/stt/gpuDetector.ts, which solves the same "resolve a bundled native binary per platform" problem. - candidateFfmpegPaths(): env override -> appPath -> resourcesPath -> cwd, same layering as gpuDetector, all inputs injected so tests never touch process state. - parseAvailableEncoders() / selectVideoEncoder(): win32 nvenc>qsv>amf, darwin videotoolbox, linux nvenc>vaapi; null means "keep WebCodecs". - isLgplBuild(): the gate that keeps this MIT app MIT. ffmpeg is LGPL by default and only turns GPL via --enable-gpl/--enable-nonfree, all-or- nothing. Checks the flags AND the GPL-only external list from ffmpeg.org/legal.html, plus the nonfree ones - libfdk-aac is a live trap since we encode AAC and it makes the binary unredistributable outright. Drafted by the cc-delegate worker (MiniMax-M3), which timed out on a stalled model call but salvaged usable work. Reviewed and corrected here: it had invented two justifications that contradict our measurements - that AMF has "visibly worse quality" (measured frame-identical to software) and that falling back to libx264 would "explode export time" (it is the fastest thing we measured at 201 fps; the real reason to refuse it is that it is GPL). Fabricated rationale in comments is worse than no comment, so both were replaced with what the numbers actually say. 44 tests, tsc 0, biome clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measurement left one wall standing: with native ffmpeg encoding, the
renderer->main crossing becomes the bottleneck (34 fps BGRA, 81 fps with NV12
packing) while the encoder idles at 165 fps. The crossing exists only because
the compositor runs inside a sandboxed Chromium renderer, and Electron has no
zero-copy way out (electron#34905). Removing it means compositing natively -
worth ~2x over Phase 4, i.e. 20x vs today rather than 10x.
Corrects the framing this started from ("keep Electron for UI, Tauri for the
backend"): they are both app shells, you pick one. The real question is where
the compositor lives and how the preview displays it, and that yields four
options of which only one works - a Rust/wgpu compositor serving BOTH preview
and export, with the preview drawn to a native surface. An export-only Rust
compositor is the tempting shortcut and the worst outcome: two implementations
of the whole visual identity, pixel parity owed forever. Pushing preview frames
back to the webview just reintroduces the same crossing in reverse.
Documents what ports cheaply (RenderPlan, the geometry/timing SSOT, audio - all
pure and already tested), what is the real cost (the wgpu compositor rewrite,
and the hidden one: the Electron->Tauri shell migration), and what survives
untouched (React UI, wgc-capture, cursor-sampler, whisper).
Recommends: ship Phase 4 first (10x, weeks, no architecture change, and none of
it is wasted if we later go native), measure 4K before deciding - every number
so far is 1080p and the crossing scales with pixels while the encoder degrades
more gently, so 4K could make this case much stronger - then spike one clip
with zoom+shadow+webcam frame-diffed against the current export before
committing months.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… against it
Review caught a real gap: the crossing was presented as an architectural
consequence of the sandboxed renderer. It is partly CONFIGURATION. With
sandbox:false + nodeIntegration:true the renderer IS a Node process - it can
spawn ffmpeg and write frames straight to stdin, V8 heap to pipe, no Mojo, no
structured clone. That is exactly what our own 165 fps pipebench measured. A'
reaches the same ~165 fps as the whole Rust/wgpu rewrite, for a boolean.
We still refuse A': the app deliberately runs nodeIntegration:false +
contextIsolation:true with the sandbox on, and the editor already spends its
budget on webSecurity:false. Trading defence-in-depth on the window that
handles user media for 2x is not a deal we want. (The threat model is narrower
than it looks - demux is WASM, decode is in the GPU process, ffmpeg only sees
rawvideo we produced - but an exploit would land on full Node instead of having
to escape a sandbox first.)
What A' does to this document: C is no longer "the only way to 20x", it is "the
way to 20x that keeps the sandbox". Same prize, different price. Judged against
A' rather than against A, its case is much narrower than the original draft
implied.
Also fixes two things the review was right about:
- C's parity column was a green tick. It is now a warning: one compositor gives
parity by construction only AFTER shadow, webcam mask, cursor and zoom easing
are re-implemented identically in wgpu - which is exactly where parity breaks.
The tick was a promise sold as a state.
- Retracted the "measure 4K, it makes this case stronger" argument. The encoder
slows at 4K too (~40-50 fps for h264_amf), so the gap narrows rather than
widens. It was an unmeasured extrapolation - the exact failure mode this spec
keeps warning about.
One correction back: the review's headline ("Phase 4 gives 131, so you buy ~1
second") read the IPC-only row. End-to-end with ffmpeg attached it is 81 fps,
because the receiving process also writes stdin (-40%). The stake is 2x - 3.4s
on os_parity, ~4 minutes on a 10-minute export - not one second.
Adds the free levers to try before paying for anything: N parallel MessagePorts
(untested; if the crossing reaches ~600 MB/s, Phase 4 goes encoder-bound and C
loses its reason to exist) and a worker_thread for the stdin write. A larger
credit window is already ruled out - measured 131 at window=32 vs 130 at 8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…does not solve it
Etienne caught a real inconsistency: I refused A' (sandbox:false) on security
grounds while recommending C (native Rust core) - which is also unsandboxed. The
objection does not transfer, and pretending it did was the weakest part of this
document.
Laid out honestly in a new S3.2:
A + Phase 4 : demux in WASM (confined) + decode in the GPU process (sandboxed)
-> a parser bug must be CHAINED with a sandbox escape
A' : media path unchanged; but ANY renderer bug now reaches Node
C : demux/decode become native ffmpeg C, in-process, NO sandbox
-> any ffmpeg demuxer CVE lands on the machine
Read the media column: today the riskiest parser runs in WASM. A native core
would replace it with unsandboxed C parsing untrusted containers. On the media
path specifically C is a REGRESSION against both A and A'. Media demuxers are
among the most CVE-dense code there is, which is precisely why browsers confine
them.
The honest distinction is narrower than "C keeps the sandbox" (deleted, twice):
A' removes a layer wholesale with no recourse, whereas C hands us the isolation
DESIGN - sandboxed child process for demux/decode, memory-safe Rust for our own
logic, a webview with no Node (Tauri's IPC is a command allowlist). That is a
capability, not a property. Built without it, C costs months to end up less safe
than today. So it is now a blocking requirement, not a nice-to-have.
Also records A'' (S3.1): electron#34905 says the zero-copy transfer that fails
renderer->main works renderer->renderer. That suggests keeping demux/decode in
the sandboxed renderer and transferring decoded frames - our own pixels - to a
second sandbox:false window that spawns ffmpeg, leaving the lock where the
hostile input actually arrives. Explicitly flagged as speculation: unmeasured,
and a renderer with no untrusted input still runs Blink/V8. Worth an hour before
anyone spends a month on C.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…S exception
Etienne's call: keeping WebCodecs as the primary encoder on macOS "because the
gap is smaller there" is not an option. The constraint was already in S1 ("no
platform gets a degraded export") and the shortcut never actually reached a
spec - it only came up while relaying a research report's recommendation. But
unwritten is not protected: the argument is seductive and would come back, so
it is now an explicit non-goal WITH its rebuttal, rather than an absence.
The rebuttal, recorded so nobody re-derives it: WebCodecs really is closer to
native on VideoToolbox (~3.5x off per w3c/webcodecs#492, vs ~20x on our
Windows/AMD reference), and skipping macOS would dodge real work - BtbN
publishes prebuilt LGPL binaries for Windows and Linux but NOT macOS, so macOS
is the one platform we must build and notarise ourselves. That is a convenience
argument dressed as an engineering one, and the answer is still no.
Consequence, now written into Phase 3 as a per-platform table: Windows and
Linux (x64+arm64) vendor BtbN's prebuilt LGPL builds - AMF/NVENC/QSV already
compiled in, so that is verify-and-vendor. macOS we own end to end: build, sign,
notarise. A VideoToolbox-only macOS build is inherently x264-free so the licence
gate is easy there; the cost is the pipeline, not the licence.
Also adds the verification recipe for any binary we vendor regardless of source
(banner must say "License: LGPL version 2.1 or later"; configuration: line free
of --enable-gpl/--enable-nonfree; -encoders free of libx264/libx265), and points
it at isLgplBuild() in ffmpegCapabilities.ts so CI can enforce it - a "just add
x264 for the software fallback" patch must not be able to silently relicense the
app.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second piece of the native-encode path: spawn the bundled ffmpeg, stream raw frames into its stdin, surface progress, and cancel cleanly. ffmpeg muxes and writes the file itself, so this eventually retires both the WebCodecs encoder and the JS muxer on the export path. Two details carry measured weight and are commented as such: - writeFrame does NOT copy. Buffer.from(typedArray) duplicates the frame; wrapping the caller's memory instead measured +31% end-to-end (26 -> 34 fps). At 3-8 MB a frame that copy is gigabytes of memcpy across one export. - writeFrame awaits 'drain'. The pipe sustains ~500 MB/s only because the writer paces itself; ignoring the false return just buffers frames in memory. parseProgress requires real trailing whitespace rather than end-of-string. A chunk ending in "frame=12" is the head of "frame=1234", and accepting end-of-string would report 12 and make progress jump backwards on the next chunk. Caught by its own test - the first implementation had exactly that bug. cancel() kills the process tree (taskkill /T /F on Windows, where a bare kill() leaves ffmpeg orphaned holding the output file) and leaves finish() able to settle, so a cancelled export cannot hang on a process that will never exit. stdin EPIPE after a kill is swallowed: expected, and it must not reach the main process as an unhandled error. Tests drive a fake ffmpeg fixture rather than the real binary, which is not bundled yet - hermetic and fast. Byte conservation is asserted for real: 40 MB written in frame-sized chunks, and the fake reports the exact total it received back through the stderr tail, so a broken backpressure path fails the test. Written here rather than delegated: three cc-delegate workers in a row were killed by MCP server restarts with zero output, after the spec had been written twice. 64 tests, tsc 0, biome clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Provisions ffmpeg into electron/native/bin/<platform>-<arch>/ next to wgc-capture and the whisper binaries - gitignored, shipped via extraResources, so it is fetched at build time rather than committed. Wired into build:win; linux follows the same script. Supply chain, because this binary gets signed and shipped to every user: - Pinned to an immutable dated tag, not `latest` (an alias that moves daily). - Pinned to a release-branch build (n8.1.2), not the N-* master snapshots BtbN also publishes. Master is not a release. - 8.1.x deliberately: it is the version the 165 fps h264_amf benchmark was taken with, so what we ship is what we measured. - SHA-256 verified before the archive is opened; GitHub exposes a per-asset digest so the pin needs no trust-on-first-use. - Source approved by Etienne. BtbN is linked from ffmpeg.org's download page and published on winget as BtbN.FFmpeg.LGPL. The licence gate is the point of the script. ffmpeg is LGPL by default and turns GPL only via --enable-gpl/--enable-nonfree, all-or-nothing - one GPL component relicenses the whole binary and would contaminate this MIT app. So we take the *-lgpl asset AND verify it before vendoring, on three independent signals: `-L` must print "Lesser General Public License"; the configure flags must carry no --enable-gpl/--enable-nonfree nor any GPL/nonfree external (list from ffmpeg.org/legal.html); and `-encoders` must not expose libx264/libx265. Verified both directions: ACCEPT on the vendored LGPL build, REJECT on the system gyan.dev build, which trips all three signals. The gate also caught its own first version - it looked for a "License:" line in `-version`, which does not exist (only `configuration:` does; the licence text is behind `-L`). It failed closed and refused to vendor, which is the right direction to fail in. Result on this machine: n8.1.2 LGPL, 139 MB, exposing h264_nvenc, h264_qsv and h264_amf. macOS is deliberately not handled: BtbN has no macOS target, so it must be built and notarised separately (spec Phase 3). The script exits with that message rather than silently leaving darwin without an encoder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The encode session was video-only (-an), which would have shipped a silent export once ffmpeg started writing the file. Give it the assembled PCM as a second input instead, and it encodes AAC and muxes in the same pass. Raw f32le, not WAV: the PCM is already in memory in exactly that layout, it needs no header, and it skips the int16 round-trip a WAV would force. (The existing electron/stt/wav.ts is deliberately mono/16k for whisper - reusing it would mean either quantising or generalising an STT file for no gain.) This is what lets ffmpeg own the output, which in turn retires the JS AAC encoder AND mediabunny: the JS side ends up producing frames and PCM, nothing else. Argument order matters and is now asserted: every input must precede any codec option, and pipe:0 before the PCM is what makes video stream 0 and audio stream 1. Verified against the real vendored binary, not just the arg builder - NV12 frames on stdin plus f32le PCM produced: stream 0: h264, 2.000000s stream 1: aac, 48000 Hz, 2ch, 2.000000s A/V exactly aligned, one pass, exit 0. 66 tests, tsc 0, biome clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…coders selectVideoEncoder picked from what `ffmpeg -encoders` reports, which is what was COMPILED IN, not what the machine can run. Our bundled portable build carries every vendor at once, so on the AMD reference machine it elected h264_nvenc. Verified against the real binary: h264_nvenc FAIL Cannot load nvcuda.dll h264_qsv FAIL Error creating a MFX session: -9 h264_amf OK h264_mf OK libopenh264 OK That would have broken export on every AMD and every Intel machine while passing all its unit tests, because the fixtures encoded the same wrong assumption. The spec called for "a 1-frame smoke encode" and the module simply did not do it; testing against the real binary rather than my own fixtures is what surfaced it. selectVideoEncoder is replaced by three pieces that keep the decision provable: - candidateVideoEncoders(): the shortlist, explicitly NOT a choice. - smokeTestArgs(): a ~100ms one-frame encode from lavfi to null - no files touched - that answers the only real question. - pickWorkingEncoder(): first candidate that survives it, with the probe injected so the logic stays pure and testable. Also drops the null-means-keep-WebCodecs contract, per Etienne: there is no WebCodecs fallback any more. Every list now ends in libopenh264 (BSD, bundled, LGPL-safe), so "no hardware" still exports through the same code and mux, just slowly. null now means the bundled ffmpeg is broken - a hard error, not a fallback. libx264 stays refused even though it is the fastest thing we measured (201 fps): GPL would relicense this MIT app. Adds h264_mf (Media Foundation) ahead of software: it reaches AMD/Intel/NVIDIA through the OS without a vendor SDK, so it can save a machine whose vendor path is missing. Proven end to end on the AMD box: shortlist nvenc>qsv>amf>mf>openh264, probes nvenc FAIL, qsv FAIL, amf OK, elected h264_amf. 68 tests, tsc 0, biome clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Composes the two pure modules into something the renderer can drive: resolve the bundled ffmpeg, prove an encoder works on this machine, own the live sessions, and clean up. The renderer stays sandboxed, so it cannot spawn ffmpeg itself. Frames cross to main and we feed stdin. That crossing costs ~390 MB/s and it is not fixable here - the compiler caught me trying: Electron's transfer list takes MessagePort[] only, and an ArrayBuffer transferred renderer->main silently drops the whole message (electron#34905). It is send() and its copy, which is exactly what the 34/81 fps measurements assume. Comment left at the call site so the next person does not retry it. Capability resolution is cached per process and driven by a real one-frame smoke encode, not by -encoders: our portable build lists nvenc/qsv/amf everywhere, so on this AMD box nvenc is "available" and dies with "Cannot load nvcuda.dll". No usable encoder is a hard error, not a fallback - there is no WebCodecs path behind this any more, and libopenh264 is bundled, so failing here means the binary itself is broken. Audio rides as a temp f32le file handed to ffmpeg as its second input, deleted with the session. will-quit cancels every live session, or quitting mid-export leaves an orphaned ffmpeg holding the output file. Frames are send() rather than invoke(): one-way, nothing to return, and flow control belongs to the renderer's credit window - acked per frame, including the last, or the window stalls short of the end. A write failure reports on the ack channel rather than throwing inside ipcMain.on, where it would be an unhandled rejection and the renderer would wait forever. 231 tests, tsc 0, biome clean. Renderer wiring next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion A Driving the export through the UI cost ~5 minutes a run and kept injecting confounds: one A/B ran with DevTools open on one arm only, another ran on a laptop at 5% battery whose SoC budget drifted 26% between two IDENTICAL runs -- enough to invert the conclusion. So: one command, real pipeline. npm run bench:export -- --project=<id> --arms=webcodecs,native --runs=2 It simulates nothing. It opens the real editor window (same webPreferences, same preload, same sandbox), loads a real saved project through the same bridge the editor uses, and calls exportAxcutDocument -- ExportDialog's own entry point. Only React is skipped, so nothing renders alongside the export. Arms interleave (A,B,A,B) and same-arm spread is reported: over 10% the run declares itself VOID rather than let drift pose as an effect. Every guard in the runner exists because its absence already produced a confident wrong answer: a stale main bundle (twice -- once the export IPC was "not registered", once the bench flag silently did nothing), a leftover instance holding the single-instance lock (launch exits 0 in silence), a dead dev server, and a runner that swallowed the app's own stderr while it was explaining the failure. What it measured, at 3-4% spread on proj_a7468696 (1418 frames, 1080p60): webcodecs 38.5s 36.9 fps readback 158ms native 80.8s 17.5 fps readback 55207ms native-cpu 83.6s 17.0 fps readback 57989ms webcodecs-cpu 38.0s 37.3 fps readback 166ms Native ffmpeg is 2.1x SLOWER, and willReadFrequently does not move it (the arm is real: the renderer now reports what Chromium GRANTED, not what we asked for -- granted=true, unchanged). ffmpeg is not the problem: it consumes frames FASTER (encodeWait -29%, flush -94%, 1444ms -> 80ms). The problem is the price of admission. WebCodecs encodes straight off the GPU texture and never brings a frame to the CPU; ffmpeg needs the pixels in RAM, so we pay a descent per frame that WebCodecs never pays. Two resolutions give that descent a shape: 38.9ms/frame at 1080p (8.29MB), 21.0 at 720p (3.69MB) => ~6.6ms fixed + 3.89ms/MB. So NV12 (3.11MB) extrapolates to ~18.7ms -- halving the readback and still landing under WebCodecs. Phase 4 cannot save this path either. The v2 spec's premise is now inverted twice over: readback was never the cost until we forced it, and forcing it is exactly what feeding a native encoder requires. A native encoder only pays off if the frame never descends -- which means compositing and encoding on one device, i.e. the Tauri/Rust core (option C), not a bundled ffmpeg fed from a sandboxed renderer (option A). Measured on one machine (Ryzen 5 7520U iGPU, h264_amf). 952 tests, tsc 0, biome clean.
…annot win Writes up the end-to-end measurement of option A (bundled ffmpeg fed from the renderer) and marks the specs it refutes, so nobody implements Phase 3 or Phase 4 from a plan the evidence has already overtaken. The finding: 2.1x SLOWER than WebCodecs (38.5s -> 80.8s, 1418 frames, 1080p60, same-arm spread 3-4%). ffmpeg is not at fault -- it consumes frames faster (encodeWait -29%, flush -94%). It needs the pixels in RAM, and that descent costs 38.9ms/frame that WebCodecs never pays, because it encodes straight off the GPU texture. The v2 spec's "readback is 0.1% of wall" holds only while nothing forces the descent; feeding a native encoder is what forces it. Two hypotheses closed without guessing: - willReadFrequently does nothing (granted=true, verified -- the renderer now logs what Chromium GRANTED, not what we asked, so a no-op arm can't pass for a tested one). - NV12 cannot rescue it. Two resolutions give the descent a shape (6.7ms fixed + 3.9ms/MB, 257 MB/s marginal => a sync, not a copy), which extrapolates Phase 4 to ~29.5 fps vs 36.9 measured for WebCodecs today -- parity at best, after building GPU shader packing. Consequences recorded where they'll be read: v2 spec header + Phase 3 + Phase 4 now carry the refutation and link here; the Tauri spec is re-framed -- C is not "the last x2", it is the only arrangement where native encode can pay, and its load-bearing assumption (composite and encode on one device, frame never descends) is UNMEASURED and must be prototyped before any migration. The report also documents the two runs discarded on the way -- a 26% and a 62% same-arm spread that inverted the conclusion between them -- because they are why the bench interleaves arms and gates on spread, and why a single A/B on this machine was never capable of answering the question. Also recorded (unrelated, data-loss): os_parity's project file is corrupt -- 3485 bytes of valid JSON followed by the tail of a longer version, i.e. two concurrent writeFile calls. writeProject truncates correctly; what's missing is save serialisation + atomic write. Not repaired: the recoverable prefix has lost the speed and zoom regions, so "repairing" it would have silently stripped the timeline. Byte-exact backup left beside it.
Open question from the report: we never tested A' (sandbox: false, renderer spawns ffmpeg and writes its stdin directly), which removes the IPC structured clone. Nor shared memory, nor a zero-copy transfer. None of them needed building. They all keep copyTo() -- ffmpeg needs the pixels in RAM whoever spawns it -- so one arm bounds every one of them at once: descend every frame, then throw it away. No IPC, no ffmpeg, no muxer, no audio, no file. webcodecs 32.2s 44.0 fps readback 170ms encodeWait 20997ms readback-ceiling 35.0s 40.5 fps readback 29210ms encodeWait 2ms native 69.1s 20.5 fps readback 34389ms encodeWait 26787ms (spread 1-2%; encodeWait=2ms over 1418 frames confirms the crossing really is zero -- the arm does what it claims) The ceiling LOSES: 40.5 vs 44.0 -- and WebCodecs is also muxing and writing the file, while the ceiling writes nothing. The descent alone, with nothing behind it, is slower than WebCodecs doing the whole job. So no engineering on the crossing can save this shape. A' is strictly worse than that ceiling anyway: it does not remove the crossing, it swaps a structured clone (~390 MB/s) for a pipe write (~500 MB/s) -- ~1.2x on one leg, bought by giving up the sandbox that protects demux/decode of untrusted user media. Even A' + NV12 projects to ~37.6 fps, still under 44.0. This corrects the previous commit's own conclusion: the wall is NOT the crossing. Set the crossing to zero and the shape still loses. The wall is the DESCENT -- any design where a composited frame reaches system RAM has already spent more than WebCodecs spends on the entire export. That is the whole case for option C (composite and encode on one device, frame never descends) and the reason its load-bearing assumption must be prototyped before any migration. Report §5 + §1 + roadmap updated; v2 spec banner and Tauri spec now carry the exclusion. 952 tests, tsc 0, biome clean.
… is the compositor The whole investigation was chasing the wrong stage. Measured, in one run: composite-ceiling 24.6 ms/frame <- decode + composite, NOTHING downstream readback-ceiling 24.6 ms/frame <- the same + a full copyTo() webcodecs 31.1 ms/frame <- the same + encode + mux + file Adding the entire GPU->CPU descent moves the wall by 0.03 ms/frame. The 32 SECONDS of "readback" this branch has been reporting, documenting and building reports around was never a transfer: it was the GPU finishing the composite. new VideoFrame(canvas) is lazy, renderFrame only SUBMITS, so the first stage to force a sync gets billed for everything queued before it. `render` claims 2.06 ms/frame while the compositor costs 24.6 -- it under-reports by 13x because it times submission, not execution. That one fact explains every dead end on this branch: latencyMode, the GPU flags, willReadFrequently, the GPU-resident composite, and native ffmpeg at 0.48x. None of them could move a wall none of them touched. And it retires "~13 ms of Chromium overhead we do not control" -- it is our compositor, and we control all of it. So: the compositor is 79% of the export's wall. Three things it rebuilds every frame that never change: - Texture.from(videoFrame) + oldTexture.destroy(true): allocate, upload and free a 1080p GPU texture per frame. Every VideoFrame is a new object so nothing caches. Now the resource is swapped into the texture we already have; a size change (segments differ) still rebuilds. - maskGraphics.clear()/roundRect()/fill(): retessellates a rounded rect that is identical frame to frame unless the layout moves. Now keyed on the shape. (A zoom easing changes it per frame -- that case this cannot help.) - clearRect(w,h) before a drawImage(w,h) that covers the whole canvas: two full-frame passes where "copy" does one. Kept the clear where nothing is drawn over it, which is the only branch that needed it. Measured interleaved, old vs new, one thermal state (composite-ceiling): legacy 23594 / 26929 / 28646 ms new 20085 / 20862 / 21511 ms The bench VOIDS itself on spread (19% on the legacy arm; this machine has been under load for hours), so the exact figure is not claimable -- but the samples are disjoint: the worst new run beats the best legacy run. Median -22%, bounded [-9%, -30%]. Direction certain, magnitude approximate. Said plainly rather than picked from. Parity is not argued, it is gated: exporting the same timeline through the same encoder with each compositor produces files that are IDENTICAL BYTE FOR BYTE (SSIM 1.000000 across all 1418 frames). 952 tests, tsc 0, biome clean. Also adds the arms that made this measurable (composite-ceiling, *-legacy) and fixes a bench guard that only looked for electron.exe: the INSTALLED app runs as openscreen.exe and holds the same single-instance lock, so the bench launched, exited 0 and reported nothing -- silently, which is the one thing the guards exist to prevent. Not touched, and next: the shadow (three chained drop-shadows over 2M pixels per frame, to compute a function of (rect, radius) alone -- drop-shadow reads only SourceAlpha, and videoCanvas's alpha IS the rounded rect) and the wallpaper (a static image re-blurred 1418 times). Both are OFF in this project, so fixing them here would have measured nothing -- they need a project that enables them.
The recording's shadow is three chained CSS drop-shadows applied to the full 1920x1080 frame, every frame. Measured on the real compositor: ~30 ms/frame, twice the cost of everything else in it put together. It computes nothing that changes. drop-shadow reads SourceAlpha only, and videoCanvas's alpha is just the sprite masked by a roundRect — so the filter blurs 2M pixels of VIDEO, three times, to produce a function of (rect, radius, intensity) and nothing else. The video pixels do not reach the result. So it now runs against a silhouette instead, cached on the geometry. The silhouette is taken from videoCanvas's own alpha (drawImage + source-in over black), not re-derived from the layout: same pixels, same anti-aliasing, nothing to keep in sync as the layout code evolves. The filter output IS `silhouette OVER shadow`, so drawing videoCanvas over it covers the silhouette exactly — including the anti-aliased corners, which is the part I expected to lose and did not. Measured interleaved, one thermal state, shadow+radius on (bench --effects): composite-ceiling-legacy 39.45 ms/frame 25.4 fps spread 2% composite-ceiling 20.22 ms/frame 49.5 fps spread 8% The compositor is 2x faster (-48.8%), and the cache does what it says: 1417 hits / 1 miss over 1418 frames — the filter runs once. Parity is gated, not argued: the same timeline exported through the same encoder with each compositor gives files that are IDENTICAL BYTE FOR BYTE, SSIM 1.000000 across all 1418 frames. Not a pixel moved. Also folds the filter chain into one helper. It was duplicated verbatim on the flat and the 3D path, which is how they would drift. The 3D path is NOT cached and stays at full price on purpose: it blurs foregroundCanvas, whose alpha carries the webcam, cursor and annotations, so it has no stable geometry to key on. Restructuring that is separate work. The bench grew what was needed to measure this honestly: - --effects=shadow,blur,radius,zoom patches an in-memory COPY of the document. Saved projects carry no appearance at all (shadowIntensity defaults to 0), so entire effects never execute and "fixing" them measured exactly zero. Nothing reaches disk. - zoom matters beyond its own cost: it is the one effect that changes the composited geometry per frame, so it is what invalidates a geometry-keyed cache. A parity test without it would pass with a broken key, because nothing would ever ask the cache to invalidate. 952 tests, tsc 0, biome clean. Per-effect cost on the real compositor, for whoever picks this up (isolated, spread 3-4%): shadow 43.5 ms/frame, blur 17.9, radius 13.3. The wallpaper blur (a static image re-blurred 1418 times) is the next one, and it is ~5 ms/frame.
backgroundSprite is rasterised once at load and never touched again — it is a still image. It was then cleared, re-blurred and re-blitted on every frame. Measured at ~5 ms/frame for a result that is identical every time. Hoisted: the blur runs once into a w*h canvas and the frames blit it. Identical by construction — the original blurs while scaling to w*h, so the blur already landed in destination space; doing it once and blitting 1:1 is the same operation, moved. Keyed on the sprite's identity, not just its size, so loading a different wallpaper cannot keep serving the previous one's blur. Parity gated as before: same timeline, same encoder, blur+shadow+radius on, old compositor vs new -> files IDENTICAL BYTE FOR BYTE, SSIM 1.000000. On the numbers, and this matters more than the number: the absolute fps of this machine is NOT reproducible across runs. The same arm, same project, same settings measured 44.0, 36.8, 32.3, 31.8, 22.2 and 11.9 fps across sessions. The 11.9 run reported 0% spread over its two samples -- it looked airtight and did not survive a re-run 40 minutes later (22.2). A stable measurement is not a true one, which is the same lesson this branch has now learned from the pipe probe, from copyTo, and from `render`. So only WITHIN-RUN ratios are claimable, and those agree: composite-ceiling, shadow+radius 39.45 -> 20.22 ms/frame 1.95x webcodecs, blur+shadow+radius 67717 -> 33707 ms 2.01x webcodecs, shadow+radius 63825 -> 39839 ms 1.60x The compositor fixes are worth ~1.6-2.0x on the export, output unchanged. What is cached now, classified by what invalidates it rather than by layer: never wallpaper blur geometry drop shadow, rounded-rect mask, video texture every frame video, cursor, webcam, annotations The residue is honest: a zoom region changes the geometry every frame, so the shadow cache misses for its whole duration and pays full price there. That is the case a single-pass shader would fix, and it should be argued on what remains after this, not before. 952 tests, tsc 0, biome clean.
Three architecture docs disagreed with each other and, on the performance half,
with reality — each was written on top of the previous one's wrong conclusion,
and all three still asserted in bold that the wall was somewhere it is not.
Replaced by docs/architecture/export-pipeline.md.
export-pipeline-v2-spec.md deleted
native-core-tauri-spec.md deleted
export-native-encode-measurement.md deleted
What survives, because it is true and shipped: the RenderPlan and segment loop,
the two-clock split (output time to the encoder, source time to the renderer —
virtual-time-everywhere does not survive speed regions), the A/V lock sized from
real frame counts, per-segment cursor, and the LGPL gate.
What changed: the compositor is 79% of the export's wall. Not the encoder
("~90% of wall"), not the readback ("0.1%"), not the descent ("38.9 ms/frame"),
not the crossing. One wall, and every earlier number was it wearing the name of
whichever stage happened to force the first GPU sync. `render` reported 2.06
ms/frame for work costing 24.6 — under by 13x, and by 41x with a shadow on.
§5 records what that mistake cost, so the arguments are not re-run: native
ffmpeg from the renderer (2.1x slower), A' sandbox:false (excluded by a ceiling
measurement, not by opinion), NV12 (cannot rescue it), the CPU compositor
(parity at best — and anchored on an fps figure that has since been retired),
Worker+OffscreenCanvas, GPU-resident compositing. Plus the Tauri framing: it is
not Node vs Rust and not Electron vs Tauri, it is "compositor in a browser
engine vs in our own GPU code" — reachable from Electron, and the shell is a
consequence, not a cause.
§4.2 is the one to read before quoting any number from this machine: the same
arm has measured 44.0, 36.8, 32.3, 31.8, 22.2 and 11.9 fps across sessions, and
the 11.9 run reported 0% spread. Only within-run ratios are claimable.
§6 lists the nine traps that each cost hours: the windowless GPU probe, the MSYS
pipe, lazy VideoFrame, isolated benchmarks that cannot price their own
connection, -encoders vs what runs, electron#34905, Buffer.from copying, stale
dist-electron, and the installed app holding the dev lock.
Also repoints the code comments that outlived the docs — one still claimed
dropping the sandbox "buys ~2x", which is refuted — and marks fetch-ffmpeg.mjs
as bench-only, since the plan it was vendored for is dead while the binary is
still what the bench's native arms run.
952 tests, tsc 0, biome clean.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
EtienneLescot
force-pushed
the
feat/ai-edition
branch
2 times, most recently
from
July 26, 2026 15:25
b43adda to
92fbe07
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This Draft explores editable cursor choreography as an AI Edition idea/prototype.
recorded,1x) and create an editable range from the playhead to the next click, split at detected rests and clicks.recorded,straight,arc,wave,loop, andovershootpresets, with 1x-4x speed, easing, and cycle controls.Why
The AI Edition transition retained some zoom primitives but removed the previous automatic placement flow and the global cursor-follow mode from the primary V4 workflow. It also had no first-class way to author cursor movement without changing the underlying recording.
This prototype tests whether cursor movement can become a non-destructive, per-clip timeline effect while keeping untouched footage exactly as recorded.
Prototype UX
Architecture
cursorMotionRegionsis backward-compatible with a default empty array. Old assets default toautoZoomState: processed; newly imported assets are markedpendingso automatic zoom runs once without modifying old documents on load.clipIdandassetId. It stores both virtual and source boundaries, and regions that cross or disagree with their render segment are rejected.FrameRenderer; source changes clear the previous asset's telemetry and focus state.Validation
npx tsc --noEmit- passednpm run test:browser- 2 files, 6 tests passednpm run i18n:check- passed; all 12 translations match English across 7 namespacesnpm run test- 98/99 files and 1,010/1,012 tests passed. The two failures are existing Windows timeouts inelectron/media/ffmpegEncodeSession.test.ts(cancel()/post-cancel writes); the implementation and test files are unchanged from base commit4c2f842and also fail in isolation.Known limitations and open questions
Type of change
Release impact
Desktop impact
Screenshots / video
No attachment yet. The Windows Electron smoke confirmed the toolbar and timeline behavior; a polished demo should follow after the native cursor sampler is available in the test checkout.