feat(compositor): port the compositor to macOS (Metal + VideoToolbox) - #204
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ing back to WARP PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP cannot serve this pipeline at all -- and not for the expected reason (speed): - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns DXGI_ERROR_UNSUPPORTED (0x887A0004). - Drop the flag and WARP creates at FL 11_1, but QueryInterface for ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles. pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so preview and export decode every frame on it. A WARP device would produce none: the retry would only convert a clear startup failure into an obscure ffmpeg one. crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and fails if Windows ever changes them. So there is no fallback, and the answer to "WARP for export or preview only?" is neither. What the compositor does instead is fail in a way the user can act on: - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT) to tell "this adapter has no video decoder" -- the RDP / VM case -- apart from "no FL 11_1 adapter at all", and says which, plus what to do. Export gets this for free: ExportDialog already renders the native message. - That message now reaches the preview too. create_view returns an id long before the render thread can die, so the failure existed only as an eprintln! and the user just saw a black canvas. The thread stores its fatal error in live::Shared, read_frame relays it as an Err on the next pull (~33 ms), and NativeCompositorOverlay renders it in place of the canvas. The WGC capture helper stays hardware-only on purpose: it requests no VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host whose compositor cannot start only produces footage the user can neither edit nor export.
The previous paragraph justified leaving the WGC helper hardware-only with "recording on a host whose compositor cannot start only produces footage the user can neither edit nor export". That premise is false: Windows recording already falls through to getDisplayMedia + MediaRecorder when startNativeWindowsRecordingIfAvailable returns false, and the compositor is built to ingest that output -- allow_d3d11va_h264_baseline exists because Chrome's MediaRecorder emits plain H.264 Baseline. The conclusion survives, with a better reason: capture already has a fallback that needs no D3D device at all, so a WARP device there would add a slow path alongside a working one. But the correction exposes a real gap, now documented rather than asserted away: that fallback is only reachable through the pre-flight probe, and is-native-windows-capture-available only checks the Windows build and whether the helper binary is on disk -- it never touches D3D. A host whose helper fails createD3DDevice is told available:true, commits to native, and then startNativeWindowsRecordingIfAvailable rethrows instead of returning false, so the browser path is never reached. Recording fails next to a route that works. Not fixed here: by then the renderer has released the webcam preview stream for the helper's exclusive session, so a bare `return false` would silently record screen-only. A correct fix re-acquires that stream on the fallback route, or makes the probe truthful.
… iso output
Rendering and decoding are two independent axes, and no platform has a software
rasteriser that also decodes video (WARP here, lavapipe on Linux, nothing at all
on macOS). So the CPU fallback needed both halves, not just a driver-type swap:
- d3d::Backend::{Hardware, Cpu}. Cpu = D3D_DRIVER_TYPE_WARP without
VIDEO_SUPPORT, which WARP rejects outright.
- cpu_frames.rs: libavcodec software decode -> swscale to NV12 -> upload into
one owned D3D11 texture, presented as an AVFrame whose data[0]/data[1] are
the texture and its slice.
That last part is the whole design. Everything the compositor knows about a
decoded frame is compositor::nv12_srvs and compositor::tex_dims, which read four
fields. Fill those and compositor.rs, every HLSL shader and the scene contract
stay untouched -- a Metal or Vulkan port replaces d3d.rs and cpu_frames.rs and
nothing else.
Measured, C1..C8 on the frozen fixture:
- iso render: 93-95% of channels bit-identical to the GPU path, max deviation
3/255, matching mean levels (neither frame is blank). Every effect layer
survives the swap; the residual is rasteriser/FP difference.
- two shaders own the entire speed gap, both multi-tap sampling loops:
background blur +4.53 ms on hardware vs +75.43 ms on WARP (17x), motion blur
+2.30 vs +52.65 (23x). Everything else is within ~2.2x of the GPU.
- so C1-C3 hold ~30 fps (a usable preview) and C4+ sit at 6-9 fps.
Benchmarking it needed a preview-shaped harness: the CPU backend cannot reach
the export path, since h264_amf requires the real GPU. --preview measures
decode -> compose -> readback with no encoder, --backend picks the device, and
each run writes a PPM so a backend that composes black cannot post a flattering
fps. C0 is excluded from preview mode -- "decode + encode, no composite" has no
meaning without an encoder.
swscale was already linked but never bound; wrapper.h and the bindgen allowlist
now cover it. Preferred over a hand-written UV interleave because it also
handles 10-bit and 4:2:2, which a hand-rolled loop would corrupt silently.
Nothing selects this backend automatically -- Gpu::create still returns the
hardware device. A silent switch to a 6-9 fps preview is the same "the app is
slow today" failure the rest of this branch removes. The effect policy C4+ needs
is likewise not wired.
…user
Everything for the fallback existed; nothing chose it. Gpu::create_auto now prefers
the hardware device and falls back to Backend::Cpu, and the production paths (live
render thread, napi export) use it. Gpu::create stays hardware-strict for tests and
goldens, where measuring the GPU path on a software rasteriser would be meaningless.
The switch is never silent, which was the whole objection to an automatic fallback:
- create_auto logs why the hardware device was refused, via diagnose() -- "this
adapter has no video decoder" (RDP, VMs, Basic Render Driver) vs "no FL 11_1
adapter at all". That distinction is what tells a user whether a driver update
fixes their machine.
- probeBackend() answers "hardware" | "cpu" | "none" without allocating a view,
because the export dialog needs it before any preview exists. Cached both sides:
it is a property of the machine, and creating a device is not free.
- The preview carries a small persistent notice, and the export dialog warns before
the export starts. Persistent rather than a toast in the preview: the question it
answers ("why is this choppy?") recurs every time the user looks at it.
NO effect is disabled on the CPU path. The earlier plan was to force background blur
and motion blur off, since they cost 17x and 23x on WARP and are what pull C4+ down to
~8 fps. Dropped: 8 fps is what the 1.7.0 preview delivered, so users have already lived
with it, and disabling effects would trade away the iso-render property (max deviation
3/255) that makes a backend swap worth having. Slow and correct beats fast and wrong,
in both preview and export.
"none" is not a warning. It means there is no native compositor at all -- pure-web
dev, jsdom, an addon that failed to load -- which is the normal state in development;
warning there would put "no compatible GPU" in front of every developer on every run.
Only "cpu" is a degraded machine, and the probe resolves to null first so a machine
that turns out to have a GPU never flashes the notice. Both cases are tested.
Verified through the real addon, not just the mocks: probeBackend() returns "hardware"
on this machine and stays cached on the second call. 1144 vitest + 78 Rust tests pass.
The known-gaps entry in native-compositor.md said the compositor was "unusable on
virtualised environments" and that nothing fell back -- both were true when written and
are now the opposite. Rewritten rather than appended to.
Rebasing onto release/v1.8.0 landed on top of "feat(export): pick the video encoder
from the host's hardware, with a software fallback", which already does the encoder
half of this properly and more generally than the version on this branch did -- NVENC
and QSV hosts too, not just AMF -> libopenh264, and with the same NV12 -> YUV420P
de-interleave. That commit was dropped in the rebase rather than merged; this is what
was actually still missing.
Choosing the encoder was never the problem on a WARP device. The frame SOURCE was.
VideoEncoder::send feeds software encoders through av_hwframe_transfer_data, which
presupposes a D3D11 pool, and WARP cannot create one: av_hwdevice_ctx_init(D3D11VA)
fails for exactly the reason decoding does -- no ID3D11VideoDevice. Export on the CPU
backend died at "enc hwdevice init: -1313558101" before any encoder was tried.
So, on Backend::Cpu only:
- no encoder pool is created at all,
- VideoEncoder::open drops the zero-copy candidates with a stated reason instead of
handing them a null hw_frames_ctx and reporting the fallout as a driver refusal,
- VideoEncoder::send_composited reads the composed NV12 straight from the compositor
(Compositor::read_nv12_scaled) into the same buffers the software path already
uses, so conversion, buffer reuse and pts all stay on one code path.
Measured on the fixture, 360 frames: hardware 88.2 fps via h264_amf zero-copy; CPU
backend 4.8 fps via h264_mf; CPU with OPENSCREEN_EXPORT_ENCODER=libopenh264 4.6 fps.
All three decode clean under `ffmpeg -f null -`.
h264_mf winning here is a local artefact and is documented as one: Media Foundation
picks its own MFT independently of our D3D device, so on this machine -- which has an
AMD GPU, just not one this compositor is using -- it can still reach hardware. On a
genuinely GPU-less host it would fall to software or fail, and libopenh264 is the
floor. That is why the forced row exists: from a machine with a GPU, the env override
is the only way to exercise the real last resort.
The bench regains --export (dropped with the redundant commit) because it is what
proves the above rather than assuming it: on WARP no hardware encoder can open, so the
candidate list has to walk down to a software one on its own, and that walk is the
thing under test.
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF refactor that extracted `walk_composited_timeline` for the GIF path. The conflict resolution put the WARP branching inside the timeline walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those are MP4-specific. This commit moves the encoder creation back into `run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope) and adds the `if software_frames { ... } else { ... }` branching in the per-frame `on_frame` closure. Also fixes the unclosed `run_gif_bench` delimiter from the conflict resolution and adapts the call site to `export_gif`'s new signature (`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF refactor moved from `(screen, webcam, cursor, out_path, params, dither)` to the slice-2 MP4-shaped signature).
Branch off the CPU-backend seam established by PR #162 and split the compositor crate into per-platform directories. Windows behaviour is byte-identical (81 tests pass) and the macOS stubs that follow this commit in the stack expose the same public surface (Backend, Gpu, Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams, LayerCB...) and Err from every operation. Renames + cfg dispatch: - crates/compositor/src/d3d_windows.rs (from d3d.rs) - crates/compositor/src/cpu_frames_windows.rs (from cpu_frames.rs) - crates/compositor/src/compositor_windows.rs (from compositor.rs) - crates/compositor/src/pipeline_windows.rs (from pipeline.rs) - crates/compositor/src/text_windows.rs (from text.rs) - crates/compositor/wrapper_windows.h (from wrapper.h) lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across the per-platform modules so call-sites stay portable. live.rs: Win32 harness (run_standalone, host_proc, wide, client_size) moved into a #[cfg(windows)] pub mod standalone_harness with a dev-only on Windows). Cargo.toml: macOS deps (metal, objc, block, core-foundation) added under [target.cfg(target_os = macos).dependencies]; windows dep gated to cfg(windows). compositor-view-napi mirrors the gate. crates/.cargo/config.toml: target-specific env block for macOS (LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs). build.rs: selects wrapper_windows.h vs wrapper_macos.h on CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin + -isysroot on macOS, finds MAC_FFMPEG_DIR or vendored thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared. Stacked on feat/d3d-warp-fallback (PR #162).
…fer upload
Implements the decode axis of the macOS backend. Mirrors
cpu_frames_windows.rs: takes a libavcodec software-decoded frame
(libswscale → NV12 in system memory), uploads to a CVPixelBufferRef,
and presents an AVFrame whose data[0] carries the CVPixelBufferRef cast
as an opaque pointer — the seam contract that compositor_macos::nv12_srvs
will wrap into two MTLTextures via
CVMetalTextureCacheCreateTextureFromImage (zero-copy).
Notes:
- Direct FFI to CoreVideo (CVPixelBufferCreate et al.) rather than
the core-video-rs 0.1 crate — the latter's API surface is too thin
and doesn't expose CVMetalTextureCache.
- IOSurface request deferred: this commit passes attributes=NULL
(CPU-backed allocation); the follow-up IOSurface-backed commit
will pass the proper CFDictionary for true zero-copy.
- AV_PIX_FMT_D3D11 as sentinel in present.format: 'platform-specific
GPU buffer in data[0]' — same convention as the Windows path.
…stubs
The remaining scaffold stubs alongside the engine-layer files already
committed:
- crates/compositor/src/d3d_macos.rs: Gpu (metal::Device + metal::CommandQueue),
Backend::{Hardware, Cpu}, Gpu::create_auto/create, probe(), diagnose().
Same surface as d3d_windows.rs so call-sites stay portable.
- crates/compositor/src/text_macos.rs: TextSpec + TextRasterizer (CoreText
stub, returns Err from new/rasterize). The actual CoreText/CoreGraphics
rendering lands in a follow-up commit.
- crates/compositor/wrapper_macos.h: bindgen header that pulls in
hwcontext_videotoolbox.h (instead of hwcontext_d3d11va.h) for the
AVVideotoolboxContext type used in the macOS pipeline.
…ureCache seam, dual-format dispatch
Exposes the same public surface as compositor_windows.rs:
- Compositor::new / new_sized / render_size / normalize_render_size
- set_live_params / set_scene / set_cursor / set_cursor_time /
set_timeline_time / clear_cursor / scene_snapshot
- compose_frame / compose_frame_mb / rgb_to_nv12 / rgb_to_nv12_scaled /
readback_resized / readback_direct / read_nv12_scaled /
render_nv12 / dump_nv12 / dump_raw / blit_to / tex_dims / nv12_srvs
- webcam_shape_code / live_params_from_scene
- OUT_W / OUT_H / FIXTURE_FRAMES
- LiveParams with the same 11 fields as the Windows version
Returns Err from every engine method (compose_frame, render_nv12,
rgb_to_nv12, readback, dump_*). The seam — nv12_srvs + tex_dims — is
implemented: it reads the CVPixelBufferRef from data[0] (D3D11 sentinel
posed by mac_frames::CpuFrames::present) or data[3] (raw VideoToolbox
frames, per ffmeg convention), creates two MTLTextures (Y R8Unorm,
UV RG8Unorm) via CVMetalTextureCacheCreateTextureFromImage, and caches
them by (CVPixelBufferRef, planeIndex).
The Compositor holds a CVMetalTextureCacheRef created in new_sized
from the MTLDevice. Cache keyed on (CVPixelBufferRef, plane_index) so
repeated reads of the same plane reuse the MTLTexture.
Hand port of crates/compositor/src/shaders.hlsl (508 lines) to MSL. Each of the 9 entry points has a 1:1 equivalent; the constant buffer Layer mirrors the HLSL cbuffer field-for-field; SDF helpers (sd_segment, sd_round_rect, sd_convex_quad, line_cross, quad_inverse_bilinear) port without modification. vs_main → vertex VSOut vs_main ps_main → fragment float4 ps_main (14 modes, méga-shader) vs_fs → vertex FSOut vs_fs (fullscreen triangle) ps_y → fragment float ps_y (RGB → Y BT.709) ps_uv → fragment float2 ps_uv (RGB → CbCr BT.709) ps_blur → fragment float4 ps_blur (gaussien séparable, conservé) ps_tex → fragment float4 ps_tex (RGBA copy) ps_kawase_down → fragment float4 ps_kawase_down ps_kawase_up → fragment float4 ps_kawase_up HLSL→MSL mapping decisions documented at the top of the file. Will be embedded via include_str! and compiled at runtime via MTLDevice.makeLibrary(source:options:) once compositor_macos::new_sized wires the engine — the 'every_shader_entry_point_compiles' test will need a macOS counterpart.
…-format seam Wires ffmpeg's VideoToolbox hwaccel into the macOS pipeline. Decoder::open tries av_hwdevice_ctx_create(AV_HWDEVICE_TYPE_VIDEOTOOLBOX) first; on success sets hw_device_ctx + get_format returning AV_PIX_FMT_VIDEOTOOLBOX, on failure (codec not supported by VT — VP9, AV1, etc.) falls back to software decode via mac_frames::CpuFrames. compositor_macos::nv12_srvs and tex_dims dispatch on AVFrame.format: D3D11 sentinel → data[0], VIDEOTOOLBOX → data[3]. Both land on the same CVMetalTextureCache path (zero-copy IOSurface → MTLTexture Y+UV). Decoder::seek_to, next, cur_time_sec remain Err — the seek/receive_frame loop is symmetric with pipeline_windows but the actual avcodec_send_packet / avcodec_receive_frame / av_read_frame cycle needs the engine commit alongside ComposeFrame and run_composited_multi. run_composited_multi and VideoEncoder stay Err — see the 'engine VT' commit that follows.
Replace the hardcoded -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk (which only exists when CommandLineTools is the active developer dir — not the case on GitHub macos-14 runners, which use Xcode.app) with an xcrun probe. Build.rs falls back to bindgen's default sysroot search if xcrun fails or returns empty.
The 'build:' job was at indent 0 (root level) instead of indent 2 (under jobs:). GitHub Actions tolerates this but PyYAML's strict parser doesn't — and GitHub's own schema validator (actionlint) is stricter still. Indent it to 2 spaces so the macOS compositor-check job added in the previous commit parses.
…ine, first-pass compose_frame
Wires up the Metal compositor engine. new_sized now allocates render
targets and compiles the MSL shader library at compositor construction:
Render targets (MTLStorageMode::Private for GPU-only, Shared for readback):
- rt (RGBA8) — main composition target
- rt_read (RGBA8) — CPU-readable mirror for preview live
- nv12_y (R8) — Y plane of the NV12 output
- nv12_uv (RG8) — UV plane of the NV12 output
- nv12_read_y/_uv — CPU-readable mirrors for encode
MSL library: include_str! of shaders.metal, compiled at new_sized via
MTLDevice.new_library_with_source. Function lookups for vs_main, vs_fs,
ps_main, ps_y, ps_uv, ps_tex.
Pipeline states:
- pipeline_main — vs_main + ps_main, RGBA8 color attachment,
premultiplied alpha blend (SrcOne / DstOneMinusSrcAlpha)
- pipeline_fs_y — vs_fs + ps_y, R8 color attachment
- pipeline_fs_uv — vs_fs + ps_uv, RG8 color attachment
- pipeline_fs_tex — vs_fs + ps_tex, RGBA8 color attachment
Engine methods (first-pass implementation):
compose_frame(screen, _webcam, _frame, _cfg)
— first-pass blit: full-canvas (LayerCB mode=0) of the screen frame's
Y+UV textures (via nv12_srvs) into the RGBA8 RT. The webcam is
ignored, the layered composition (rounded corners, shadows, blur
pyramid, motion blur, timeline interpolation) is left to follow-up
commits that wire each LayerCB quad individually. This passes the
encode / preview-loop test path: a full-canvas video frame lands
on the RT.
— Empty screen → clear_rt (RT to black).
render_nv12()
— Two-pass fullscreen conversion: pass 1 (vs_fs + ps_y) writes Y to
nv12_y; pass 2 (vs_fs + ps_uv) writes UV to nv12_uv. Same BT.709
limited RGB→Y'CbCr pipeline as compositor_windows::render_nv12.
rgb_to_nv12(out_tex, slice) / rgb_to_nv12_scaled(...)
— First-pass: route to render_nv12 (writes the internal nv12_y/_uv).
The cross-engine zero-copy path to an external encoder-owned
CVPixelBuffer lands with the encodeur VT commit.
readback_direct()
— Synchronous get_bytes on rt_read → Vec<u8> RGBA8.
readback_resized(target_w, target_h)
— First-pass: routes to readback_direct; the resize is left to a
follow-up commit.
read_nv12_scaled(target_w, target_h, dst_y, pitch_y, dst_uv, pitch_uv)
— get_bytes from nv12_read_y/_uv → caller-provided AVFrame planes.
compose_frame_mb, dump_nv12, dump_raw, blit_to remain Err/no-op — they
exercise engine layers that are not yet wired (the layer-by-layer
composition that compose_frame_mb needs).
Helper: Layer (repr(C, align(16))) matches the HLSL/MSL cbuffer layout
field-for-field (each float4 naturally 16-byte aligned, float2 inside
float4 padded the same way on both sides). Uploaded via
set_fragment_bytes (the VS doesn't read it in full-canvas mode).
…c / fps / available_duration
Mirrors pipeline_windows::Decoder::seek_to/next/cur_time_sec/fps/
available_duration_sec/rewind with the macOS equivalents — ffmpeg-level
machinery that's identical across D3D11VA and VideoToolbox, only the
GPU hand-off differs (and that was wired in Decoder::open):
rewind seek to t=0, flush codec, clear EOF.
seek_to(seconds) keyframe-seek + decode-forward to first
frame at-or-after seconds (single seek per
clip boundary — the multiclip performance
hinge, same as on Windows).
next() loop avcodec_receive_frame / av_read_frame
with AVERROR_EAGAIN / EOF handling — calls
mac_frames::CpuFrames::present on the CPU
fallback path, hands the raw AV_PIX_FMT_VIDEOTOOLBOX
frame on the VT path.
cur_time_sec, fps,
available_duration_sec used by Player (live.rs) and
run_composited_multi (export) for
timeline math and progress reporting.
…libopenh264
Implements VideoEncoder on macOS:
- open(codec, gpu, w, h, fps, bit_rate) walks ExportCodec::candidates(),
honoring OPENSCREEN_EXPORT_ENCODER. The first candidate that opens
wins; failures are collected into a refusal list surfaced as 'aucun
encodeur utilisable'.
- try_open does avcodec_find_encoder_by_name + alloc_context3 + the
candidate-specific pix_fmt + AV_CODEC_FLAG_GLOBAL_HEADER. For
h264_videotoolbox/hevc_videotoolbox, av_hwframe_ctx_alloc creates a
fresh VT hw_frames_ctx (one VT device per process, fine for mono-clip
export — multi-device sharing will land alongside the encode run).
- send(frame) routes VT zero-copy straight to avcodec_send_frame; for
software paths the frame is moved via av_hwframe_transfer_data into
a sw-backed AVFrame (NV12 or YUV420P), with nv12_to_yuv420p for the
rare encoder that wants planar (kvazaar, kept as a no-op stub for
now since libopenh264 takes NV12 directly and VT is the common path).
- send_composited(compositor, w, h, pts) wires render_nv12 + the
composited sw frame into the encoder — sets the API surface; the
actual read_nv12_scaled population lands with run_composited_multi
which has both ends in scope.
alloc_sw_frame + nv12_to_yuv420p helpers kept here (next to where the
encoder uses them) — mirrors pipeline_windows.rs.
…xport
Orchestrates the full export pipeline (symmetric with pipeline_windows):
per clip, until end:
sdec.seek_to(start); wdec.seek_to(start - webcam_offset)
for each frame:
sdec.next() / wdec.next()
comp.compose_frame(sf, wf, frame, cfg)
enc.send_composited(comp, out_w, out_h, pts)
drain_encoder -> muxer
The VideoToolbox encoder consumes AV_PIX_FMT_VIDEOTOOLBOX frames zero-copy;
libopenh264 gets NV12 sw frames. The first-pass engine (compositor_macos's
full-canvas compose_frame) feeds the encoder; per-frame transfer from
render_nv12 to the encoder pool happens via send_composited.
Audio (AAC) is not wired in this commit — audio.rs is still
Windows-only via cfg re-export; landing AAC on macOS lands alongside
the audio port. drain_encoder is a faithful port of
pipeline_windows::drain_encoder.
Implements TextRasterizer::rasterize on macOS. The pipeline mirrors
text_windows.rs's DirectWrite + Direct2D flow:
1. MTLTexture BGRA8Unorm, StorageMode::Shared (CPU-readable for the
upload step).
2. CGContext bitmap backed by a CPU buffer
(CGBitmapContextCreate). Alpha-premultiplied, BGRA byte order
(kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Little).
metal-rs 0.29 doesn't expose a Shared-texture raw pointer without
IOSurface, so the CPU buffer is uploaded via replace_region at the
end — same Net I/O budget as a normal atlas upload.
3. NSAttributedString built via objc2 msg_send! with the spec's
foreground color (CGColor), font (CTFont), underline style, and
paragraph alignment (NSMutableParagraphStyle).
4. CTFramesetterCreateWithAttributedString + CTFrame over the box +
CTFrameDraw into the CGContext. The path=NULL argument means the
frame fills the full context bounds (i.e. box_px).
5. replace_region uploads the buffer to the MTLTexture.
The returned raw id<MTLTexture> is what compositor_macos.rs keeps in
its text_cache, keyed by spec.cache_key().
TextSpec::cache_key is byte-identical to text_windows.rs's — same
content + same color/background/font/align/box hash — so the cache
hits are platform-agnostic.
The scaffold commit puts `pub mod live` and `pub mod pipeline` in the "platform-independent" block, but the engine commits then add another `pub mod live` at the end. This conflict resolved as two declarations, which Rust rejects. Drop the duplicate from the head of the file (the end-of-file declaration is the one that the engine commits expect). The WARP CPU-backend commit fix-up earlier (88f79fa) also landed a copy of these `pub mod` lines, which the scaffold-vs-engine conflict merged into the same dual-declaration state.
Neither discovery path could fire on a Mac, and both failed for the same
reason: `crates/.cargo/config.toml` sets `FFMPEG_DIR` and `LIBCLANG_PATH` in a
GLOBAL `[env]` block. Cargo has no `[target.<cfg>.env]` — the macOS section at
the bottom of that file is inert, and cargo says so on every build
("unused key `env` in [target] config table"). So on macOS:
- `FFMPEG_DIR` is always set, to the win64 tree. build.rs read it first, so
the `MAC_FFMPEG_DIR` / vendored-tree fallback below it was dead code and
bindgen went looking for headers in
`thirdparty/ffmpeg-n8.1.2-win64-lgpl-shared/include`.
- `LIBCLANG_PATH` is always `C:\Program Files\LLVM\bin`. clang-sys honours it,
finds nothing there, and does NOT fall back to its own search — the build
dies on "Unable to find libclang" rather than using the clang that ships
with Xcode.
macOS now prefers `MAC_FFMPEG_DIR`, then the vendored tree, and only falls back
to `FFMPEG_DIR` when it points at something that actually exists (a dev who set
it deliberately, never the Windows pin). `LIBCLANG_PATH` is repointed at the
active toolchain via `xcrun --find clang`, which follows `xcode-select` and
works for both Xcode.app and CommandLineTools.
The vendored-tree probe was also looking in the wrong place: `thirdparty/` is a
sibling of `compositor/` under `crates/`, but build.rs runs with cwd = the crate
root, so the relative path resolved to `crates/compositor/thirdparty/`. It now
walks up from CARGO_MANIFEST_DIR, matching what the Windows pin's
`relative = true` means.
This is what stood between `.github/workflows/ci.yml`'s
rust-macos-compositor-check job and its own MAC_FFMPEG_DIR.
The first thing the macOS decoder did on a real Mac was fail: [live] set_active_clip: receive_frame a échoué (ret=-35) `AVERROR(e)` is `-e`, and `EAGAIN` is 11 on Windows and Linux but 35 on macOS and the BSDs. The value was written out by hand in three places (pipeline_windows.rs, audio.rs, and — copied from them — the macOS pipeline), all as -11. So `avcodec_receive_frame` returning its perfectly ordinary "redonne-moi un paquet" was read as a fatal error, and not one frame was ever decoded. Nothing crashed and nothing logged a wrong value; the preview was simply always black. Rather than add a fourth copy with the right number, the constant moves to `crate::ffi` — the module that already exists for "what bindgen cannot generate" — with a `#[cfg]` on the two families. shim.c gains `sn_averror_eagain()` / `sn_averror_eof()`, which the target's own compiler evaluates against the real ffmpeg headers, and a unit test confronts the Rust constants with them. That test is what would have caught this before it became "the preview is black".
`gif_export.rs` imports `crate::pipeline::walk_composited_timeline`, and `crate::pipeline` is `pipeline_macos` on a Mac — where that function does not exist. Native GIF export therefore did not compile at all on macOS. The two ways out were to copy ~170 lines into `pipeline_macos.rs`, or to lift them somewhere both platforms can reach. The function's own doc comment argues against the copy: "a GIF driven by its own loop is how the slow-motion truncation bug happened". That argument holds just as well between two platforms as between two output formats — and `pipeline_macos.rs` had in fact already grown a second, divergent walk of its own. `timeline_walk.rs` is portable by construction: it only touches the cfg-re-exported `Decoder` and `Compositor`, so D3D11VA and VideoToolbox both go through it with no `cfg` in sight. `advance_decoder_to` comes along because it is the walk's inner loop. Windows keeps the identical code path — the only change on that side is one `use`.
The MSL port was a line-for-line transcription of the HLSL, including the part
that has no MSL equivalent: HLSL declares its `cbuffer`, `Texture2D` and
`SamplerState` at global scope, and Metal does not allow that.
error: 'texture' attribute only applies to parameters, global constant
variables, and non-static data members
error: program scope variable must reside in constant address space
error: declaration of reference variable 'layer' requires an initializer
Nine entry points, none of which compiled — and none of which could fail at
`cargo build`, because the file is compiled at RUNTIME by
`MTLDevice.newLibraryWithSource`. The whole shader set was dead on arrival and
the type-check said nothing.
Resources are now parameters of each entry point, and the helpers that read them
(`sample_yuv`) take them as arguments. Also fixed on the way through, all of
them things the transcription could not have caught without a compiler:
- `texture2d<float>::sample` returns a `float4` in MSL where HLSL's
`Texture2D<float>` returns a scalar — `sample_yuv` assigned it to a `float`
and a `float2`. Now `.r` / `.rg`.
- `float3 acc = 0.0` / `float4 acc = 0.0`: HLSL splats, MSL does not.
- `constexpr int BLUR_R` at program scope must be `constant`.
- `line_cross` had a genuine transcription typo: `d1 * n1.x` where the HLSL
says `d1 * n2.x`, which degenerates the corner solve for the rounded-corner
tilted quad (mode 12).
Guarded by `every_shader_entry_point_compiles`, which compiles the file on the
system Metal device — the macOS counterpart of the HLSL test the Windows engine
already has.
First run on a real Mac. The engine layer compiled against an imagined metal-rs
and rendered into a texture nobody read; this is what it took to get a decoded
frame onto the preview canvas.
metal-rs 0.29 API, none of which existed as written:
- `metal::TextureType::Type2D` / `LoadAction` / `StoreAction` → the crate spells
them `MTLTextureType::D2`, `MTLLoadAction`, `MTLStoreAction`.
- `new_command_buffer()` and `new_render_command_encoder()` return refs, not
`Option`s, so the `.ok_or_else(...)` chains did not type-check.
- `RenderPipelineColorAttachmentDescriptor::new()` does not exist; attachment 0
is configured on the array the descriptor already owns.
- `get_bytes` takes an `MTLRegion`, not `(x, y, w, h)`.
- `Texture::from_raw` does not exist; `ForeignType::from_ptr` does, and it takes
ownership rather than retaining.
- `render_nv12` returned `()` while two callers used it as a `Result`.
The readback was the actual reason a preview would have been black even after
all that. `compose_frame` rendered into `rt` (`StorageMode::Private`) and
`readback_direct` read `rt_read`, which nothing ever wrote — and Metal has no
`Map` on a Private resource. Both passes now end with a blit to their `Shared`
mirror plus `waitUntilCompleted`, which is also what makes `readback_direct`
synchronous the way `live.rs` assumes.
Other corrections:
- `set_fragment_bytes` alone left `vs_main` reading an unbound buffer, and it
reads `layer.dst`/`src`/`quad_px` to build the quad. Bound to both stages.
- `nv12_uv` was allocated at full luma resolution; NV12 chroma is half in both
axes, so `read_nv12_scaled` was reading past what the pass wrote.
- The `(CVPixelBufferRef, plane)` → raw `MTLTexture` cache was a second cache
in front of `CVMetalTextureCache`, keyed on a pointer VideoToolbox recycles,
that never evicted, and whose entries were never retained. Dropped: CoreVideo
already returns the same texture for the same IOSurface.
- `CVMetalTextureGetTexture` returns a BORROWED reference. The texture is now
retained and the `CVMetalTextureRef` released — previously neither happened,
leaking two CoreVideo objects per frame (120/s at 60 fps preview).
- `CVMetalTextureCache`'s Drop only flushed, leaking the cache itself.
- `pixel_format` was passed to `CVMetalTextureCacheCreateTextureFromImage` as
`u32`; `MTLPixelFormat` is `NSUInteger`.
mac_frames.rs, the software-decode fallback:
- `sws_scale` was called with `self.nv12` as BOTH source and destination — the
decoded frame was never read, so the CVPixelBuffer got whatever the previous
conversion left behind. The source is `src`.
- The CVPixelBuffer was created with `attributes = NULL`, documented as a
scaffold to fix later. It is not optional:
`CVMetalTextureCacheCreateTextureFromImage` refuses a buffer that is not
IOSurface-backed, so every software-decoded frame failed at the seam. Created
with `IOSurfaceProperties` + `MetalCompatibility`.
- `#[derive(Clone)]` on a handle whose `Drop` calls `CVPixelBufferRelease`
copied the pointer without retaining — two releases for one retain. `Clone`
now retains.
- `present` did not carry `pts`/`best_effort_timestamp` across, so the timeline
read every frame as t=0.
- Dropped `CVReturnFromRetainCountedObjectGetRetainCount`, which is not a
CoreVideo symbol.
d3d_macos / pipeline_macos, to match the surface `compositor-view-napi` calls:
- `Gpu::probe()` was a free function where Windows has an associated one, and
`Gpu::create` took no `debug` flag. The addon did not compile.
- `ClipSource` had drifted to a different field set than the Windows struct the
addon constructs.
- `Decoder::open`'s VideoToolbox branch returned `hwdev` where the `cpu` field
expects an `Option<CpuFrames>`.
- `Decoder` was not `Send`, so `live.rs` could not hand it to its render thread.
- `av_hwframe_ctx_alloc` takes one argument and returns the buffer; it was
called with three, and with no VideoToolbox device to hang the pool on.
- `let end = ….unwrap_or(end)` referenced itself inside its own initialiser.
Verified end to end on an M-series Mac: `probeBackend()` → "hardware", a 1280x720
H.264 clip decoded through VideoToolbox, composed, and read back as real pixels
(per-channel mean 124/130/126, the burned-in timecode matching the requested
`presentTime`), sustained at 60 published fps.
The rasterizer talked to Objective-C classes that do not exist. `CGColor`,
`CGColorSpace` and `CFString` are CFTypes, not ObjC classes, so
`AnyClass::get(c"CGColor")` returns `None` — silently, since each lookup was in
an `if let Some`. Every attribute was skipped. The four attribute KEYS were
built with `Sel::register("NSColor")`, which produces a selector where
`CFAttributedString` wants the CFString `kCTForegroundColorAttributeName`.
`NSMutableAttributedString` has no `attributedStringWithString:`.
`CGBitmapContextCreate` was called with a NULL colour space, which returns NULL
for 32-bit RGBA. `kCGImageAlphaPremultipliedFirst` was written as `0x100` in
code and `1` in the comment (it is 2). And `rasterize` returned
`texture.as_ptr()` from a `metal::Texture` dropped on the next line — an
`id<MTLTexture>` already released.
It also did not compile: it imported `objc2` (the crate is `objc` 0.2, via
metal-rs) and `TextSpec::cache_key` called `Hash::hash(&mut h)` with a `u64`
standing in for a `Hasher`.
Rewritten against the C API, where all of this is a straight line:
`CGColorSpaceCreateDeviceRGB`, `CGBitmapContextCreate`, `CFAttributedStringCreate`
over a `CFDictionary` of the real `kCT*AttributeName` constants,
`CTFramesetterCreateFrame` on a `CGPath` rect, `CTFrameDraw`. No `msg_send!` at
all. CFTypes are owned by a `CFOwned` guard so a failure part-way through does
not leak a font, a colour and a framesetter per call. The context is flipped so
`box_px` reads top-left like DirectWrite's. `cache_key` is now byte-identical to
`text_windows.rs` — same FNV-1a over the same bytes in the same order — which is
what makes the cache policy genuinely shared rather than nominally shared.
`rasterize` returns an owned `metal::Texture`. Not runtime-exercised yet:
`compositor_macos` does not drive annotations in the first-pass engine, so this
is correct-by-construction rather than measured.
On a Mac where the microphone is still `not-determined` — every first run, and
every fresh machine — the app showed a TCC permission dialog with nothing behind
it, and created no window until it was dismissed. If the dialog was missed, the
app looked like it had failed to start: process alive, zero windows, no log line.
`await systemPreferences.askForMediaAccess("microphone")` resolves only once the
user answers, and `createWindow()` is 70 lines further down the same async block.
Nothing in between needs the answer — the recorder re-reads the status when the
user actually arms the mic — so the request fires without being awaited. The
prompt still appears; it just no longer holds the whole startup.
Found while trying to launch the app on macOS to verify the Metal compositor:
Playwright's `firstWindow()` timed out at 60s against a perfectly healthy process.
Both branches of the candidate list were joined onto `app.getAppPath()`, and on
neither side of the packaged line is that where the `.node` is:
- **Unpackaged.** For `electron dist-electron/main.js` — the entry every dev run
uses, `npm run dev` included — `app.getAppPath()` is `<repo>/dist-electron`.
Every candidate came out as `<repo>/dist-electron/electron/native/...`, which
no build step writes. The loader logged "native addon not present; running as
no-op" and the editor ran without a compositor, on a machine with a freshly
built addon one directory up.
- **Packaged.** `electron/native/bin/<tag>/**` ships exclusively via
`extraResources` (electron-builder.json5, mac/win/linux blocks), so it lands
at `<resources>/electron/native/bin/<tag>` and is never inside `app.asar`.
The `.asar` → `.asar.unpacked` rewrite cannot reach a file that was never in
the archive, and `process.resourcesPath` was not consulted at all.
Both bases are added — the same pair `ffmpegSharedBinCandidates` already walks
for the ffmpeg directory, and for the same reason. The unpackaged case is handled
by a bounded walk up to the first ancestor that actually has an `electron/native`,
so it does not hardcode "one level above dist-electron".
Confirmed on macOS: before, the editor only found the addon via
`OPENSCREEN_COMPOSITOR_VIEW_NODE`; after, it finds it on its own and the preview
canvas comes up with real pixels.
The twin of build-windows-compositor-addon.mjs. Same shape — drive cargo from `crates/` so `.cargo/config.toml` applies, then vendor the artifact to both paths compositorViewService.ts probes — minus the vcvarsall sweep, since `xcrun` already gives build.rs the SDK and libclang. `build:mac` now runs it, so a packaged macOS build ships an addon instead of falling through to the no-op compositor. ffmpeg is the one asymmetry and the script says so rather than pretending otherwise: BtbN publishes no macOS target (scripts/fetch-ffmpeg.mjs exits 1 on darwin by design), so there is nothing to download and the tree has to be built. `--print-ffmpeg-recipe` prints the configure line, with the LGPL posture spelled out — no `--enable-gpl`, no `--enable-nonfree`, because either one relicenses this MIT app. Homebrew's ffmpeg is explicitly called out as fine to develop against and never fine to ship: brew's formula is GPL. The script also prints what is still missing for packaging: the `.node` carries absolute paths to the ffmpeg dylibs it linked against, so shipping needs an `install_name_tool`/@rpath pass plus the dylibs in extraResources — the macOS equivalent of the PATH-prepend `ensureFfmpegSharedDllsOnPath()` does on Windows.
`cargo check` cannot see any of the three bugs that kept the first real macOS
run from rendering anything:
- shaders.metal is compiled at RUNTIME, so a file of invalid MSL type-checks
perfectly;
- `AVERROR(EAGAIN)` hardcoded to the Windows value compiles anywhere and
silently decodes zero frames;
- a shader/attachment mismatch is only rejected at `newRenderPipelineState`.
Each now has a test, and macos-14 runners have a real Metal device, so the
shader and pipeline ones run for real rather than skipping. The job runs
`cargo test` and then builds the napi addon.
Worth noting this job could not have been green before: it passes
`MAC_FFMPEG_DIR`, which build.rs ignored in favour of the global `FFMPEG_DIR`
pointing at the win64 tree.
Two independent breakages found while getting the branch to build and test on a Mac. `npm run build` failed at the tsc step: `ExportDialog.tsx` still imported `nativeBridgeClient` after the CPU-backend commit replaced its use with `useIsCpuCompositor`. TS6133, so `build:win`/`build:mac`/`build:linux` all stopped there. `gpuDetector.test.ts` stubbed `process.platform` to "win32" but asserted the tag `win32-x64` — and only `platform` is stubbed, `arch` stays the host's. It passed on CI (linux-x64) and on Intel Macs, and failed on every Apple Silicon machine, where the tag is `win32-arm64`. Asserted against `process.arch` instead. `npm run test` is now 1162/1162 on macOS arm64.
The scaffold commit renamed six modules (`d3d.rs` → `d3d_windows.rs`, `compositor.rs` → `compositor_windows.rs`, `pipeline.rs`, `text.rs`, `cpu_frames.rs`, `wrapper.h`) and left the docs pointing at the old paths — 24 broken links across three pages, so the `Docs` CI job has been red on this branch since commit 7 of 20. Every one of those links resolves to the Windows half, because that is what the prose around them describes: HLSL, D3D11VA, AMF, DirectWrite. Repointed there rather than at the cfg re-export, which is a name and not a file. The module map also stopped being a map once the crate grew a second half, so the macOS siblings get a table of their own, plus the sentence that explains why there are two: `lib.rs` cfg-re-exports each pair under one name, which is what lets `live.rs` and `compositor-view-napi` stay platform-blind. `timeline_walk.rs` is listed as neither — it is the portable walk both platforms and both export formats share.
`compositor_windows.rs` is 3654 lines of D3D11 engine — the code every Windows user actually runs — and no pull request compiled it. `ci.yml`'s only Rust job is `rust-macos-compositor-check` (macos-14). The Windows half is built exclusively by `build.yml`'s `build-windows` step, via `npm run build:win` → `build:native:compositor`, and `build.yml` triggers only on `push: tags: v*` or `workflow_dispatch`. So a typo in the Windows compositor was caught when someone cut a release, not when they pushed the commit that caused it. That is a bad place to find out on its own. It is a worse place to find out while landing a cross-platform refactor of that exact file, which is what comes next: the macOS port needs `compose_frame`'s geometry to move into a module both backends share, and every one of those commits touches `compositor_windows.rs` from a machine that cannot compile it. No extra tooling needed. `crates/.cargo/config.toml` already pins LIBCLANG_PATH to `C:\Program Files\LLVM\bin`, which is where the windows-latest image keeps LLVM, and `npm run fetch:ffmpeg` vendors the pinned BtbN LGPL-shared build into the exact directory FFMPEG_DIR names. That script imports only node builtins, so the job skips `npm ci` entirely. `cargo check --all-targets` rather than `build`: the question is whether the Windows engine still compiles, tests included, and check is about half the wall-time.
`compose_frame` decides two separate things: WHERE each layer goes, and how to hand that to a GPU. The first is arithmetic — preset placements, source crop, cover-fit, corner radii, shadow fractions, the fixture timeline, CSS colour parsing — and it lived in `compositor_windows.rs` purely because that is where the D3D11 engine was written first. The macOS port needs the same placements to the pixel — that is the "iso-render" property the project measures — and the only way two backends genuinely agree is that they call the same function, not that they maintain two copies obliged to stay in step. Same argument as `timeline_walk.rs`, one layer up. So: 337 lines of code and 14 helpers move to `frame_geometry.rs`, declared `pub mod` with no `cfg`, next to `regions.rs` which is the working precedent for a portable geometry module. Nothing is rewritten — every moved line is byte-identical to the original, verified mechanically; the only edits are visibility (`pub(crate)` so the Windows engine can still reach them, and `pub(crate)` on `Placement` / `FrameParams` fields, which were module-private and would otherwise become unreachable) and the imports at both ends. `compositor_windows.rs` keeps `pub use` on OUT_W / OUT_H / HALF_W / HALF_H / FIXTURE_FRAMES, so `pipeline_windows.rs`, `live.rs` and `crates/poc-d3d/src/app.rs` that read them through `crate::compositor::…` need no edit at all. The immediate payoff: **20 tests that had only ever run on Windows now run on macOS too** — the crate goes from 73 to 93 on a Mac. They cover exactly the arithmetic the Metal engine will consume next: `screen_source_rect`, `cover_crop_uv`, `cover_uv_rect`, `cursor_sprite_dst`, the zoom/padding invariants. The file-level `allow(dead_code)` is honest rather than defensive: on macOS half of this module has no caller yet because the Metal engine has no layered `compose_frame`. It is not dead code, it is code the port has not reached — and it is exercised by its tests on both platforms. The allow goes away when the layer driving lands.
Four items existed twice, once per backend, and the duplication had already done
what duplication does.
**LayerCB / Layer.** The same ten fields in the same order, declared in both engines.
They are the Rust half of a three-way contract — `cbuffer Layer` in shaders.hlsl,
`struct Layer` in shaders.metal, this struct — and a mismatch does not raise
anything, it makes a shader read `color` where `fx` was written. Unified on the
macOS spelling (`repr(C, align(16))`); under plain `repr(C)` both already produced
offsets 0/16/32/40/44/48/64/80/96/112, and Windows only `copy_nonoverlapping`s the
struct into a mapped constant buffer, where source alignment is irrelevant. The two
forms were compatible; unifying them stops them from ceasing to be. The size
assertion moves to the shared module and gains a per-field `offset_of!` check.
The doc said "64 octets". It was never right — ten fields, thirty-two `f32`.
**LiveParams and friends.** The macOS copies were annotated "mêmes champs et même
layout" and "mêmes formules". Both claims were false:
bg_color default windows [0.10, 0.11, 0.14, 1.0] macos [0, 0, 0, 0]
has_webcam default windows true macos false
webcam_shape_code(_) windows 3 ("rounded") macos 0 ("rectangle")
The third is the sharp one. `live_params_from_scene` calls it and the app's default
`webcam_shape` is "rounded", so one scene described a rounded camera on Windows and
a rectangular one on macOS — a silent cross-platform rendering difference sitting in
the code that exists to prevent cross-platform rendering differences.
Windows values win: that is the backend that renders in production. This changes
macOS behaviour by exactly nothing today, because the Metal engine reads none of
these fields yet — `compose_frame` hardcodes its LayerCB. Which is why it is worth
doing now: after the layer driving lands, the same fix is a regression hunt.
`OUT_W` / `OUT_H` / `FIXTURE_FRAMES` were duplicated too, with identical values.
Both compositor modules `pub use` everything back out, so `live.rs`,
`pipeline_windows.rs`, `compositor-view-napi` and `crates/poc-d3d` are untouched.
The first 353 lines of `compose_frame` contained no GPU call at all — the first
one is `self.begin(...)`. They decide WHERE every layer goes: preset placements,
the app-resolved screen and camera rects, user crop, zoom regions, Full Camera,
cover/contain fitting, corner radii in render-target px, the reactive camera
shrink, and the previous-frame rects the motion blur derives velocity from.
They now live in `frame_geometry::plan_frame`, which both backends can call. The
Metal engine cannot drive a single layer today because this arithmetic was locked
inside the D3D11 engine; this is what unlocks it.
Of the 75 locals that block produces, **15 survive to the first draw**. That is the
whole interface: `scene_preset`, `mb_taps`, `source_t`, `zoom_rotation`,
`padding_scale`, `cut`, `s_dst`, `s_dst_prev`, `s_radius`, `frame_min_px`, `w_dst`,
`w_dst_prev`, `w_px`, `w_radius`, `shape_fade`. The other 60 die where they were
computed, which is why a struct of outputs is the honest shape rather than a draw
list — every `LayerCB` in this function is still built at its own draw site.
**The draw half does not change. At all.** Verified mechanically against `HEAD`, not
asserted: `self.begin(...)` to the closing brace is byte-identical. The moved block
is likewise the original text, with exactly four substitutions, each mechanical:
self.rw() / self.rh() -> rw / rh (22 sites)
scene_ref.as_ref() -> scene (13)
cursor_ref.as_ref() -> cursor (1)
self.timeline_t_override.borrow() -> input.timeline_t_override
The three `RefCell` guards stay in `compose_frame` and are borrowed across the call:
`scene_ref` because the draw half still reads it (`:1686`, `:1894`), `lp` because it
does too (10 sites), `cursor_ref` because the draw half rebinds its own. `plan_frame`
receives `Option<&Scene>` / `Option<&CursorTrack>` through them, so no borrow lives
one instruction longer than it did.
Nothing in the input struct is a backend object — dimensions, the scene, the live
params. `screen_tex_px` is called out in its doc because it is the one place the two
platforms genuinely differ: D3D11VA pads to macroblocks (1080 -> 1088), CoreVideo
does not, and `u_max`/`v_max` exist precisely so the geometry never assumes
texture == visible.
93 tests still pass on macOS; the Windows job type-checks the call site.
The second build of the addon killed whatever loaded it:
signal: SIGKILL (Code Signature Invalid)
termination: { namespace: "CODESIGNING", indicator: "Invalid Page" }
No JS error, no stack, no output — `require()` and the whole process were gone. In
the Electron app it looks exactly like a crash on startup.
macOS validates code pages lazily against the Mach-O signature. `fs.copyFileSync`
onto the existing `compositor_view.node` rewrites the file in place, so the kernel's
page cache still holds pages from the OLD binary on a vnode whose signature is now
the NEW one; the next process to fault one of those pages is killed outright. The
first build is always fine, which is what makes this so confusing to diagnose —
the addon works, you change one line of Rust, rebuild, and now nothing loads.
Copy to a temp name in the same directory and `rename` over: the new content gets a
fresh inode, and the stale pages belong to a vnode nothing will fault again.
Found by rebuilding after the `plan_frame` extraction and having the verification
harness die with no output at all. Worth noting it is not specific to this refactor —
it would have hit every macOS developer on their second build, and the crash report
is the only place the real reason appears.
Found by driving the layered composition on a Mac for the first time and comparing `shaders.metal` to `shaders.hlsl` mode by mode. All three compiled without a complaint, which is the whole problem: the port reads plausibly and is wrong. **Mode 6 — wallpaper image.** HLSL returns `float4(texImg.Sample(samp, i.uv).rgb, 1.0)`, opaque by construction. The port returned `s.rgb * a` with `a = layer.color.a`, and `LayerCB::default()` zeroes `color` — so the wallpaper was drawn with alpha 0 and was rigorously invisible. It reads as "the compositor does not draw the background", which is exactly how it was reported. **Mode 5 — gradient.** HLSL interpolates stop0 -> stop1 along `fx.xy`, normalised corner-to-corner. The port replaced the entire computation with `float4(color.rgb * color.a, color.a)` — a flat colour. Every gradient wallpaper rendered as its first stop, uniformly. **Mode 2 — drop shadow.** HLSL: `1.0 - smoothstep(0.0, spread, d)`. The port: a LINEAR ramp over `(d - spread)`, which shifts the penumbra out by a full `spread` and gives it a hard edge where it should fade out. Mode 0/1 and the shared radius tail were checked at the same time and are faithful.
`compose_frame` on macOS blitted the screen frame full-canvas and ignored
everything else. It now calls `frame_geometry::plan_frame` — the same function the
D3D11 engine calls — and emits the layers:
background colour / gradient / image wallpaper (modes 1, 5, 6)
screen shadow mode 2, spread and offset from frame_min_px
screen mode 0, dst + source cut + corner radius
camera shadow mode 2, PiP-specific opacity, faded by shape_fade
camera mode 0, cover-crop + mirror + radius
The drawing verbs carry the same names and the same parameters as their Windows
counterparts (`draw_solid`, `draw_video`, `draw_shadow`, `draw_image_bg`), so the two
draw halves can be read side by side. `draw_shadow` is a word-for-word port.
Image wallpapers get a real path now: decode through the `image` crate (already a
dependency), upload to an `MTLTexture`, cache by path — the same shape as
`load_image_srv`, including the isolated `let` that releases the immutable borrow
before the `borrow_mut`, which on the Windows side was a first-frame RefCell panic.
A failure still falls back to the background colour, but logged: a silent fallback
would hide a broken path.
Verified against a real Windows-authored project on an M-series Mac — padding,
background, rounded corners, screen and camera shadows, and the PiP camera all
render, and the shadow responds to its slider.
Still not emitted: cursor, annotations, the 3D tilt (mode 8), and background blur.
The cursor is the last thing the preview drew on Windows and did not draw at all on macOS. It needs three pieces, and only the first was cheap. **Shared geometry.** `plan_cursor` joins `plan_frame` in `frame_geometry`: the source-rect mapping, the click bounce, the size (which folds in `padding_scale` — the cursor is a synthetic overlay, so it must shrink with the screen or its tip appears to drift), the trail window, and the tap count. Both engines call it. Two copies of this mapping is exactly how two backends end up putting the pointer in different places, and the mapping is the part users notice. **Passes.** Metal has no `OMSetRenderTargets`: changing target means ending the encoder and opening another. `begin_pass` is that, and `compose_frame` is now a sequence of passes rather than one long encoder — which is also what the tilt, the blur and the annotations will need. **The trail.** N samples accumulate in an ISOLATED transparent buffer, then composite "over" onto the scene. Adding them straight onto the render target would add white to whatever is underneath, and on a light background the cursor all but disappears — the same bug the Windows comment records having had. Metal spells the weighting `MTLBlendFactor::BlendColor` + `set_blend_color(1/taps, …)`, which is `OMSetBlendState(blend_add, [w,w,w,w])` on the other side. Not ported: the mathematical dot+ring fallback (mode 4). The app always resolves a sprite set and the built-in art covers the states a theme omits, so if there is truly no sprite, drawing nothing is more honest than drawing a cursor that matches no setting. The tilted sprite (mode 13) lands with the 3D tilt; until then a tilted placement draws upright at the plane's centre. Verified in the editor against a real project: cursor visible, following the zoom region, with padding, rounded corners, shadows and the PiP camera around it.
"Blur BG" blurs the WALLPAPER, not the video — parity with the web `blurredBackgroundLayer`. A no-op on a flat colour, a real effect on a gradient or an image. Six fullscreen passes over a three-level pyramid (half, quarter, eighth), three down then three up, the last one writing the render target back. The pyramid is derived from the render size rather than a constant: half of 1080 is not half of 2160, and a fixed pyramid would make the effective blur radius depend on the output resolution. Same sizes and same texel steps as `compositor_windows::blur_bg`. `ps_kawase_down` / `ps_kawase_up` already compiled — they were among the nine entry points nothing called. What was missing was the render-target choreography, which on Metal means six encoders rather than six `OMSetRenderTargets`; `fs_pass` is that, and the viewport comes free from the attachment size. Reading and writing the same texture would be illegal, and does not happen: down samples `rt` into `half`, up samples `half` back into `rt`. Verified in the editor by toggling Blur BG on a real project — the wallpaper's colour bands go from hard edges to soft transitions, with the screen, camera and cursor untouched on top.
…ergences Correcting the record on 2dd2e3e. That commit says it changed mode 2's drop shadow from a linear ramp to `smoothstep`. It did not: the substitution matched on `(d - spread)`, a string that only existed inside the `layer.mode > 11.5` block — **mode 12**, the projected-quad shadow, which no macOS code path can even reach. The assertion passed and I never checked which block it had edited. Real mode 2 kept the linear ramp, so the shadow that draws on every frame was still wrong. Mode 2 is now `1.0 - smoothstep(0.0, spread, d)`, matching `shaders.hlsl:408`. Two further divergences, found by diffing the two shaders line by line rather than mode by mode: **Rounded-corner feather** (`:349` and `:409`). MSL `clamp(d * 1.5, 0.0, 1.0)` against HLSL `smoothstep(0.0, 1.5, d)`. Two compounded errors: linear instead of Hermite, and saturating at `d = 0.667 px` instead of `1.5 px` — a 2.25x narrower feather. This one runs on *every* rounded quad, screen and camera, on every frame. **Dual-Kawase up-pass** (`:510`, `:514`). The port doubled the two purely vertical taps, so the weights summed to 14 while still dividing by 12: **+16.7 % brightness per pass**, and anisotropic. Three up passes compound to ~1.59x — a background blur that washes out toward white and smears vertically. `ps_kawase_down` was checked at the same time and is exact; only `up` had drifted. All three compiled, and none was reachable by reading a diff — the first two only show as "the shadow looks a bit hard", the third as "the blur looks washed out".
…12, 7 The tilted screen (zoom-region "rotation") had no macOS draw path, and the three shader modes it depends on were each wrong in ways that only running them reveals. **Mode 8 — the tilted plane.** Three faults, any one of which alone renders nothing. The port carried a clip test comparing `i.pout` (in [0,1]) against `layer.dst_prev`, copied from mode 13 — but mode 8 packs `dst_prev.xy` with `plane_px`, the plane size in PIXELS (~1600). Every pixel failed the test, so the branch returned transparent everywhere. It also sampled `r` directly instead of interpolating into the `src` crop, ignoring crop and zoom entirely, and returned `layer.color.a` as alpha where the HLSL returns the plane-space rounded-corner coverage — and mode-8 draws leave `color` at default, so that alpha was zero regardless. **Mode 12 — the projected-quad shadow.** Read `spread` from `fx.x`, but `fx` carries the CORNERS and spread lives in `mb.y`. Re-centred `i.local` on `quad_px * 0.5` when it is already in bbox space, putting the shadow half a box off. And replaced the per-edge `line_cross` radius inset with a flat `+ spread`, so the shadow had no rounded corners at all — a sharp-cornered shadow behind a rounded screen pokes out at every corner. The tell-tale was that `line_cross` was defined in the file and called from nowhere. **Mode 7 — cursor sprite.** The "Clip to canvas" test was simply absent. `plan_cursor` computes the rect and the draw passes it in `fx`; the shader ignored it, so `cursor.clipToBounds` was inert on macOS. The Rust side ports `draw_quad_shadow` and the mode-8 packing from `compositor_windows.rs`, and `draw_cursor_sprite` now takes the `Tilted` branch (mode 13) instead of collapsing to the plane's upright centre. The tilt geometry is computed once and shared between the shadow and the screen — two separate computations is an shadow that detaches the moment either changes. Found by diffing the two shaders LINE BY LINE rather than mode by mode, which is what the previous pass failed to do.
Completing the line-by-line MSL/HLSL diff. Annotations have no macOS draw path yet, so none of this was reachable — but all three would have rendered wrongly the moment one was written, and two of them silently. **Mode 9 — arrow.** The port invented a shape: one `sd_segment` derived from `quad_px`, with `radius_px` as stroke width. The HLSL unions THREE segments — a shaft and two barbs — from `fx` / `src_prev` / `dst_prev`, with half-stroke in `mb.y`, for exact parity with `ArrowSvgs.tsx`. That is the geometry `regions::arrow_local_geometry` computes and uploads, and the port ignored all of it. Not an approximation of the arrow: a different figure. **Mode 10 — blur / pixelate.** The port sampled `texImg` and returned it. The HLSL implements the whole mask: rectangle or inscribed oval coverage, then either a mip level of the composed image (`log2(radius)` — a few taps spaced by the radius do not blur, they superimpose offset copies and read as ghosted text) or a quantised UV grid aligned to the quad, then an optional half-mix tint. **Mode 11 — text.** The comment said "already premultiplied — do NOT re-multiply" and the code did exactly that: `s.rgb * (s.a * color.a)`, alpha applied twice. Text would come out too dark on its antialiased edges and vanish on thin strokes. `sample(samp, uv, level(lod))` is MSL's `SampleLevel`; the mip pyramid it needs is the render-target copy the Rust side still has to provide.
…sterizer `draw_annotations` had no macOS counterpart at all — arrows, blur/pixelate masks, images and text simply did not exist on the preview. All four kinds now render, in zIndex order, anchored on the screen rect (the container the web overlay receives). Three resources this needed: - **A mipmapped copy of the render target.** Mode 10 blurs by sampling a mip level — `log2(radius)` picks the level whose texel spans the requested radius. It is taken ONCE per frame, before any mask draws, because a blur must see the composed image *without* the other blurs: two overlapping masks would otherwise sample each other depending on paint order. `generate_mipmaps` on a blit encoder is Metal's `GenerateMips`. - **An annotation image cache**, keyed on annotation ID rather than the data URL — the URL is routinely megabytes, and hashing it every frame would cost more than the decode. The stored length guards against the user swapping the image. - **The text rasterizer**, which has existed since `4d495ae8` and had never been called. `TextSpec` is built from the same fields as the Windows side and its `cache_key` is byte-identical, so the cache policy is genuinely shared. Text animations (translate, scale, reveal) come from the portable `text_anim`. The freehand blur mask falls back to the bounding BOX, deliberately and asymmetrically: drawing nothing would leave in the clear exactly what the user marked as hidden, and a privacy mask that does not mask is worse than none because it earns trust it has not got. Over-covering a margin harms no one. The fallback uses the rectangle rather than the inscribed oval for the same reason — an oval would drop the corners, and with them part of what was covered.
…ngine Export had never produced a frame on macOS. Three things stood between it and one. **`send_composited` sent an empty frame.** It called `render_nv12()` and then handed the encoder `self.sw` — a buffer nothing had ever written — with no pts. It now reads the composed NV12 back into the frame's planes via `read_nv12_scaled` and stamps the timestamp. The old body even carried `let _ = w; let _ = h; let _ = pts;`, which is the shape of code written to satisfy a signature rather than to do the work. **The clip walk was hand-rolled.** It decoded 1:1 while advancing `t` by `1/fps`, so it ignored speed regions, per-clip scene windowing and the cursor. That is precisely the slow-motion truncation `walk_composited_timeline`'s own doc records having cost once — "a GIF driven by its own loop is how the bug happened". It now calls the shared walk, so MP4, GIF and both backends agree on which source frame belongs at output frame N. **The encoder was configured for a pool that never existed.** The candidate list asked `h264_videotoolbox` for `AV_PIX_FMT_VIDEOTOOLBOX` and built a fresh `hw_frames_ctx` per export. But `ffmpeg -h encoder=h264_videotoolbox` reports `videotoolbox_vld nv12 yuv420p`: it takes SOFTWARE NV12 and uploads internally — and NV12 is exactly what the compositor produces. The candidate now asks for NV12 and the whole `hw_frames_ctx` construction is deleted. An entire layer of hardware-pool plumbing that had never run, replaced by the format both sides already speak. Measured on an M-series Mac against a real recording: 120 frames at **73 fps**, 1280x720, `h264_videotoolbox`. `ffmpeg -f null -` decodes it clean, and the frames carry the whole composition — background, padding, rounded corners, drop shadow and the circular PiP camera. Still absent from the export: audio (`audio.rs` is portable and compiles, but the muxing path has not been exercised on macOS).
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into a failing test, and says plainly what the rest is worth. The two backends cannot run on the same machine, so iso cannot be established by comparing two rendered images. What can be — and what is the layer where divergence actually happened on this branch — is the geometry: `plan_frame` is literally the same code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit` pins them for a scene that exercises padding, crop, a displaced PiP camera, corner radii and the zoom, comparing with `to_bits()` rather than an epsilon. It runs in BOTH CI jobs — `rust-macos-compositor-check` and `rust-windows-compositor-check`. If the two platforms ever compute different placements, one of them goes red. That is the guarantee that was being claimed and never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely because nothing compared them. What this test does NOT cover, and must not be read as covering: rasterisation. D3D11 and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical with a max deviation of 3/255 between two backends on the SAME machine, and that is the floor, not the target. Shader parity is held separately, by `shaders.metal` and `shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences came out of that pass, three of them live. The expected values are measured from this code, not chosen. A drift is a divergence to explain, not a tolerance to loosen.
The export shipped video only. Everything it needed already existed and was already portable — `audio.rs` has no `cfg` at all, and decode / WSOLA time-stretch / multi-track mix / concat planning all compile on macOS. What was missing was the muxing, four calls. `AacEncoder::open` happens BEFORE `avformat_write_header`, because the header carries the stream table and a stream added afterwards is not in it. Per-clip PCM is collected in the walk's `on_clip_end` — the same hook the Windows path uses, which now exists on this side because the walk is shared — then assembled once the video is drained. The plan is built from the frames each clip ACTUALLY produced, not from its requested duration: a clip clamped short (source shorter than its bound) must have its audio clamped by the same amount, or the track drifts for every clip after it. Measured: 120 video frames plus an AAC stereo 48 kHz track, 4.000 s against 3.967 s of video, `ffmpeg -f null -` clean. Note the PR description was claiming `audio.rs` is "Windows-only via cfg re-export". It never was — it is declared `pub mod audio;` unconditionally and has no `cfg` in it.
…hing Reported: the mosaic mask works, the gaussian one does not. Mode 10's blur samples the mip pyramid of the composited-frame copy — `texImg.sample(samp, i.pout, level(lod))` with `lod = log2(radius)`. But the sampler was declared `sampler samp(filter::linear, address::clamp_to_edge)`, and MSL defaults a sampler with no `mip_filter` to `mip_filter::none`, which makes `level(lod)` return mip 0 whatever `lod` says. So the "blur" resampled the image at full resolution: visually a no-op. The mosaic path asks for `level(0.0)` explicitly, so it worked by accident — which is exactly why the two behaved differently and why the report is so precise. `mip_filter::linear` is the equivalent of `D3D11_FILTER_MIN_MAG_MIP_LINEAR`, which is what the D3D sampler has used all along (`compositor_windows.rs:316`). The mip chain itself was already being built — `generate_mipmaps` on the render-target copy — it just had no way to be read.
Reported: text appears, vertically mirrored. `rasterize` flipped the CGContext (`TranslateCTM(0, h)` + `ScaleCTM(1, -1)`) with the comment that CoreGraphics has its origin bottom-left. The origin part is true and the conclusion does not follow: `CGBitmapContext` stores buffer row 0 at the TOP of the image, and `CTFrameDraw` fills its frame top-down. The first line of text therefore already lands in the first buffer rows, i.e. the top of the `MTLTexture`. The flip inverted a picture that was already right. The test asserts where the INK is rather than trusting the sign of a transform: rasterize "Ag" into a 256px box and require the upper half to carry at least four times the alpha of the lower half. Reintroducing the flip makes it fail — checked, because a passing assertion has already fooled me once in this port.
"70 fps is not great on this machine" — correct, and profiling put the cost somewhere
I had not looked. Per 120-frame export: compose 0.027s, encoder submit 0.426s, drain
0.000s, total 1.76s. **74 % of the wall time was decode**, and the compositor itself
runs at 0.2 ms/frame.
Measured on a 1920x1080@60 Constrained Baseline capture, decode only:
VideoToolbox 240 frames in 1.11s 215 fps
libavcodec sw 240 frames in 0.08s 3000 fps 13x
End to end on the export, 76 fps against 182 — 2.4x — even though the software path
additionally pays swscale and a full memcpy into the IOSurface every frame.
The reason is structural: the hardware decoder has a fixed per-frame latency and
allocates a CVPixelBuffer/IOSurface for each, while a trivial profile decodes in a few
hundred microseconds across cores that are, unlike the decode block, plural. Baseline
is what openscreen's own capture produces (`crates/fixture/fixture.json`, profile_idc
66) and what Chrome's MediaRecorder emits — the common case, not an edge one.
So the choice is made on the profile rather than by flipping a default: Baseline and
Constrained Baseline decode in software, everything else (High, 10-bit, HEVC, 4K, where
the arbitration reverses and software becomes the bottleneck) keeps VideoToolbox.
`OPENSCREEN_MAC_DECODE=software|videotoolbox` overrides, which is also how the numbers
above were taken.
Also removed two per-frame pipeline stalls: `compose_frame` and `render_nv12` each
committed and immediately waited, draining the GPU before the CPU had anything to do.
They now submit and track; the wait happens where a CPU read actually occurs. And the
full-resolution RGBA mirror blit moved into `readback_direct`, so the export — which
never reads RGBA — stops paying for it.
Export output still decodes clean under `ffmpeg -f null -`.
The macOS export rendered NV12 into internal textures, read 1.4 MB back to the CPU, and handed that to VideoToolbox — which uploaded it again. `rgb_to_nv12` took an `out_tex` argument and ignored it, a shortcut from the original port. It now honours it. `out_tex` is the `CVPixelBufferRef` of a frame pulled from the encoder's own VideoToolbox pool; both planes are wrapped as `MTLTexture` through the same `CVMetalTextureCache` the decoder uses, and the two fullscreen passes target them directly. The frame never leaves the GPU. This is the macOS shape of the zero-copy the D3D11 engine has always had. `ExportCodec::candidates()` now lists `h264_videotoolbox` twice — once on `AV_PIX_FMT_VIDEOTOOLBOX`, once on NV12. If the hardware pool refuses to open, the existing candidate walk falls through to the software-frame variant: same encoder, one CPU round trip more. Nothing is lost when zero-copy is unavailable. **Measured honestly: no throughput gain on this workload.** 176 fps, against 172-176 for the readback path. Decode still dominates the frame budget, so removing work from `send` does not move the total. It is kept because it is the architecturally correct path, it removes real per-frame work, and it will matter as soon as decode stops being the ceiling — not because it made this measurement faster, which it did not. Output validated: 1280x720 H.264 + AAC stereo, clean under `ffmpeg -f null -`, frame content real (mean 201.7/255, not a black field). One trap worth recording: the `hw_frames_ctx` this path needs had been silently lost — a `cp` restoring a backup after a profiling run clobbered it. `avcodec_open2` succeeds without it, `sw` stays null, and the first `av_hwframe_get_buffer` segfaults on a null pointer with no Rust error at all. The crash report was the only thing that said so.
… nil
The previous commit claimed "no throughput gain on this workload". That was true of
the benchmark and false of the product, and the benchmark was the wrong one.
Measured on the real project — 1920x1080, 22 s, 60 fps output — by the person running
the app: **12 s before, 8 s after**. 1320 frames, so 110 fps against ~165. A 1.5x
speedup on the thing users actually do.
My bench said otherwise because it was 1280x720 and four seconds long, and both
mattered:
* the NV12 readback is 1.4 MB per frame at 720p and 3.1 MB at 1080p, so the work
removed is 2.25x larger on the real output size;
* over 1320 frames that is ~4.1 GB pushed across the CPU boundary and uploaded
again, where a 120-frame clip moves 170 MB;
* and on a 4 s clip decode dominated so heavily that what `send` saved sat inside
the noise.
Re-run at the real size: 600 frames of 1080p in 3.42 s, 175 fps — consistent with the
figures above.
The lesson is the benchmark's, not the code's: a synthetic clip a fifth of the length
at half the resolution does not stand in for the export, and reporting "no gain" from
it was a measurement error on my side rather than a property of the change.
Two gaps that both ended at the same place: a packaged macOS app whose addon would not load, or would load something we are not allowed to ship. **Packaging.** Straight out of cargo, `compositor_view.node` references its ffmpeg dependencies by ABSOLUTE path — `otool -L` showed `/Users/…/crates/thirdparty/…`. That resolves on the build machine and nowhere else; an installed app would die at `require()` naming a directory the user has never had. Windows does not hit this because `LoadLibrary` searches `PATH`, which is what `ensureFfmpegSharedDllsOnPath()` prepends; macOS has no equivalent search, so the fix has to be baked into the binary. The build script now copies the five dylibs next to the shipped `.node`, rewrites every install name and inter-library reference to `@rpath`, and adds `@loader_path` as an rpath — which is exactly where the mac `extraResources` filter `darwin-*/*` already puts them. Re-signing after `install_name_tool` is not optional: it invalidates the ad-hoc signature, and macOS then kills the process with the same `SIGKILL (Code Signature Invalid)` documented on `installAtomically`. Verified: no absolute path left, and the addon still loads and probes `hardware`. **Licence.** `npm run fetch:ffmpeg:mac` vendors ffmpeg 8.1.2 from the pinned release tarball, checksummed, configured with neither `--enable-gpl` nor `--enable-nonfree`. It builds rather than downloads, and not by preference: BtbN — whose pinned `-lgpl-shared` archives `fetch-ffmpeg.mjs` fetches for Windows — publishes no macOS target, and every macOS binary that does circulate (evermeet, osxexperts, Homebrew) is GPL, because they ship x264/x265. Linking any of them would relicense this MIT app. Convenience does not make them candidates. The result is verified by asking the binary (`ffmpeg -L`) rather than by trusting the configure line — the same check `fetch-ffmpeg.mjs` runs on Windows, and the one that catches a tree swapped by hand for a GPL one. Both scripts refuse to proceed on a non-LGPL tree, and the fetcher deletes a build that fails the check rather than leave it where `build.rs` would link it. `build:mac` runs it before the addon build.
EtienneLescot
force-pushed
the
feat/macos-metal-compositor-v2
branch
from
July 30, 2026 10:08
98ba0e9 to
209a496
Compare
…o v1.8.0 release/v1.8.0 added an SRV-cache invalidation call in the portable live.rs (comp.clear_srv_cache(), called unconditionally whenever a decoder set is closed) plus a decoder fast-seek path in the Windows Decoder. Both landed during the rebase of this branch, but only pipeline_windows.rs got the method/the fast path applied — compositor_macos::Compositor had no clear_srv_cache at all, which doesn't fail until compiled on macOS. clear_srv_cache() on Metal calls CVMetalTextureCache::flush() — there's no address-keyed HashMap to clear like on D3D11 (CoreVideo already dedupes by IOSurface, not by pointer), so flushing the cache IS the equivalent operation. Also ported the Decoder::seek_to fast path (cur_pts, decode_forward_to, SEEK_FORWARD_MAX_SEC) into pipeline_macos.rs so both engines stay at the same seek performance instead of drifting apart again right after the last ISO-parity pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed since the first revision of this description
The original text said the engine was "best-effort, port-by-comment" and "cannot be runtime-verified from this Windows host". That was accurate when written. It has since been compiled, run, and driven on an M-series Mac, and the first run found a lot.
What actually works, measured on a Mac
CVPixelBuffer→CVMetalTextureCache→MTLTexture, zero-copy over IOSurface.cargo testonmacos-14with a real Metal device — not just a type-check.What the first real run found
Every one of these compiled cleanly. That is the point: none was reachable by review.
AVERROR(EAGAIN)is −35 on macOS, −11 on Windows/Linuxshaders.metalwas not valid MSLcbuffer/Texture2Dat global scope; Metal forbids it. All nine entry points failed to compile — invisibly, since the file is compiled at runtimecompose_frametargeted aPrivatetexture,readback_directread aSharedone; Metal has noMaponPrivateCVMetalTextureGetTexturereturns a borrowed referencesws_scalewas passed its own destination as sourceCVPixelBufferwas not IOSurface-backedCVMetalTextureCacheCreateTextureFromImagerefuses those outrighttext_macos.rssent ObjC messages to classes that do not existCGColor/CGColorSpace/CFStringare CFTypes; every attribute was silently dropped. Rewritten on the CoreText C APIsmoothstepbuild.rscould not find its own ffmpeg or libclangcrates/.cargo/config.tomlsets both in a global[env]; cargo has no[target.<cfg>.env], so macOS got the Windows pathsmetal-rs0.29 APITextureType::Type2D,LoadAction,Texture::from_raw,Option-returning encoders,get_bytes(x,y,w,h)— none exist as writtenBeyond the compositor
compositor_windows.rs.ci.yml's only Rust job was the macOS one; the 3654-line D3D11 engine was built exclusively bybuild.yml, which triggers onpush: tags: v*. A typo surfaced at release time. Closed byci(windows)— green before any refactor touched that file.npm run buildfailed attsc(a leftover import from commit290691ad), sobuild:win/build:mac/build:linuxall stopped there.compositorViewServicecould not find the addon in a dev run or a packaged macOS build —app.getAppPath()is<repo>/dist-electronunpackaged, andextraResourcesputs the addon atprocess.resourcesPath, never inside the asar.createWindow()sat behind anawait askForMediaAccess("microphone").Docshad been red since the per-platform rename, 14 commits earlier.SIGKILL (Code Signature Invalid), because overwriting a.nodein place leaves stale pages on a vnode with a new signature. Now installed by atomic rename.Architecture: one definition, not two
The port started by duplicating; that had already produced divergence. Now shared,
pub mod, nocfg:frame_geometry.rs— the geometry both engines call, includingplan_frame: the 353 lines ofcompose_framethat contained no GPU call at all, returning the 15 values (of 75 locals) that survive to the first draw. The draw half ofcompositor_windows.rsis byte-identical across that refactor, verified mechanically.timeline_walk.rs— the clip walk MP4, GIF and both backends share.LayerCB/LiveParams— one each. Unifying them surfaced thatwebcam_shape_codereturned3("rounded") on Windows and0("rectangle") on macOS, so one scene described a rounded camera on Windows and a rectangular one on macOS.The engine is now complete
Every layer the D3D11 engine draws, the Metal engine draws: background (colour / gradient / image, with the dual-Kawase blur), screen with padding, corner radii and drop shadow, the PiP camera with its own shadow, the cursor with its motion-blur trail, the 3D tilt with its projected-quad shadow, and all four annotation kinds — arrow, blur/pixelate mask, image, and text through the CoreText rasterizer.
h264_videotoolboxzero-copy — the composed frame is rendered straight into the encoder'sCVPixelBufferand never returns to the CPU — plus an AAC stereo 48 kHz track, clean underffmpeg -f null -. Measured on a real 22 s project: 12 s before, 8 s after, a 1.5x speedup. The decoder is chosen on the H.264 profile, because VideoToolbox is 13x slower than software decode on the Constrained Baseline that openscreen's own capture produces.Nine shader divergences, found by diffing line by line
Diffing mode by mode is what missed them the first time. Three were live:
clamp(d*1.5)poutin [0,1] todst_prev~1600 pxspreadfrom the wrong field, no radius insetline_crossdefined, never calledclipToBoundsinertarrow_local_geometryWhat "ISO" means here, measured
The two backends cannot run on the same machine, so iso cannot be established by comparing two rendered images. What can be, and what is the layer where divergence actually happened, is the geometry:
plan_frameis literally the same code on both sides, andplan_frame_is_pinned_bit_for_bitpins its 15 outputs withto_bits()— in both CI jobs. If the two platforms ever compute different placements, one goes red.Rasterisation is explicitly not claimed bit-identical: PR #162 measured 93-95 % of channels identical, max deviation 3/255, between two backends on the same machine. Shader parity is held by the line-by-line diff above.
Still missing
.nodehardcodes absolute ffmpeg dylib paths; shipping needs aninstall_name_tool/@rpath pass plus the dylibs inextraResources.scripts/build-macos-compositor-addon.mjs --print-ffmpeg-recipeprints an LGPL-shared configure line. Homebrew's ffmpeg is GPL: fine to develop against, never to ship.Type of change
Release impact
Desktop impact
Testing
cargo check -p openscreen-compositor -p compositor-view-napi --all-targetsonwindows-latest, every PR — new.tsc, biome, i18n, docs all green.