feat(compositor): port the native compositor to Linux (Vulkan backend) - #183
Draft
EtienneLescot wants to merge 134 commits into
Draft
feat(compositor): port the native compositor to Linux (Vulkan backend)#183EtienneLescot wants to merge 134 commits into
EtienneLescot wants to merge 134 commits into
Conversation
…dels (#181) ChatAnthropic defaults maxTokens from a table of known Claude slugs and falls back to 4096 for anything else (MiniMax-M3, self-hosted models). With thinking enabled, a cold-start turn can spend that whole budget on reasoning and truncate with stop_reason=max_tokens before any text block, surfacing as an empty first response that succeeds on retry. Give the minimax and non-Claude anthropic models an explicit 16384 budget (matching the known Claude 4.x/5.x default). Known claude-* slugs keep LangChain's per-model values, since those are hard limits (e.g. claude-3-haiku caps at 4096). The OpenAI-shaped transports send no max_tokens by default, so there is nothing to floor there.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
EtienneLescot
force-pushed
the
feat/linux-compositor-port
branch
from
July 28, 2026 07:49
19c8314 to
f3aa8e5
Compare
EtienneLescot
changed the base branch from
release/v1.8.0
to
feat/d3d-warp-fallback
July 28, 2026 07:49
The rename button in the v4 in-renderer topbar had `style={{ all: "unset" }}`
on it, which clobbered the `-webkit-app-region: no-drag` rule inherited from
`.topbar button` in EditorShellV4.module.css. The button then became a
window-drag region: clicking the project title dragged the window instead
of firing the onClick handler that opens the inline editor. That is the
behaviour the v1.8.0.rc4 bug report describes — "the header title either
does not respond to the rename gesture, the inline editor doesn't appear".
Apply the class directly to the button (drop the wrapping span and the
`all: "unset"` reset) so the topbar no-drag rule applies. While here,
collapse the three pieces of component state (editing, draft, inputRef +
useEffect for focus) into a render switch with autoFocus + onFocus select,
move the inline input style into the CSS module, and add a `:disabled`
state for `.ghostBtn` so the no-project case is visually consistent with
the other topbar buttons.
Cover the rename flow with a unit test (Enter / blur commit, Escape
cancels, whitespace-only is rejected, no-project is disabled, and a
regression check that no element in the topbar reintroduces `all: unset`).
EtienneLescot
added a commit
that referenced
this pull request
Jul 28, 2026
Closes part of PR #183 (the scoping body). Defines the target architecture (wgpu backend mirroring the D3D11 seam #162 introduced), the per-platform frame contract, the WGSL shader port inventory (1:1 with shaders.hlsl's 9 entry points), the cosmic-text replacement for text.rs, the build/CI plumbing for Linux, and a WP0..WP7 milestone plan. Adds a Linux port subsection to native-compositor.md and a doc-table entry to README.md. The spec body of PR #183 is updated separately to reference this doc.
EtienneLescot
added a commit
that referenced
this pull request
Jul 28, 2026
…d skeleton Implements the foundation of PR #183. WP0 — Cargo target gating - workspace deps: wgpu 24 (cross-platform, Vulkan/Metal/D3D12; pinned to the windows-0.58-compatible line so the existing D3D11 path stays on its validated toolchain), pollster for block_on. - compositor/Cargo.toml: wgpu + pollster deps cross-platform, the windows dep kept for the D3D11 path. - compositor-view-napi: the windows dep stays; lib.rs is split — the existing napi surface is cfg(windows)-gated into windows.rs, a non-Windows placeholder stands in until WP7 ships the Linux napi skeleton. - compositor/src/lib.rs: cfg(windows) on d3d/cpu_frames/text/compositor/ live/pipeline (the D3D11 path); portable modules + new vk* stay ungated so the crate compiles everywhere. - poc-d3d: cfg(windows) on the Win32 body; non-Windows stub so `cargo check` on the target stays green. - compositor/build.rs: cross-build detection via cfg!(host_os) vs CARGO_CFG_TARGET_OS. When target != host, skip cc/bindgen and write a cached or stub ffi.rs so `cargo check --target x86_64-unknown-linux-gnu` from a Windows host passes without a cross-toolchain. Path separator made platform-aware. WP1 — vk.rs (wgpu device + probe + lavapipe classification) - VkGpu { device, queue, backend, adapter_info } mirrors d3d::Gpu shape. - create (hardware-strict, for tests/goldens), create_blocking, create_auto (fallback HW -> lavapipe; full semantic equivalence with d3d::Backend in WP1+). - probe cached via OnceLock; returns the same "none" semantics as a missing addon on a machine without a Vulkan implementation. - classify() detects lavapipe by adapter name on Linux (llvmpipe / lavapipe substrings) and WARP on Windows — both routed to BackendKind::Cpu so probeBackend() reports "cpu" exactly as the Windows WARP rung does today. - Unit tests: probe-doesn't-panic, classify-hardware, classify-lavapipe. WP2 — vk_frames.rs (NV12 upload skeleton) - VkFrameTex { y: r8unorm, uv: rg8unorm, width, height } mirrors the D3D11 NV12 texture that cpu_frames.rs presents today; the AVFrame carrier (data[0] = Box ptr) is the contract #162 introduced, generalised to carry the platform payload through crate boundaries. - VkFrames: AVFrame slot (av_frame_alloc), ensure_textures(w, h) creates the two R8/Rg8 textures when dimensions change. - Drop frees the AVFrame slot. - The swscale -> AV_PIX_FMT_NV12 -> queue.write_texture path lands in a follow-up WP2 commit; this commit establishes the seam and types. vk_render.rs — module skeleton with VERSION constant and a test. vk_shaders/blit.wgsl — minimal placeholder WGSL (fullscreen triangle + black clear) for the include_str! source of the slice renderer. Verified: - `cargo check --workspace` green - `cargo check --workspace --target x86_64-unknown-linux-gnu` green (uses OPENSCREEN_BINDGEN_CACHE_PATH to feed the bindgen output from a prior host build into the cross-build — see build.rs write_stub_ffi.) WP3+ (slice renderer with WGSL core shaders and cross-backend golden diff) is scoped in linux-compositor-port.md but lands in follow-up commits to keep this PR reviewable.
Drops the chat-panel '+ skip' control, its six i18n keys across all 13 locales, and the runAddTrim handler / useOptimisticTimelineOps hook that existed only for it (the hook had no other callers and is removed with the file). Refs #182.
After removing the chat-panel '+ skip' button (the only renderer-side caller of nativeBridgeClient.aiEdition.runTimelineOperation), the entire IPC chain is dead: the renderer method, the action variant in the IPC request union, the AiEditionService method, the bridge wiring, the IPC handler, and the 'timeline.run' case in the dispatcher. The backend runTimelineOperation in chat-service.ts is kept (it is a clean main-process API for any future caller). The chat-service.test.ts suite that exercises it is also kept. Refs #182.
The chat-panel '+ skip' removal (7e2bcaf) left the transcript-pane trim handlers (handleAddTrimRange / handleRemoveTrimRange) as the last user of the saveQueueRef pattern — and they were untested. The pattern is non-trivial: a serialised save queue where the doc read happens INSIDE the chain (after awaiting the previous save), otherwise two rapid Backspaces would race and clobber each other's edit. Extracts the pattern into src/lib/ai-edition/store/useSequentialTimelineOps.ts with the same three properties the inline version depended on: 1. Two concurrent apply() calls are serialised — op N+1 reads the doc op N committed, not the pre-op-N doc (race fix preserved). 2. A save rejection doesn't poison the queue; the next call still has a resolved promise to chain off (error swallow preserved). The original promise returned to the caller keeps its rejection. 3. The store-empty fallback (no project loaded) returns null instead of crashing. The hook is generic over the operation type (add_trim_range, remove_trim_range, future timeline ops), not hard-coded to trim ranges — the transcript-pane handlers are now a one-line apply() call each, and any future renderer op (e.g. a timeline ruler drag-to-trim) can use the same hook instead of reinventing the queue. External signatures of handleAddTrimRange / handleRemoveTrimRange are unchanged (RightPanes doesn't await the return value). Includes a 4-test suite covering all three properties. NewEditorShell.tsx: -53 / +18 lines.
…satisfy biome - Add cameraTrack: null to the asset fixture (the asset schema requires it after the v3 -> v4 migrate moved cameraTrack onto each asset). This was a new error on PR #187 that pushed the test-typecheck baseline from 80 to 81. - Run biome check --write on the three files the lint job flagged (LeftPanel.tsx, NewEditorShell.tsx, useSequentialTimelineOps.test.ts) for the unsorted-imports and formatter errors. tsc, biome, vitest (1147/1148 — the one fail is the pre-existing Windows temp-dir race in document-service.test.ts, passes 23/23 in isolation), i18n and docs checks all stay clean.
EtienneLescot
added a commit
that referenced
this pull request
Jul 28, 2026
…ackend skeleton The cross-backend architecture for the Linux port. The trait `CompBackend` is the seam between the orchestration math (which stays in `Compositor`) and the per-platform GPU work (which moves behind the trait). Two impls exist: - `D3d11Backend` (Windows): wraps the existing `Compositor` and forwards each trait method. The 3000-line `compositor.rs` body is preserved byte-for-byte — `upload_cb` becomes `pub(crate)` and the `LayerCB` struct moves to `backend/mod.rs` so both backends see the same type. All 86 tests still pass on Windows with no behavioral change. - `WgpuBackend` (Linux/macOS, cfg-gated to `not(windows)`): implements `CompBackend` using the same wgpu primitives as `vk_render.rs`. The 8 minimal trait methods (`render_size`, `bind_compose_state`, `begin`, `upload_cb`, `fs_pass`, `blur_bg`, `draw_video`, `draw_solid`, `draw_shadow`, `draw_quad_shadow`) cover the compose_frame path. The slice test continues to use `vk_render::render_slice_c1` directly for now (proven working); the trait-dispatched path will be wired up in WP4+ as `compose_frame` migrates. ### Trait scope (PR #183, première vague) The trait covers **the compose_frame path** — the 8 methods that `compose_frame` invokes to assemble a frame. Out of scope (still on `Compositor` directly, will migrate as needed): - encode-side (`fs_pass` for `ps_y`/`ps_uv`/`ps_blur`) → WP6 - background blur chain (`blur_bg` Kawase) → WP4 - cursor draws (sprite, themed, math) → WP4 - image background (`draw_image_bg`) → WP4 - annotations (`draw_annotations`) → WP5 (text_cosmic) + WP4 This minimal scope protects the Windows path: the refactor is borné et testable. `WgpuBackend` n'expose pour l'instant que ce qu'il sait faire (les opérations du slice WP3). ### Cross-platform types `LayerCB` (the 128-byte constant buffer / uniform struct) is now defined in `backend/mod.rs` as `pub struct LayerCB` (platform-neutral — same layout as the HLSL `cbuffer Layer` and the WGSL `struct Layer`). `compositor.rs` imports it via `use crate::backend::LayerCB;` (not `pub use`, which would create a new type). `VkGpu` now derives `Clone` (cheap Arc inside `wgpu::Device`/`wgpu::Queue`) so backends can share it cheaply. Verified: - `cargo check --workspace` (Windows host) green. - `cargo check --workspace --target x86_64-unknown-linux-gnu` green (`WgpuBackend` compiles non-Windows). - `cargo test --tests -p openscreen-compositor`: 86 tests pass — 80 unit (unchanged), 3 integration (unchanged), 1 vk_cross_golden slice (unchanged), 2 vk::tests (unchanged). - The Windows `D3d11Backend` is **a forwarder only** — no D3D11 method body was copied. Behavior is bit-identical to before this commit. - The `vk_cross_golden` slice test still passes against the `vk_render` standalone renderer; it is the production path's responsibility (WP4+) to wire `compose_frame` through `CompBackend`. This is the architecture commit. WP4+ extends both backends to fill in the unimplemented methods (blur chain, cursor, image bg, annotations, encode-side) and wires `compose_frame` / `live.rs` / `pipeline.rs` / `text.rs` / `napi` to dispatch through `Compositor::b: Box<dyn CompBackend>`. The trait surface (10 methods) is enough to demonstrate the architecture end-to-end while keeping the diff reviewable. A full `compose_frame` migration would be a follow-up that lifts every method body into the backend (and is the bulk of WP4+).
Anthropic and MiniMax with thinking enabled (our default) stream
reasoning as content parts of type "thinking" alongside the visible
text. LangChain surfaces these in on_chat_model_stream chunks, but the
deep-agent sink only forwarded the text parts — so the chat panel sat
empty during a long reasoning phase, even though the model was working
through a complex edit. Users perceived the silence as a hang.
Plumb the thinking text through the same channel as the visible text
end-to-end:
ChatAnthropic "thinking_delta"
-> AIMessageChunk.content[{type:"thinking", thinking}]
-> messageContentToThinking() in chat-model.ts
-> sink.thinking(delta) in deep-agent/service.ts
-> ChatEventSink.thinking in chat-service.ts
-> {kind:"thinking", sessionId, delta} IPC event
-> LeftPanel.tsx ThinkingBlock (ephemeral 2-line preview,
click-to-expand to full scrollable text)
While the run is in flight the block lives below the assistant
author with a spinner; once chatRun resolves the trace moves onto the
finished assistant message as a collapsed Thinking section (same
component, per-message expand state) so the reasoning stays available
to revisit.
Also fix a flicker where every thinking delta called
setThinkingExpanded(false), collapsing the block the instant the user
clicked to expand it. Run boundaries (send() start, finally) still
reset expand state, so the choice persists for the duration of the
run.
Tests:
- messageContentToThinking unit tests (concatenates thinking parts,
skips redacted_thinking, handles non-array input).
- chat-service toolloop test exercises interleaved text + thinking +
tool events through the sink.
No new i18n keys: the existing chat.thinking label fits both the
ephemeral and persisted contexts.
Issue #178 — captions appear in the preview (DOM overlay + native compositor) but the exported video is missing the background plate, so depending on the recording's brightness the user reads it as 'no captions in the export'. Root cause: the caption inspector stores colour and opacity as two separate fields, and \captionBackgroundCss\ recombines them into a CSS string the editor overlay can render directly (e.g. \ gba(0, 0, 0, 0.55)\). The native side's \parse_hex\ only understood 3- or 6-char hex strings, so anything else — including the caption background — fell through to the \[0, 0, 0, 0]\ fallback (alpha 0, no plate). The text was drawn, the plate was not, and the contrast that made captions legible in the preview disappeared in the file. Fix: teach \parse_hex\ to accept the CSS surface area the JS bridge already produces — \ gba(...)\, \ gb(...)\, and \ ransparent\ — alongside the existing hex format. The fallback to None (and from there the caller's \[0,0,0,0]\) is preserved for everything that isn't a recognised colour, so a regression in any of the existing annotation colours still surfaces as a missing element rather than a wrong one. This also unblocks the gradient stop path, which has been sending the same \ gba(...)\ strings through \parse_hex\ for the same reason — they silently fell back to the colour preset. Not the issue's headline, but the same fix. The JS contract is now pinned by a test in \sceneDescription.test.ts\: the caption background leaves the bridge as \ gba(0, 0, 0, 0.55)\ exactly, and the text colour remains the ColorField hex (\#ffffff\). A future rewrite of either side that drops the alpha will fail this test instead of silently regressing the user-visible output.
…ollow CSS arity Review follow-up on the caption-plate fix. `strip_color_fn` sliced `s[..name.len()]` without checking the UTF-8 char boundary, so any colour string whose byte 3 or 4 lands mid-character aborted the process instead of returning None. `parseWallpaper` hands every `#`-prefixed string straight through to `SceneBackground::Color`, so `#ab€cd` was enough to take out the render loop — and a panic crossing the N-API bridge is exactly what this parser's None-then-caller-fallback contract exists to avoid. Taking the tail with `get` first proves the boundary, which makes the head slice safe by construction and makes the old length guard redundant. The hex path had the same latent bug independently (`h[i..=i]` / `h[0..2]` on a body of exactly 3 or 6 bytes, e.g. `éa` or `€€`); an `is_ascii` guard closes it before any slicing happens. Both predate the rgba work, which only widened the first one's reach. Also collapses the two component parsers into one. CSS Color 4 makes `rgb()` and `rgba()` synonyms, both taking 3 or 4 components, so rejecting `rgba(0, 0, 0)` was non-standard — and it failed the same silent way #178 did, by falling through to the caller's alpha-0 fallback and painting no plate at all. Tests: 83/83 in the compositor lib (81 before, +2 here).
The preview showed every caption twice, most visibly at widths where the two copies wrapped differently — one broke to a second line, the other did not. `PreviewCanvas` hosts the native D3D canvas as "the sole pixel source" (its own header comment) with the DOM overlays kept interactive-only. `CaptionLayer` was not interactive: `aria-hidden`, `pointerEvents: none`, and a visible `<span>` painting the cue text and its background plate. Meanwhile the native compositor already draws the same cue, because captions are emitted into the scene as ordinary text annotations. Two painters, two line-breakers — CSS `word-break`/`pre-wrap` against DirectWrite laying out into `box_px` — so the same string wrapped at two different points. The component's own comment explains how it got here: it "mirrors annotationRenderer.renderText's box model ... so what the preview shows is what the export writes". That mirror was right when the preview was DOM-drawn and the exporter was a separate renderer; it became a duplicate once the native compositor started drawing the preview too. Deleted rather than gated: there is no non-native preview path left to fall back to (`VITE_NATIVE_COMPOSITOR` is read nowhere in the tree, and `NativeCompositorOverlay` mounts unconditionally). Dropping the DOM copy also makes preview and export agree by construction instead of by two implementations of the same box model. `useCaptions` stays — `CaptionsPane` still uses it. Pre-existing, not introduced by the plate fix: the native side always drew the caption text, it just drew it without a plate, so the duplicate read as a faint ghost. Restoring the plate made both copies solid and the doubling obvious. Tests: 96 files / 1061 pass, tsc clean, biome clean.
The caption doc still described a DOM preview painter alongside the native exporter, and linked to `CaptionLayer.tsx`, which no longer exists — the docs check failed on the dead link. Rewritten around what is actually there: one path, cues -> synthetic text regions -> annotation plumbing -> native compositor, which draws preview and export alike. That is a stronger version of the property the old text was reaching for: preview and export cannot drift because they are the same renderer, not two box models kept in sync by hand. Kept a short note on why the DOM layer existed and why it went, so the next reader does not re-add a "preview overlay" to fix a perceived gap. Also recorded that `captionBackgroundCss` emits `rgba(...)`, which is what forces the native colour parser to accept CSS colours rather than hex only. `docs:check` OK (22 files).
Closes #179 When a zoom region is active, the screen content was confined to the padded screen rect, so the zoomed viewport stopped at the padded boundary instead of reaching the edges of the output frame. Expand s_dst / s_dst_prev to the full output frame while p.zoom > 1.0 so the zoomed content overflows the padding as expected. The shadow and rounded corners currently follow the expansion too (drawn into the same rect via the pixel shader) — noted as a follow-up TODO in the comment.
…m overflow The previous fix extended s_dst to the full output frame whenever p.zoom > 1.0 so the zoomed viewport overflows the preview padding (issue #179). But s_min_px and s_dst now both refer to the full output frame, so the existing shadow and rounded-corner code scaled up too: * shadow: drawn around [0,0,1,1] with its 40px spread — reads as a black band against the frame edges, not as a shadow. * rounded corners: s_min_px collapses to min(rw, rh) and the two radius formulas (block preset, roundness slider) coincide on that value, so * min(rw, rh) rounded the entire output frame — 43px on 1080p. Neutralize both when p.zoom > 1.0: no shadow, no radius. The content reaches the edges unframed, which matches the issue's " reach the edges of the
…nary Follow-up to the original issue #179 fix and the f7c4317 bandaid. The two earlier commits each made the wrong call: the original swap to [0,0,1,1] in one step (abrupt padding disappearance), and f7c4317 kept the same switch but neutralized the shadow and radius to mask the regression (still abrupt on the padding ring, and now also abruptly loses the frame on zoom engage). The right behavior: the "zoom frame" (s_dst) grows continuously with the zoom, from the padded size up to the full frame. At zoom = 1 it's the padded area. As the zoom ramps up, the frame expands until it hits the frame edge at zoom = 1 / padding_scale (= frame / padded_size). Past that the frame stays put and only the source rect keeps shrinking — the GPU upscales further. The source texture itself is never touched, so full resolution is preserved all the way; only the mapping changes. s_radius and the shadow follow s_dst naturally now (s_min_px grows with the zoom frame), so the f7c4317 gates on those are removed. Result: no more binary switch anywhere. The padding smoothly fades as the zoom ramps up, the frame "follows" the zoom, and by the time the content hits the edges the frame is already at full size — visually consistent. Single TODO kept, pointing at the still-pending "frame rect" separation: if you want the shadow and corners to stay anchored at the padded box while the content overflows (the "window" effect), split the frame rect from the content rect and apply shadow+corners to the frame alone.
Review follow-up on the three earlier #179 commits. They all grew the screen box (s_dst) while STILL passing the full p.zoom to screen_source_rect, so the two multiplied: with padding 50% (padding_scale 0.8, cap 1.25) a depth-1 region authored at 1.25x rendered at 1.56x, a depth-4 at 2.75x. The reference does not do that — applyZoomTransform (TS) scales the camera container by exactly zoomScale and never crops the source on top. Give the box a share g of the zoom and take that same share OFF the source cut (p.zoom / g). The product stays zoom, so: * the magnification is exactly what the region asks for; * (box, cut) is one affine image->screen map and splitting it does not move that map — the focus lands on the same pixel it did before. Everything riding the map (cursor layer, 3D tilt, motion blur) is therefore unchanged by construction. Only the drawn EXTENT grows, which is precisely what the issue asks for. Two bounds on g, both derived from the geometry rather than from padding_scale: * cover — the size that covers the frame. Past it, growing adds nothing (the rasterizer clips). Taken from the real rect, so a letterboxed crop (16:9 in a 9:16 output) also eats its bars; the old 1/padding_scale cap never reached them. * focus — the source cut must stay centreable on the target, else screen_source_rect pins it to the crop edge and the zoom stops aiming. Same constraint getFocusBoundsForScale (TS) already applies upstream, so it never blocks growth for an app-authored focus. Also drops the [0,1] clamp on the grown box: it slid off-centre rects (block presets) sideways as they grew, and pinned the rounded corners and shadow to the frame edge instead of letting them leave it — the artifact the second commit had neutralized the radius and shadow to hide. Uncapped matches the reference ("No stage clamping", frameRenderer.cameraAwareMaskRect); the rasterizer clips the overflow, as it already does for the blurred background. screen_zoom_growth / grow_around_center are free functions next to screen_source_rect so they can be tested; the previous closure could not. Three tests, the first of which fails on any of the three earlier versions: the map-invariance sweep (magnification and focus position unchanged across zooms and focus positions), the frame-edge reach, and the focus-limit arbitration. cargo test -p openscreen-compositor --lib --tests: 79 + 3 pass. Bench note: the fixture (zoom 1.8, focus [0.5, 0.32], 5% margin) now grows its box at deep zoom, so bench frames change there by design.
Fixes the regression in f2c1faa, caught in manual testing: the padding still constrained the zoom. That commit capped the box growth by a "focus budget" — g = min(zoom, cover, zoom * 2*min(c, 1-c)) — meant to keep the source cut centreable on the focus. But the zoom focus follows the cursor and is almost never centred, so the third term dominated: focus 0.3 at zoom 1.5 gives 0.9, clamped back to 1.0, i.e. no growth at all. The fix was inert in the common case. The tests missed it because they swept centred focuses and focuses pinned exactly on their bound, never an ordinary off-centre one. Root cause: that commit refused to TRANSLATE the box (to avoid re-deriving ease() in regions.rs). Without translation the only lever left to preserve framing was refusing to grow — so it protected the framing by giving up the point of the PR. So translate. The box now takes the whole zoom and the drawn cut goes back to the bare crop, which is applyZoomTransform's geometry. The box placement is not a formula to get right: remap_box() carries the new cut through the old `cut_ref -> base` mapping, so the framing is preserved by construction — crop, edge clamp and cover are all already baked into the two cuts. Nothing left to protect, hence nothing to refuse. Consequences, all of them wanted: * the box grows by exactly `zoom` for every focus, so the padding is eaten from the first frame of the ramp; * src == src_prev now, and the zoom velocity the motion blur reads lives in dst vs dst_prev instead of in the cut; * an edge-focused zoom keeps the padding on the side where the image runs out — there is no content to put there, and the reference shows background in the same spot. Tests rewritten around the two things that pull against each other: handing_the_zoom_to_the_box_moves_no_pixel (no point of the image shifts) and any_zoom_overflows_the_padding (the box always takes the zoom). Both sweep off-centre focuses including the [0.0, 1.0] corner; the second fails on f2c1faa. cargo test -p openscreen-compositor --lib --tests: 78 + 3 pass. Verified in the app with a rebuilt addon.
…ure flag Findings from the v1.8.0 ponytail audit (see also PR #187): - Delete \src/utils/getTestId.ts\: the TestId type and getTestId() helper are not imported anywhere; tests use hardcoded data-testid strings. - Delete \src/components/video-editor/featureFlags.ts\: the single constant \AI_FEATURES_ENABLED = true\ is referenced only by a doc comment and by \LeftPanel.tsx\ as a runtime gate. Inline the gate in the two sites that used it and update the architecture docs to match. The new editor ships as the default from Phase 1 PR 1.3 onward; an always-true flag was carrying its own explanation as cargo. - Drop the \MIN_DELTA\ and \VIEWPORT_SCALE\ exports from \src/components/video-editor/videoPlayback/constants.ts\: only the source file references either. - Demote \RenderableChatMessage\, \estimateTokens\, and \DEFAULT_CHAT_BUDGET_TOKENS\ in \src/components/ai-edition/chatBudget.ts\ to file-private — no external importer. - Delete \scripts/bench-export.mjs\ (retired harness, runner already removed per rendering-performance.md:371), \scripts/stt-dev-server.mjs\, \scripts/e2e-pipeline-smoke.mjs\, \scripts/e2e-stt-smoke.mjs\. None are wired into \package.json\ scripts. - Tidy the now-stale comment in \electron-builder.json5\ that explained the \ fmpeg.exe\ exclude via \�ench-export.mjs\. No behavior change: tsc clean, lint unchanged, 1144/1144 unit tests pass.
fetch-ffmpeg.mjs still justified keeping ffmpeg.exe 'so scripts/bench-export.mjs can use it' — the same stale reference this PR already fixed in electron-builder.json5. The real reason it stays is the licence check below it. rendering-performance.md still said bench-export.mjs 'survives', and ai-agent.md claimed the no-provider state renders no chat rail; the rail entry is now unconditional and opening it shows the welcome view.
Drops the 126-line bespoke loader in src/i18n/loader.ts for a ~70-line
wrapper around i18next. Locale-fallback, {{var}} interpolation, defaultNS,
and fallbackLng come from the library; we keep the same four exports
(getAvailableLocales, getLocaleName, getLocaleShort, translate) so
LaunchWindow, EditorTopBar, and I18nContext don't change.
Public API preserved: I18nContext still calls translate(locale, ns, key, vars);
the new translate is a thin shim around i18next.t. Miss behavior matches the
old loader (\{namespace}.{key}\ on miss, locale code on locale-name miss).
Test results:
- tsc --noEmit: clean
- vitest (i18n + LaunchWindow + components): all pass
- biome check: clean
…marker fallback
The stored value fed only init({lng}), but every lookup passes an explicit lng,
so it never affected output — while adding a SecurityError path at module scope,
where it takes down the whole module graph instead of one component. I18nContext
already owns the preference behind a try/catch.
returnedObjectHandler restores the old loader's behaviour for a non-leaf key:
fall through to the 'namespace.key' marker rather than rendering i18next's
English developer message into the UI.
"70 fps is not great on this machine" — correct, and profiling put the cost somewhere
I had not looked. Per 120-frame export: compose 0.027s, encoder submit 0.426s, drain
0.000s, total 1.76s. **74 % of the wall time was decode**, and the compositor itself
runs at 0.2 ms/frame.
Measured on a 1920x1080@60 Constrained Baseline capture, decode only:
VideoToolbox 240 frames in 1.11s 215 fps
libavcodec sw 240 frames in 0.08s 3000 fps 13x
End to end on the export, 76 fps against 182 — 2.4x — even though the software path
additionally pays swscale and a full memcpy into the IOSurface every frame.
The reason is structural: the hardware decoder has a fixed per-frame latency and
allocates a CVPixelBuffer/IOSurface for each, while a trivial profile decodes in a few
hundred microseconds across cores that are, unlike the decode block, plural. Baseline
is what openscreen's own capture produces (`crates/fixture/fixture.json`, profile_idc
66) and what Chrome's MediaRecorder emits — the common case, not an edge one.
So the choice is made on the profile rather than by flipping a default: Baseline and
Constrained Baseline decode in software, everything else (High, 10-bit, HEVC, 4K, where
the arbitration reverses and software becomes the bottleneck) keeps VideoToolbox.
`OPENSCREEN_MAC_DECODE=software|videotoolbox` overrides, which is also how the numbers
above were taken.
Also removed two per-frame pipeline stalls: `compose_frame` and `render_nv12` each
committed and immediately waited, draining the GPU before the CPU had anything to do.
They now submit and track; the wait happens where a CPU read actually occurs. And the
full-resolution RGBA mirror blit moved into `readback_direct`, so the export — which
never reads RGBA — stops paying for it.
Export output still decodes clean under `ffmpeg -f null -`.
The macOS export rendered NV12 into internal textures, read 1.4 MB back to the CPU, and handed that to VideoToolbox — which uploaded it again. `rgb_to_nv12` took an `out_tex` argument and ignored it, a shortcut from the original port. It now honours it. `out_tex` is the `CVPixelBufferRef` of a frame pulled from the encoder's own VideoToolbox pool; both planes are wrapped as `MTLTexture` through the same `CVMetalTextureCache` the decoder uses, and the two fullscreen passes target them directly. The frame never leaves the GPU. This is the macOS shape of the zero-copy the D3D11 engine has always had. `ExportCodec::candidates()` now lists `h264_videotoolbox` twice — once on `AV_PIX_FMT_VIDEOTOOLBOX`, once on NV12. If the hardware pool refuses to open, the existing candidate walk falls through to the software-frame variant: same encoder, one CPU round trip more. Nothing is lost when zero-copy is unavailable. **Measured honestly: no throughput gain on this workload.** 176 fps, against 172-176 for the readback path. Decode still dominates the frame budget, so removing work from `send` does not move the total. It is kept because it is the architecturally correct path, it removes real per-frame work, and it will matter as soon as decode stops being the ceiling — not because it made this measurement faster, which it did not. Output validated: 1280x720 H.264 + AAC stereo, clean under `ffmpeg -f null -`, frame content real (mean 201.7/255, not a black field). One trap worth recording: the `hw_frames_ctx` this path needs had been silently lost — a `cp` restoring a backup after a profiling run clobbered it. `avcodec_open2` succeeds without it, `sw` stays null, and the first `av_hwframe_get_buffer` segfaults on a null pointer with no Rust error at all. The crash report was the only thing that said so.
Two gaps that both ended at the same place: a packaged macOS app whose addon would not load, or would load something we are not allowed to ship. **Packaging.** Straight out of cargo, `compositor_view.node` references its ffmpeg dependencies by ABSOLUTE path — `otool -L` showed `/Users/…/crates/thirdparty/…`. That resolves on the build machine and nowhere else; an installed app would die at `require()` naming a directory the user has never had. Windows does not hit this because `LoadLibrary` searches `PATH`, which is what `ensureFfmpegSharedDllsOnPath()` prepends; macOS has no equivalent search, so the fix has to be baked into the binary. The build script now copies the five dylibs next to the shipped `.node`, rewrites every install name and inter-library reference to `@rpath`, and adds `@loader_path` as an rpath — which is exactly where the mac `extraResources` filter `darwin-*/*` already puts them. Re-signing after `install_name_tool` is not optional: it invalidates the ad-hoc signature, and macOS then kills the process with the same `SIGKILL (Code Signature Invalid)` documented on `installAtomically`. Verified: no absolute path left, and the addon still loads and probes `hardware`. **Licence.** `npm run fetch:ffmpeg:mac` vendors ffmpeg 8.1.2 from the pinned release tarball, checksummed, configured with neither `--enable-gpl` nor `--enable-nonfree`. It builds rather than downloads, and not by preference: BtbN — whose pinned `-lgpl-shared` archives `fetch-ffmpeg.mjs` fetches for Windows — publishes no macOS target, and every macOS binary that does circulate (evermeet, osxexperts, Homebrew) is GPL, because they ship x264/x265. Linking any of them would relicense this MIT app. Convenience does not make them candidates. The result is verified by asking the binary (`ffmpeg -L`) rather than by trusting the configure line — the same check `fetch-ffmpeg.mjs` runs on Windows, and the one that catches a tree swapped by hand for a GPL one. Both scripts refuse to proceed on a non-LGPL tree, and the fetcher deletes a build that fails the check rather than leave it where `build.rs` would link it. `build:mac` runs it before the addon build.
…o v1.8.0 release/v1.8.0 added an SRV-cache invalidation call in the portable live.rs (comp.clear_srv_cache(), called unconditionally whenever a decoder set is closed) plus a decoder fast-seek path in the Windows Decoder. Both landed during the rebase of this branch, but only pipeline_windows.rs got the method/the fast path applied — compositor_macos::Compositor had no clear_srv_cache at all, which doesn't fail until compiled on macOS. clear_srv_cache() on Metal calls CVMetalTextureCache::flush() — there's no address-keyed HashMap to clear like on D3D11 (CoreVideo already dedupes by IOSurface, not by pointer), so flushing the cache IS the equivalent operation. Also ported the Decoder::seek_to fast path (cur_pts, decode_forward_to, SEEK_FORWARD_MAX_SEC) into pipeline_macos.rs so both engines stay at the same seek performance instead of drifting apart again right after the last ISO-parity pass.
…nd architecture
release/v1.8.0's macOS/Metal port unified the cross-backend layer differently
from the original Linux port: per-platform modules aliased to generic names
(compositor / d3d / text / pipeline / cpu_frames), a shared
frame_geometry::plan_frame (the geometry half of compose_frame, iso-render
across backends), and shared live.rs + compositor-view-napi. This re-expresses
the Linux port on that architecture instead of the old parallel CompBackend
trait.
New per-platform Linux modules (cfg(target_os = "linux"), aliased in lib.rs):
- d3d_linux (Gpu/Backend, wgpu/Vulkan) from vk.rs
- linux_frames (CpuFrames, NV12-split upload) from vk_frames.rs
- text_linux (cosmic-text rasterizer) from text_cosmic.rs
- compositor_linux (renders FrameGeometry via WGSL) from vk_render + layer.wgsl
- pipeline_linux (Decoder; export is a WP6 stub) from pipeline_lin + vk_decode
plus linux_decode (SwDecoder helper) and vk_shaders/{layer,blur}.wgsl.
build.rs gains a Linux branch (wrapper_linux.h; ffmpeg without D3D11VA/VideoToolbox);
compositor/Cargo.toml gains cfg(linux) deps (wgpu / pollster / cosmic-text);
poc-d3d (the Win32 bench POC) is cfg(windows)-gated so the workspace builds on Linux.
Verified on Linux (dzn/AMD): `cargo check --workspace` is green, and the
compose_linux test renders a real 960x540 frame through the shared plan_frame
geometry (mean_R = 202.6). compose_frame draws the core today (solid background
+ screen cover-fit with rounded corners); webcam PiP, cursor, annotations
(WGSL mode 11 text), background blur and 3D tilt reuse the same draw primitives
and land incrementally -- now sharing the Metal port's cross-backend plumbing
instead of a parallel Linux one.
…e 11) compose_frame now draws the scene's text annotations over the screen layer, mirroring the "text" branch of compositor_macos: each annotation is placed relative to the screen rect (g.s_dst), its TextSpec is rasterised by text_linux (cosmic-text -> R8 coverage atlas), and drawn with layer.wgsl mode 11 tinted by the annotation colour. Bind groups are built before the render pass (one uniform buffer per draw). Differs from the Metal path only in the texture format: macOS bakes colour into a BGRA texture; here the atlas is R8 coverage and the shader tints it (cb.color) -- same visual result, and the iso-render contract is on the shared plan_frame geometry, not on text rasterisation. Image/figure/blur annotations and the text_anim reveal/opacity animation remain incremental.
EtienneLescot
force-pushed
the
feat/linux-compositor-port
branch
from
July 30, 2026 14:35
e03428e to
69158fe
Compare
Webcam layer (mode 0) placed at plan_frame g.w_dst/g.w_radius, gated by g.shape_fade>0, drawn over the screen. cover-crop UV, mirror, shadow and circle shape remain incremental; compose_linux test now writes a PPM for visual inspection.
Restructure compose_frame into bg-pass (clear + gradient mode 5) -> blur_bg (3 down / 3 up Kawase pyramid, blur.wgsl) -> fg-pass (screen + webcam + annotations, LoadOp::Load). Gated by cfg.bg_blur. Adds the WGSL mode 5 linear gradient to layer.wgsl. 1:1 port of the Windows/macOS blur_bg. compose_linux test now renders a gradient background + padding so the blurred background is visible around the inset screen.
Draw the cursor as a themed RGBA sprite on top of the composite: plan_cursor -> upright placement -> load_image_texture (PNG/data-URI via the image crate, cached in img_cache) -> LayerCB mode 7. Adds WGSL mode 7 (sprite sampled on texY, Clip-to-canvas via fx). 1:1 port of the macOS draw_cur_themed upright path; tilted (mode 13) and motion-blur (taps>1) are follow-up increments. compose_linux test now also renders a green data-URI sprite at screen center and asserts the green pixels appear.
Implement run_composited_multi for Linux: software VideoEncoder (libopenh264 / libkvazaar, the LGPL-build encoders that need no HW device), the composited RGBA frame read back (readback_direct) and converted to YUV420P via sws_scale, fed through the SHARED walk_composited_timeline into an MP4 muxer (avformat + the sn_fmt_set_pb C shim, as on Windows/macOS). Video-only for now; AAC audio mux is the next increment (audio.rs / AacEncoder are already shared), as is VAAPI/Vulkan hardware encode. Verified: export_linux_mp4 test renders ~1s of the fixture -> a valid 30-frame H264 MP4 (ffprobe: h264 640x360, 30 packets), content correct.
Wire the shared audio machinery (audio.rs: AacEncoder, decode_clip_audio, stretch_clip_pcm_by_speed, build_audio_concat_plan, assemble_concatenated_pcm) into run_composited_multi: add the AAC stream before the header, collect one PCM per clip in on_clip_end (speed-stretched to the frames the clip actually produced), then a single AAC encode after the video walk. Silent track when a clip has no audio — parity with Windows/macOS which always mux AAC. Verified: export_linux_mp4 (has_audio=true) yields an MP4 with h264 (30 pkts) + aac (48 pkts), duration 1.0s.
…tor_linux Handle SceneBackground::Image in compose_frame: load the wallpaper via load_image_texture (reused from the cursor), cover-fit its UV rect to the output aspect, draw it as WGSL mode 6 (opaque RGBA on texY) in the bg-pass. The bg handling is refactored into a BgLayer/BgDraw pair covering gradient (mode 5) and image (mode 6) uniformly. 1:1 port of the macOS draw_image_bg cover math. Verified: compose_linux_fond_image renders an orange data-URI wallpaper filling the background around the inset screen (orange pixels 487 -> 152970 once the scene padding is wired through set_live_params).
`Math.round` returns negative zero for any delta in [-0.5, 0), which is routine under fractional scaling where screenY deltas are fractional. V8's IsInt32() rejects -0, so gin refuses the conversion and BrowserWindow's native setPosition throws "Error processing argument at index 1, conversion failure from" — fatal, because installMainProcessErrorGuards re-throws. `Number.isFinite(-0)` is true, so the existing finiteness guard could not catch it. `| 0` collapses -0 to 0 and pins the value to int32. Reproduced on Electron 41.2.1: flipping which axis carries the -0 flips the reported argument index, and the empty tail after "from " identifies the bad value as a Number rather than undefined/null. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The HUD root carried `-webkit-app-region: drag` while the grab handle explicitly opted out with `no-drag` — exactly inverted. One cause, both reported symptoms on Wayland: - pressing empty space beside the bar dragged the HUD, because the root is the whole 820x560 window and a compositor honours a drag region whether or not anything is painted there. It reads as "dragging the window behind". - pressing the grab handle did nothing, because it fell through to the IPC/setPosition path, which Wayland prohibits: getPosition() answers [0,0] and setPosition() only updates Electron's own cache. Windows and macOS never saw this: setIgnoreMouseEvents makes the transparent area input-transparent at the OS level there, so the root region was never pressable. That call is a no-op on Wayland. The handle now takes the native drag region only where the platform cannot position itself; Windows and macOS keep the pointer-handler path unchanged, since a drag region swallows pointer events and the two cannot coexist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`crates/thirdparty/` holds the LGPL ffmpeg build the Rust compositor links against (see crates/.cargo/config.toml). It is gitignored but still watched: several thousand files, including a doc/ directory full of HTML. Extracting it fires the same reload storm the neighbouring worktree entries already guard against, leaving the running app stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
blur.wgsl, layer.wgsl and linux_decode.rs were written from Windows and carried CRLF plus double-encoded French comments: an em dash (U+2014) had been decoded as Mac Roman and re-encoded as UTF-8, so "—" read as "ÔÇö" and the accented characters were mangled with it. .editorconfig already mandates `end_of_line = lf` and `charset = utf-8`, but nothing enforced it on checkout. .gitattributes now does, so this cannot silently come back the next time someone builds on Windows. Comment text and line endings only — no code or shader logic changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
blur.wgsl's vs_main read `pos[vid]` twice. naga 24 keys its `spilled_composites` map by the base expression handle, so the second dynamic access overwrites the first entry: a single OpVariable is emitted while the first access's OpStore/OpAccessChain still reference an id that has no definition anywhere in the module. The module therefore violates VUID-VkShaderModuleCreateInfo-pCode-08737 and driver behaviour is undefined. RADV dereferences the dangling id and segfaults inside vkCreateGraphicsPipelines, killing the process before the compositor finishes construction; lavapipe happens to survive, which is why the crash looked adapter-specific. No Mesa version fixes this and no Mesa bug exists for it — the SPIR-V we hand every Vulkan driver is simply malformed. Doing the index once and reusing the value is pure common-subexpression elimination, so the output is pixel-identical. layer.wgsl was never affected: its vs_main derives the quad corners arithmetically and has no array literal, which is why only the blur pipeline ever crashed. Fixed upstream by gfx-rs/wgpu#7239 (wgpu 25). There is no naga 24.0.x patch release, so wgpu 24 can never pick it up and the shader must avoid the construct until that upgrade lands. Verified on AMD Radeon 610M (RADV RAPHAEL_MENDOCINO, Mesa 25.2.8): the four compose_linux tests went from dumping core to 4/4 passing on the default adapter, with no VK_DRIVER_FILES override. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Linux counterpart of the Windows and macOS addon builds, which had no equivalent — so compositor_view.node was never produced on Linux and compositorViewService found nothing to load. The whole native compositor was unreachable from the app there, however well the Rust crate itself worked. Two things differ from Windows, and both are load-bearing: - FFMPEG_DIR is not supplied by crates/.cargo/config.toml, which pins the win64 tree only (cargo's `[env]` has no per-target form). The script resolves an explicit FFMPEG_DIR or a vendored thirdparty/ tree, and says what to do when neither exists. LIBCLANG_PATH and clang's missing stddef.h are discovered the same way rather than left to each contributor. - Shared libraries are found by RUNPATH, not PATH. Windows prepends the DLL directory to PATH at runtime (ensureFfmpegSharedDllsOnPath), but glibc reads LD_LIBRARY_PATH once at process start, so that trick cannot work once Electron is running. The addon is instead linked with `-rpath,$ORIGIN` and the five ffmpeg sonames are copied beside it, which makes the .node self-contained wherever it lands — no env var, no PATH surgery. Verified on AMD Radeon 610M (RADV, Mesa 25.2.8): the produced addon loads under Electron, exposes all twelve entry points, and probeBackend() answers "hardware" — a real GPU device, not a software fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On Linux the native compositor called into `libffmpeg.so` — Chromium's own
heavily stripped ffmpeg — instead of the LGPL build shipped beside the addon.
ELF has a single flat symbol namespace and `libffmpeg.so` is a DT_NEEDED
dependency of the Electron binary, so it is loaded into the global scope
before any addon can be dlopen'd. Every `avformat_*` / `avcodec_*` / `av_*`
symbol the addon imports was already satisfied by the time its own RUNPATH
was consulted.
Confirmed with LD_DEBUG=bindings:
binding file .../compositor_view.node
to .../electron/dist/libffmpeg.so: symbol `avformat_open_input'
The visible symptom was the editor showing "Preview unavailable on this
machine" with avformat_open_input returning -1330794744, which decodes to
AVERROR_PROTOCOL_NOT_FOUND: Chromium's ffmpeg has no `file` protocol, since
Chromium feeds it buffers rather than filenames. The quieter danger was that
the addon ran against an entirely different ffmpeg than the one it was
compiled against — no libopenh264, no libkvazaar, and no guarantee the two
agree on anything beyond a shared soname. It did not fail louder only because
the major versions happen to match.
RTLD_DEEPBIND puts the addon's own dependency scope ahead of the global one,
so the `$ORIGIN` RUNPATH wins and it binds to the ffmpeg it was built with.
After the change the same trace reads:
binding file .../compositor_view.node
to .../electron/native/bin/linux-x64/libavformat.so.62
Windows and macOS are structurally immune — PE resolves imports by DLL name
and Mach-O records the defining dylib per symbol — so they keep the plain
require and are untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This reverts commit c005c06.
Electron links libffmpeg.so — Chromium's own stripped ffmpeg — as a DT_NEEDED dependency, so it occupies the global symbol scope before any addon is dlopen'd. ELF has a single flat namespace, so every av*/sws_*/swr_* import in the compositor addon bound to Chromium's build regardless of the addon's $ORIGIN RUNPATH: the symbols were already satisfied. The visible symptom was the editor reporting "Preview unavailable on this machine", avformat_open_input returning AVERROR_PROTOCOL_NOT_FOUND on an ordinary path — Chromium's ffmpeg has no `file` protocol because Chromium feeds it buffers, not filenames. Video export, GIF export and audio muxing were broken by the same defect. The resting state was worse than it looked: avformat_open_input bound to Chromium while sws_getContext bound to ours, so two libavutil instances with separate global state coexisted in one process. The build now stages renamed copies of the five vendored libraries — the map is derived from the libraries themselves with `nm -D`, never a checked-in list — and bindgen's declarations get a matching `#[link_name]`. The Rust identifiers are unchanged, so no call site moves, and Windows/macOS never set the env var, so their bindings stay byte-identical. Renaming was chosen over reordering the lookup scope. RTLD_DEEPBIND fixes the binding too, but it was measured to crash the process: Electron interposes malloc/free globally, deep-binding flips the addon's free() to glibc's while libc keeps allocating through PartitionAlloc, and the first std::fs::canonicalize frees a PartitionAlloc pointer with glibc's free. A ten-line C addon with no ffmpeg reproduces it. Symbol versioning is not an option either: our references are already versioned and still bind to Chromium's definition. A post-build assertion fails the build if any unprefixed ffmpeg symbol remains imported — without it the failure mode is silent, since the addon still loads and only misbehaves at runtime. Verified on this machine: with a plain require() and no loader flags, the addon binds every ffmpeg symbol to the libraries shipped beside it, zero bindings to Chromium's libffmpeg.so remain, probeBackend() still answers "hardware", and a 900-tick soak with seeks decodes 596 frames at 1280x720 before a clean teardown. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`Decoder::next` called `decode_present(idx)`, which seeks to the enclosing keyframe and decodes forward every single time. Sequential playback therefore re-walked half a GOP per frame. It carried its own admission of guilt: "decode_at par index re-seek a chaque frame -- OK pour le scrub/preview, a optimiser en WP si le bench de lecture l'exige". The bench now exists: a seek measures 18 ms per call. `SwDecoder::next_frame` is the ordinary receive/read/send pump with a persistent packet and frame, so a sequential read costs one packet per frame and the decoder keeps its state — the same shape pipeline_windows and pipeline_macos have always used. Two things found while taking this over: - `thread_count = 0` was never set on the Linux decoder, so ffmpeg ran single-threaded. Windows (pipeline_windows.rs:526) and macOS (pipeline_macos.rs:178) both set it. - The guard meant to skip a misaligned packet after a seek compared against a hardcoded `-0x2A2A2A2A`, commented as AVERROR_INVALIDDATA. That value is the tag `****` and is not an ffmpeg error at all; AVERROR_INVALIDDATA is -0x41444E49. The guard could therefore never match, so genuinely invalid data aborted the whole decode instead of being skipped — precisely the scrub path, which seeks constantly. The real constant now lives in ffi.rs beside AVERROR_EOF. `cur_time_sec` now prefers the decoder's real pts, falling back to the index counter. With sequential decode the frame index no longer maps to time on a variable-framerate stream, which every MediaRecorder capture is. Measured on AMD Radeon 610M / RADV: the compose_linux suite goes from 6.85s to 1.53s, and a 900-tick preview soak with periodic seeks goes from 596 to 891 frames, tearing down cleanly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ater `readback_direct` allocated its output with `vec![0u8; total]`, which memsets the whole frame — 8 MB at 1080p — and then overwrote every byte of it from the mapped buffer. It also copied row by row unconditionally, to strip a padding that is usually not there: wgpu aligns bytes_per_row to 256, and an RGBA width that is a multiple of 64 px is already aligned, so at both 1280 and 1920 the loop produced a byte-identical buffer in `h` memcpys instead of one. `Vec::with_capacity` + `extend_from_slice` skips the zero-fill, and the unpadded case is now a single contiguous copy. The padded path is kept for widths that actually need it. This matters more than it looks: profiling put readback at 82% of the preview frame (8.9 ms of 10.8 ms at 1080p) and 37% of the export frame, and it reads from a MAP_READ buffer at ~1.5 GB/s against 5.65 GB/s for the same copy in ordinary RAM — so every avoidable byte touched is expensive. Two larger wins in the same function are left for later, both measured: there is no double-buffered staging ring, so the CPU stalls ~3.2 ms per frame in device.poll(Maintain::Wait) while the GPU finishes; and the export still does a full RGBA readback plus an sws_scale to YUV420P (11.3 ms, 53% of the frame) where macOS renders NV12 straight into the encoder's buffer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Linux recording is unseekable. Capture falls back to getDisplayMedia/MediaRecorder — there is no native Linux capture helper, where Windows has wgc-capture and macOS has screencapturekit — and MediaRecorder writes WebM as a live stream, so the file carries neither Cues nor SeekHead. Verified by scanning the bytes of a real recording: both absent. `av_seek_frame` therefore fails for any non-zero timestamp, and `decode_at` turned that into a hard error. Scrubbing the preview died on it, and export failed partway through — around 85% of the file, which read as "the export crashes at the end". Rewinding to 0 stays possible without an index, and the loop below already scans forward to the target. The cost is linear, which is only acceptable because the decoder no longer re-seeks per frame: at ~0.07 ms/frame, reaching second 14 costs tens of milliseconds instead of failing outright. Verified on the real unindexed recording: a 900-tick soak with periodic seeks now runs 894 frames with no seek error and a clean teardown. This is a fallback, not the fix. Re-muxing through the Matroska muxer writes the index in ~0.06s with no re-encode and no change of extension, and a native PipeWire capture helper would produce indexed files at the source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…Cast portal
Every Linux recording shipped a cursor track pinned to the top-left corner.
`telemetryRecordingSession` samples `screen.getCursorScreenPoint()`, which
returns {0,0} under Wayland — a real recording contained 473 samples, all
`{cx:0, cy:0, visible:true}`. The file looked valid, so the compositor drew its
cursor sprite (WGSL mode 7) in the corner for the whole video. Silent output
corruption.
Wayland exposes the pointer only through the ScreenCast portal's METADATA
cursor mode, where the compositor omits the cursor from the pixels and attaches
SPA_META_Cursor per frame. This adds a helper that negotiates that session and
emits nothing else — no video, no encoding. Video still goes through the
existing path.
`electron/native/pipewire-capture/` is a standalone Rust package, deliberately
outside the crates/ workspace: it is a stdio sidecar like the Swift and C++
helpers, shares nothing with the compositor, and would otherwise pull
ashpd/zbus/png into every addon build. The portal half uses ashpd (pure-Rust
D-Bus, so no libdbus-1-dev); PipeWire is reached through a C shim that dlopens
libpipewire-0.3.so.0, because two thirds of its consumer API is static inline
in headers. The binary links only libc and libgcc — no dev package is needed to
build it and none to run it, which also means it survives a distro shipping a
different PipeWire.
Three bugs were found against a real GNOME session rather than assumed:
- SPA_META_Cursor never arrived. mutter 46.2 declares a FIXED
CURSOR_META_SIZE(384,384) = 589872 bytes; our declared range topped out at
256x256 = 262192. PipeWire intersects the two, the intersection was empty, so
the whole ParamMeta object was silently discarded — stream negotiated, ran,
and reported nothing. The ceiling now matches OBS at 1024x1024, and a test
runs spa_pod_filter over the real POD to keep it that way.
- The stream died after one buffer with "target not found". That string is
WirePlumber's, not PipeWire's: PW_STREAM_FLAG_DONT_RECONNECT sets
node.dont-reconnect, and policy-node.lua destroys the node on that branch
instead of reconnecting. Removed; OBS does not set it either.
- on_process drained to the newest buffer and requeued the rest unread, so
cursor updates arriving on separate zero-chunk buffers were dropped. Every
dequeued buffer is now inspected in arrival order.
Verified on GNOME 46 / mutter 46.2 / PipeWire 1.0.5: metas report
`Cursor:589872`, coordinates track the pointer across the screen, four distinct
cursor shapes were captured with their own hotspots, and sprites are cached by
content hash so repeat samples carry only an id.
There is no telemetry fallback: if the helper is missing or the portal fails,
recording proceeds with no cursor file. No file beats a file full of zeros.
Known limitation, and it is permanent: mouse CLICKS cannot be observed on
Wayland. No portal API reports them and /dev/input is not readable, so
interactionType is always "move" and the click-ripple effect cannot work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`decode_at` passed `SEEK_SET | 4` to av_seek_frame, with a comment asserting "BACKWARD = 4". AVSEEK_FLAG_BACKWARD is 1; 4 is AVSEEK_FLAG_ANY. This is the only seek in the crate that did not use ffi::AVSEEK_FLAG_BACKWARD — pipeline_windows.rs, pipeline_macos.rs and audio.rs all pass 1. Without BACKWARD, ffmpeg lands on the first indexed position at or after the target rather than the keyframe before it. The forward-scan loop then stops on its own first decoded frame, since `pts >= target` is already true, so it does nothing and decode_at returns the NEXT keyframe. The error is a full GOP. That is why the symptom was asymmetric. The screen track carries a keyframe every ~1.78 s and the webcam every ~6.73 s, so the webcam lands ~4x further out. And live::Player::step catches the webcam up with a monotone forward-only loop, so once it is parked in the future it can never come back — it stops as a still frame until the screen clock reaches it. That is the reported freeze. Third wrong constant found in this file today, after -0x2A2A2A2A standing in for AVERROR_INVALIDDATA and the missing thread_count. All three were hardcoded magic numbers with comments asserting the wrong value; ffi.rs already held the right ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`compose_frame` submitted and returned without waiting, then `readback_direct` submitted a second command buffer and blocked in `device.poll(Maintain::Wait)` — which drains the WHOLE queue, so the wait absorbed the entire Kawase blur chain and every layer draw, not just the copy. Meanwhile avcodec_send_frame spent ~8 ms of pure CPU with the GPU idle. The two never overlapped. A ring of staging buffers now keeps frame N's copy in flight while the encoder works on N-1, waiting on the SubmissionIndex that Queue::submit already returns rather than on the whole queue. Measured on AMD Radeon 610M / RADV, 1920x1080, 180 frames, paired A/B in the same binary (depth 1 reproduces the old path byte for byte): light scene 23.98 -> 19.90 ms/frame 41.7 -> 50.3 fps 1.20x heavy scene 24.80 -> 19.62 ms/frame 40.3 -> 51.0 fps 1.26x end to end through the addon, heavy 34.9 -> 42.4 fps 1.22x The poll(Wait) line itself goes from 3.81 ms to 0.008 ms on the light scene and 6.17 ms to 0.007 ms on the heavy one: it has left the critical path entirely. Depth 2, not 3 — depth 3 measured slower (20.48 ms vs 19.62). The render target is deliberately NOT double-buffered: frame N's copy is submitted before frame N+1's compose commands on the same queue, so wgpu's barriers order them without a CPU stall. Correctness is checked by output, not by inspection: the exported MP4s are md5-identical before and after on both scenes, through both the Rust path and the addon, and at ring depth 3 as well — which covers frame ordering, pts contiguity and the flush of the last frame in one check. The three test PPMs are byte-identical too. Two things worth recording for whoever optimises this next. The wall-clock gain is slightly LESS than the wait it removes (heavy: 5.2 ms saved for 6.2 ms of wait); the likely cause is DRAM contention, since a 610M APU has no dedicated VRAM and the overlapped DMA now competes with sws_scale and the encoder for one memory controller — plausible but not instrumented, and it predicts a discrete GPU would do better rather than worse. And the heavy scene GAINS more than the light one (1.26x vs 1.20x), because a heavier GPU frame simply gives the ring more to hide. What remains is now unambiguous: ~12.6 ms of CPU per frame, sws_scale RGBA->YUV420P plus software H.264. macOS avoids the first half by rendering straight into the encoder's buffer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MediaRecorder writes WebM as a live stream, so the file has no Duration
element — Chromium then reports `duration=Infinity` and `seekable=0..Infinity`
for every Linux recording. `webm-duration.ts` already existed to patch that
header after finalization; this replaces it on Linux with a full remux through
the matroska muxer, which computes the duration from real packet timestamps
instead of trusting a header. Forcing a bogus input Duration of 999999
confirmed it: the output came out 16977.
The remux also writes Cues and SeekHead, which the file lacked. That turns out
to be a nicety rather than a fix — see below.
Same I/O as before (both rewrite the file exactly once) and the existing header
patch stays as the fallback if the remux fails, so a recording can never be
lost to this. Measured: 0.055 s for 7.8 MB / 819 packets, 0.074 s for 44 MB /
18000 packets.
It runs through the compositor addon rather than a spawned ffmpeg:
libavformat.so.62 already ships beside the addon on Linux, and
electron-builder.json5 explicitly refuses to ship a standalone static ffmpeg
("a large binary no runtime code opens", plus LGPL redistribution).
CORRECTION TO 3b61766. That commit added a rewind-and-scan fallback for
`av_seek_frame` failing on index-less recordings, and its message asserted the
seek could not work without Cues. That was wrong. Measured: av_seek_frame
returns 0 for every flag at every timestamp on all four real recordings —
libavformat's matroska demuxer binary-searches by byte position when there is
no index, and Chromium's <video> seeks these files correctly too. An index
takes a seek from 0.229 ms to 0.002 ms on a 10-minute file, which is not
user-visible. The real cause of the original symptom was the seek-flag bug
fixed afterwards in 46b6448 (AVSEEK_FLAG_BACKWARD is 1, the code passed 4).
The fallback in linux_decode.rs is therefore dead code — its `r < 0` branch
cannot fire — and is left in place only as a guard for genuinely unseekable
input.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate looked like an oversight now that Linux has real cursor telemetry, and I was about to lift it. Measurement says the gate is right and lifting it would have made the reported "two cursors" bug worse. `cursor: "never"` is not a constraint Chromium implements — it is absent from `getSupportedConstraints()` (verified on Electron 42 / Chromium 148, where the only screen-capture constraints are `displaySurface` and `suppressLocalAudioPlayback`), so WebIDL drops it silently. Below the web API, `DesktopCaptureDevice::Create` wraps every capturer in a `DesktopAndCursorComposer` unconditionally, with no cursor branch; on Linux `prefer_cursor_embedded` is false, so WebRTC asks the portal for METADATA mode and then paints the cursor back in itself. Linux does not even reach the `cursor:` key in useScreenRecorder — that line is inside the win32 branch. Windows and macOS honour the toggle through their NATIVE HELPERS (`captureCursor` to wgc-capture, `hideSystemCursor` to ScreenCaptureKit), not through the constraint, which is a no-op there too. So the gate is precisely "platforms with a native capture helper", and lifting it would ship a control that changes nothing in the pixels while switching the editor's overlay on — producing exactly the two cursors it was meant to fix. `hideSystemCursor` (useScreenRecorder.ts) is inert on Linux for the same reason: it sits inside startNativeMacRecordingIfAvailable, which returns false off darwin. The real fix needs the cursor out of the captured pixels, which needs a native Linux capture helper. Until then the lever that works is `scene.cursor.show`, and defaulting it to false on Linux is a UX decision, not a bug fix — Linux would lose themed, smoothed and click-bounce cursors. Left for a deliberate call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's in this branch (12 commits on top of
#162's tip)What's in each commit
Spec —
linux-compositor-port.md(614 lines): target architecture, three-axis decision matrix (render/decode/encode), the AVFrame carrier contract generalising the four-field seam #162 introduced, the WGSL shader port inventory (1:1 with HLSL's 9 entry points), thecosmic-textdecision fortext.rs, build/CI plumbing for Linux, verification methodology (3/255 cross-backend tolerance), WP0–WP7 milestone plan.WP0 — Cargo gating.
wgpu = "24"pinned to thewindows = "0.58"-compatible line. D3D11 modules gated#[cfg(windows)]incompositor/src/lib.rs.compositor-view-napisplit: a small platform-agnostic stub plus a#[cfg(windows)]-gatedwindows.rs.build.rscross-build detection viacfg!(host_os)vsCARGO_CFG_TARGET_OS; on cross-build, skipscc/bindgenand writes a cachedffi.rs(CI on Linux regenerates the real bindings).WP1 —
vk.rs.VkGpu { device, queue, backend, adapter_info },create/create_blocking/create_auto,probe()cached viaOnceLock,classify()detecting lavapipe by adapter name on Linux and WARP on Windows — both routed toBackendKind::CpusoprobeBackend()keeps its three-state semantics for free.WP2 + WP3 —
vk_frames.rsand the slice renderer.VkFrameTex { y: R8Unorm, uv: Rg8Unorm, width, height }.VkFrames::present(src)does the swscale → upload of NV12 split into two wgpu textures.vk_decode.rs::SwDecoderis a software decoder.vk_render.rs::render_slice_c1runs the full pipeline:vk_frames::present→ WGSLlayer.wgsl(modes 0/1/2, BT.709 limited YUV→RGB, rounded-corner SDF, smoothstep feather) → copy_texture_to_buffer + map_async + pollster::block_on → RGBA8 → PPM.vk_cross_goldenintegration test: decodes frame 180 of the fixture, renders, writes PPM, logs stats. The slice produces a real, recognisable GitHub README frame on this Windows machine — mean RGB ≈ 232 over 6.22 M channels, dimensions 1920×1080.WP4 arch —
CompBackendtrait. Trait defined inbackend/mod.rswith 10 methods covering thecompose_framepath.LayerCBextracted tobackend/mod.rsas the platform-neutral constant buffer (matches HLSLcbuffer Layerand WGSLstruct Layerbyte-for-byte).D3d11Backend(Windows) wraps the existingCompositorand forwards — zero behavioural change on Windows, all 80 unit + 3 integration tests still pass.WgpuBackend(Linux) implements the trait, withvk_frames/vk_renderprimitives for the rendering.WP4 effects — partial.
WgpuBackendimplementsdraw_solid(mode 1),draw_shadow(mode 2, SDF-based),draw_quad_shadow(mode 12 wrapper for the AABB case), andblur_bg(the full Kawase chain — 3 down passes + 3 up passes, 2.2 spread, all 6 render passes share aRenderCtxwith 3 pyramid textures and 2 dedicated pipelines). The shader isvk_shaders/blur.wgsl(5-tap average down + 5-tap weighted up — matches HLSLps_kawase_down/ps_kawase_up). The remaining HLSL modes (image_bg / cursor / motion blur / tilt / annotations) are marked as WP4+ follow-ups in commit messages.WP5 —
text_cosmic.rs. Linux text rasterizer usingcosmic-text0.19 (pure Rust: rustybuzz shaping, swash rasterization, fontdb font discovery). SameTextSpecshape and same FNV-1acache_key()algorithm astext.rs(Windows DirectWrite).TextRasterizer::new(device, queue)+rasterize(&mut self, &TextSpec) -> Result<RasterizedGlyphs>builds aBuffer, shapes it, iterateslayout_runs()to pull each glyph image fromSwashCache::get_image(...), and uploads the per-glyph alpha into aR8Unormwgpu texture. Text is consumed byWgpuBackend::draw_annotations(still unimplemented, lands with thecompose_frameport).WP6 —
pipeline_lin.rs. Linux pipeline arm.Decoder::open(path, &VkGpu)opens the fixture, preparesvk_framesfor upload.next(frame_idx)decodes + uploads.VideoEncoder::open(codec, w, h, fps, bit_rate)opensh264_vaapi(NV12) when/dev/dri/renderD128is present, elselibopenh264(YUV420P) — the Linux counterpart of the Windows vendor stack.VideoEncoder::send_composited(carrier)reads the carrier's NV12 split, copies to NV12 contigu (for VAAPI) or swscales to YUV420P (for libopenh264), thenavcodec_send_frame.run_compositedis a skeleton — the full MP4 muxer (avformat_new_stream + write_header + write_trailer) needs the Linux shim forAVFormatContext.streamsaccess, deferred to a follow-up.WP6 — module restructuring.
pipeline.rs→pipeline_win.rs,live.rs→live_win.rs, with cfg gates.pipeline_lin.rsandlive_lin.rsare the Linux counterparts.poc-d3dandcompositor-view-napi/src/windows.rsupdated to use the renamed paths.WP7 partial — Linux napi arm.
crates/compositor-view-napi/src/linux.rsexposes the same#[napi]surface as the Windows arm (create_view / read_frame / destroy_view / set_scene / set_live_params / dump_ppm / probe_backend). It uses the slice renderer (one frame per call) as the production path —vk_cross_goldenproves this works end-to-end.probeBackend()returns"none"until the fullLiveViewlands; the slice path is real but not exposed as a live preview stream yet. The compositor-view-napiwindowsdependency is nowcfg(windows)-gated so the Linux build doesn't pull it.Decisions baked in (mirror the spec §4)
poc-nativePOC rancomposite.wgslunchanged on Vulkan with pixel-identical output. The 2026-07 wgpu rejection indecisions.mdwas a Windows-encoder problem, not a Linux-render problem.cosmic-textfortext.rs— pure Rust, 13-locale shaping parity, no system C deps.Backendenum stays semantic —probeBackend()is platform-agnostic on the Electron side; no UI work needed.CompBackendtrait covers thecompose_framepath. Out-of-trait methods (encode-sidefs_pass, readback, blur_bg of secondary effect sets) stay onCompositordirectly until migrated.Verified
cargo check --workspace(Windows host) — green. 86 unit + integration tests pass — the D3D11 path is unchanged.cargo check --workspace --target x86_64-unknown-linux-gnu— green. WGPU + the Linux pipeline arm + the Linux napi addon + all module splits compile on Linux (usingOPENSCREEN_BINDGEN_CACHE_PATHfor the cross-build; CI on Linux regenerates the real bindings).What's deliberately still in follow-up commits (within this same PR if review permits)
The branch ships a real Linux compositor path: decode → upload → render → readback, all proven on real hardware via the slice. The remaining work is filling in the production
compose_frameorchestration math on Linux (which today still goes throughcompositor.rs::compose_frame, a Windows-only path) and the corresponding readback + export side:compose_frameorchestration port — 800 lines of pure Rust incompositor.rs(placements, zoom state, cursor follow, motion-blur supersampling). The dispatcher toWgpuBackenddraws is the next commit; once it's in, the full C1..C8 effect set works on Linux end-to-end.live_lin.rsLiveView — the render thread that produces frames continuously, not one-shot. The skeleton exists; wiring the napi to it (withSend/Syncfor the GPU handle and a generation counter) is the next commit.pipeline_lin::run_compositedis a skeleton today. The full muxer (avformat_new_stream, write_header, write_trailer) needs the Linux shim forAVFormatContext.streams— same pattern as the Windows shim, just with the Linux AVFormat bindings.WgpuBackend::draw_layerinfrastructure. They follow in subsequent commits; the trait architecture is in place to absorb them.build-linux-compositor-addon.mjs,fetch-ffmpeg.mjsLinux arm, electron-builder extras for Linux. These are CI plumbing, not blocking the core port.Each follow-up commit is small (100-300 LoC) and reviewable; they keep landing in this same PR per the user's "all WP should be on this PR" directive.
Cross-backend parity verification path
vk_cross_goldenslice test produces a valid frame on this Windows AMD iGPU (mean RGB ≈ 232 over 6.22 M channels, dimensions 1920×1080, written tocrates/compositor/target/vk-slice-c1.ppm). That's the proof the WGSL+vk+wgpu+swscale+YUV→RGB+SDF+feather chain works on real hardware.poc-d3d --release --cfg C1 --backend hardware --frames 1, deferred to the bench-side follow-up). The slice produces correct output in isolation; the diff infrastructure (OPENSCREEN_VK_GOLDEN,diff_stats, the 3/255 gate) is in place and CI runs it once both PPMs are committed.Branch / PR state
Related
Backendseam,cpu_frames.rs,probeBackend(), the iso-render tolerance baseline this port is measured against.technical-documentation/architecture/native-compositor.md— the engine today; "Linux port" subsection now links the spec.technical-documentation/architecture/linux-compositor-port.md— the spec.technical-documentation/engineering/rendering-performance.md— C1..C8 measurement methodology, reused verbatim for the Linux bench numbers when the bench lands.