fix(setup): substitute Vulkan layer manifest template vars and detect duplicate layers

[BANANASJIM]

Addresses #256 (Parts 1 and 2; Part 3 deferred).

Problem

On Windows, rdc's installed Vulkan layer manifest (renderdoc.json) kept unsubstituted CMake template vars, because upstream's configure_file runs only in the UNIX CMake branch. The installer rewrote only library_path, so @VULKAN_ENABLE_VAR@ survived as the enable-environment key and rdc's own implicit layer could never self-enable. Combined with a system RenderDoc install (a second VK_LAYER_RENDERDOC_Capture layer), Vulkan captures silently timed out, and rdc doctor stayed green.

Fix

Part 1 — manifest substitution (_build_renderdoc.py:_install_vulkan_layer): thread the in-scope version (RDOC_TAG / --version) through and resolve every templated field — force name=VK_LAYER_RENDERDOC_Capture, enable_environment={ENABLE_VULKAN_RENDERDOC_CAPTURE: "1"} (matching the env var capture_core already sets for the child), derive the DISABLE_VULKAN_RENDERDOC_CAPTURE_<major>_<minor> key and implementation_version from the version, keep library_path=.\renderdoc.dll. A post-serialization guard raises if any @ remains, so a future template change fails loudly instead of regressing silently.

Part 2 — duplicate-layer detection (doctor.py:_check_win_vulkan_layer): enumerate HKLM+HKCU implicit layers (mockable winreg helper), resolve each manifest's layer.name, dedup by resolved path, and warn when more than one distinct layer resolves to VK_LAYER_RENDERDOC_Capture — listing each manifest path, DLL, and version. A single layer stays green.

Deferred (Part 3)

Forcing rdc's own layer for the spawned child via VK_ADD_LAYER_PATH / VK_LOADER_LAYERS_ENABLE / VK_LOADER_LAYERS_DISABLE is documented in the proposal as a follow-up: it needs a real Windows box with a dual RenderDoc install to verify, and because both layers share the exact name VK_LAYER_RENDERDOC_Capture, a name-based VK_LOADER_LAYERS_DISABLE would disable both — an unresolved design question.

Tests

New cases in test_build_renderdoc.py (real-template substitution → zero @, correct values; @ guard raises; version threading) and test_doctor.py (duplicate warn; single green; same-manifest-both-hives dedup; both-hive enumeration), all cross-platform via mocked winreg.

pixi run lint && pixi run typecheck && pixi run test: all green (2998 passed, 6 skipped, 93.09% coverage).

Verification note

Parts 1 and 2 are fully covered by deterministic unit tests. Their end-to-end effect (capture succeeds with a dual install) is only fully confirmable on a real Windows machine with two RenderDoc installs, which the deferred Part 3 also requires.

Summary by CodeRabbit

• Bug Fixes

  • Fixed Vulkan capture layer manifest configuration on Windows—all template fields now properly substitute during installation, preventing layer self-enable failures.
  • Enhanced rdc doctor to detect and report duplicate Vulkan layer registrations across system registries that could interfere with capture operations.

• Tests

  • Added test coverage for manifest substitution validation and duplicate layer detection scenarios.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

Fixes Issue #256 by fully resolving @...@ template placeholders in the Windows Vulkan layer manifest during installation (adding _parse_version, extending _install_vulkan_layer with a version parameter, and a post-serialization guard) and refactoring _check_win_vulkan_layer to enumerate all hives, detect duplicate VK_LAYER_RENDERDOC_Capture registrations, and fail with per-manifest details. Includes spec documents and unit tests for both parts.

Changes

Vulkan Layer Manifest Fix (#256)

Layer / File(s)	Summary
Proposal, tasks, and test-plan specs openspec/changes/2026-06-22-fix-256-vulkan-layer-manifest/proposal.md, ...tasks.md, ...test-plan.md	Design proposal identifies the two root causes (unsubstituted manifest placeholders, hidden duplicate registrations), tasks checklist drives implementation, and test plan covers unit cases and manual Windows VM steps.
_parse_version and _install_vulkan_layer template resolution src/rdc/_build_renderdoc.py	Adds _parse_version(version) -> (major, minor), extends _install_vulkan_layer with a version parameter, substitutes all manifest fields (name, enable_environment, disable_environment, implementation_version), asserts no @ placeholders survive, and main() forwards args.version.
Registry enumeration helpers and duplicate-layer detection src/rdc/commands/doctor.py	Introduces _RENDERDOC_LAYER_NAME, _enumerate_implicit_layers, and _read_layer_manifest. Rewrites _check_win_vulkan_layer to filter candidates by resolved layer name, deduplicate across hives, fail with per-manifest DLL/version details on duplicates, and validate library_path existence for the single-match path.
Unit tests: manifest substitution and version forwarding tests/unit/test_build_renderdoc.py	Adds test_main_windows_forwards_version_to_vulkan_layer; refactors existing vulkan layer test with shared _fake_winreg/_write_layer_template helpers; adds tests for full placeholder substitution and RuntimeError on unhandled placeholders.
Unit tests: duplicate detection and implicit layer enumeration tests/unit/test_doctor.py	Adds _make_fake_winreg/_write_manifest helpers and four new tests: duplicate-layer FAIL, single-layer OK, same-manifest-in-both-hives non-duplication, and enumeration returning only RenderDoc manifest paths from HKLM/HKCU.

Sequence Diagram(s)

sequenceDiagram
  participant CLI as rdc main()
  participant IVL as _install_vulkan_layer
  participant PV as _parse_version
  participant FS as renderdoc.json

  CLI->>IVL: _install_vulkan_layer(install_dir, build_dir, args.version)
  IVL->>PV: _parse_version(version)
  PV-->>IVL: (major, minor)
  IVL->>FS: read template JSON
  IVL->>IVL: set name, enable_environment, disable_environment, implementation_version
  IVL->>IVL: serialize JSON → assert "@" not in output
  IVL->>FS: write resolved renderdoc.json

  Note over CLI,FS: At runtime: rdc doctor check

  participant REG as Windows Registry (HKCU/HKLM)
  participant CWV as _check_win_vulkan_layer
  participant MF as manifest files

  CWV->>REG: _enumerate_implicit_layers()
  REG-->>CWV: list of manifest paths
  CWV->>MF: _read_layer_manifest(path) for each
  MF-->>CWV: layer objects
  CWV->>CWV: filter layer.name == VK_LAYER_RENDERDOC_Capture
  CWV->>CWV: deduplicate by resolved path
  alt multiple matches
    CWV-->>CLI: FAIL – lists each manifest path, DLL, version
  else single match
    CWV->>MF: check library_path exists on disk
    CWV-->>CLI: OK or FAIL (missing DLL)
  end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• Windows: rdc capture silently times out when a system RenderDoc is also installed (duplicate implicit layer) + unsubstituted @VULKAN_*@ vars in generated renderdoc.json #256: This PR directly implements the fix for Issue #256, addressing both the unsubstituted @...@ manifest template placeholders in _install_vulkan_layer and the hidden duplicate-layer split-brain behavior in _check_win_vulkan_layer.

Possibly related PRs

• BANANASJIM/rdc-cli#175: Previously introduced _check_win_vulkan_layer-based Vulkan validation in doctor.py, which this PR refactors into helper-based enumeration with duplicate detection.

Poem

🐰 Hop hop, the placeholders flee,
No more @version@ haunting me!
Two hives checked, duplicates found,
The manifest's clean, the layer is sound.
VK_LAYER_RENDERDOC_Capture stands alone—
One bunny, one layer, one happy home! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 47.62% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the two main changes: substituting Vulkan layer manifest template variables and detecting duplicate layers on Windows.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/256-vulkan-layer-manifest

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.