Widen the kernel's SIMD: mask-driven BVH4 traversal and 4-wide triangle packets - #115
Merged
Conversation
…le packets The BVH4 node test already built `Vec4` lanes, then threw the vectorization away by reading lanes back one at a time; leaf intersection was fully scalar, one `Box<dyn Prim>` call per triangle. Both are now vertical SIMD, and the codegen settings let the vector code actually inline across crates. - Release profile: `lto = "thin"` + `codegen-units = 1`. The hot loop spans crust-core -> crust-rt -> glam as small `#[inline]` functions; without LTO those calls survive and the vector bodies never fuse. Free 21% on the triangle traversal benchmark. - Traversal (`bvh.rs`): `RaySlab` pre-splats the ray once per query instead of rebuilding seven splats per visited node; `safe_inv3` does all three reciprocals branch-free; lane verdicts come out as a nibble via `cmple(..).bitmask()`; lane validity moved into a `flags` bitmask ANDed into that nibble, replacing four per-lane branches. Unused lanes cannot simply be given empty bounds -- the slab test's per-axis min/max un-inverts an inverted box, and +INF/+INF passes when `t_max == INF`. - Leaf packets (`triangle.rs`): the Woop 2013 transform splits into a per-ray part (`RayShear`, derived once per traversal rather than once per triangle) and per-triangle arithmetic that maps onto `Vec4` lanes. `Tri4` holds four triangles in SoA and intersects them in one vector round, with a two-compare fast path for per-lane visibility masks. Leaf payloads moved to a side `Leaf` table so `WideNode` still fits two cache lines (pinned by a size test). Watertightness is preserved exactly: the f64 tie-break for exactly-zero edge functions is inherently scalar, so those lanes are reported in a `fallback` mask and re-tested through the scalar path. The two intersectors are bit-identical -- `simd_matches_scalar_bitwise` asserts `to_bits()` equality per lane over ~8000 lanes, not an epsilon. - Leaf size retuned to the SIMD width: packet occupancy was 1.66 of 4 lanes under `MIN_LEAF = 2`. `MIN_LEAF_PACKED = 4` applies only to all-triangle ranges (raising the floor for everything cost instanced scenes ~12%); occupancy is now 2.89 of 4 and every scene kind gets its best number. - `material/brdf.rs`: three `powf(_, 5.0)` in Schlick Fresnel became `powi(5)`, matching the `powi(6)` already in the same function. `powf` is an opaque libm call the surrounding vector code cannot be optimized across. cornellbox 9.09s -> 8.41s; image differs by at most one ULP. Measured (min of 40, fixed 4096-ray batch, both sides with LTO): tri_spheres intersect 4.48 -> 3.43 ms occluded 3.13 -> 1.93 ms sphere_grid intersect 1.96 -> 1.66 ms occluded 1.45 -> 0.93 ms instances intersect 4.86 -> 3.79 ms occluded 3.55 -> 2.31 ms End to end, the 383k-triangle stress scene renders in 6.35s against 8.03s. The checked-in samples move far less (1.5-9%): they are shading-bound, not traversal-bound -- worth knowing before benchmarking a kernel change on them. Verification: whole workspace green; every sample scene rendered before and after and compared with the new `exr_diff` example. Seven of eight are pixel-identical. cornellbox differs in 1 pixel of 230400 by 4.6e-4, at the seam where the inner box's bottom face is exactly coplanar with the floor -- a genuine `t` tie whose winner depends on leaf visit order (confirmed independent of the retuned tree). Adds `crust-rt` traversal benchmarks, a min-of-N `ray_throughput` probe (criterion means drift 10%+ on a shared machine), the `exr_diff` regression check, and `docs/simd.md` -- including why `std::simd` is not used: it is still nightly-only on the stable toolchain crust builds with. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Q7RXTdHvPFM5weuxWYLYU
…packets Follow-up from Kerkour's "SIMD programming in pure Rust", whose advice is to skip SSE2 and target AVX2. Everything vectorized here is 128-bit, so that guidance needed either acting on or answering. Measured, it is answered: - Enabling AVX2+FMA codegen on the existing 4-lane code is worth 2-4% (tri_spheres intersect 3.450 -> 3.386 ms). LLVM cannot widen a 4-lane algorithm; real 8-wide throughput needs the data structures restructured, which is the article's "chunk into X = lane-count blocks" recipe. - 8-wide *leaf packets* would buy nothing at all. On a dense mesh no leaf holds more than four triangles -- the SAH splits anything above MIN_LEAF_PACKED -- so a Tri8 would run the same 2319 vector rounds with half the lanes idle. `eight_wide_packets_would_not_reduce_vector_rounds` pins that, and fails if a future retune makes leaves big enough to change the answer. That leaves BVH8 nodes as the one place 256-bit vectors would still pay, and it is a project decision rather than an optimization: reaching 256 bits at runtime needs unsafe `core::arch` intrinsics (against this crate's "100% safe Rust" claim), a new dependency (`wide`/`multiversion`), or a non-distributable `-C target-cpu`. Documented with the trade-offs rather than picked unilaterally. Also adopts the article's testing practice, which turned out to matter for a claim this branch makes. The kernel's correctness is stated in exact float bits (`simd_matches_scalar_bitwise` asserts `to_bits()` equality, and watertightness needs adjacent triangles to compute *exactly* equal edge functions), so FMA contraction would break it. `scripts/test_simd_matrix.sh` runs the suite with AVX/AVX2/AVX-512 disabled, with AVX2+FMA, and with target-cpu=native. All four pass, so the "no FMA contraction" claim is verified rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Q7RXTdHvPFM5weuxWYLYU
Contributor
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
…ypes Neither USD instancing mechanism was consumed by the importer, which is the gate on loading any production-scale scene: instancing is how a scene fits billions of primitives into memory, and without it the importer's cost scales with the instance count rather than the unique geometry. Three defects, all reproducible on the new sample scene: - `UsdGeomPointInstancer` was ignored entirely. Its instances never appeared, and its prototypes were instead drawn once each at their authored location -- wrong in both directions, silently. - `class` prims were rendered. An abstract prim describes geometry that exists only to be referenced or instanced; drawing it put a phantom copy of every prototype at the origin. - Native instancing (`instanceable = true`) worked only by accident, via the mesh content-hash dedup: BVH sharing happened, but every instance still re-read and re-hashed the prototype's full point array. Both mechanisms now share one path. A prototype becomes a `Vec<ProtoPart>` -- one part per bound leaf geometry, each a committed local-space kernel scene plus its prototype-relative transform, material and ray mask -- built once, memoized by path, and placed by transform. The split is per *part* rather than per prototype because `World` maps materials by top-level `geom_id`: a prototype binding two materials has to become two instances or one material is lost. PointInstancer follows the spec: `translate ∘ orient ∘ scale` under the instancer's world matrix, `orientationsf` preferred over `orientations`, `invisibleIds` pruning by `ids` (array index where absent), and the instancer's children left untraversed since prototypes are authored beneath it and drawn only through it. Nested instancers and volumes inside prototypes warn rather than disappear. Non-invertible placements are skipped -- a zero scale is the usual "hide this instance" idiom, not an error. Measured on generated scenes: - 4000 PointInstancer instances of a 6240-triangle prototype: 11 MB peak RSS, 0.3 s. Baking those 25M triangles would cost ~8 GB at the ~330 bytes/triangle this renderer currently spends. Before, the scene loaded as two geometries -- the instances were simply missing. - 2000 `instanceable` prims: load 0.672 s -> 0.473 s (min of 5, -30%), from not re-reading and re-hashing each instance's geometry. Peak RSS is unchanged at 23 MB, because the content hash was already sharing the BVH -- so for native instancing the win is load time and correctness, not memory. (An earlier note in this branch claimed memory; that was wrong.) Verification: all 8 existing sample scenes render pixel-identical to HEAD (`exr_diff`), so nothing outside the instancing paths moved. Four new integration tests cover the geometry count, the class-prototype regression, per-part materials and non-uniform instance scale (both by ray-casting at known positions), and `invisibleIds`. `World::primitive_count` is exposed so a test can assert geometry is instanced rather than baked -- an instanced import stays flat as placements multiply. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Q7RXTdHvPFM5weuxWYLYU
Nested instancing is how a production scene stays tractable -- Moana's
vegetation is instancers inside instancers -- and the docs said the kernel
could not do it. Testing that claim first turned out to matter: it is wrong.
`crust-rt` already nests to arbitrary depth. Transforms compose, normals map
back through every level, and ray masks gate per level; three new kernel
tests pin all three, and they passed against the unmodified kernel. The
limitation was only ever in the importer, and CLAUDE.md, crust-rt's docs and
the Embree comparison table are corrected accordingly.
A `PointInstancer` inside a prototype now expands into real nested
sub-scenes rather than being skipped. The grouping is per (prototype, part),
not per instance, because a part carries one material: M nested placements
of a K-part prototype become K parts, each a committed scene holding M
instances. Flattening instead would multiply the outer instance count by the
inner one, which is precisely the blow-up instancing exists to prevent -- on
the scaling probe, 400 outer x 25 nested placements stay 400 top-level
geometries, not 10 000. That scene is 22M triangles if baked and imports in
8 MB / 0.29 s.
Two bugs found on the way, one mine and one upstream.
Mine: a prototype root reports `is_instance()` true with `prototype()`
pointing at *itself* -- it carries the instance's `instanceable` metadata and
composition arc -- so expanding a prototype re-resolved its own root and
recursed until the depth cap tripped. Guarded by not treating the walk's own
root as an instance; the root is the content we are already there to walk.
Upstream: an `instanceable` prim *inside another instance's prototype*
cannot be read at all with openusd 0.5.0. Resolving its prototype, or
reading the type name of any prim beneath it, reaches
`pcp/instancing.rs::materialize_prototype`, whose debug_assert ("materialized
prototype root's instanceable must be inert") fires -- debug builds abort,
release builds have it compiled out. The instance prim itself is safe to
inspect (`is_instance`, `children`, `type_name` all succeed); only its
contents are unreachable, so proxy traversal is not a fallback either. Four
lines of USDA reproduce it, with `class` and `def` prototypes alike, and it
is independent of crust; single-level native instancing and nested
PointInstancers are unaffected. `collect_proto_parts` therefore tests
`is_instance()` *before* any schema lookup -- a schema `get()` reads the type
name, which is what aborts -- and skips such prims with a warning rather than
taking the process down. `nested_native_instance_degrades_gracefully` pins
that, and will start failing when upstream is fixed, which is when the arm
should be replaced by splicing the inner prototype's parts in.
`MAX_INSTANCE_NESTING` (8) backstops a malformed stage describing a cycle.
Verification: all 8 pre-existing sample scenes plus samples/instancing.usda
render pixel-identical to HEAD, so nothing outside the nesting paths moved.
Four new integration tests cover the geometry count, that nesting does not
flatten, that two levels of transform compose (an outer non-uniform scale
reaching an inner placement) while both of a nested prototype's materials
survive, and that a multi-part prototype keeps every part.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Q7RXTdHvPFM5weuxWYLYU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The BVH4 node test already built
Vec4lanes, then threw the vectorizationaway by reading lanes back one at a time; leaf intersection was fully scalar,
one
Box<dyn Prim>call per triangle. Both are now vertical SIMD, and thecodegen settings let the vector code actually inline across crates.
Release profile:
lto = "thin"+codegen-units = 1. The hot loop spanscrust-core -> crust-rt -> glam as small
#[inline]functions; without LTOthose calls survive and the vector bodies never fuse. Free 21% on the
triangle traversal benchmark.
Traversal (
bvh.rs):RaySlabpre-splats the ray once per query insteadof rebuilding seven splats per visited node;
safe_inv3does all threereciprocals branch-free; lane verdicts come out as a nibble via
cmple(..).bitmask(); lane validity moved into aflagsbitmask ANDedinto that nibble, replacing four per-lane branches. Unused lanes cannot
simply be given empty bounds -- the slab test's per-axis min/max
un-inverts an inverted box, and +INF/+INF passes when
t_max == INF.Leaf packets (
triangle.rs): the Woop 2013 transform splits into aper-ray part (
RayShear, derived once per traversal rather than once pertriangle) and per-triangle arithmetic that maps onto
Vec4lanes.Tri4holds four triangles in SoA and intersects them in one vector round, with
a two-compare fast path for per-lane visibility masks. Leaf payloads moved
to a side
Leaftable soWideNodestill fits two cache lines (pinned bya size test).
Watertightness is preserved exactly: the f64 tie-break for exactly-zero
edge functions is inherently scalar, so those lanes are reported in a
fallbackmask and re-tested through the scalar path. The twointersectors are bit-identical --
simd_matches_scalar_bitwiseassertsto_bits()equality per lane over ~8000 lanes, not an epsilon.Leaf size retuned to the SIMD width: packet occupancy was 1.66 of 4 lanes
under
MIN_LEAF = 2.MIN_LEAF_PACKED = 4applies only to all-triangleranges (raising the floor for everything cost instanced scenes ~12%);
occupancy is now 2.89 of 4 and every scene kind gets its best number.
material/brdf.rs: threepowf(_, 5.0)in Schlick Fresnel becamepowi(5), matching thepowi(6)already in the same function.powfisan opaque libm call the surrounding vector code cannot be optimized
across. cornellbox 9.09s -> 8.41s; image differs by at most one ULP.
Measured (min of 40, fixed 4096-ray batch, both sides with LTO):
tri_spheres intersect 4.48 -> 3.43 ms occluded 3.13 -> 1.93 ms
sphere_grid intersect 1.96 -> 1.66 ms occluded 1.45 -> 0.93 ms
instances intersect 4.86 -> 3.79 ms occluded 3.55 -> 2.31 ms
End to end, the 383k-triangle stress scene renders in 6.35s against 8.03s.
The checked-in samples move far less (1.5-9%): they are shading-bound, not
traversal-bound -- worth knowing before benchmarking a kernel change on them.
Verification: whole workspace green; every sample scene rendered before and
after and compared with the new
exr_diffexample. Seven of eight arepixel-identical. cornellbox differs in 1 pixel of 230400 by 4.6e-4, at the
seam where the inner box's bottom face is exactly coplanar with the floor --
a genuine
ttie whose winner depends on leaf visit order (confirmedindependent of the retuned tree).
Adds
crust-rttraversal benchmarks, a min-of-Nray_throughputprobe(criterion means drift 10%+ on a shared machine), the
exr_diffregressioncheck, and
docs/simd.md-- including whystd::simdis not used: it isstill nightly-only on the stable toolchain crust builds with.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_019Q7RXTdHvPFM5weuxWYLYU