Stream the USD import: 117 GiB / 13:20 -> 44 GiB / 09:19 on Moana - #119
Merged
Conversation
Adds the measurement needed to find where a render's memory actually
goes, then fixes the two things it showed were wasteful in the kernel.
Measurement:
- each phase now samples resident and peak RSS at its own boundary, so
a phase that allocates far more than it keeps is visible as a gap
between the two;
- Scene::memory_footprint walks the committed scene and reports exact
resident bytes by structure, deduplicating shared prototypes.
Fixes:
- InstancePrim::l2w_end is boxed. Geometry::Instance's equivalent was
already boxed, but this is the struct that stays resident for the
whole render, and the field is None for every static placement;
- BVH partitioning counts each side before allocating and takes its
input by value, instead of sizing both children at the full input
length and holding the parent's buffer alive through a parallel
subtree build.
On the complete Moana island: 119.74 -> 117.21 GiB, runtime unchanged.
The modest size of that win is itself the finding. The footprint report
puts the kernel at 39.12 GiB of a 115.67 GiB peak, and isolating the
importer shows openusd's stage accounts for 75.74 GiB and 6m19s on its
own -- roughly two thirds of the memory and half the runtime. Further
work inside the kernel is bounded by the remaining third.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pure refactor, no behaviour change: the traversal loop moves out of load_scene into traverse_into, with the state it accumulates gathered into an ImportCtx. The function borrows the stage and retains nothing from it, so a caller can walk one stage, drop it, and walk another into the same scene. That is the shape a streaming importer needs. Composing the whole Moana island at once costs openusd 75.74 GiB and 6m19s, where a stage masked to a single element costs 1.10 GiB and 2.6s -- so importing element by element should bound peak memory at roughly one element's worth. This commit only makes that possible; it does not yet do it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Composing the whole Moana island costs openusd 75.74 GiB and 6m19s, against 1.10 GiB and 2.6s for a stage masked to a single element. So load_scene can now walk one masked stage per top-level subtree and drop each when done: an index stage opened with InitialLoadSet::LoadNone supplies the render settings and the list of subtrees, then each is composed, traversed and released in turn. Where it works the win is large -- 117.10 -> 42.53 GiB peak and 13:20 -> 08:57 on the island. It is nonetheless **off by default**, behind CRUST_STREAM_IMPORT=1, because it loses geometry: 50 990 392 triangles against 56 825 650, and 1.5% of pixels differ. Each element's root is instanceable, so its content lives in a prototype at a stage-root path outside the element's subtree, and under a per-element mask nine of the island's prototypes resolve empty. openusd's mask_includes is meant to cover that case, so the cause is not simply an unlisted mask path. The full diagnosis, and what has been ruled out, is on stream_roots. Default behaviour is unchanged and verified pixel-identical. Also adds ImportCaches::epoch, which separates per-stage prototype namespaces without clearing the cache -- MeshKey identifies a material by Arc address, so freeing material Arcs mid-import would let a later allocation reuse an address a stale mesh-cache entry still matches. That hazard is real regardless of streaming; keying MeshKey by the material's authored path instead would remove it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A production stage is dominated by USD itself rather than by anything the
renderer builds: composing all of Moana costs openusd 75.74 GiB and
6m19s, against 1.10 GiB and 2.6s for a stage masked to a single element.
So load_scene now opens a cheap index stage with InitialLoadSet::LoadNone
for the render settings and the list of top-level subtrees, then composes,
traverses and drops one masked stage per subtree. That bounds the composed
set at roughly one element instead of all twenty. On the island:
peak 117.10 GiB -> 43.76 GiB
total 13:20 -> 09:19
with the image pixel-identical (0 of 230400 pixels differ) and every scene
counter unchanged. MIN_STREAM_CHUNKS keeps the single-stage path for scenes
too small to repay the re-opens, and CRUST_STREAM_IMPORT=0 forces it.
The subtlety is which cache keys survive a change of stage. Prototype paths
are numbered per composition, so each masked stage has its own
/__Prototype_0, and MaterialCache keyed on the bare path handed one chunk's
materials to the next chunk's geometry. Since MeshKey identifies a material
by Arc address, distinct meshes then collapsed onto shared cache entries:
50 990 392 triangles instead of 56 825 650, and 1.5% of pixels wrong. Only
prototype-internal paths are now epoch-scoped, so authored paths still
deduplicate across chunks -- which is what stops streaming costing memory.
No single element reproduces that bug; all twenty are identical in both
modes on their own. It takes two chunks that both carry prototype-internal
materials, which is why the elimination had to run to completion before the
cause was visible.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Records what the numbers are, and the trap: prototype paths are numbered per composition, so anything cached under one must be scoped per stage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Cuts peak memory on the complete Moana island by 63% and runtime by
30%, with output pixel-identical.
How it was found
The per-phase profiling added in #118 said parsing was 75% of runtime, so
the first attempts were aimed at the kernel. Measurement then closed the
accounting and redirected the work:
Two thirds of the memory and about half the runtime were USD, not us. The
kernel-side fixes here (boxing
InstancePrim::l2w_end, exact-size BVHpartitions) are real but worth only 2.53 GiB — that ceiling is the point,
and it is what motivated the rest.
Three hypotheses were tested and disproved rather than assumed: BVH
build churn (predicted ~85 GiB, actual 0.49), glibc arena fragmentation
(1.43 GiB), and a prototype-cache/
Arc-lifetime theory (left the countsbyte-identical).
The change
Composing the whole island costs openusd 75.74 GiB; composing a stage
masked to one element costs 1.10 GiB. So
load_sceneopens a cheap indexstage (
InitialLoadSet::LoadNone) for the render settings and the list oftop-level subtrees, then composes, traverses and drops one masked stage
per subtree, bounding the composed set at roughly one element.
MIN_STREAM_CHUNKSkeeps the single-stage path for scenes too small torepay the re-opens;
CRUST_STREAM_IMPORT=0forces it, and is how the twopaths were A/B'd against each other.
The bug this surfaced, and why it was hard
Prototype paths (
/__Prototype_N) are numbered per composition, soevery masked stage has its own
/__Prototype_0.MaterialCachekeyed onthe bare path handed one chunk's material to the next chunk's geometry;
because
MeshKeyidentifies a material by itsArcaddress, distinctmeshes then collapsed onto shared cache entries — 50 990 392 triangles
instead of 56 825 650, and 1.5% of pixels wrong.
It is invisible to targeted testing: all twenty elements are identical in
both modes individually. It needs two chunks that both carry
prototype-internal materials. Running the elimination to completion is what
proved it had to be an interaction, which is what pointed at the one cache
still keyed on an unstable path.
Only prototype-internal paths are now epoch-scoped, so authored paths still
deduplicate across chunks — which is what stops streaming costing memory.
An intermediate commit (
05a6fb9) lands the feature disabled with the bugdocumented; the next fixes it and enables it. Kept deliberately so the
hazard is on record, but happy to squash.
Testing
scripts/test_simd_matrix.sh -p crust-rtbit-exact across all fourcodegen configurations.
Follow-ups (not in this PR)
MeshKeystill identifies materials byArcaddress, making "no materialArcmay be freed during import" an unwritten invariant. Both this bug andthe earlier prototype one were instances of that hazard class; keying on the
authored path would remove it.
Stage::prototypes()returns[]even unmasked whileprim.prototype()resolves correctly — an openusd inconsistency noticed but not chased.
🤖 Generated with Claude Code