Add SANA-WM integration (_bidirectional and _streaming variants) - #392
Add SANA-WM integration (_bidirectional and _streaming variants)#392aidanfnv wants to merge 70 commits into
Conversation
Add a workspace integration package for the upstream NVlabs/Sana SANA-WM bidirectional world model. The package registers the `sana-wm-bidirectional` runner, resolves upstream Sana assets and Hugging Face checkpoints, loads images, prompts, camera trajectories, and intrinsics, then writes generated videos through the upstream pipeline. Port CPU-safe camera trajectory and intrinsics helpers so the integration can be checked without running GPU inference. Add smoke and camera tests covering runner registration, config derivation, action parsing, and geometry transforms. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Document that the SANA-WM integration is expected to run inside the upstream Sana environment created by `environment_setup.sh sana`. The setup notes keep Sana's pinned PyTorch and model dependencies intact, then install FlashDreams and the local integration in editable mode without dependency resolution. Include the expected checkout layout and call out Transformer Engine as an optional dependency for quantized inference while BF16 remains the default validated path. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Update the SANA-WM README examples to pass explicit values for boolean Tyro options such as `--no-refiner True`. This matches the generated CLI contract and avoids the confusing "Missing value" error when users treat those fields as flag-style booleans. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Record the BF16 commands used to validate both the Stage-1-only and refined SANA-WM paths, including the expected output locations under `outputs/sana_wm*/sana_wm_generated.mp4`. Note the upstream warnings that appeared during successful runs so they are not mistaken for failures: the null-embed load warning, missing `pos_embed`, and the optional Apex RMSNorm fallback. These commands establish BF16 as the known-good baseline for the integration. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Add first-class `stage1_precision` and `refiner_precision` runner options for BF16, FP8, and FP4. The runner maps those options onto upstream SANA-WM Transformer Engine environment selectors inside a scoped environment so external NVFP4 variables cannot accidentally change the default BF16 path. Validate quantized precision requests before loading checkpoints. FP8 now requires a CUDA device, Hopper-or-newer GPU capability, CUDA 12.9 or newer as reported by PyTorch, and Transformer Engine's Float8BlockScaling recipe. FP4 requires a CUDA device, Blackwell GPU capability, and Transformer Engine's NVFP4 recipe. Extend the SANA README with precision requirements and an FP4 example command, and add CPU-safe tests for the env mapping, cleanup behavior, and early capability errors. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Route the public SANA-WM runner through FlashDreams pipeline, diffusion-model, transformer, and scheduler boundaries instead of the upstream inference harness. Keep the upstream harness only as an explicit reference runner with separate documentation. Replace the vendored upstream diffusion package with a FlashDreams-owned Stage-1 DiT module, direct checkpoint loading, local YAML config parsing, direct diffusers LTX2 VAE encode/decode, and native runner wiring for Stage-1 BF16 execution. Add Transformer Engine-free native FP8 and FP4 Stage-1 Linear replacements using PyTorch scaled-mm and Triton NVFP4 activation quantization, with precision/backend validation before checkpoint loading. Keep the native scope honest: require no-refiner for native runs, keep action overlays and upstream TE quantization on the separate upstream-reference harness, and document the remaining refiner and GDN parity gaps separately from the native runner docs. Expand CPU-safe SANA tests to pin native/reference runner separation, config loading without upstream types, native Stage-1 checkpoint schema, Stage-1 forward shape, runtime release, VAE tiling safeguards, and native quant replacement behavior. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Run classifier-free guidance as sequential negative and positive Stage-1 forwards instead of doubling the full model batch. This reduces peak activation memory for long native SANA-WM rollouts. Release large Stage-1 block temporaries earlier and make the VAE decode OOM retry reduce temporal tile size as well as spatial tile size. Add CPU coverage for sequential CFG kwargs and the lower-memory VAE retry configuration. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Keep retry temporal tiling compatible with LTX2's temporal compression ratio so diffusers does not derive a zero latent stride during OOM retry. Update the SANA smoke coverage for the clamped retry stride and sync the uv lock metadata for the integration dependencies declared by the native SANA package. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Wrap native SANA conditioning and VAE decode in torch.inference_mode so full-length runs do not retain autograd activations during first-frame encode, prompt encode, or LTX2 VAE decode. Extend the VAE OOM retry smoke test to assert both decode attempts run under inference mode. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Replace the Stage-1 placeholder attention path with checkpoint-compatible GDN, softmax, camera UCPE, output-gate, and FFN behavior. Add a native LTX-2 refiner that loads diffusers transformer/connectors and Gemma text conditioning directly from the SANA-WM checkpoint, then wire the native runner to refine by default before VAE decode. Keep BF16/FP8/FP4 on FlashDreams/PyTorch native paths, add accelerate for low-memory refiner loading, update docs, and cover the new refiner construction plus latent packing in CPU tests. Validation: uv run --package flashdreams-sana-wm pytest integrations/sana/tests/test_smoke.py integrations/sana/tests/test_camera.py; uv run --package flashdreams-sana-wm flashdreams-run --no-instantiate sana-wm-bidirectional; no-instantiate checks for fp8, fp4, and --no-refiner True.
The native LTX-2 refiner prompt encoder returns inference tensors. Keep the whole refiner denoise path inside torch.inference_mode so diffusers Linear calls do not try to save those tensors for autograd. Add a CPU regression assertion that the refiner prediction callback executes with inference mode enabled. Validation: uv run --package flashdreams-sana-wm pytest integrations/sana/tests/test_smoke.py integrations/sana/tests/test_camera.py.
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Move SANA-WM prompt, first-frame, camera, decode, and refiner orchestration into explicit pipeline components instead of keeping it inside the transformer or runner. The public runner now drives the normal StreamInferencePipeline contract, with a Sana conditioning encoder, Stage-1 transformer, LTX-style scheduler, and video decoder/refiner boundary. Keep the native Stage-1 DiT implementation but make SanaWMDiffusionModelConfig instantiate the shared FlashDreams DiffusionModel. Add generic infra hooks for transformer-provided initial noise and scheduler access to per-step encoder context so Sana can pin the first-frame latent and run per-token LTX Euler sampling without owning a custom diffusion loop. Replace direct diffusers FlowMatch Euler use with a SanaWMLTXEulerScheduler that matches the LTX per-token behavior and keeps the scheduler-specific timestep handling isolated behind the FlashDreams scheduler interface. Restore batched CFG as the default Stage-1 path and remove manual early temporary deletions added during the earlier OOM investigation. Keep inference-mode execution and default-false offload/tiling fallback paths where they remain useful inference hygiene or explicit user controls. Expand CPU coverage for runner registration, component config routing, prompt/first-frame/camera conditioning shapes, scheduler parity, base DiffusionModel instantiation, CFG batching, VAE decode/refiner boundaries, and Stage-1 checkpoint shape coverage. Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Make the SANA-WM pipeline reuse-friendly: the Stage-1 DiT, LTX-2 refiner, and Gemma text encoder are no longer torn down and rebuilt on every generate(). Previously each rollout unconditionally released Stage-1 before decode, released the refiner after refine, and reloaded the ~20 GB Gemma encoder from disk per prompt, so a reused pipeline re-paid a full checkpoint reload and re-quantization (~10 s bf16, ~17 s fp8) plus an ~8-15 s Gemma reload on every call. - transformer: gate the Stage-1 teardown behind a new offload_stage1 flag (default keeps it resident); still drop per-rollout conditioning. - decoder: drop the unconditional refiner release (refine_latents already releases when offload_refiner is set). - refiner: cache the Gemma tokenizer/encoder in CPU RAM instead of reloading per call, and move it to the GPU only for the encode forward so it never inflates peak memory during denoise/decode. Fix deprecated torch_dtype and drop dead _empty_cuda_cache/gc. - runner: expose --offload-stage1 and route it to the transformer. - Add [timing] instrumentation for the Stage-1 build, refiner build, and Gemma load. Measured on Blackwell: warm-reuse pass drops from ~26 s to ~5 s (fp8); full-res fp8 demo_0 peaks at ~54/98 GB with no OOM. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make --intrinsics-path optional. When omitted, intrinsics are derived from the first-frame size with a centered principal point and a focal length set by a horizontal field of view (new --intrinsics-hfov-deg, default 90 to match the public demo intrinsics). This lets a run be driven by just an image and a prompt (with --action supplying the camera trajectory). - camera: add default_intrinsics_vec4(), returning [F, 4] intrinsics in source-image pixels so they flow through transform_intrinsics_for_crop identically to a loaded file. - runner: derive intrinsics when intrinsics_path is None, log the derivation, and expose --intrinsics-hfov-deg; explicit files behave as before. - README: document optional intrinsics and a minimal image+prompt run. - tests: cover the helper (centering, FOV math, bad-FOV rejection) and the runner derive path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move SANA-WM first-frame timestep pinning and zero-flow behavior out of the scheduler context path and into the SANA transformer. The shared diffusion model now calls schedulers through the generic initial_noise/predict_flow/rng interface again, and the SANA LTX Euler scheduler only owns timestep construction and update math. Make the decoder/refiner boundary conform to FlashDreams video decoder contracts by subclassing StreamingVideoDecoder and publishing the SANA 8k+1 temporal sizing. Remove runner, cache, and refiner AR block-size/KV-frame knobs that were accepted but ignored, leaving the supported sink-bidirectional LTX-2 refiner path explicit. Route runner step and flow_shift overrides into the scheduler config instead of through SANA conditioning, and update SANA smoke coverage for scheduler isolation, transformer first-frame pinning, decoder sizing, CFG, and refiner behavior. Tests:\n- uv run --package flashdreams-sana-wm --extra dev pytest integrations/sana/tests Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Make the SANA-WM runner choose the snapped rollout length from --num-frames before preparing camera conditioning. Action-derived trajectories now repeat or truncate to that snapped length, so a short action prompt no longer caps the generated video. Fit explicit --camera-path trajectories to the same frame count instead of treating the file length as an upper bound. This matches the existing intrinsics fitting behavior and prevents a 321-frame pose file from truncating a longer requested rollout. Document the updated behavior and add CPU coverage for action repetition, explicit camera-path fitting, and runner input preparation. Tests: - uv run --package flashdreams-sana-wm --extra dev pytest integrations/sana/tests Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Greptile SummaryAdds independently packaged SANA-WM bidirectional and streaming integrations.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; the previously reported oversized action-duration allocation is bounded by the requested frame count for all production call paths. Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant Runner as SANA-WM Runner
participant Camera as Camera Conditioning
participant Stage1 as Stage-1 DiT
participant Refiner as LTX-2 Refiner
participant VAE as VAE Decoder
participant Output as Video Output
User->>Runner: Select bidirectional or streaming runner
Runner->>Camera: Resolve pose file or bounded action DSL
Camera-->>Runner: Fitted trajectory and intrinsics
Runner->>Stage1: Generate conditioned latent sequence
Stage1-->>Runner: Stage-1 latents
Runner->>Refiner: Refine full sequence or causal prefix
Refiner-->>Runner: Refined latents
Runner->>VAE: Decode full clip or streaming chunks
VAE-->>Runner: Video frames
Runner->>Output: Write clip or progressively emit chunks
Reviews (18): Last reviewed commit: "Minor model card update" | Re-trigger Greptile |
This reverts commit 3fede3f.
…danf/sana_integration # Conflicts: # uv.lock
|
/ok to test 1485061 |
|
/ok to test 83e5ec4 |
|
/ok to test 7d486ca |
|
/ok to test bc58ce3 |
jarcherNV
left a comment
There was a problem hiding this comment.
This was pretty long so admittedly I only skimmed it, but I didn't notice any glaring issues so I think it is fine to merge.
|
An additional comment: There are a lot of commits in this PR, so it may produce a pretty long commit message when merged. You may want to consider squashing it and creating a reasonable commit message for everything, but I'll leave it up to you to decide how to proceed. |
Summary
Adds the
flashdreams-sana-wmintegration for both public NVlabs/Sana camera-controlled world-model releases:SANA-WM_bidirectionalsana-wm-bidirectionalSANA-WM_streamingsana-wm-streamingThe two upstream releases have distinct runner slugs, configs, checkpoint roots, parity helpers, benchmark scripts, benchmark data, and model cards.
Implementation
SANA-WM_bidirectionalis wired through a native FlashDreams Stage-1 DiT, the LTX-2 refiner, VAE decode, camera conditioning, output writing, and the shared FlashDreams runner/pipeline/diffusion contracts.SANA-WM_streamingis wired as its own chunk-causal path with streaming Stage-1 conditioning, streaming checkpoint roots, chunk boundaries, streaming LTX-2 refiner settings, causal VAE decode, and progressive output emission._scaled_mmplus integration-local Triton helpers where needed; no upstream fused kernels are vendored.SANA-WM_bidirectionalexposes FlashDreams FP8/FP4 options, but the matched upstream comparison is BF16-only because upstreamSANA-WM_bidirectionaldoes not expose the streaming release's FP8/FP4 precision flags.SANA-WM_streamingowns the matched BF16/FP8/FP4 upstream comparisons.User-Facing Changes
integrations/sana/, published asflashdreams-sana-wm(Python packagesana_wm).sana-wm-bidirectionalsana-wm-streamingdocs/source/models/sana_wm_bidirectional.rstdocs/source/models/sana_wm_streaming.rstdocs/source/_static/performance/sana_wm_bidirectional/perf-0801-bf16.mddocs/source/_static/performance/sana_wm_streaming/perf-0801-bf16.mddocs/source/_static/performance/sana_wm_streaming/perf-0801-fp8.mddocs/source/_static/performance/sana_wm_streaming/perf-0801-fp4.md"w-100,dw-60,w-101".--intrinsics-pathis optional. When omitted, intrinsics are derived from the first-frame geometry and--intrinsics-hfov-deg.8 * --num-frame-per-block * k + 1.Implementation Notes
integrations/sana/sana_wm/and covers both bidirectional and chunk-causal streaming transformer variants._stage1_state_dict()normalizes the two release formats into one state dict: bidirectional ships flat safetensors, streaming ships a.ptnesting the tensors undergenerator/state_dictwithmodel.-prefixed keys. Both load understrict=True.DiffusionModel.run()delegates initial latent creation toTransformer.initial_noise(), so SANA-WM first-frame pinning can live in the transformer boundary without a SANA-only diffusion-model subclass.integrations/sana/tests/parity_check/adds a correctness harness that dispatches throughSANA_WM_VARIANT=bidirectional|streaming, with separate FlashDreams helpers for each release.integrations/sana/tests/performance/holds the timing harness (bench.sh,bench_summary.py,bench_sweep_summary.py). It clones and pins the upstream Sana checkout, applies instrumentation-only patches, and runs both sides from an isolateduvenvironment. The benchmark summary code rejects low-precision upstream comparisons forSANA-WM_bidirectionaland treats streaming metrics as steady-state milliseconds per produced chunk.quantextra, used to run the upstream streaming FP8/FP4 side. FlashDreams low-precision rows use the base harness environment, and the FlashDreams runtime path never imports it.Differences From Official SANA-WM Code
te.Linear,te.fp8_autocast,NVFP4BlockScaling, andFloat8BlockScaling). The FlashDreams integration uses Torch/Triton replacement layers instead.Benchmarks
Published on the model cards. One NVIDIA GB300 GPU, upstream Sana at
6298508, FlashDreams atbd0816e(later commits have been smoke tested to ensure they still produce the same results), one discarded warmup run and three measured runs. Model construction, checkpoint loading, video writing, and frame dumps are outside the timing boundary.SANA-WM_bidirectional, BF16, 121-frame full-pipeline clip (Stage-1 DiT + LTX-2 refiner + VAE decode):SANA-WM_streaming, steady-state generation latency per produced chunk (241 requested frames,w-80,dw-40,w-80,aw-40action path):The streaming rows are published as a measured baseline rather than a claimed speedup: the FlashDreams streaming path is functional and benchmarked, but still slower than the official implementation.
Test Plan
Setup:
CPU tests (113 total):
uv run --package flashdreams-sana-wm --extra dev pytest \ integrations/sana/tests/test_smoke.py \ integrations/sana/tests/test_camera.py \ integrations/sana/tests/test_performance_benchmark_summary.py \ integrations/sana/tests/test_parity_quality_tools.pytest_smoke.pytest_camera.pytest_performance_benchmark_summary.pyquantextra routing, and streaming steady-state metricstest_parity_quality_tools.pyGPU tests (27 total, run on RTX PRO 6000 Blackwell):
uv run --package flashdreams-sana-wm --extra dev pytest \ integrations/sana/tests/test_stage1_cuda.py \ integrations/sana/tests/test_quant_cuda.pytest_stage1_cuda.pytest_quant_cuda.py_scaled_mmand Triton quantization kernelsEnd-to-end generation, both runners, BF16,
demo_0assets:sana-wm-bidirectionalwrites a clip; Stage-1 weights load understrict=Truewith no missing or unexpected keys.sana-wm-streamingwrites a clip; same cleanstrict=Trueload of the.ptrelease.Full benchmark sweeps (BF16/FP8/FP4, both variants, against the pinned upstream Sana checkout at
6298508) were run on RTX PRO 6000 Blackwell and on GB300. The model cards publish the GB300 rows.