Skip to content

Add SANA-WM integration (_bidirectional and _streaming variants) - #392

Open
aidanfnv wants to merge 70 commits into
NVIDIA:mainfrom
aidanfnv:dev/aidanf/sana_integration
Open

Add SANA-WM integration (_bidirectional and _streaming variants)#392
aidanfnv wants to merge 70 commits into
NVIDIA:mainfrom
aidanfnv:dev/aidanf/sana_integration

Conversation

@aidanfnv

@aidanfnv aidanfnv commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the flashdreams-sana-wm integration for both public NVlabs/Sana camera-controlled world-model releases:

upstream release FlashDreams runner inference contract benchmark scope
SANA-WM_bidirectional sana-wm-bidirectional full-sequence I2V clip generation in one bidirectional pass BF16 upstream-vs-FlashDreams clip latency
SANA-WM_streaming sana-wm-streaming chunk-causal autoregressive generation with streaming refiner/VAE decode BF16, FP8, and FP4 upstream-vs-FlashDreams chunk latency

The two upstream releases have distinct runner slugs, configs, checkpoint roots, parity helpers, benchmark scripts, benchmark data, and model cards.

Implementation

  • SANA-WM_bidirectional is wired through a native FlashDreams Stage-1 DiT, the LTX-2 refiner, VAE decode, camera conditioning, output writing, and the shared FlashDreams runner/pipeline/diffusion contracts.
  • SANA-WM_streaming is wired as its own chunk-causal path with streaming Stage-1 conditioning, streaming checkpoint roots, chunk boundaries, streaming LTX-2 refiner settings, causal VAE decode, and progressive output emission.
  • Runner configs select separate bidirectional and streaming checkpoint roots for Stage-1, refiner, VAE, and text-encoder assets.
  • Runtime quantization does not depend on TransformerEngine. FP8/FP4 paths use Torch _scaled_mm plus integration-local Triton helpers where needed; no upstream fused kernels are vendored.
  • SANA-WM_bidirectional exposes FlashDreams FP8/FP4 options, but the matched upstream comparison is BF16-only because upstream SANA-WM_bidirectional does not expose the streaming release's FP8/FP4 precision flags.
  • SANA-WM_streaming owns the matched BF16/FP8/FP4 upstream comparisons.

User-Facing Changes

  • New integration package under integrations/sana/, published as flashdreams-sana-wm (Python package sana_wm).
  • Two registered runner slugs:
    • sana-wm-bidirectional
    • sana-wm-streaming
  • Separate docs pages:
    • docs/source/models/sana_wm_bidirectional.rst
    • docs/source/models/sana_wm_streaming.rst
    • Both are linked from the landing page and model index, each with a sample AVIF clip.
  • Separate chart data:
    • docs/source/_static/performance/sana_wm_bidirectional/perf-0801-bf16.md
    • docs/source/_static/performance/sana_wm_streaming/perf-0801-bf16.md
    • docs/source/_static/performance/sana_wm_streaming/perf-0801-fp8.md
    • docs/source/_static/performance/sana_wm_streaming/perf-0801-fp4.md
  • Camera inputs can come from a pose file or the action DSL, for example "w-100,dw-60,w-101".
  • --intrinsics-path is optional. When omitted, intrinsics are derived from the first-frame geometry and --intrinsics-hfov-deg.
  • Explicit camera paths and generated action trajectories are fitted to the requested frame count. Streaming requests are snapped to 8 * --num-frame-per-block * k + 1.

Implementation Notes

  • The Stage-1 DiT implementation lives in integrations/sana/sana_wm/ and covers both bidirectional and chunk-causal streaming transformer variants.
  • Bidirectional and streaming use distinct constants for Hugging Face repos, config roots, Stage-1 checkpoints, refiner roots, and VAE roots.
  • _stage1_state_dict() normalizes the two release formats into one state dict: bidirectional ships flat safetensors, streaming ships a .pt nesting the tensors under generator / state_dict with model.-prefixed keys. Both load under strict=True.
  • Streaming uses a FlashDreams-owned config literal instead of vendoring the upstream YAML.
  • DiffusionModel.run() delegates initial latent creation to Transformer.initial_noise(), so SANA-WM first-frame pinning can live in the transformer boundary without a SANA-only diffusion-model subclass.
  • integrations/sana/tests/parity_check/ adds a correctness harness that dispatches through SANA_WM_VARIANT=bidirectional|streaming, with separate FlashDreams helpers for each release.
  • integrations/sana/tests/performance/ holds the timing harness (bench.sh, bench_summary.py, bench_sweep_summary.py). It clones and pins the upstream Sana checkout, applies instrumentation-only patches, and runs both sides from an isolated uv environment. The benchmark summary code rejects low-precision upstream comparisons for SANA-WM_bidirectional and treats streaming metrics as steady-state milliseconds per produced chunk.
  • TransformerEngine appears only in the benchmark harness's optional quant extra, used to run the upstream streaming FP8/FP4 side. FlashDreams low-precision rows use the base harness environment, and the FlashDreams runtime path never imports it.

Differences From Official SANA-WM Code

  • Low-precision backend: official SANA-WM uses TransformerEngine for FP8/FP4 paths (te.Linear, te.fp8_autocast, NVFP4BlockScaling, and Float8BlockScaling). The FlashDreams integration uses Torch/Triton replacement layers instead.
  • Intrinsics fallback: omitted intrinsics are derived from a fixed-HFoV assumption instead of Pi3X. This is lighter-weight, but less accurate when the true camera HFoV differs from the assumption.
  • No prepared-module cache: upstream caches the pickled post-quantization Stage-1 module and reloads it on subsequent FP4 runs. The integration re-quantizes on every launch. This affects FP4 startup only — BF16 and FP8 do not use the cache upstream, and preparation happens outside the benchmark timing boundary either way.

Benchmarks

Published on the model cards. One NVIDIA GB300 GPU, upstream Sana at 6298508, FlashDreams at bd0816e (later commits have been smoke tested to ensure they still produce the same results), one discarded warmup run and three measured runs. Model construction, checkpoint loading, video writing, and frame dumps are outside the timing boundary.

SANA-WM_bidirectional, BF16, 121-frame full-pipeline clip (Stage-1 DiT + LTX-2 refiner + VAE decode):

metric official FlashDreams
post-load generation latency / clip 56,932.83 ms 34,182.39 ms

SANA-WM_streaming, steady-state generation latency per produced chunk (241 requested frames, w-80,dw-40,w-80,aw-40 action path):

precision official FlashDreams
BF16 1,170.29 ms/chunk 1,957.93 ms/chunk
FP8 1,482.92 ms/chunk 2,392.67 ms/chunk
FP4 1,594.33 ms/chunk 4,118.36 ms/chunk

The streaming rows are published as a measured baseline rather than a claimed speedup: the FlashDreams streaming path is functional and benchmarked, but still slower than the official implementation.

Test Plan

Setup:

uv sync --package flashdreams-sana-wm --extra dev

CPU tests (113 total):

uv run --package flashdreams-sana-wm --extra dev pytest \
    integrations/sana/tests/test_smoke.py \
    integrations/sana/tests/test_camera.py \
    integrations/sana/tests/test_performance_benchmark_summary.py \
    integrations/sana/tests/test_parity_quality_tools.py
suite tests covers
test_smoke.py 76 runner/pipeline/config wiring, both transformer variants, conditioning, decode, refiner adapter, scheduler, quant layer construction, HF path resolution, and the pinned artefact defaults
test_camera.py 14 pose-file and action-DSL trajectories, bounded action materialization, frame-count fitting, and intrinsics derivation
test_performance_benchmark_summary.py 20 benchmark summary parsing, precision-row rejection rules, SANA benchmark quant extra routing, and streaming steady-state metrics
test_parity_quality_tools.py 3 parity diff and streaming-continuity helpers

GPU tests (27 total, run on RTX PRO 6000 Blackwell):

uv run --package flashdreams-sana-wm --extra dev pytest \
    integrations/sana/tests/test_stage1_cuda.py \
    integrations/sana/tests/test_quant_cuda.py
suite tests covers
test_stage1_cuda.py 24 Stage-1 CUDA execution, chunk-causal streaming windows, and KV-cache behaviour
test_quant_cuda.py 3 FP8/FP4 _scaled_mm and Triton quantization kernels

End-to-end generation, both runners, BF16, demo_0 assets:

  • sana-wm-bidirectional writes a clip; Stage-1 weights load under strict=True with no missing or unexpected keys.
  • sana-wm-streaming writes a clip; same clean strict=True load of the .pt release.

Full benchmark sweeps (BF16/FP8/FP4, both variants, against the pinned upstream Sana checkout at 6298508) were run on RTX PRO 6000 Blackwell and on GB300. The model cards publish the GB300 rows.

aidanfnv and others added 19 commits July 16, 2026 08:22
Add a workspace integration package for the upstream NVlabs/Sana SANA-WM
bidirectional world model. The package registers the `sana-wm-bidirectional`
runner, resolves upstream Sana assets and Hugging Face checkpoints, loads
images, prompts, camera trajectories, and intrinsics, then writes generated
videos through the upstream pipeline.

Port CPU-safe camera trajectory and intrinsics helpers so the integration can
be checked without running GPU inference. Add smoke and camera tests covering
runner registration, config derivation, action parsing, and geometry
transforms.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Document that the SANA-WM integration is expected to run inside the upstream
Sana environment created by `environment_setup.sh sana`. The setup notes keep
Sana's pinned PyTorch and model dependencies intact, then install FlashDreams
and the local integration in editable mode without dependency resolution.

Include the expected checkout layout and call out Transformer Engine as an
optional dependency for quantized inference while BF16 remains the default
validated path.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Update the SANA-WM README examples to pass explicit values for boolean Tyro
options such as `--no-refiner True`. This matches the generated CLI contract
and avoids the confusing "Missing value" error when users treat those fields
as flag-style booleans.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Record the BF16 commands used to validate both the Stage-1-only and refined
SANA-WM paths, including the expected output locations under
`outputs/sana_wm*/sana_wm_generated.mp4`.

Note the upstream warnings that appeared during successful runs so they are
not mistaken for failures: the null-embed load warning, missing `pos_embed`,
and the optional Apex RMSNorm fallback. These commands establish BF16 as the
known-good baseline for the integration.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Add first-class `stage1_precision` and `refiner_precision` runner options for
BF16, FP8, and FP4. The runner maps those options onto upstream SANA-WM
Transformer Engine environment selectors inside a scoped environment so
external NVFP4 variables cannot accidentally change the default BF16 path.

Validate quantized precision requests before loading checkpoints. FP8 now
requires a CUDA device, Hopper-or-newer GPU capability, CUDA 12.9 or newer as
reported by PyTorch, and Transformer Engine's Float8BlockScaling recipe. FP4
requires a CUDA device, Blackwell GPU capability, and Transformer Engine's
NVFP4 recipe.

Extend the SANA README with precision requirements and an FP4 example command,
and add CPU-safe tests for the env mapping, cleanup behavior, and early
capability errors.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Route the public SANA-WM runner through FlashDreams pipeline, diffusion-model, transformer, and scheduler boundaries instead of the upstream inference harness. Keep the upstream harness only as an explicit reference runner with separate documentation.

Replace the vendored upstream diffusion package with a FlashDreams-owned Stage-1 DiT module, direct checkpoint loading, local YAML config parsing, direct diffusers LTX2 VAE encode/decode, and native runner wiring for Stage-1 BF16 execution.

Add Transformer Engine-free native FP8 and FP4 Stage-1 Linear replacements using PyTorch scaled-mm and Triton NVFP4 activation quantization, with precision/backend validation before checkpoint loading.

Keep the native scope honest: require no-refiner for native runs, keep action overlays and upstream TE quantization on the separate upstream-reference harness, and document the remaining refiner and GDN parity gaps separately from the native runner docs.

Expand CPU-safe SANA tests to pin native/reference runner separation, config loading without upstream types, native Stage-1 checkpoint schema, Stage-1 forward shape, runtime release, VAE tiling safeguards, and native quant replacement behavior.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Run classifier-free guidance as sequential negative and positive Stage-1 forwards instead of doubling the full model batch. This reduces peak activation memory for long native SANA-WM rollouts.

Release large Stage-1 block temporaries earlier and make the VAE decode OOM retry reduce temporal tile size as well as spatial tile size.

Add CPU coverage for sequential CFG kwargs and the lower-memory VAE retry configuration.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Keep retry temporal tiling compatible with LTX2's temporal compression ratio so diffusers does not derive a zero latent stride during OOM retry.

Update the SANA smoke coverage for the clamped retry stride and sync the uv lock metadata for the integration dependencies declared by the native SANA package.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Wrap native SANA conditioning and VAE decode in torch.inference_mode so full-length runs do not retain autograd activations during first-frame encode, prompt encode, or LTX2 VAE decode.

Extend the VAE OOM retry smoke test to assert both decode attempts run under inference mode.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Replace the Stage-1 placeholder attention path with checkpoint-compatible GDN, softmax, camera UCPE, output-gate, and FFN behavior.

Add a native LTX-2 refiner that loads diffusers transformer/connectors and Gemma text conditioning directly from the SANA-WM checkpoint, then wire the native runner to refine by default before VAE decode.

Keep BF16/FP8/FP4 on FlashDreams/PyTorch native paths, add accelerate for low-memory refiner loading, update docs, and cover the new refiner construction plus latent packing in CPU tests.

Validation: uv run --package flashdreams-sana-wm pytest integrations/sana/tests/test_smoke.py integrations/sana/tests/test_camera.py; uv run --package flashdreams-sana-wm flashdreams-run --no-instantiate sana-wm-bidirectional; no-instantiate checks for fp8, fp4, and --no-refiner True.
The native LTX-2 refiner prompt encoder returns inference tensors. Keep the whole refiner denoise path inside torch.inference_mode so diffusers Linear calls do not try to save those tensors for autograd.

Add a CPU regression assertion that the refiner prediction callback executes with inference mode enabled.

Validation: uv run --package flashdreams-sana-wm pytest integrations/sana/tests/test_smoke.py integrations/sana/tests/test_camera.py.
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Move SANA-WM prompt, first-frame, camera, decode, and refiner orchestration into explicit pipeline components instead of keeping it inside the transformer or runner. The public runner now drives the normal StreamInferencePipeline contract, with a Sana conditioning encoder, Stage-1 transformer, LTX-style scheduler, and video decoder/refiner boundary.

Keep the native Stage-1 DiT implementation but make SanaWMDiffusionModelConfig instantiate the shared FlashDreams DiffusionModel. Add generic infra hooks for transformer-provided initial noise and scheduler access to per-step encoder context so Sana can pin the first-frame latent and run per-token LTX Euler sampling without owning a custom diffusion loop.

Replace direct diffusers FlowMatch Euler use with a SanaWMLTXEulerScheduler that matches the LTX per-token behavior and keeps the scheduler-specific timestep handling isolated behind the FlashDreams scheduler interface.

Restore batched CFG as the default Stage-1 path and remove manual early temporary deletions added during the earlier OOM investigation. Keep inference-mode execution and default-false offload/tiling fallback paths where they remain useful inference hygiene or explicit user controls.

Expand CPU coverage for runner registration, component config routing, prompt/first-frame/camera conditioning shapes, scheduler parity, base DiffusionModel instantiation, CFG batching, VAE decode/refiner boundaries, and Stage-1 checkpoint shape coverage.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Make the SANA-WM pipeline reuse-friendly: the Stage-1 DiT, LTX-2 refiner,
and Gemma text encoder are no longer torn down and rebuilt on every
generate(). Previously each rollout unconditionally released Stage-1 before
decode, released the refiner after refine, and reloaded the ~20 GB Gemma
encoder from disk per prompt, so a reused pipeline re-paid a full checkpoint
reload and re-quantization (~10 s bf16, ~17 s fp8) plus an ~8-15 s Gemma
reload on every call.

- transformer: gate the Stage-1 teardown behind a new offload_stage1 flag
  (default keeps it resident); still drop per-rollout conditioning.
- decoder: drop the unconditional refiner release (refine_latents already
  releases when offload_refiner is set).
- refiner: cache the Gemma tokenizer/encoder in CPU RAM instead of reloading
  per call, and move it to the GPU only for the encode forward so it never
  inflates peak memory during denoise/decode. Fix deprecated torch_dtype and
  drop dead _empty_cuda_cache/gc.
- runner: expose --offload-stage1 and route it to the transformer.
- Add [timing] instrumentation for the Stage-1 build, refiner build, and
  Gemma load.

Measured on Blackwell: warm-reuse pass drops from ~26 s to ~5 s (fp8);
full-res fp8 demo_0 peaks at ~54/98 GB with no OOM.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make --intrinsics-path optional. When omitted, intrinsics are derived from
the first-frame size with a centered principal point and a focal length set
by a horizontal field of view (new --intrinsics-hfov-deg, default 90 to match
the public demo intrinsics). This lets a run be driven by just an image and a
prompt (with --action supplying the camera trajectory).

- camera: add default_intrinsics_vec4(), returning [F, 4] intrinsics in
  source-image pixels so they flow through transform_intrinsics_for_crop
  identically to a loaded file.
- runner: derive intrinsics when intrinsics_path is None, log the derivation,
  and expose --intrinsics-hfov-deg; explicit files behave as before.
- README: document optional intrinsics and a minimal image+prompt run.
- tests: cover the helper (centering, FOV math, bad-FOV rejection) and the
  runner derive path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move SANA-WM first-frame timestep pinning and zero-flow behavior out of the scheduler context path and into the SANA transformer. The shared diffusion model now calls schedulers through the generic initial_noise/predict_flow/rng interface again, and the SANA LTX Euler scheduler only owns timestep construction and update math.

Make the decoder/refiner boundary conform to FlashDreams video decoder contracts by subclassing StreamingVideoDecoder and publishing the SANA 8k+1 temporal sizing. Remove runner, cache, and refiner AR block-size/KV-frame knobs that were accepted but ignored, leaving the supported sink-bidirectional LTX-2 refiner path explicit.

Route runner step and flow_shift overrides into the scheduler config instead of through SANA conditioning, and update SANA smoke coverage for scheduler isolation, transformer first-frame pinning, decoder sizing, CFG, and refiner behavior.

Tests:\n- uv run --package flashdreams-sana-wm --extra dev pytest integrations/sana/tests
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Make the SANA-WM runner choose the snapped rollout length from --num-frames before preparing camera conditioning. Action-derived trajectories now repeat or truncate to that snapped length, so a short action prompt no longer caps the generated video.

Fit explicit --camera-path trajectories to the same frame count instead of treating the file length as an upper bound. This matches the existing intrinsics fitting behavior and prevents a 321-frame pose file from truncating a longer requested rollout.

Document the updated behavior and add CPU coverage for action repetition, explicit camera-path fitting, and runner input preparation.

Tests:

- uv run --package flashdreams-sana-wm --extra dev pytest integrations/sana/tests

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@greptile-apps

greptile-apps Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds independently packaged SANA-WM bidirectional and streaming integrations.

  • Registers separate runner configurations, checkpoints, camera conditioning, diffusion, refinement, decoding, and output paths for both variants.
  • Adds bounded action-DSL trajectory generation, optional derived intrinsics, and streaming frame-count snapping.
  • Adds BF16/FP8/FP4 quantization support, parity and benchmark harnesses, CPU/CUDA tests, model documentation, sample clips, and performance data.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported oversized action-duration allocation is bounded by the requested frame count for all production call paths.

Important Files Changed

Filename Overview
integrations/sana/sana_wm/camera.py Implements camera action parsing, bounded trajectory generation, pose fitting, intrinsics handling, and Plücker conditioning; the previous unbounded-duration issue is fixed for all production callers.
integrations/sana/sana_wm/runner.py Defines separate bidirectional and streaming execution paths, input preparation, checkpoint loading, generation, and output emission.
integrations/sana/sana_wm/stage1_model.py Adds the native Stage-1 transformer implementation for bidirectional and chunk-causal SANA-WM inference.
integrations/sana/sana_wm/transformer.py Integrates SANA-WM conditioning, initial latent construction, denoising, and streaming cache behavior with FlashDreams transformer contracts.
integrations/sana/sana_wm/quant.py Adds integration-local FP8 and FP4 linear and quantization paths based on Torch scaled matrix multiplication and Triton helpers.
flashdreams/flashdreams/infra/diffusion/model/base.py Delegates initial latent creation to the transformer while preserving Gaussian noise as the default behavior.
flashdreams/flashdreams/infra/diffusion/transformer/base.py Adds an overridable initial-noise extension point for model-specific latent initialization.

Sequence Diagram

sequenceDiagram
  participant User
  participant Runner as SANA-WM Runner
  participant Camera as Camera Conditioning
  participant Stage1 as Stage-1 DiT
  participant Refiner as LTX-2 Refiner
  participant VAE as VAE Decoder
  participant Output as Video Output
  User->>Runner: Select bidirectional or streaming runner
  Runner->>Camera: Resolve pose file or bounded action DSL
  Camera-->>Runner: Fitted trajectory and intrinsics
  Runner->>Stage1: Generate conditioned latent sequence
  Stage1-->>Runner: Stage-1 latents
  Runner->>Refiner: Refine full sequence or causal prefix
  Refiner-->>Runner: Refined latents
  Runner->>VAE: Decode full clip or streaming chunks
  VAE-->>Runner: Video frames
  Runner->>Output: Write clip or progressively emit chunks
Loading

Reviews (18): Last reviewed commit: "Minor model card update" | Re-trigger Greptile

@aidanfnv

Copy link
Copy Markdown
Collaborator Author

/ok to test 1485061

@aidanfnv

Copy link
Copy Markdown
Collaborator Author

/ok to test 83e5ec4

@aidanfnv

Copy link
Copy Markdown
Collaborator Author

/ok to test 7d486ca

@aidanfnv

Copy link
Copy Markdown
Collaborator Author

/ok to test bc58ce3

@jarcherNV jarcherNV left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was pretty long so admittedly I only skimmed it, but I didn't notice any glaring issues so I think it is fine to merge.

@jarcherNV

Copy link
Copy Markdown
Collaborator

An additional comment: There are a lot of commits in this PR, so it may produce a pretty long commit message when merged. You may want to consider squashing it and creating a reasonable commit message for everything, but I'll leave it up to you to decide how to proceed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants