Skip to content

feat(pi05): complete the native C++ runtime interface#144

Open
LiangSu8899 wants to merge 66 commits into
mainfrom
feat/pi05-cpp-runtime-interface
Open

feat(pi05): complete the native C++ runtime interface#144
LiangSu8899 wants to merge 66 commits into
mainfrom
feat/pi05-cpp-runtime-interface

Conversation

@LiangSu8899

@LiangSu8899 LiangSu8899 commented Jul 11, 2026

Copy link
Copy Markdown
Member

Summary

  • add a Python-free Pi0.5 producer through frt_model_runtime_open_v1
  • support native SM120 BF16 and Thor SM110 FP8 E4M3 backends behind explicit build gates
  • add model-specific SM110 calibration sessions for one observation or repeated dataset observations, with one to three named camera views
  • implement native checkpoint/tokenizer loading, prompt and state staging, image preprocessing, graph capture, replay, and logical action output
  • optimize safetensors loading with mmap-backed typed views, fused final-dtype transforms, bounded parallel materialization, and concurrent full-file hashing
  • preserve the backend-neutral runtime/model-runtime ABI and keep all Pi0.5 dimensions, tensor names, camera names, prompt semantics, and calibration sites model-local

Dependency and merge order

This PR depends on #145 for the shared Python-free FA2 runtime used by the SM120 backend. Merge #145 first, then rebase this branch before merging so the shared kernel change lands once through its owner review.

Design boundaries

  • runtime/ and exec/ remain mechanism-only and contain no model, backend, dataset, or calibration policy
  • Python and native producers publish the same frt_model_runtime_v1 face; consumers do not branch on model or backend kind
  • SM110 calibration is a model-specific setup API under cpp/models/pi05/; dataset traversal, decoding, sampling, and camera synchronization remain host policy
  • intermediate prompt, attention, RoPE, KV, and diffusion storage remains context-owned buffers; only complete declared capsule regions are restorable
  • no backend registry or speculative generic model framework was added

SM110 calibration contract

frt_pi05_calibration_create_v1, observe_v1, and finalize_v1 accept complete observations and publish one atomic safetensors artifact. Calls may be repeated for dataset calibration. Camera arrays are canonicalized by the Pi0.5 names image, wrist_image, and wrist_image_right; missing, duplicate, unknown, or non-RGB inputs reject the observation without changing the successful sample count.

Artifacts bind schema/reducer version, observed hardware, checkpoint and tokenizer SHA-256, precision, fixed model shapes, successful sample count, and percentile. Runtime open requires a compatible artifact on SM110, verifies the checkpoint digest again, and includes the artifact SHA-256 in producer identity. Different calibration bytes therefore produce a different fingerprint and cannot silently restore an old capsule.

Loading and execution

The native loader accepts F32, BF16, and F16 safetensors sources, including valid unaligned payload offsets. It emits final BF16 or F16/FP8 layouts directly without a checkpoint-sized float dictionary or a second implicit cache format. Full checkpoint hashing runs concurrently with independent materialization, and the runtime is published only after both succeed.

precision="auto" resolves to BF16 on SM120 and FP8 E4M3 on SM110. Explicit hardware/precision mismatches fail setup. Native C++ NVFP4 is not advertised by this producer; existing Python NVFP4 paths are unchanged.

The hot model face declares real STAGED verbs for prompt, state, images, and logical actions, with SWAP windows for noise and raw actions. Capture produces one fixed graph variant. The measured service-loop range includes input staging, replay, and output readback.

Compatibility

  • no changes under frozen runtime/include or exec/include
  • SM110 support is opt-in through FLASHRT_CPP_WITH_THOR_FP8; existing defaults remain unchanged
  • SM120 publishes BF16 tensor windows; SM110 publishes F16 windows and identifies FP8 E4M3 math explicitly
  • the new calibration C API is model-specific and does not add fields or verbs to the generic deployment ABI
  • the strict native model face accepts RGB8; legacy Pi0.5 image entry points retain their existing format conversion behavior
  • port, stage, binding, region, hardware, precision, checkpoint, tokenizer, or calibration changes intentionally change deployment identity

Validation

  • CUDA-off Debug CTest: 12/12
  • CUDA-off ASAN/UBSAN CTest: 12/12
  • changed native libraries and focused tests compile with strict warnings as errors
  • SM120 CTest regression: 26/26
  • SM110 Release CTest: 26/26
  • SM110 Debug CTest: 26/26
  • SM110 configure rejects missing CUDA staging or exec dependencies
  • one-view, one-observation native calibration and full runtime lifecycle
  • two-view, three-observation dataset calibration with reversed caller camera order
  • missing, duplicate, unknown, non-RGB, malformed-noise, and incomplete samples reject atomically
  • deterministic generated-noise path exercised independently
  • all 72 encoder scales, all 720 ten-step decoder scales, and all 320 raw action values are bit-exact against the shipped Torch producer
  • native runtime schema, one graph variant, finite logical actions, identity, and teardown verified
  • 1,000-loop profiler range contains exactly 1,000 graph launches and no CUDA device/host allocation, mempool, VMM, graph instantiation, or capture API
  • 1,000 prompt/state updates keep one graph variant; p99 is 266 microseconds against a 1 ms gate
  • representative SM110 setup: 7.90 s; complete open/infer/output/teardown lifecycle: 8.62 s
  • frozen ABI diff, generic-layer model leakage scan, public-document privacy scan, git diff --check, and static analysis pass

The public build, calibration, artifact, runtime-port, identity, and validation procedures are documented in docs/pi05_thor_native_fp8.md and linked from USAGE.md.

@LiangSu8899

Copy link
Copy Markdown
Member Author

Shared FA2 ownership and packaging has been split into prerequisite PR #145. The model runtime continues to consume the same fvk_* C entries and does not carry a private kernel implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Track Jetson-PI multi-backend provider integration

1 participant