[Tracking] Verified structures — composing HF Kernels into net-win building blocks

[LiangSu8899]

Following up on the discussion with the HF Kernels team — this issue tracks the verified-structures integration between HF Kernels and FlashRT. cc @sayakpaul

Why

Individual fused/quantized kernels are often not a net win when swapped in
alone: quantization pays conversion at every boundary, and fusion changes
operator boundaries so it doesn't map to a single module swap. The
performance lives in composition. Our benchmarks: composed structurally,
the same Hub kernels reach 6–9x over the torch reference — gains that
isolated per-op swaps largely miss.

kernels already solves the atomic problems (build, versioning,
multi-hardware distribution). What's missing is the layer above: a way to
adopt composed, calibrated blocks — incrementally, with proof.

The idea

Distribute verified structures instead of raw kernels or whole runtimes.

A structure is a small spec of one model sub-block (e.g. a decoder FFN):

• boundary: the tensors in/out (names, shapes, dtypes);
• weight slots: which parameters it needs (slot names, framework-neutral);
• reference: ~20 lines of plain torch — the ground truth;
• gates: pass criteria — numerical parity vs the reference on real data,
  and a net speed win vs the compiled baseline, boundary costs included.

It is not code and not an IR. Existing pipelines are never rewritten:
a small mapping (adapter) tells the checker where the same computation lives in
each implementation — a transformers module path, a native pipeline
segment, or a Hub-kernel composition. The checker feeds the boundary,
compares against the reference, and reports PASS/FAIL. Implementations
that don't pass don't activate.

What we're after

1. Incremental adoption: plug one structure into an existing pipeline,
   verified as a net win on your data, or it refuses to activate.
2. Reuse across models and hardware: decoder_ffn is the same
   definition for Pi0.5 / GROOT / Qwen-class models; implementations vary
   per architecture (different quant scheme per GPU generation is normal),
   gates stay identical.
3. Agent-friendly adaptation: adapting a new model becomes
   lookup + diff + gate instead of re-derivation — each structure carries
   its worked examples, calibration recipe and pre-tuned configs, so an
   agentic workflow can assemble and validate a pipeline quickly.

Already demonstrated (ecosystem channel)

The integration pattern was first proven by hand through LeRobot — as
hot-pluggable structural embeddings of Hub kernels — inference in
huggingface/lerobot#3870 and training in huggingface/lerobot#3974 (both merged):

Platform	upstream	+ structures	speedup	cosine
RTX 4090	96.84 ms	53.79 ms	1.80x	0.99952
RTX 5090	89.40 ms	46.70 ms	1.91x	0.99976
RTX PRO 6000	57.62 ms	39.27 ms	1.47x	0.99993
Thor (aarch64)	131.19 ms	98.95 ms	1.33x	0.99992
Spark (aarch64)	151.10 ms	106.74 ms	1.42x	0.99982
AMD MI300	66.90 ms	42.15 ms	1.59x	0.9999988

One structure set (sync-free denoise loop + FP8 MLP with calibration),
1.33–1.91x across six platforms — NVIDIA desktop / workstation /
edge aarch64 and AMD ROCm — no forks, delivered through the official
pipeline. This is the existence proof for "composed structures travel
across hardware while per-op swaps don't".

Standalone composed-runtime demos built only from public Hub packages
(FlashRT-HF-kernels/demos)
show the same principle end-to-end: up to 11x over the torch
reference on model-shaped hot paths.

This tracking issue (FlashRT structural layer — planned)

Formalize what the LeRobot PRs did by hand into the mechanism above:

• First target — Pi0.5, full pipeline: FlashRT kernels + the
  structure abstraction → a single multi-platform pipeline (one set
  of structure definitions) replacing the three pipelines we maintain
  today (RTX / Thor / ROCm); low development cost, agent-workflow-
  friendly, cross-hardware (NVIDIA, AMD) by relying on kernels'
  distribution. Starting from the structures already proven in LeRobot
  (FP8 MLP; fused cross-entropy on the training side).
• The same structure definitions carry into the full FlashRT machinery:
  model-runtime contract stages (captured CUDA graphs, cut at real
  scheduling seams), scheduling, and Nexus serving.
• The shared structures then get plug-in embedding tests in other
  same-family model pipelines (GROOT / Qwen-class), reusing the common
  definitions unchanged.

Acceptance

• Parity: cosine vs reference ≥ floor, on real-distribution inputs.
• Performance: net win vs the compiled baseline, including boundary
  conversion costs, reported separately per M-class (small-M blocks must
  win on small M, not just on large shapes).
• Serving path: the composed pipeline passes the existing runtime
  adoption/tick validation.
• All numbers posted in this thread, with pinned kernel revisions and environments.

Relationship to existing work

• The llama.cpp/GGML provider work is ongoing (Track Jetson-PI multi-backend provider integration #143/feat: integrate Jetson-PI (public repo) as Pi0/LLM/MLLM provider #148) and unchanged; both provider
  families sit behind the same runtime contract.
• transformers/diffusers official APIs (quantization_config,
  kernelize, attn_implementation) remain the preferred delivery
  channel for whatever they can express; structures fill the multi-op /
  calibrated / boundary-sensitive gap, and ship pre-verified evidence.
• With the kernels team: reviews of the consumption path (locking,
  variants) welcome as offered; trustedKernelPublisher for the
  flashrt org; pre-tuned configs per structure are complementary to the
  JIT-autotune roadmap.