feat(sim): add BEHAVIOR-1K challenge support

.gitignore

@@ -78,6 +78,10 @@ tmp/
 example_videos/
 outputs/
 
+# Per-episode benchmark rollout MP4s (`openral benchmark run` records by
+# default next to the eval JSON; the JSON is checked in, the videos are not).
+rskills/*/eval/videos/
+
 # Optional project-local LIBERO config dir — useful when overriding
 # LIBERO_CONFIG_PATH to bypass ~/.libero (see tools/fix_libero_config.py
 # + docs/reference/vla_compatibility.md). The auto-fix targets ~/.libero

CLAUDE.md

@@ -64,7 +64,7 @@ Adding, removing, renaming, or moving a responsibility between layers requires r
 
 **rSkill packaging.** One HF Hub repo per skill: `rskill.yaml` (name, version, license, embodiment_tags, capabilities_required, runtime, quantization, latency budgets, fallback_skill_id; plus `state_contract.dim` + `action_contract.dim` for dataset bridge users), weights (`model.safetensors`), optional `engine.plan`/`Dockerfile`, `README.md` per [`rskills/template/README.md`](rskills/template/README.md) (publish gate enforced by `rskill_publisher` validator), `eval/<benchmark>.json` validating against `openral_core.SkillEvalResult`. Canonical eval producer: `openral benchmark run --suite <id> --vla <vla_id>:rskills/<this_skill>`. Paper-cited numbers allowed with `reproduced_locally: false` + `reproduction_cli`. **Provenance:** sigstore signing/verification is the planned control but is **not yet implemented** (no manifest `signature` field, no verification in the loader). Until it lands, `rSkill.from_pretrained`/`from_yaml` emit an `rskill.unverified_provenance` warning and honor `OPENRAL_REQUIRE_SIGNED_SKILLS=1` to fail closed; `*.pt` weights are treated as untrusted code and require `OPENRAL_ALLOW_UNSAFE_PICKLE=1` to load (prefer `model.safetensors`); `trust_remote_code` models (e.g. MolmoAct2) execute repo-shipped code and require `OPENRAL_ALLOW_REMOTE_CODE=1`. Do not describe skills as "signed/verified" until the control exists (§1.2).
 
-**VLA license matrix.** SmolVLA, OpenVLA/OFT, Octo, ACT, DP/DP3, UnifoLM-VLA-0/WMA-0, LingBot-VLA 1/2 — Apache-2.0 / MIT (free; LingBot-VLA is Apache-2.0 code **and** weights, but runs out-of-process via an auto-provisioned Py3.12 + torch-2.8 ZMQ sidecar because `lingbotvla` pins torch==2.8.0 + custom kernels — **v2** 6B Qwen3-VL MoE on `transformers==4.57.3`; **v1** 4B Qwen2.5-VL dense expert on `transformers==4.51.3` / `lerobot==0.4.2`, the RoboTwin post-train, driven by the same sidecar via `--variant v1`). π0 / π0.5 / π0.6 / π0.7 — code Apache-2.0; weights permissive research (flag in manifest). GR00T N1 / N1.5 / N1.6 — NVIDIA OneWay **Noncommercial** (install-time guard); GR00T **N1.7+** — NVIDIA Open Model License, **commercial OK** (`nvidia_open_model` posture); N2 announced, unreleased. GR00T N1.7 runs **in-process** under the workspace's Python 3.12 via lerobot 0.6.0's native `GrootPolicy` (backbone-only NF4); only RLDX-1 (a GR00T-N1.5 finetune) still runs out-of-process via a ZMQ sidecar (Py3.10) through the `rldx` adapter. InternVLA-N1 / DualVLN (InternRobotics, arXiv:2512.08186) — the **navigation** family: code MIT, weights **CC-BY-NC-SA-4.0 noncommercial** (`cc-by-nc-sa-4.0` posture, install-time guard). Dual-system (Qwen2.5-VL-7B S2 planner + NavDP DiT S1); runs out-of-process via a Py3.11 ZMQ sidecar (upstream pins transformers 4.51) through the `internvla_n1` adapter; consumes RGB-D + instruction, emits `BODY_TWIST`. Helix / Gemini Robotics / Skild Brain — closed, API-only. Quantize for target; action chunks not single actions; embodiment tags must match `RobotCapabilities.embodiment_tags`; latency budget is contractual.
+**VLA license matrix.** SmolVLA, OpenVLA/OFT, Octo, ACT, DP/DP3, UnifoLM-VLA-0/WMA-0, LingBot-VLA 1/2 — Apache-2.0 / MIT (free; LingBot-VLA is Apache-2.0 code **and** weights, but runs out-of-process via an auto-provisioned Py3.12 + torch-2.8 ZMQ sidecar because `lingbotvla` pins torch==2.8.0 + custom kernels — **v2** 6B Qwen3-VL MoE on `transformers==4.57.3`; **v1** 4B Qwen2.5-VL dense expert on `transformers==4.51.3` / `lerobot==0.4.2`, the RoboTwin post-train, driven by the same sidecar via `--variant v1`). π0 / π0.5 / π0.6 / π0.7 — code Apache-2.0; weights permissive research (flag in manifest). GR00T N1 / N1.5 / N1.6 — NVIDIA OneWay **Noncommercial** (install-time guard); GR00T **N1.7+** — NVIDIA Open Model License, **commercial OK** (`nvidia_open_model` posture); N2 announced, unreleased. Standard GR00T N1.7 checkpoints run **in-process** under the workspace's Python 3.12 via lerobot 0.6.0's native `GrootPolicy` (backbone-only NF4). The official 2026 BEHAVIOR-1K organizer checkpoint is the exception: its pinned `wensi-ai/Isaac-GR00T` runtime stays in an external Python 3.10 sidecar, and its Drive artifact remains `license: unknown` until the organizers publish checkpoint-specific terms. Its `deploy sim` path preserves the official 61-D state and commits the six individually safety-approved action slots atomically as one 23-D OmniGibson step. RLDX-1 (a GR00T-N1.5 finetune) also runs out-of-process via its own ZMQ sidecar through the `rldx` adapter. InternVLA-N1 / DualVLN (InternRobotics, arXiv:2512.08186) — the **navigation** family: code MIT, weights **CC-BY-NC-SA-4.0 noncommercial** (`cc-by-nc-sa-4.0` posture, install-time guard). Dual-system (Qwen2.5-VL-7B S2 planner + NavDP DiT S1); runs out-of-process via a Py3.11 ZMQ sidecar (upstream pins transformers 4.51) through the `internvla_n1` adapter; consumes RGB-D + instruction, emits `BODY_TWIST`. Helix / Gemini Robotics / Skild Brain — closed, API-only. Quantize for target; action chunks not single actions; embodiment tags must match `RobotCapabilities.embodiment_tags`; latency budget is contractual.
 
 **Safety.** Touching `packages/openral_safety/` or `cpp/openral_safety_kernel/` requires (a) safety-WG reviewer, (b) an update to the safety hazard log (private `OpenRAL/management` repo), (c) tests proving the new behavior is at least as conservative. **Never** add a flag that disables safety. **Never** add a debug mode that bypasses E-stop. **Never** add a path where a Python crash leaves motors energized. Python proposes; C++ disposes.
 

README.md

@@ -80,9 +80,9 @@ Live status: [docs/roadmap/index.md](docs/roadmap/index.md). Per-module canvas:
 | rSkill (S1) runtime | `Skill` ABC, `rSkill` loader (HF Hub), PyTorch / ONNX adapters (engine cache), async action chunks; Pro runtimes attach through entry-point hooks | `python/rskill/`, `rskills/` |
 | Inference runtimes | One `InferenceRunner` Protocol shared by `openral sim run`, `openral benchmark run`, and `openral deploy`; open-core runners are PyTorch / ONNX | `python/runner/`, `python/rskill/`, `python/sim/` |
 | Sim rollouts | One YAML → reproducible sim rollout; video + metrics + `SkillEvalResult` JSON out | `python/sim/`, `scenes/benchmark/` |
-| Simulation engines | MuJoCo (LIBERO, MetaWorld, ManiSkill3, SimplerEnv, gym-aloha, gym-pusht), RoboCasa, RoboTwin 2.0 (SAPIEN), Isaac Sim, RLBench/CoppeliaSim (PyRep, py3.10 sidecar) | `python/sim/`, `docs/reference/sim-environments.md` |
+| Simulation engines | MuJoCo (LIBERO, MetaWorld, ManiSkill3, SimplerEnv, gym-aloha, gym-pusht), RoboCasa, RoboTwin 2.0 (SAPIEN), Isaac Sim, BEHAVIOR-1K/OmniGibson, RLBench/CoppeliaSim | `python/sim/`, `docs/reference/sim-environments.md` |
 | Observability | OpenTelemetry SDK + OTLP exporter, span helpers, structlog bridge, live `openral dashboard`, read-only Foxglove live-scene surface | `python/observability/` |
-| CLI (`openral`) | `doctor`, `detect`, `connect`, `calibrate`, `check`, `install`, `rskill`, `sensor`, `sim`, `benchmark`, `deploy`, `dashboard`, `prompt`, `record`, `replay`, `dataset`, `collision`, `robot`, `profile`. Bare `openral` → interactive REPL. | `python/cli/` |
+| CLI (`openral`) | `doctor`, `detect`, `connect`, `calibrate`, `check`, `install`, `rskill`, `sensor`, `sim`, `behavior`, `benchmark`, `deploy`, `dashboard`, `prompt`, `record`, `replay`, `dataset`, `collision`, `robot`, `profile`. Bare `openral` → interactive REPL. | `python/cli/` |
 | Schemas | Pydantic v2 + JSON Schema export; manifests at `schema_version: "0.1"` | `python/core/`, `tools/schema_export.py` |
 | ROS 2 IDL | `openral_msgs` (.msg, .action) — normative across the runtime | `packages/msgs/` |
 
@@ -284,6 +284,11 @@ just sim-libero                              # SmolVLA × LIBERO
 just sim-pi05-libero                         # π0.5 × LIBERO (≥8 GB VRAM)
 just sim-act-aloha                           # ACT × gym-aloha bimanual
 
+# BEHAVIOR Challenge — OmniGibson runs in its official conda environment
+uv run openral behavior serve \
+  --rskill rskills/gr00t-n17-b1k-turning-on-radio \
+  --task turning_on_radio
+
 # Observability
 uv run openral dashboard                     # OTLP receiver at :4318
 
@@ -305,11 +310,11 @@ Full toolchain: [docs/contributing/toolchain.md](docs/contributing/toolchain.md)
 
 ## Robot descriptions
 
-17 robot platforms are supported, from low-cost manipulators to mobile manipulators, bimanual arms and humanoids. Each is a typed `RobotDescription` manifest under `robots/<robot_id>/robot.yaml`.
+18 robot platforms are supported, from low-cost manipulators to mobile manipulators, bimanual arms and humanoids. Each is a typed `RobotDescription` manifest under `robots/<robot_id>/robot.yaml`.
 
 → **Full table:** [docs/reference/robots.md](docs/reference/robots.md)
 
-Quick examples: SO-100/SO-101 (HW + sim), Franka Panda, UR5e/UR10e, ALOHA bimanual/AgileX, OpenArm v2, Anvil OpenARM 2.0, Google Robot, Rizon4, Unitree H1/G1, Rethink Sawyer, Fourier GR1.
+Quick examples: SO-100/SO-101 (HW + sim), Franka Panda, UR5e/UR10e, ALOHA bimanual/AgileX, OpenArm v2, Anvil OpenARM 2.0, Galaxea R1 Pro (BEHAVIOR sim), Google Robot, Rizon4, Unitree H1/G1, Rethink Sawyer, Fourier GR1.
 
 ---
 
@@ -346,7 +351,7 @@ rSkills come in several **kinds**, all installed and run the same way:
 - **`kind: ros_action`** — classical-control skills wrapping MoveIt (`rskill-moveit-multi-joints-none` / `-eef-pose` / `-look-at`) and Nav2 (`rskill-nav2-mobile_base-navigate_to_pose-none`).
 - **`kind: playbook`** — human-authored Markdown SOPs the S2 reasoner reads as content (decompose-mission, verify-outcome, clarify-ambiguity, preflight-reach, stage-for-manipulation, find-object); no weights, no actuation.
 
-Most are published under `OpenRAL/rskill-*` on HuggingFace Hub. LocateAnything is private and non-commercial; the GR00T N1.7 policy (`gr00t-n17-libero`, NVIDIA Open Model License) runs in-process through lerobot 0.6.0's `GrootPolicy`. The OpenVLA-OFT policy (`openvla-oft-simpler-widowx-nf4`, MIT) is an in-process transformers custom-code NF4 model validated on the SimplerEnv WidowX carrot-on-plate task.
+Most are published under `OpenRAL/rskill-*` on HuggingFace Hub. LocateAnything is private and non-commercial; the GR00T N1.7 LIBERO/SO-101 policies run in-process through lerobot 0.6.0's `GrootPolicy`, while the official BEHAVIOR-1K R1Pro checkpoint uses its pinned Isaac-GR00T Python 3.10 sidecar. The OpenVLA-OFT policy (`openvla-oft-simpler-widowx-nf4`, MIT) is an in-process transformers custom-code NF4 model validated on the SimplerEnv WidowX carrot-on-plate task.
 
 → **Full table + license notes:** [docs/reference/rskills.md](docs/reference/rskills.md)
 

benchmarks/README.md

@@ -59,6 +59,7 @@ for the pattern.
 | `gr1_tabletop.yaml`                | gr1            | robocasa/gr1/PnPCupToDrawerClose (MuJoCo via robosuite + GR1 fork)       |  1 | 10 | `is_success` | 720 |  10 |
 | `robotwin.yaml`                    | aloha_agilex   | robotwin (SAPIEN via lerobot, py3.10 sidecar)                           |  5 | 100 | `is_success` | 300 | 500 |
 | `rlbench.yaml`                     | franka_panda   | RLBench PerAct subset: open_drawer / meat_off_grill / close_jar (CoppeliaSim/PyRep, py3.10 sidecar) |  3 | 25 | `is_success` | 25 |  75 |
+| `behavior.yaml`                    | r1pro          | BEHAVIOR-1K 2026 challenge, all 100 public tasks (OmniGibson / Isaac Sim, sidecar) | 100 |  1 | `is_success` | 3224–39090 | 100 |
 
 Per-suite `max_steps` mirrors the upstream `lerobot.envs.libero.TASK_SUITE_MAX_STEPS`
 table for the LIBERO suites and the ACT / Diffusion Policy paper protocols for
@@ -102,6 +103,24 @@ typed `ROSConfigError` with the full provisioning recipe when the sidecar venv /
 `COPPELIASIM_ROOT` are absent. Verified live on an 8 GB Ada GPU host
 (open_drawer 4/4, meat_off_grill 3/3, close_jar solved).
 
+`behavior.yaml` carries **all 100 public tasks of the BEHAVIOR-1K 2026
+challenge** on **OmniGibson / Isaac Sim** (BEHAVIOR-1K defines 1,000
+activities; the challenge evaluator exposes this 100-task subset). The
+official environment lives in its own venv (never vendored) and the scene
+runs out-of-process through `tools/behavior_scene_sidecar.py`; the organizer
+GR00T N1.7 baseline policy (`rskills/gr00t-n17-b1k-turning-on-radio`) runs
+in a second sidecar (`tools/behavior_groot_sidecar.py`, whole-model NF4 to
+co-reside with the simulator on 8 GB). The openral side needs
+`just sync --group behavior-groot`. Like `maniskill3_panda`, `--rskill`
+auto-filters the suite to a policy's `evaluated_tasks` — the only released
+organizer checkpoint covers `turning_on_radio`, so today's runs execute that
+single task; the suite stays runnable as multi-task BEHAVIOR policies land.
+Per-task `max_steps` is the official cap (`int(1.5 x mean human-demo
+length)`) read from the installed challenge stats. One instance, one episode
+per task: closed-loop protocol at ~1 step/s makes an episode ~1–2 h
+wall-clock; the official leaderboard metric is q_score partial credit
+averaged over all public instances, which this suite does not reproduce.
+
 ## Adding a new benchmark
 
 1. Pick the published protocol you want to reproduce. Don't invent protocols —