diff --git a/.gitignore b/.gitignore index 74407d7..dc84676 100644 --- a/.gitignore +++ b/.gitignore @@ -78,6 +78,10 @@ tmp/ example_videos/ outputs/ +# Per-episode benchmark rollout MP4s (`openral benchmark run` records by +# default next to the eval JSON; the JSON is checked in, the videos are not). +rskills/*/eval/videos/ + # Optional project-local LIBERO config dir — useful when overriding # LIBERO_CONFIG_PATH to bypass ~/.libero (see tools/fix_libero_config.py # + docs/reference/vla_compatibility.md). The auto-fix targets ~/.libero diff --git a/CLAUDE.md b/CLAUDE.md index d7017f2..affcb81 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -64,7 +64,7 @@ Adding, removing, renaming, or moving a responsibility between layers requires r **rSkill packaging.** One HF Hub repo per skill: `rskill.yaml` (name, version, license, embodiment_tags, capabilities_required, runtime, quantization, latency budgets, fallback_skill_id; plus `state_contract.dim` + `action_contract.dim` for dataset bridge users), weights (`model.safetensors`), optional `engine.plan`/`Dockerfile`, `README.md` per [`rskills/template/README.md`](rskills/template/README.md) (publish gate enforced by `rskill_publisher` validator), `eval/.json` validating against `openral_core.SkillEvalResult`. Canonical eval producer: `openral benchmark run --suite --vla :rskills/`. Paper-cited numbers allowed with `reproduced_locally: false` + `reproduction_cli`. **Provenance:** sigstore signing/verification is the planned control but is **not yet implemented** (no manifest `signature` field, no verification in the loader). Until it lands, `rSkill.from_pretrained`/`from_yaml` emit an `rskill.unverified_provenance` warning and honor `OPENRAL_REQUIRE_SIGNED_SKILLS=1` to fail closed; `*.pt` weights are treated as untrusted code and require `OPENRAL_ALLOW_UNSAFE_PICKLE=1` to load (prefer `model.safetensors`); `trust_remote_code` models (e.g. MolmoAct2) execute repo-shipped code and require `OPENRAL_ALLOW_REMOTE_CODE=1`. Do not describe skills as "signed/verified" until the control exists (§1.2). -**VLA license matrix.** SmolVLA, OpenVLA/OFT, Octo, ACT, DP/DP3, UnifoLM-VLA-0/WMA-0, LingBot-VLA 1/2 — Apache-2.0 / MIT (free; LingBot-VLA is Apache-2.0 code **and** weights, but runs out-of-process via an auto-provisioned Py3.12 + torch-2.8 ZMQ sidecar because `lingbotvla` pins torch==2.8.0 + custom kernels — **v2** 6B Qwen3-VL MoE on `transformers==4.57.3`; **v1** 4B Qwen2.5-VL dense expert on `transformers==4.51.3` / `lerobot==0.4.2`, the RoboTwin post-train, driven by the same sidecar via `--variant v1`). π0 / π0.5 / π0.6 / π0.7 — code Apache-2.0; weights permissive research (flag in manifest). GR00T N1 / N1.5 / N1.6 — NVIDIA OneWay **Noncommercial** (install-time guard); GR00T **N1.7+** — NVIDIA Open Model License, **commercial OK** (`nvidia_open_model` posture); N2 announced, unreleased. GR00T N1.7 runs **in-process** under the workspace's Python 3.12 via lerobot 0.6.0's native `GrootPolicy` (backbone-only NF4); only RLDX-1 (a GR00T-N1.5 finetune) still runs out-of-process via a ZMQ sidecar (Py3.10) through the `rldx` adapter. InternVLA-N1 / DualVLN (InternRobotics, arXiv:2512.08186) — the **navigation** family: code MIT, weights **CC-BY-NC-SA-4.0 noncommercial** (`cc-by-nc-sa-4.0` posture, install-time guard). Dual-system (Qwen2.5-VL-7B S2 planner + NavDP DiT S1); runs out-of-process via a Py3.11 ZMQ sidecar (upstream pins transformers 4.51) through the `internvla_n1` adapter; consumes RGB-D + instruction, emits `BODY_TWIST`. Helix / Gemini Robotics / Skild Brain — closed, API-only. Quantize for target; action chunks not single actions; embodiment tags must match `RobotCapabilities.embodiment_tags`; latency budget is contractual. +**VLA license matrix.** SmolVLA, OpenVLA/OFT, Octo, ACT, DP/DP3, UnifoLM-VLA-0/WMA-0, LingBot-VLA 1/2 — Apache-2.0 / MIT (free; LingBot-VLA is Apache-2.0 code **and** weights, but runs out-of-process via an auto-provisioned Py3.12 + torch-2.8 ZMQ sidecar because `lingbotvla` pins torch==2.8.0 + custom kernels — **v2** 6B Qwen3-VL MoE on `transformers==4.57.3`; **v1** 4B Qwen2.5-VL dense expert on `transformers==4.51.3` / `lerobot==0.4.2`, the RoboTwin post-train, driven by the same sidecar via `--variant v1`). π0 / π0.5 / π0.6 / π0.7 — code Apache-2.0; weights permissive research (flag in manifest). GR00T N1 / N1.5 / N1.6 — NVIDIA OneWay **Noncommercial** (install-time guard); GR00T **N1.7+** — NVIDIA Open Model License, **commercial OK** (`nvidia_open_model` posture); N2 announced, unreleased. Standard GR00T N1.7 checkpoints run **in-process** under the workspace's Python 3.12 via lerobot 0.6.0's native `GrootPolicy` (backbone-only NF4). The official 2026 BEHAVIOR-1K organizer checkpoint is the exception: its pinned `wensi-ai/Isaac-GR00T` runtime stays in an external Python 3.10 sidecar, and its Drive artifact remains `license: unknown` until the organizers publish checkpoint-specific terms. Its `deploy sim` path preserves the official 61-D state and commits the six individually safety-approved action slots atomically as one 23-D OmniGibson step. RLDX-1 (a GR00T-N1.5 finetune) also runs out-of-process via its own ZMQ sidecar through the `rldx` adapter. InternVLA-N1 / DualVLN (InternRobotics, arXiv:2512.08186) — the **navigation** family: code MIT, weights **CC-BY-NC-SA-4.0 noncommercial** (`cc-by-nc-sa-4.0` posture, install-time guard). Dual-system (Qwen2.5-VL-7B S2 planner + NavDP DiT S1); runs out-of-process via a Py3.11 ZMQ sidecar (upstream pins transformers 4.51) through the `internvla_n1` adapter; consumes RGB-D + instruction, emits `BODY_TWIST`. Helix / Gemini Robotics / Skild Brain — closed, API-only. Quantize for target; action chunks not single actions; embodiment tags must match `RobotCapabilities.embodiment_tags`; latency budget is contractual. **Safety.** Touching `packages/openral_safety/` or `cpp/openral_safety_kernel/` requires (a) safety-WG reviewer, (b) an update to the safety hazard log (private `OpenRAL/management` repo), (c) tests proving the new behavior is at least as conservative. **Never** add a flag that disables safety. **Never** add a debug mode that bypasses E-stop. **Never** add a path where a Python crash leaves motors energized. Python proposes; C++ disposes. diff --git a/README.md b/README.md index 8c2a575..5644b3d 100644 --- a/README.md +++ b/README.md @@ -80,9 +80,9 @@ Live status: [docs/roadmap/index.md](docs/roadmap/index.md). Per-module canvas: | rSkill (S1) runtime | `Skill` ABC, `rSkill` loader (HF Hub), PyTorch / ONNX adapters (engine cache), async action chunks; Pro runtimes attach through entry-point hooks | `python/rskill/`, `rskills/` | | Inference runtimes | One `InferenceRunner` Protocol shared by `openral sim run`, `openral benchmark run`, and `openral deploy`; open-core runners are PyTorch / ONNX | `python/runner/`, `python/rskill/`, `python/sim/` | | Sim rollouts | One YAML → reproducible sim rollout; video + metrics + `SkillEvalResult` JSON out | `python/sim/`, `scenes/benchmark/` | -| Simulation engines | MuJoCo (LIBERO, MetaWorld, ManiSkill3, SimplerEnv, gym-aloha, gym-pusht), RoboCasa, RoboTwin 2.0 (SAPIEN), Isaac Sim, RLBench/CoppeliaSim (PyRep, py3.10 sidecar) | `python/sim/`, `docs/reference/sim-environments.md` | +| Simulation engines | MuJoCo (LIBERO, MetaWorld, ManiSkill3, SimplerEnv, gym-aloha, gym-pusht), RoboCasa, RoboTwin 2.0 (SAPIEN), Isaac Sim, BEHAVIOR-1K/OmniGibson, RLBench/CoppeliaSim | `python/sim/`, `docs/reference/sim-environments.md` | | Observability | OpenTelemetry SDK + OTLP exporter, span helpers, structlog bridge, live `openral dashboard`, read-only Foxglove live-scene surface | `python/observability/` | -| CLI (`openral`) | `doctor`, `detect`, `connect`, `calibrate`, `check`, `install`, `rskill`, `sensor`, `sim`, `benchmark`, `deploy`, `dashboard`, `prompt`, `record`, `replay`, `dataset`, `collision`, `robot`, `profile`. Bare `openral` → interactive REPL. | `python/cli/` | +| CLI (`openral`) | `doctor`, `detect`, `connect`, `calibrate`, `check`, `install`, `rskill`, `sensor`, `sim`, `behavior`, `benchmark`, `deploy`, `dashboard`, `prompt`, `record`, `replay`, `dataset`, `collision`, `robot`, `profile`. Bare `openral` → interactive REPL. | `python/cli/` | | Schemas | Pydantic v2 + JSON Schema export; manifests at `schema_version: "0.1"` | `python/core/`, `tools/schema_export.py` | | ROS 2 IDL | `openral_msgs` (.msg, .action) — normative across the runtime | `packages/msgs/` | @@ -284,6 +284,11 @@ just sim-libero # SmolVLA × LIBERO just sim-pi05-libero # π0.5 × LIBERO (≥8 GB VRAM) just sim-act-aloha # ACT × gym-aloha bimanual +# BEHAVIOR Challenge — OmniGibson runs in its official conda environment +uv run openral behavior serve \ + --rskill rskills/gr00t-n17-b1k-turning-on-radio \ + --task turning_on_radio + # Observability uv run openral dashboard # OTLP receiver at :4318 @@ -305,11 +310,11 @@ Full toolchain: [docs/contributing/toolchain.md](docs/contributing/toolchain.md) ## Robot descriptions -17 robot platforms are supported, from low-cost manipulators to mobile manipulators, bimanual arms and humanoids. Each is a typed `RobotDescription` manifest under `robots//robot.yaml`. +18 robot platforms are supported, from low-cost manipulators to mobile manipulators, bimanual arms and humanoids. Each is a typed `RobotDescription` manifest under `robots//robot.yaml`. → **Full table:** [docs/reference/robots.md](docs/reference/robots.md) -Quick examples: SO-100/SO-101 (HW + sim), Franka Panda, UR5e/UR10e, ALOHA bimanual/AgileX, OpenArm v2, Anvil OpenARM 2.0, Google Robot, Rizon4, Unitree H1/G1, Rethink Sawyer, Fourier GR1. +Quick examples: SO-100/SO-101 (HW + sim), Franka Panda, UR5e/UR10e, ALOHA bimanual/AgileX, OpenArm v2, Anvil OpenARM 2.0, Galaxea R1 Pro (BEHAVIOR sim), Google Robot, Rizon4, Unitree H1/G1, Rethink Sawyer, Fourier GR1. --- @@ -346,7 +351,7 @@ rSkills come in several **kinds**, all installed and run the same way: - **`kind: ros_action`** — classical-control skills wrapping MoveIt (`rskill-moveit-multi-joints-none` / `-eef-pose` / `-look-at`) and Nav2 (`rskill-nav2-mobile_base-navigate_to_pose-none`). - **`kind: playbook`** — human-authored Markdown SOPs the S2 reasoner reads as content (decompose-mission, verify-outcome, clarify-ambiguity, preflight-reach, stage-for-manipulation, find-object); no weights, no actuation. -Most are published under `OpenRAL/rskill-*` on HuggingFace Hub. LocateAnything is private and non-commercial; the GR00T N1.7 policy (`gr00t-n17-libero`, NVIDIA Open Model License) runs in-process through lerobot 0.6.0's `GrootPolicy`. The OpenVLA-OFT policy (`openvla-oft-simpler-widowx-nf4`, MIT) is an in-process transformers custom-code NF4 model validated on the SimplerEnv WidowX carrot-on-plate task. +Most are published under `OpenRAL/rskill-*` on HuggingFace Hub. LocateAnything is private and non-commercial; the GR00T N1.7 LIBERO/SO-101 policies run in-process through lerobot 0.6.0's `GrootPolicy`, while the official BEHAVIOR-1K R1Pro checkpoint uses its pinned Isaac-GR00T Python 3.10 sidecar. The OpenVLA-OFT policy (`openvla-oft-simpler-widowx-nf4`, MIT) is an in-process transformers custom-code NF4 model validated on the SimplerEnv WidowX carrot-on-plate task. → **Full table + license notes:** [docs/reference/rskills.md](docs/reference/rskills.md) diff --git a/benchmarks/README.md b/benchmarks/README.md index 49164f8..60cbc1e 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -59,6 +59,7 @@ for the pattern. | `gr1_tabletop.yaml` | gr1 | robocasa/gr1/PnPCupToDrawerClose (MuJoCo via robosuite + GR1 fork) | 1 | 10 | `is_success` | 720 | 10 | | `robotwin.yaml` | aloha_agilex | robotwin (SAPIEN via lerobot, py3.10 sidecar) | 5 | 100 | `is_success` | 300 | 500 | | `rlbench.yaml` | franka_panda | RLBench PerAct subset: open_drawer / meat_off_grill / close_jar (CoppeliaSim/PyRep, py3.10 sidecar) | 3 | 25 | `is_success` | 25 | 75 | +| `behavior.yaml` | r1pro | BEHAVIOR-1K 2026 challenge, all 100 public tasks (OmniGibson / Isaac Sim, sidecar) | 100 | 1 | `is_success` | 3224–39090 | 100 | Per-suite `max_steps` mirrors the upstream `lerobot.envs.libero.TASK_SUITE_MAX_STEPS` table for the LIBERO suites and the ACT / Diffusion Policy paper protocols for @@ -102,6 +103,24 @@ typed `ROSConfigError` with the full provisioning recipe when the sidecar venv / `COPPELIASIM_ROOT` are absent. Verified live on an 8 GB Ada GPU host (open_drawer 4/4, meat_off_grill 3/3, close_jar solved). +`behavior.yaml` carries **all 100 public tasks of the BEHAVIOR-1K 2026 +challenge** on **OmniGibson / Isaac Sim** (BEHAVIOR-1K defines 1,000 +activities; the challenge evaluator exposes this 100-task subset). The +official environment lives in its own venv (never vendored) and the scene +runs out-of-process through `tools/behavior_scene_sidecar.py`; the organizer +GR00T N1.7 baseline policy (`rskills/gr00t-n17-b1k-turning-on-radio`) runs +in a second sidecar (`tools/behavior_groot_sidecar.py`, whole-model NF4 to +co-reside with the simulator on 8 GB). The openral side needs +`just sync --group behavior-groot`. Like `maniskill3_panda`, `--rskill` +auto-filters the suite to a policy's `evaluated_tasks` — the only released +organizer checkpoint covers `turning_on_radio`, so today's runs execute that +single task; the suite stays runnable as multi-task BEHAVIOR policies land. +Per-task `max_steps` is the official cap (`int(1.5 x mean human-demo +length)`) read from the installed challenge stats. One instance, one episode +per task: closed-loop protocol at ~1 step/s makes an episode ~1–2 h +wall-clock; the official leaderboard metric is q_score partial credit +averaged over all public instances, which this suite does not reproduce. + ## Adding a new benchmark 1. Pick the published protocol you want to reproduce. Don't invent protocols — diff --git a/benchmarks/behavior.yaml b/benchmarks/behavior.yaml new file mode 100644 index 0000000..22ad6a1 --- /dev/null +++ b/benchmarks/behavior.yaml @@ -0,0 +1,1553 @@ +# BEHAVIOR-1K 2026 challenge suite — all 100 public challenge tasks. +# +# BEHAVIOR-1K (Li et al., 2403.09227) defines 1,000 household activities; the +# 2026 challenge (and its OmniGibson evaluator) exposes this 100-task subset +# on the Galaxea R1 Pro. Per-task max_steps is the official cap the evaluator +# computes itself: int(1.5 x mean human-demo length), read from the installed +# challenge human-demo stats. Instance 0 of the public_test split per task. +# +# Run (auto-filters to the rSkill's evaluated_tasks, like maniskill3_panda): +# openral benchmark run --suite behavior --rskill rskills/gr00t-n17-b1k-turning-on-radio +# +# Setup: OmniGibson owns the scene physics in its official Python 3.10 +# environment (never vendored); the scene runs out-of-process through +# tools/behavior_scene_sidecar.py (OPENRAL_BEHAVIOR_SIDECAR_PYTHON). Policy +# checkpoints from the pinned wensi-ai/Isaac-GR00T runtime run in their own +# sidecar (tools/behavior_groot_sidecar.py, whole-model NF4 on 8 GB). +# +# env_wrapper: the official DefaultWrapper (rgb-only, 224x224) — the wrapper +# every verified rollout used. RGBDFullResWrapper crashes at boot on the +# pinned OmniGibson build (reads joint state before og.sim.update_handles()). +# +# Protocol: the official evaluator scores partial credit (q_score) over final +# goal predicates averaged across public instances; OpenRAL's runner records +# the binary `is_success` from the same evaluator wrapper. +# +# GENERATED from the evaluator's own load_available_tasks() + +# load_human_stats() (cap = int(1.5 x mean human-demo length)); regenerate +# from those helpers rather than hand-editing task rows. + +- scene: &scene + id: behavior + backend: isaacsim + observation_height: 224 + observation_width: 224 + cameras: ["head", "left_wrist", "right_wrist"] + backend_options: &opts + task: turning_on_radio + instance_index: 0 + mode: public_test + env_wrapper: "omnigibson.eval.wrappers.DefaultWrapper" + boot_timeout_s: 1200 + timeout_ms: 120000 + task: + id: behavior/turning_on_radio + scene_id: behavior + instruction: "turning on radio" + max_steps: 3224 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: &meta + paper: "https://arxiv.org/abs/2403.09227" + display_name: "BEHAVIOR-1K 2026 challenge (100 tasks)" + simulator: "OmniGibson / Isaac Sim (sidecar)" + honest_scope: > + All 100 public tasks of the BEHAVIOR-1K 2026 challenge, one episode on + public_test instance 0 each, with the official per-task step cap + (1.5x mean human-demo length). The official leaderboard metric is + q_score partial credit averaged over all public instances; this suite + records the evaluator's binary is_success on one instance per task + because a single episode takes ~1-2 h wall-clock on the 8 GB reference + host (closed-loop protocol). The runner auto-filters to the rSkill's + evaluated_tasks; the only released organizer baseline checkpoint + covers turning_on_radio, and that checkpoint is itself flaky + (Isaac-GR00T issue #515: q_score 0.0 in 9/10 episodes vs the claimed + 0.70) — numbers reflect the checkpoint, not an OpenRAL integration + ceiling (NF4-vs-bf16 action MAE 0.003). +- scene: + <<: *scene + backend_options: + <<: *opts + task: assembling_gift_baskets + task: + id: behavior/assembling_gift_baskets + scene_id: behavior + instruction: "assembling gift baskets" + max_steps: 39090 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: attach_a_camera_to_a_tripod + task: + id: behavior/attach_a_camera_to_a_tripod + scene_id: behavior + instruction: "attach a camera to a tripod" + max_steps: 5867 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: boxing_books_up_for_storage + task: + id: behavior/boxing_books_up_for_storage + scene_id: behavior + instruction: "boxing books up for storage" + max_steps: 36341 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: boxing_food_after_dinner + task: + id: behavior/boxing_food_after_dinner + scene_id: behavior + instruction: "boxing food after dinner" + max_steps: 10046 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: bringing_in_wood + task: + id: behavior/bringing_in_wood + scene_id: behavior + instruction: "bringing in wood" + max_steps: 20303 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: bringing_paper_to_recycling + task: + id: behavior/bringing_paper_to_recycling + scene_id: behavior + instruction: "bringing paper to recycling" + max_steps: 16821 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: bringing_water + task: + id: behavior/bringing_water + scene_id: behavior + instruction: "bringing water" + max_steps: 14157 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: can_meat + task: + id: behavior/can_meat + scene_id: behavior + instruction: "can meat" + max_steps: 17770 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: canning_food + task: + id: behavior/canning_food + scene_id: behavior + instruction: "canning food" + max_steps: 34463 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: carrying_in_groceries + task: + id: behavior/carrying_in_groceries + scene_id: behavior + instruction: "carrying in groceries" + max_steps: 21412 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: carrying_out_garden_furniture + task: + id: behavior/carrying_out_garden_furniture + scene_id: behavior + instruction: "carrying out garden furniture" + max_steps: 16588 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: chop_an_onion + task: + id: behavior/chop_an_onion + scene_id: behavior + instruction: "chop an onion" + max_steps: 9599 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: chopping_wood + task: + id: behavior/chopping_wood + scene_id: behavior + instruction: "chopping wood" + max_steps: 16127 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: clean_a_keyboard + task: + id: behavior/clean_a_keyboard + scene_id: behavior + instruction: "clean a keyboard" + max_steps: 5814 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: clean_a_patio + task: + id: behavior/clean_a_patio + scene_id: behavior + instruction: "clean a patio" + max_steps: 18106 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: clean_a_trumpet + task: + id: behavior/clean_a_trumpet + scene_id: behavior + instruction: "clean a trumpet" + max_steps: 7960 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: clean_boxing_gloves + task: + id: behavior/clean_boxing_gloves + scene_id: behavior + instruction: "clean boxing gloves" + max_steps: 12353 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: clean_up_broken_glass + task: + id: behavior/clean_up_broken_glass + scene_id: behavior + instruction: "clean up broken glass" + max_steps: 13037 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: clean_up_your_desk + task: + id: behavior/clean_up_your_desk + scene_id: behavior + instruction: "clean up your desk" + max_steps: 32126 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: clean_your_rusty_garden_tools + task: + id: behavior/clean_your_rusty_garden_tools + scene_id: behavior + instruction: "clean your rusty garden tools" + max_steps: 22736 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cleaning_up_branches_and_twigs + task: + id: behavior/cleaning_up_branches_and_twigs + scene_id: behavior + instruction: "cleaning up branches and twigs" + max_steps: 18790 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cleaning_up_plates_and_food + task: + id: behavior/cleaning_up_plates_and_food + scene_id: behavior + instruction: "cleaning up plates and food" + max_steps: 20544 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: clearing_food_from_table_into_fridge + task: + id: behavior/clearing_food_from_table_into_fridge + scene_id: behavior + instruction: "clearing food from table into fridge" + max_steps: 19602 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: collecting_aluminum_cans + task: + id: behavior/collecting_aluminum_cans + scene_id: behavior + instruction: "collecting aluminum cans" + max_steps: 15304 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: collecting_childrens_toys + task: + id: behavior/collecting_childrens_toys + scene_id: behavior + instruction: "collecting childrens toys" + max_steps: 28779 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: composting_waste + task: + id: behavior/composting_waste + scene_id: behavior + instruction: "composting waste" + max_steps: 5457 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cook_a_brisket + task: + id: behavior/cook_a_brisket + scene_id: behavior + instruction: "cook a brisket" + max_steps: 11002 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cook_a_frozen_pie + task: + id: behavior/cook_a_frozen_pie + scene_id: behavior + instruction: "cook a frozen pie" + max_steps: 13001 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cook_bacon + task: + id: behavior/cook_bacon + scene_id: behavior + instruction: "cook bacon" + max_steps: 11519 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cook_broccolini + task: + id: behavior/cook_broccolini + scene_id: behavior + instruction: "cook broccolini" + max_steps: 6926 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cook_brussels_sprouts + task: + id: behavior/cook_brussels_sprouts + scene_id: behavior + instruction: "cook brussels sprouts" + max_steps: 23492 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cook_cabbage + task: + id: behavior/cook_cabbage + scene_id: behavior + instruction: "cook cabbage" + max_steps: 21217 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: cook_hot_dogs + task: + id: behavior/cook_hot_dogs + scene_id: behavior + instruction: "cook hot dogs" + max_steps: 13717 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: dispose_of_batteries + task: + id: behavior/dispose_of_batteries + scene_id: behavior + instruction: "dispose of batteries" + max_steps: 21642 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: dispose_of_glass + task: + id: behavior/dispose_of_glass + scene_id: behavior + instruction: "dispose of glass" + max_steps: 13792 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: freeze_fruit + task: + id: behavior/freeze_fruit + scene_id: behavior + instruction: "freeze fruit" + max_steps: 18928 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: freeze_pies + task: + id: behavior/freeze_pies + scene_id: behavior + instruction: "freeze pies" + max_steps: 18682 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: getting_organized_for_work + task: + id: behavior/getting_organized_for_work + scene_id: behavior + instruction: "getting organized for work" + max_steps: 23506 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: halve_an_egg + task: + id: behavior/halve_an_egg + scene_id: behavior + instruction: "halve an egg" + max_steps: 9565 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: hanging_pictures + task: + id: behavior/hanging_pictures + scene_id: behavior + instruction: "hanging pictures" + max_steps: 3581 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: hiding_Easter_eggs + task: + id: behavior/hiding_Easter_eggs + scene_id: behavior + instruction: "hiding Easter eggs" + max_steps: 11429 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: installing_a_fax_machine + task: + id: behavior/installing_a_fax_machine + scene_id: behavior + instruction: "installing a fax machine" + max_steps: 6115 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: installing_a_modem + task: + id: behavior/installing_a_modem + scene_id: behavior + instruction: "installing a modem" + max_steps: 3619 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: installing_a_scanner + task: + id: behavior/installing_a_scanner + scene_id: behavior + instruction: "installing a scanner" + max_steps: 6387 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: installing_smoke_detectors + task: + id: behavior/installing_smoke_detectors + scene_id: behavior + instruction: "installing smoke detectors" + max_steps: 3853 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: laying_tile_floors + task: + id: behavior/laying_tile_floors + scene_id: behavior + instruction: "laying tile floors" + max_steps: 21526 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: loading_the_car + task: + id: behavior/loading_the_car + scene_id: behavior + instruction: "loading the car" + max_steps: 28839 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: make_cabinet_doors + task: + id: behavior/make_cabinet_doors + scene_id: behavior + instruction: "make cabinet doors" + max_steps: 5163 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: make_gift_bags_for_baby_showers + task: + id: behavior/make_gift_bags_for_baby_showers + scene_id: behavior + instruction: "make gift bags for baby showers" + max_steps: 14476 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: make_microwave_popcorn + task: + id: behavior/make_microwave_popcorn + scene_id: behavior + instruction: "make microwave popcorn" + max_steps: 4856 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: make_pizza + task: + id: behavior/make_pizza + scene_id: behavior + instruction: "make pizza" + max_steps: 28780 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: make_rose_centerpieces + task: + id: behavior/make_rose_centerpieces + scene_id: behavior + instruction: "make rose centerpieces" + max_steps: 6798 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: moving_boxes_to_storage + task: + id: behavior/moving_boxes_to_storage + scene_id: behavior + instruction: "moving boxes to storage" + max_steps: 21894 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: organizing_art_supplies + task: + id: behavior/organizing_art_supplies + scene_id: behavior + instruction: "organizing art supplies" + max_steps: 9291 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: organizing_school_stuff + task: + id: behavior/organizing_school_stuff + scene_id: behavior + instruction: "organizing school stuff" + max_steps: 21887 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: outfit_a_basic_toolbox + task: + id: behavior/outfit_a_basic_toolbox + scene_id: behavior + instruction: "outfit a basic toolbox" + max_steps: 15956 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: packing_meal_for_delivery + task: + id: behavior/packing_meal_for_delivery + scene_id: behavior + instruction: "packing meal for delivery" + max_steps: 13289 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: picking_up_toys + task: + id: behavior/picking_up_toys + scene_id: behavior + instruction: "picking up toys" + max_steps: 28335 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: picking_up_trash + task: + id: behavior/picking_up_trash + scene_id: behavior + instruction: "picking up trash" + max_steps: 7901 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: polishing_shoes + task: + id: behavior/polishing_shoes + scene_id: behavior + instruction: "polishing shoes" + max_steps: 16106 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: preparing_lunch_box + task: + id: behavior/preparing_lunch_box + scene_id: behavior + instruction: "preparing lunch box" + max_steps: 12367 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: put_together_a_basic_pruning_kit + task: + id: behavior/put_together_a_basic_pruning_kit + scene_id: behavior + instruction: "put together a basic pruning kit" + max_steps: 16896 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: putting_away_Halloween_decorations + task: + id: behavior/putting_away_Halloween_decorations + scene_id: behavior + instruction: "putting away Halloween decorations" + max_steps: 20682 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: putting_away_toys + task: + id: behavior/putting_away_toys + scene_id: behavior + instruction: "putting away toys" + max_steps: 16945 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: putting_dirty_dishes_in_sink + task: + id: behavior/putting_dirty_dishes_in_sink + scene_id: behavior + instruction: "putting dirty dishes in sink" + max_steps: 18222 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: putting_dishes_away_after_cleaning + task: + id: behavior/putting_dishes_away_after_cleaning + scene_id: behavior + instruction: "putting dishes away after cleaning" + max_steps: 16430 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: putting_shoes_on_rack + task: + id: behavior/putting_shoes_on_rack + scene_id: behavior + instruction: "putting shoes on rack" + max_steps: 11589 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: putting_up_Christmas_decorations_inside + task: + id: behavior/putting_up_Christmas_decorations_inside + scene_id: behavior + instruction: "putting up Christmas decorations inside" + max_steps: 20578 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: re_shelving_library_books + task: + id: behavior/re_shelving_library_books + scene_id: behavior + instruction: "re shelving library books" + max_steps: 14292 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: rearrange_your_room + task: + id: behavior/rearrange_your_room + scene_id: behavior + instruction: "rearrange your room" + max_steps: 19282 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: rearranging_kitchen_furniture + task: + id: behavior/rearranging_kitchen_furniture + scene_id: behavior + instruction: "rearranging kitchen furniture" + max_steps: 13414 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: scrubbing_bathroom_floor + task: + id: behavior/scrubbing_bathroom_floor + scene_id: behavior + instruction: "scrubbing bathroom floor" + max_steps: 4730 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: set_up_a_coffee_station_in_your_kitchen + task: + id: behavior/set_up_a_coffee_station_in_your_kitchen + scene_id: behavior + instruction: "set up a coffee station in your kitchen" + max_steps: 9399 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: setting_mousetraps + task: + id: behavior/setting_mousetraps + scene_id: behavior + instruction: "setting mousetraps" + max_steps: 15294 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: setting_the_fire + task: + id: behavior/setting_the_fire + scene_id: behavior + instruction: "setting the fire" + max_steps: 13677 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: setting_the_table + task: + id: behavior/setting_the_table + scene_id: behavior + instruction: "setting the table" + max_steps: 26712 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: setup_a_bar_for_a_cocktail_party + task: + id: behavior/setup_a_bar_for_a_cocktail_party + scene_id: behavior + instruction: "setup a bar for a cocktail party" + max_steps: 20128 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: slicing_vegetables + task: + id: behavior/slicing_vegetables + scene_id: behavior + instruction: "slicing vegetables" + max_steps: 22267 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: sorting_books_on_shelf + task: + id: behavior/sorting_books_on_shelf + scene_id: behavior + instruction: "sorting books on shelf" + max_steps: 11609 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: sorting_bottles_cans_and_paper + task: + id: behavior/sorting_bottles_cans_and_paper + scene_id: behavior + instruction: "sorting bottles cans and paper" + max_steps: 15395 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: sorting_household_items + task: + id: behavior/sorting_household_items + scene_id: behavior + instruction: "sorting household items" + max_steps: 23711 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: sorting_vegetables + task: + id: behavior/sorting_vegetables + scene_id: behavior + instruction: "sorting vegetables" + max_steps: 17855 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: spraying_for_bugs + task: + id: behavior/spraying_for_bugs + scene_id: behavior + instruction: "spraying for bugs" + max_steps: 9719 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: spraying_fruit_trees + task: + id: behavior/spraying_fruit_trees + scene_id: behavior + instruction: "spraying fruit trees" + max_steps: 12518 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: stacking_wood + task: + id: behavior/stacking_wood + scene_id: behavior + instruction: "stacking wood" + max_steps: 23909 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: store_batteries + task: + id: behavior/store_batteries + scene_id: behavior + instruction: "store batteries" + max_steps: 11556 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: store_honey + task: + id: behavior/store_honey + scene_id: behavior + instruction: "store honey" + max_steps: 10150 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: store_produce + task: + id: behavior/store_produce + scene_id: behavior + instruction: "store produce" + max_steps: 14283 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: storing_food + task: + id: behavior/storing_food + scene_id: behavior + instruction: "storing food" + max_steps: 29803 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: sweeping_garage + task: + id: behavior/sweeping_garage + scene_id: behavior + instruction: "sweeping garage" + max_steps: 6709 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: thawing_frozen_food + task: + id: behavior/thawing_frozen_food + scene_id: behavior + instruction: "thawing frozen food" + max_steps: 12344 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: tidying_bathroom + task: + id: behavior/tidying_bathroom + scene_id: behavior + instruction: "tidying bathroom" + max_steps: 19504 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: tidying_bedroom + task: + id: behavior/tidying_bedroom + scene_id: behavior + instruction: "tidying bedroom" + max_steps: 16556 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: tidying_living_room + task: + id: behavior/tidying_living_room + scene_id: behavior + instruction: "tidying living room" + max_steps: 18848 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: turning_out_all_lights_before_sleep + task: + id: behavior/turning_out_all_lights_before_sleep + scene_id: behavior + instruction: "turning out all lights before sleep" + max_steps: 15009 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: unloading_the_car + task: + id: behavior/unloading_the_car + scene_id: behavior + instruction: "unloading the car" + max_steps: 16985 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: vacuuming_floors + task: + id: behavior/vacuuming_floors + scene_id: behavior + instruction: "vacuuming floors" + max_steps: 3617 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: wash_a_baseball_cap + task: + id: behavior/wash_a_baseball_cap + scene_id: behavior + instruction: "wash a baseball cap" + max_steps: 12524 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta +- scene: + <<: *scene + backend_options: + <<: *opts + task: wash_dog_toys + task: + id: behavior/wash_dog_toys + scene_id: behavior + instruction: "wash dog toys" + max_steps: 16834 + success_key: is_success + robot_id: r1pro + n_episodes: 1 + seed: 0 + metadata: *meta diff --git a/docs/architecture/repo-state-map.html b/docs/architecture/repo-state-map.html index ca8c63e..b681d1f 100644 --- a/docs/architecture/repo-state-map.html +++ b/docs/architecture/repo-state-map.html @@ -868,8 +868,8 @@

Schemas registry

title: "ROS lifecycle node", pkg: "packages/world_state · lifecycle_node.py", status: "green", - desc: "Subscribes to /joint_states + configurable sensor topics. Publishes typed openral_msgs/WorldStateStamped on /openral/world_state_fast (30 Hz) and /openral/world_state_slow (5 Hz) from one in-memory snapshot per fast tick. JSON /world_state topic deleted in a hard wire break (no migrator, openral_msgs still at 0.1.0 — the pre-F2 topic was never persisted on disk). configure/activate/deactivate/cleanup. It also subscribes /openral/perception/objects (PromptStamped, BEST_EFFORT) and /openral/world_voxels (OccupancyVoxels, RELIABLE), wires a tf2_ros.Buffer+TransformListener, and runs a memory tick timer (default 2 Hz) feeding ObjectMemory → aggregator.update_detected_objects(). Best-effort: stale/absent voxel grid → detected_objects == []; never fabricates a pose. A follow-up landed: the memory is now serialised onto WorldStateStamped's detected_object_* parallel arrays (via _fill_detected_objects in build_world_state_stamped_msg) and reconstructed by world_state_from_idl, so cross-process consumers of /openral/world_state_slow see it.", - inputs: ["/joint_states", "/openral/perception/objects", "/openral/world_voxels"], + desc: "Subscribes to /joint_states + configurable sensor topics + /openral/policy_state (std_msgs/Float32MultiArray — a backend-native policy state vector such as BEHAVIOR's official 61-D R1Pro proprio, routed verbatim to WorldStateAggregator.update_policy_state and carried on WorldState.policy_state; never inferred from joint state). Publishes typed openral_msgs/WorldStateStamped on /openral/world_state_fast (30 Hz) and /openral/world_state_slow (5 Hz) from one in-memory snapshot per fast tick. JSON /world_state topic deleted in a hard wire break (no migrator, openral_msgs still at 0.1.0 — the pre-F2 topic was never persisted on disk). configure/activate/deactivate/cleanup. It also subscribes /openral/perception/objects (PromptStamped, BEST_EFFORT) and /openral/world_voxels (OccupancyVoxels, RELIABLE), wires a tf2_ros.Buffer+TransformListener, and runs a memory tick timer (default 2 Hz) feeding ObjectMemory → aggregator.update_detected_objects(). Best-effort: stale/absent voxel grid → detected_objects == []; never fabricates a pose. A follow-up landed: the memory is now serialised onto WorldStateStamped's detected_object_* parallel arrays (via _fill_detected_objects in build_world_state_stamped_msg) and reconstructed by world_state_from_idl, so cross-process consumers of /openral/world_state_slow see it.", + inputs: ["/joint_states", "/openral/policy_state", "/openral/perception/objects", "/openral/world_voxels"], outputs: ["/openral/world_state_fast", "/openral/world_state_slow"], schemas: ["WorldState", "WorldStateStamped", "ObjectsMetadata", "msg/OccupancyVoxels"], }, @@ -1630,7 +1630,7 @@

Schemas registry

title: "Sim runner", pkg: "python/sim · openral_sim", status: "green", - desc: "Strict-by-construction (robot × scene × task × VLA) sim runner. vla.weights_uri MUST be rskill:// — there is no warn-and-proceed path. _check_rskill_compatibility resolves the manifest, looks up the registered RobotDescription, and runs rSkill.check_compatibility before make_env / make_policy; misconfiguration fails with ROSConfigError before any GPU time is spent. GH-134: make_env (MuJoCo XML compile / dataset prefetch) and make_policy (PaliGemma + NF4 quantization on π0.5, weights load on SmolVLA) run concurrently on a 2-worker ThreadPoolExecutor by default so SimRunner.activate() wall-clock collapses to max(env_ms, policy_ms); set OPENRAL_SIM_SEQUENTIAL_INIT=1 for the legacy sequential path. Scenes that hard-wire a physics robot (LIBERO → franka_panda, MetaWorld → sawyer, PushT → pusht_2d, gym-aloha → aloha_bimanual) declare it via SCENES.register(..., fixed_robot=...); the CLI raises a typed ROSConfigError on any --robot mismatch before adapter import, so misauthored YAMLs / typos fail loud instead of silently swapping the robot. The manifest's latency_budget.per_chunk_ms is plumbed into EpisodeResult.latency_budget_ms / budget_violations. Scene adapters: libero, metaworld, aloha (gym-aloha), pusht (gym-pusht), maniskill3 (SAPIEN, opt-in via --group maniskill3; default obs_mode=state_dict+rgb so the adapter's agent.qpos/qvel state extraction wires through cleanly — overridable via scene.backend_options.obs_mode), simpler_env (real-to-sim correlator on top of ManiSkill, opt-in via --group simpler-env + manual pip install of the upstream git URL; the adapter calls gym.make directly bypassing the broken upstream simpler_env.make() and auto-bumps the upstream -v0 env ids to MS3 v3.0.x's -v1, shares the MS3 obs-extraction helpers — covered by tests/sim/test_franka_panda_maniskill3_adapter.py + tests/sim/test_widowx_simpler_env_adapter.py; example configs scenes/benchmark/maniskill_pick_cube.yaml + scenes/benchmark/widowx_carrot_on_plate.yaml), isaac_sim (NVIDIA Isaac Sim — Omniverse Kit + PhysX + RTX — driven by an out-of-process py3.11 ZMQ sidecar since Isaac ships per-interpreter wheels; opt-in via --group isaacsim + an externally-provisioned sidecar venv; Isaac Sim CORE scene — no Isaac Lab — in tools/isaac_sidecar.py with two --layout scenes: lift_cube (tools/isaac_scene.py, 8-D joint-delta PoC) and bowl_plate (tools/isaac_bowl_plate_scene.py, table+YCB-bowl+plate Franka, LIBERO obs/action, end-effector control via the core Lula kinematics solver — NOT Isaac Lab's OSC term — verified driving act-libero through `openral sim run`); example scenes/sim/isaac_franka_bowl_plate.yaml, tested by tests/sim/test_franka_random_isaac.py + tests/unit/test_isaac_sim_sidecar_wire.py), rlbench (RLBench / CoppeliaSim — the PerAct 3D-keyframe benchmark — driven by an out-of-process py3.10 ZMQ sidecar since CoppeliaSim is proprietary/free-EDU/never-vendored and the released 3D policies pin the MohitShridhar/RLBench@peract fork; opt-in via --group rlbench + an externally-provisioned CoppeliaSim 4.1.0 + PyRep + peract-RLBench venv; scene sidecar tools/rlbench_sidecar.py runs the EndEffectorPoseViaPlanning retry-mover, 8-D keyframe action; scenes scenes/benchmark/rlbench_{open_drawer,meat_off_grill,close_jar}.yaml + suite benchmarks/rlbench.yaml; verified live solving open_drawer 4/4, meat_off_grill 3/3, close_jar — tests/sim/test_franka_3dda_rlbench.py), vlabench (VLABench / MuJoCo + dm_control — the OpenMOSS ICCV-2025 language-conditioned primitive-manipulation benchmark on a Franka Panda; runs NATIVE in-process on lerobot 0.6.0's VLABenchEnv, no sidecar, over PhysicsBackend.MUJOCO. Auto-provisioned by the ensure_backend_deps 'vlabench' plan — git clone + editable --no-deps + numpy-2 sim deps + a raise-on-use rrt_algorithms stub (git-only data-gen dep, off the eval path); the ~12 GB CC-BY asset bundle is a separate manual Google-Drive fetch the backend raises with the recipe for. 3 RGB views (image/second_image/wrist_image → camera1/2/3), 7-D agent_pos state, 7-D absolute-eef action; scene scenes/benchmark/vlabench_select_fruit.yaml + rSkill rskills/smolvla-vlabench (a VALIDATED 0% integration baseline — the wiring matches lerobot-eval exactly; the only >50% policy, pi0-fast-ft, is openpi/JAX pending conversion). tests/sim/test_franka_panda_smolvla_vlabench.py), mock (unit-test only) — libero / metaworld / aloha expose a duck-typed mujoco_handles() (NOT part of the SimRollout Protocol) so SimRunner can open a passive mujoco.viewer via _open_viewer AFTER env.reset (LIBERO + robosuite reallocate MjModel/MjData per reset). Policy adapters: smolvla, xvla, pi05, molmoact2, act, diffusion, zero, random. smolvla / act / pi05 / molmoact2 plumb _vla_core.apply_chunk_replay + maybe_compile_chunk_forward (pi05 forces compile_model=False to keep the nf4 path stable; molmoact2 mirrors pi05's NF4 path — `from_pretrained` → quantize_nf4_in_place → optional prequant overlay → device, no meta-init micro-opt). Robots adapter scans robots/*/robot.yaml at import time and auto-registers each (no _BUILTIN_ROBOTS tuple — drop a new manifest in and it's registered). Override the search path with $OPENRAL_ROBOTS_DIR for out-of-tree manifests. `openral sim list` walks rskills//rskill.yaml to print paste-able `:rskill://rskills/` --vla strings, replacing the legacy `policies:` line. Tested in tests/unit/test_eval_registry_and_runner.py, tests/unit/test_robots_autodiscover.py, tests/unit/test_sim_run_fixed_robot_guard.py, and tests/unit/test_sim_list_rskills.py. RLDX-1 (RLWRLD/RLDX-1 family) integrates as the `rldx` policy_id via ZMQ + msgpack sidecar — Qwen3-VL backbone in an isolated Py3.10 venv with NF4 quant; manifest-driven dispatch for LIBERO vs GR1 humanoid_everyday_g1 obs/action contracts; see python/sim/src/openral_sim/policies/rldx.py + tools/rldx_sidecar.py.", + desc: "Strict-by-construction (robot × scene × task × VLA) sim runner. vla.weights_uri MUST be rskill:// — there is no warn-and-proceed path. _check_rskill_compatibility resolves the manifest, looks up the registered RobotDescription, and runs rSkill.check_compatibility before make_env / make_policy; misconfiguration fails with ROSConfigError before any GPU time is spent. GH-134: make_env (MuJoCo XML compile / dataset prefetch) and make_policy (PaliGemma + NF4 quantization on π0.5, weights load on SmolVLA) run concurrently on a 2-worker ThreadPoolExecutor by default so SimRunner.activate() wall-clock collapses to max(env_ms, policy_ms); set OPENRAL_SIM_SEQUENTIAL_INIT=1 for the legacy sequential path. Scenes that hard-wire a physics robot (LIBERO → franka_panda, MetaWorld → sawyer, PushT → pusht_2d, gym-aloha → aloha_bimanual) declare it via SCENES.register(..., fixed_robot=...); the CLI raises a typed ROSConfigError on any --robot mismatch before adapter import, so misauthored YAMLs / typos fail loud instead of silently swapping the robot. The manifest's latency_budget.per_chunk_ms is plumbed into EpisodeResult.latency_budget_ms / budget_violations. Scene adapters: libero, metaworld, aloha (gym-aloha), pusht (gym-pusht), maniskill3 (SAPIEN, opt-in via --group maniskill3; default obs_mode=state_dict+rgb so the adapter's agent.qpos/qvel state extraction wires through cleanly — overridable via scene.backend_options.obs_mode), simpler_env (real-to-sim correlator on top of ManiSkill, opt-in via --group simpler-env + manual pip install of the upstream git URL; the adapter calls gym.make directly bypassing the broken upstream simpler_env.make() and auto-bumps the upstream -v0 env ids to MS3 v3.0.x's -v1, shares the MS3 obs-extraction helpers — covered by tests/sim/test_franka_panda_maniskill3_adapter.py + tests/sim/test_widowx_simpler_env_adapter.py; example configs scenes/benchmark/maniskill_pick_cube.yaml + scenes/benchmark/widowx_carrot_on_plate.yaml), isaac_sim (NVIDIA Isaac Sim — Omniverse Kit + PhysX + RTX — driven by an out-of-process py3.11 ZMQ sidecar since Isaac ships per-interpreter wheels; opt-in via --group isaacsim + an externally-provisioned sidecar venv; Isaac Sim CORE scene — no Isaac Lab — in tools/isaac_sidecar.py with two --layout scenes: lift_cube (tools/isaac_scene.py, 8-D joint-delta PoC) and bowl_plate (tools/isaac_bowl_plate_scene.py, table+YCB-bowl+plate Franka, LIBERO obs/action, end-effector control via the core Lula kinematics solver — NOT Isaac Lab's OSC term — verified driving act-libero through `openral sim run`); example scenes/sim/isaac_franka_bowl_plate.yaml, tested by tests/sim/test_franka_random_isaac.py + tests/unit/test_isaac_sim_sidecar_wire.py), rlbench (RLBench / CoppeliaSim — the PerAct 3D-keyframe benchmark — driven by an out-of-process py3.10 ZMQ sidecar since CoppeliaSim is proprietary/free-EDU/never-vendored and the released 3D policies pin the MohitShridhar/RLBench@peract fork; opt-in via --group rlbench + an externally-provisioned CoppeliaSim 4.1.0 + PyRep + peract-RLBench venv; scene sidecar tools/rlbench_sidecar.py runs the EndEffectorPoseViaPlanning retry-mover, 8-D keyframe action; scenes scenes/benchmark/rlbench_{open_drawer,meat_off_grill,close_jar}.yaml + suite benchmarks/rlbench.yaml; verified live solving open_drawer 4/4, meat_off_grill 3/3, close_jar — tests/sim/test_franka_3dda_rlbench.py), vlabench (VLABench / MuJoCo + dm_control — the OpenMOSS ICCV-2025 language-conditioned primitive-manipulation benchmark on a Franka Panda; runs NATIVE in-process on lerobot 0.6.0's VLABenchEnv, no sidecar, over PhysicsBackend.MUJOCO. Auto-provisioned by the ensure_backend_deps 'vlabench' plan — git clone + editable --no-deps + numpy-2 sim deps + a raise-on-use rrt_algorithms stub (git-only data-gen dep, off the eval path); the ~12 GB CC-BY asset bundle is a separate manual Google-Drive fetch the backend raises with the recipe for. 3 RGB views (image/second_image/wrist_image → camera1/2/3), 7-D agent_pos state, 7-D absolute-eef action; scene scenes/benchmark/vlabench_select_fruit.yaml + rSkill rskills/smolvla-vlabench (a VALIDATED 0% integration baseline — the wiring matches lerobot-eval exactly; the only >50% policy, pi0-fast-ft, is openpi/JAX pending conversion). tests/sim/test_franka_panda_smolvla_vlabench.py), behavior (BEHAVIOR-1K 2026 challenge — the official OmniGibson / Isaac Sim evaluator on the Galaxea R1 Pro, driven out-of-process by tools/behavior_scene_sidecar.py in the official BEHAVIOR Python environment; fixed_robot=r1pro, 61-D policy state, six typed action slots committed atomically as one 23-D evaluator step via step_action_group; env wrapper defaults to the official DefaultWrapper (rgb 224²) — RGBDFullResWrapper cannot boot on the pinned OmniGibson build; scenes scenes/sim/behavior_turning_on_radio.yaml + scenes/deploy/behavior_r1pro.yaml + scenes/benchmark/behavior_turning_on_radio.yaml, suite benchmarks/behavior.yaml (all 100 public tasks, auto-filtered per rSkill); tests/unit/test_behavior_groot_adapter.py + tests/unit/test_cli_behavior.py), mock (unit-test only) — libero / metaworld / aloha expose a duck-typed mujoco_handles() (NOT part of the SimRollout Protocol) so SimRunner can open a passive mujoco.viewer via _open_viewer AFTER env.reset (LIBERO + robosuite reallocate MjModel/MjData per reset). Policy adapters: smolvla, xvla, pi05, molmoact2, act, diffusion, zero, random, behavior_groot (the organizer's GR00T N1.7 BEHAVIOR checkpoint through tools/behavior_groot_sidecar.py — a ZMQ sidecar in the pinned wensi-ai/Isaac-GR00T py3.10 venv, whole-model NF4 for 8 GB co-residency with OmniGibson, native 61-D state + 23-D action preserved end-to-end). smolvla / act / pi05 / molmoact2 plumb _vla_core.apply_chunk_replay + maybe_compile_chunk_forward (pi05 forces compile_model=False to keep the nf4 path stable; molmoact2 mirrors pi05's NF4 path — `from_pretrained` → quantize_nf4_in_place → optional prequant overlay → device, no meta-init micro-opt). Robots adapter scans robots/*/robot.yaml at import time and auto-registers each (no _BUILTIN_ROBOTS tuple — drop a new manifest in and it's registered). Override the search path with $OPENRAL_ROBOTS_DIR for out-of-tree manifests. `openral sim list` walks rskills//rskill.yaml to print paste-able `:rskill://rskills/` --vla strings, replacing the legacy `policies:` line. Tested in tests/unit/test_eval_registry_and_runner.py, tests/unit/test_robots_autodiscover.py, tests/unit/test_sim_run_fixed_robot_guard.py, and tests/unit/test_sim_list_rskills.py. RLDX-1 (RLWRLD/RLDX-1 family) integrates as the `rldx` policy_id via ZMQ + msgpack sidecar — Qwen3-VL backbone in an isolated Py3.10 venv with NF4 quant; manifest-driven dispatch for LIBERO vs GR1 humanoid_everyday_g1 obs/action contracts; see python/sim/src/openral_sim/policies/rldx.py + tools/rldx_sidecar.py.", inputs: ["SimScene", "BenchmarkScene", "RSkillManifest", "RobotDescription"], outputs: ["EpisodeResult", "passive mujoco.viewer (optional)", "openral sim list rskill: lines"], schemas: ["SimScene", "BenchmarkScene", "SceneSpec", "TaskSpec", "PhysicsBackend", "RSkillManifest", "RobotDescription"], @@ -1666,7 +1666,7 @@

Schemas registry

title: "Benchmark runner", pkg: "python/sim · openral_sim.benchmark · openral benchmark run", status: "green", - desc: "Loops a bare `list[BenchmarkScene]` (loaded via `openral_core.load_benchmark_suite` + validated by `raise_on_invalid_suite`; suite id = filename stem) over scenes × seeds[:n_episodes], delegating each rollout to openral_sim.SimRunner so the rSkill compatibility check, OTel spans, and latency-budget reporting are identical to `openral sim run`. Emits a validated RSkillEvalResult with reproduced_locally=true and a reproduction_cli string that points back at the exact `openral benchmark run --suite … --vla …` invocation that produced it. Default output: rskills//eval/.json (overridable via --out); success can write headline avg_success_rate back into `benchmarks.`. Tested by tests/unit/test_benchmark_runner.py, tests/unit/test_rskill_benchmarks_writeback.py, and sim benchmark tests. Built-in suites live in benchmarks/: libero_spatial / libero_object / libero_goal / libero_10, metaworld_mt10 / mt50, aloha, pusht, maniskill3_panda, simpler_env_widowx, robocasa_pnp, gr1_tabletop.", + desc: "Loops a bare `list[BenchmarkScene]` (loaded via `openral_core.load_benchmark_suite` + validated by `raise_on_invalid_suite`; suite id = filename stem) over scenes × seeds[:n_episodes], delegating each rollout to openral_sim.SimRunner so the rSkill compatibility check, OTel spans, and latency-budget reporting are identical to `openral sim run`. Emits a validated RSkillEvalResult with reproduced_locally=true and a reproduction_cli string that points back at the exact `openral benchmark run --suite … --vla …` invocation that produced it. Default output: rskills//eval/.json (overridable via --out); success can write headline avg_success_rate back into `benchmarks.`. Tested by tests/unit/test_benchmark_runner.py, tests/unit/test_rskill_benchmarks_writeback.py, and sim benchmark tests. Built-in suites live in benchmarks/: libero_spatial / libero_object / libero_goal / libero_10, metaworld_mt10 / mt50, aloha, pusht, maniskill3_panda, simpler_env_widowx, robocasa_pnp, gr1_tabletop, robotwin, rlbench, behavior (all 100 BEHAVIOR-1K 2026 public tasks, per-task official step caps, auto-filtered per rSkill; `benchmark run` also records one world MP4 per episode by default).", inputs: ["benchmarks/.yaml (bare list[BenchmarkScene])", "VLASpec (CLI)", "RSkillManifest"], outputs: ["RSkillEvalResult JSON", "list[EpisodeResult]", "rskill.yaml (benchmarks. writeback)"], schemas: ["BenchmarkScene", "BenchmarkMetadata", "RSkillEvalResult", "RSkillEvalSource", "RSkillEvalBenchmark", "RSkillManifest"], diff --git a/docs/methods/00-core-schemas.md b/docs/methods/00-core-schemas.md index b3867e9..bc1b4db 100644 --- a/docs/methods/00-core-schemas.md +++ b/docs/methods/00-core-schemas.md @@ -35,17 +35,17 @@ _openral schema v0 — normative Pydantic v2 contracts for all layers._ `JOINT_POSITIONS, JOINT_VELOCITIES, DELTA_EE_6D_PLUS_GRIPPER, DELTA_EE_6D, CARTESIAN_POSE` - `class RSkillAction(str, Enum)` — Closed vocabulary of high-level action verbs an rSkill can perform; declared on `RSkillManifest.actions` and surfaced to the reasoner LLM tool palette so it can pick a skill by what it does. (L937) Manipulation primitives: `PICK, PLACE, PICK_AND_PLACE, TRANSFER, GRASP, RELEASE`; articulated / contact-rich: `OPEN, CLOSE, PUSH, PULL, SLIDE, INSERT, POUR, WIPE, ROTATE`; motion: `REACH`; mobile: `NAVIGATE`; social/expressive: `WAVE, SHAKE`; generalist marker (foundation / multi-task checkpoints): `GENERALIST`; perception producer: `DETECT` (for `kind: "detector"` rSkills); scene VLM: `QUERY` (for `kind: "vlm"` rSkills); reward monitor: `MONITOR` (for `kind: "reward"` rSkills); playbook decision procedure: `PLAN` (for `kind: "playbook"` rSkills). New entries are additive. -- `class QuantizationDtype(str, Enum)` — Weight numeric format. (L2584) +- `class QuantizationDtype(str, Enum)` — Weight numeric format. (L2592) `FP32, FP16, BF16, INT8, INT4, FP4_NVFP4` -- `class QuantizationBackend(str, Enum)` — Inference backend. (L2608) +- `class QuantizationBackend(str, Enum)` — Inference backend. (L2616) `PYTORCH, ONNX, TENSORRT, GGUF, MLX` -- `class RSkillState(str, Enum)` — Skill lifecycle. (L2680) +- `class RSkillState(str, Enum)` — Skill lifecycle. (L2688) `UNCONFIGURED, INACTIVE, ACTIVE, FINALIZED, ERROR` -- `class RSkillLicensePosture(str, Enum)` — License posture (CLAUDE §7.4). (L2760) +- `class RSkillLicensePosture(str, Enum)` — License posture (CLAUDE §7.4). (L2768) `APACHE_2_0, MIT, BSD, PERMISSIVE_RESEARCH, NVIDIA_NON_COMMERCIAL, NVIDIA_OPEN_MODEL, RLWRLD_NON_COMMERCIAL, PROPRIETARY, UNKNOWN` (NVIDIA_OPEN_MODEL = GR00T N1.7+, commercial OK) -- `class RSkillRuntime(str, Enum)` — Manifest runtime hint. (L2775) +- `class RSkillRuntime(str, Enum)` — Manifest runtime hint. (L2783) `PYTORCH, ONNX, TENSORRT, TRT_LLM, VLLM, GGUF, MLX, JAX` -- `class PhysicsBackend(str, Enum)` — Sim backend. (L6532) +- `class PhysicsBackend(str, Enum)` — Sim backend. (L6555) `MUJOCO, MUJOCO_MJX, PYBULLET, SAPIEN, ISAACSIM, COPPELIASIM, GENESIS, MOCK` (SAPIEN = ManiSkill3 / RoboTwin engine; RoboTwin uses it via a py3.10 sidecar; `COPPELIASIM` = CoppeliaSim/PyRep RLBench backend, out-of-process py3.10 sidecar) **Pydantic models — robot manifest hierarchy** @@ -66,7 +66,7 @@ _openral schema v0 — normative Pydantic v2 contracts for all layers._ - `class ComputeSpec(BaseModel)` — Compute profile for one deployment tier (edge / local / cloud). Populated by `openral_detect._enrich_compute` from GPU probe results; attached to `RobotDescription.compute_edge`, `compute_local`, or `compute_cloud`. (L667) fields: `compute_tops, system_memory_gb, num_gpus, gpu_vram_gb, cuda_compute_capability, cuda_toolkit_version, tensorrt_version, gpu_supported_runtimes, gpu_supported_dtypes, nvmm_available, endpoint, network_latency_ms`. Shared across all three tiers — `endpoint` and `network_latency_ms` are `None` for edge/local. `nvmm_available` is probed on all tiers (returns `False` gracefully when absent). `num_gpus > 1` captures cloud multi-GPU pods. GPU/runtime fields moved here from `RobotCapabilities` to separate physical robot capabilities from host compute properties. - `supports_cumotion() -> bool` — True when the host meets the cuMotion (Isaac ROS) GPU floor: compute capability >= (8, 0), CUDA toolkit major >= 13, and `gpu_vram_gb` >= `_CUMOTION_MIN_VRAM_GIB` (7.5; nominal-8 GB cards report ~7.99 GiB). The MoveIt planner gate uses it to pick cuMotion vs OMPL. False on non-CUDA hosts or when the CUDA toolkit version is unknown. -- `class ReasonerModel(BaseModel)` — Frozen curated S2 model registry entry (ADR-0088). Fields: `id`, `display_name`, `dialect (anthropic|openai)`, `hosting (cloud|managed_local|byo_local)`, `served_model_id`, `default_endpoint`, `auth_required`, `tool_choice`, `max_tokens_default`, `min_gpu_vram_gb`, `required_dtype`, `weights_license`; property `is_local`. `REASONER_MODELS` is the curated map (`claude-opus-4-8`, `gpt-5.5`, `gpt-5.6`, `cosmos3-edge`); membership means the model passed the robotics tool-calling contract. `REASONER_MANAGED_ENDPOINT="managed"` is the managed-local sentinel. (L8082) +- `class ReasonerModel(BaseModel)` — Frozen curated S2 model registry entry (ADR-0088). Fields: `id`, `display_name`, `dialect (anthropic|openai)`, `hosting (cloud|managed_local|byo_local)`, `served_model_id`, `default_endpoint`, `auth_required`, `tool_choice`, `max_tokens_default`, `min_gpu_vram_gb`, `required_dtype`, `weights_license`; property `is_local`. `REASONER_MODELS` is the curated map (`claude-opus-4-8`, `gpt-5.5`, `gpt-5.6`, `cosmos3-edge`); membership means the model passed the robotics tool-calling contract. `REASONER_MANAGED_ENDPOINT="managed"` is the managed-local sentinel. (L8105) - `class RobotCapabilities(BaseModel)` — Physical capability flags for skill compatibility. (L760) fields: `locomotion, can_lift_kg, has_dexterous_hands, has_tactile, has_force_control, has_vision, has_lidar, has_vision_slam, has_audio, bimanual, supported_control_modes, supported_vla_embodiments, embodiment_tags`. GPU/compute fields moved to `ComputeSpec` (attached at `RobotDescription.compute`). **`has_vision_slam`** gates the camera-based cuVSLAM+nvblox SLAM backend for lidar-less robots; independent of `has_lidar` (lidar backend wins when both set). - `class SafetyEnvelope(BaseModel)` — Constraints enforced by C++ safety kernel. (L808) @@ -126,7 +126,7 @@ _openral schema v0 — normative Pydantic v2 contracts for all layers._ fields: `label, confidence, pose, bbox_3d, track_id` - `class WorldCollisionPrimitive(BaseModel)` — A placed convex obstacle in the world (world-frame analogue of `LinkCollisionGeometry`); fields `shape: CollisionShape, pose: Pose6D, object_id: str | None`. (L2027) - `class OccupancyGridRef(BaseModel)` — Reference to a 2D occupancy grid for mobile-base world-collision (mirrors `nav_msgs/OccupancyGrid` metadata); fields `frame_id, resolution_m (>0), width (>=0), height (>=0), origin: Pose6D, data_topic`. (L2051) -- `class WorldState(BaseModel)` — Snapshot consumed by Reasoner and Skills. (L2196) +- `class WorldState(BaseModel)` — Snapshot consumed by Reasoner and Skills; optional `policy_state` carries a typed simulator-native checkpoint proprio vector when joint state alone cannot represent the trained observation. (L2196) fields: `stamp_ns, joint_state, base_pose, base_twist, ee_poses, contact_forces, images, image_frames, point_clouds, tactile, detected_objects, battery_pct, diagnostics, collision_primitives, occupancy_grid` - **collision_primitives / occupancy_grid** — `list[WorldCollisionPrimitive]` (default empty) + `OccupancyGridRef | None` (default `None`): the bounded world surface the kernel's world-collision phase checks robot links against; an absent/stale world is treated as unavailable (fail-closed). - **image_frames** — `dict[str, SensorFrame] | None`. Optional in-process frame carrier for no-ROS deployments; default `None` keeps the existing `images: dict[str, str]` topic-ref path unchanged. @@ -146,7 +146,7 @@ _Advisory, queryable Layer-2 world model the S2 Reasoner consults to recall wher - `class RecallObjectResult(BaseModel)` — `matches: list[RecallObjectMatch]` (empty = unknown → caller raises `ROSObjectNotInMemory`). - `class ResolvePlaceQuery(BaseModel)` — Resolve a place/room/agent reference; fields `reference, kind: SpatialNodeKind | None`. - `class ResolvePlaceResult(BaseModel)` — fields `node_id, goal: Pose6D, path_node_ids: list[str]` (a `traversable_to` path). -- `class Action(BaseModel)` — Action step or chunk produced by a Skill. (L551) +- `class Action(BaseModel)` — Action step or chunk produced by a Skill. `tick_index` preserves the shared inference-tick identity of multi-slot actions across the safety wire for atomic sim commit. (L551) fields: `control_mode, horizon, joint_targets, joint_velocities, joint_torques, cartesian_pose, cartesian_delta, cartesian_twist, body_twist, foot_placements, gripper, dex_hand_joints, confidence, stamp_ns, ee_name, frame_id, safety_overrides` - `class QuantizationConfig(BaseModel)` — Quantization recipe. (L644) fields: `dtype, backend, per_channel, calibration_dataset, extra` @@ -210,7 +210,7 @@ _Advisory, queryable Layer-2 world model the S2 Reasoner consults to recall wher - `WRAPPED_TASK_SPACE_LAYOUTS: frozenset[StateLayout]` — Subset of `StateLayout` covering Cartesian/FK-derived composites: `{rc365, human300_16d, libero_eef8d}`. These layouts REQUIRE `StateContract.bindings`; the cross-validator on `StateContract` enforces this at manifest load (`human300_16d`/`rc365` require `eef_frame`+`base_frame`; `libero_eef8d` requires `eef_frame`+`gripper_qpos_joints` — world-frame absolute EE pose, so no `base_frame`). Joint-space layouts (`smolvla_9d`, `gr1`, `simpler_*`) are excluded — they're served verbatim from raw `JointState.position`. - `StateContractBindings` (Pydantic model) — Per-robot source bindings for an rSkill's `state_contract.layout`. Fields: `eef_frame: str | None`, `base_frame: str | None`, `world_frame: str | None = "map"`, `gripper_qpos_joints: list[str]`, `quaternion_convention: Literal["xyzw","wxyz"] = "xyzw"`. Symmetric to `ControlModeSemantics` on the action side. Required when `StateContract.layout` is in `WRAPPED_TASK_SPACE_LAYOUTS`, forbidden otherwise. - `BenchmarkName` (TypeAlias = Literal[...]) — Closed canonical benchmark ids matching `benchmarks/*.yaml` suites (plus retained `aloha_insertion`/`aloha_transfer_cube` task-level ids cited by the act-aloha* manifests after the two suites were unified into `aloha.yaml`). Members: `aloha, aloha_insertion, aloha_transfer_cube, gr1_tabletop, libero_10, libero_goal, libero_object, libero_spatial, maniskill3_panda, metaworld_mt10, metaworld_mt50, pusht, rlbench, robocasa_pnp, robotwin, simpler_env_widowx`. (L2633) -- `ModelFamily` (TypeAlias = Literal["smolvla","pi05","xvla","act","diffusion","rldx","molmoact2","gr00t","diffuser_actor","openvla","lingbot_vla2","lingbot_vla","lingbot_va_a1","internvla_n1"]) — Closed VLA/policy family used by the eval/runner adapter dispatch. Required only when `RSkillManifest.kind == "vla"`. `gr00t` (NVIDIA Isaac GR00T) runs out-of-process via a ZMQ sidecar, reusing the `rldx` adapter. `diffuser_actor` (3D Diffuser Actor, RLBench keyframe policy) runs out-of-process via its own py3.10 sidecar. `openvla` (OpenVLA / OpenVLA-OFT) loads in-process as a transformers custom-code model and de-normalizes discrete action tokens via the checkpoint's `unnorm_key`. `lingbot_vla2` (Robbyant LingBot-VLA 2.0, Qwen3-VL-4B + sparse-MoE flow-matching expert, Apache-2.0) runs out-of-process via its own auto-provisioned py3.12 + torch-2.8 sidecar (`openral_sim.policies.lingbot_vla2`); `lingbot_vla` (LingBot-VLA 1.0, 4B Qwen2.5-VL + dense Qwen2 flow-matching expert, the RoboTwin post-train) shares that sidecar via `--variant v1`. `lingbot_va_a1` connects to the Runtime-owned versioned policy gateway; Runtime owns model configuration, EEF/cache semantics, and IK while OpenRAL owns candidate-action safety and HAL execution. `internvla_n1` is a dual-system vision-language navigation policy that runs out-of-process and emits a 6-D `BODY_TWIST`. (L2655) +- `ModelFamily` (TypeAlias = Literal["smolvla","pi05","xvla","act","diffusion","rldx","molmoact2","gr00t","diffuser_actor","openvla","lingbot_vla2","lingbot_vla","internvla_n1"]) — Closed VLA/policy family used by the eval/runner adapter dispatch. Required only when `RSkillManifest.kind == "vla"`. `gr00t` (NVIDIA Isaac GR00T) runs out-of-process via a ZMQ sidecar, reusing the `rldx` adapter. `diffuser_actor` (3D Diffuser Actor, RLBench keyframe policy) runs out-of-process via its own py3.10 sidecar. `openvla` (OpenVLA / OpenVLA-OFT) loads in-process as a transformers custom-code model and de-normalizes discrete action tokens via the checkpoint's `unnorm_key`. `lingbot_vla2` (Robbyant LingBot-VLA 2.0, Qwen3-VL-4B + sparse-MoE flow-matching expert, Apache-2.0) runs out-of-process via its own auto-provisioned py3.12 + torch-2.8 sidecar (`openral_sim.policies.lingbot_vla2`); `lingbot_vla` (LingBot-VLA 1.0, 4B Qwen2.5-VL + dense Qwen2 flow-matching expert, the RoboTwin post-train) shares that sidecar via `--variant v1` (a separate V1 repo + transformers==4.51.3 / lerobot==0.4.2 venv). `internvla_n1` (InternRobotics InternVLA-N1 / DualVLN, arXiv:2512.08186) is a dual-system vision-language **navigation** policy (Qwen2.5-VL-7B System-2 + NavDP DiT System-1); it runs out-of-process via a py3.11 ZMQ sidecar (upstream pins transformers 4.51) and emits a 6-D `BODY_TWIST` base velocity from RGB-D + instruction. (L2655) - `RSkillKind` (TypeAlias = Literal["vla","wam","ros_action","ros_service","detector","vlm","reward","playbook"]) — Discriminator selecting the loader / runner branch. `"vla"` is today's learnable policy path; `"ros_action"` / `"ros_service"` route through `ROSActionRskill`; `"wam"` is reserved (loader rejects at resolve time); `"detector"` is a perception producer that runs an exported ONNX/TRT detection model and publishes `ObjectsMetadata` — emits no `Action`; `"vlm"` is a video-language model answering natural-language scene queries from camera frames — emits text, no actions/boxes, `role: s2` (reached via the read-only `query_scene` tool, not `ExecuteRskill`); `"reward"` is a progress/reward monitor; `"playbook"` is a symbolic, human-authored S2 decision procedure (Markdown SOP) the reasoner *reads* into its system prompt — no weights, actuators, or `Action`, `role: s2`. (L2752) **Pydantic models — skill benchmark results (`rskills//eval/.json`)** diff --git a/docs/methods/01-hal.md b/docs/methods/01-hal.md index ee22436..00282aa 100644 --- a/docs/methods/01-hal.md +++ b/docs/methods/01-hal.md @@ -3,23 +3,15 @@ > Part of the OpenRAL [public-symbol inventory](../METHODS.md). Hand-curated; `(LNN)` markers are refreshed by `tools/refresh_methods_linenos.py`. ### `python/hal/src/openral_hal/protocol.py` -_Normative HAL protocol plus explicit optional lifecycle extensions._ +_HAL Protocol — the normative interface every HAL adapter must satisfy._ -- `class HAL(Protocol)` — Structural protocol every HAL adapter must satisfy. +- `class HAL(Protocol)` — Structural protocol every HAL adapter must satisfy. (L89) - attr `description: RobotDescription` - - `connect() -> None` — Open connection to robot/sim. - - `disconnect() -> None` — Close connection (idempotent). - - `read_state() -> JointState` — Latest joint state snapshot (hot path). - - `send_action(action: Action) -> None` — Forward action chunk to controller (hot path). - - `estop() -> None` — Trigger emergency stop, always raises `ROSEStopRequested`. -- `class LifecycleEStopHAL(Protocol)` — Opt-in propagation of the generic - lifecycle e-stop to downstream hardware or owned processes. -- `class ResettableLifecycleEStopHAL(LifecycleEStopHAL, Protocol)` — Opt-in - in-process recovery contract. -- `class EStopRecovery(StrEnum)` — Declares whether recovery is resettable or - requires a full lifecycle restart. -- `class HALHealthProvider(Protocol)` / `class HALHealthReport` — Cached, - I/O-free diagnostics consumed by the generic lifecycle heartbeat. + - `connect() -> None` — Open connection to robot/sim. (L106) + - `disconnect() -> None` — Close connection (idempotent). (L116) + - `read_state() -> JointState` — Latest joint state snapshot (hot path). (L124) + - `send_action(action: Action) -> None` — Forward action chunk to controller (hot path). (L138) + - `estop() -> None` — Trigger emergency stop, always raises `ROSEStopRequested`. (L154) ### `python/hal/src/openral_hal/_mujoco_arm.py` _Internal MuJoCo-backed HAL implementation shared by UR / Franka / SO-100 / G1 / H1 / Rizon-4 / OpenArm / ALOHA adapters. Reads its wiring from `RobotDescription.sim`._ @@ -177,249 +169,6 @@ _SO100FollowerHAL — wraps lerobot's SO-100 follower arm USB driver._ - `_rad_to_deg(rad) -> float` (L238) - const `SO100_DESCRIPTION = RobotDescription(...)` (L88) -### `python/hal/src/openral_hal/galaxea_a1.py` -_Real-only Galaxea A1 HAL. OpenRAL stays ROS 2 / Python 3.12; the operator's -official ROS 1 Noetic SDK runs out of process behind a literal IPv4-loopback -JSON-lines sidecar. No vendor source, binary, or message package is distributed._ - -- `class GalaxeaA1HAL(HALBase)` — six-axis joint-position + normalized - gripper adapter. `read_state` and `send_action` use a cached snapshot/latest - target so network I/O stays off the HAL hot path. Commands fail closed on - stale state/status, unaccepted motor bits, non-finite values, initial target - misalignment, or an excessive feedback-relative target step. Command limits - remain exact; a separately tracked 0.01 rad feedback-only endpoint tolerance - absorbs encoder zero/quantization at a nominal URDF boundary. `estop` asks - the sidecar to stop its owned ROS 1 stack and always raises - `ROSEStopRequested`. -- const `GALAXEA_A1_DESCRIPTION` — real-only `RobotDescription`, mirrored by - `robots/galaxea_a1/robot.yaml`; official A1 URDF joint names/limits, explicit - sidecar deadlines, motor masks, 0..104 mm normalized gripper mapping, and - the calibrated D455 front / D405 wrist RGB observation contracts. The six - joint origins, orientations, and axes are transcribed from the official A1 - URDF; collision primitives remain absent until their redistribution and - lowering provenance is cleared. -- `tools/galaxea_a1_ros1_sidecar.py` — Python-3.8-compatible ROS 1 process that - owns `roscore`, `signal_arm/single_arm_node.launch`, and the official - `mobiman/jointTracker_demo_node` binary. The tracker publishes to - `/openral/arm_joint_command_staged`; a sidecar-owned relay is the sole - publisher to `/arm_joint_command_host`. The relay stays `LOCKED` until the - first target, then `ARMING` until a fresh, valid tracker command is aligned - with both that target and measured joint feedback. Only `ACTIVE` forwards - unchanged tracker commands, and gripper setpoints are gated on the same - machine — a gripper command while the relay is `LOCKED`/`ARMING` is refused - fail-closed (the gripper bypasses the tracker's staged-hold interpolation, - so it must never actuate before alignment). Repeated identical joint and gripper setpoints - refresh the command lease without restarting the official tracker. A command - lease, alignment timeout, malformed command, stale feedback, motor fault, - client disconnect, or e-stop stops the complete owned process group. -- `tools/run_galaxea_a1_sidecar.sh` — Docker launcher. Requires an explicit - operator-provided image and SDK path; mounts both read-only and claims only the - selected serial device. The SDK remains read-only; the official tracker's - generated CppAD files go to - `$XDG_CACHE_HOME/openral/galaxea-a1/x1_robot` (or the equivalent path below - `~/.cache`). `--check-only` verifies the local image, required SDK files, - cache parent, serial ownership, loopback port, container name, and process - lock without opening the serial device or starting a container. - -#### Galaxea A1 hardware bring-up - -The first session is observation-only until the HAL graph is healthy. Ensure no -other process/container owns the serial device, the arm workspace is clear, and -the physical e-stop is reachable. - -```bash -# One-time: build OpenRAL's vendor-free Noetic runtime image. The official SDK -# is mounted at run time and is never copied into the image. -docker build \ - -t openral/galaxea-a1-sidecar:noetic \ - docker/galaxea_a1_sidecar - -# One-time: build OpenRAL's standard public x86 deploy image (Jazzy/Python 3.12). -just docker-build-x86 - -# Read-only gate — checks the image, SDK, serial ownership, port, and lock. -tools/run_galaxea_a1_sidecar.sh \ - --image openral/galaxea-a1-sidecar:noetic \ - --sdk-root /absolute/path/to/A1_SDK \ - --serial /dev/a1 \ - --check-only - -# Terminal 1 — isolated ROS 1 bridge network; only TCP 46011 reaches loopback. -tools/run_galaxea_a1_sidecar.sh \ - --image openral/galaxea-a1-sidecar:noetic \ - --sdk-root /absolute/path/to/A1_SDK \ - --serial /dev/a1 - -# Terminal 2 — OpenRAL's standard real-hardware path. The OpenRAL container uses -# host networking only for ROS 2 DDS and the sidecar's loopback TCP port; it owns -# no Galaxea serial device and cannot see the ROS 1 master inside the sidecar. -docker run --rm --name openral-galaxea-a1 --network host \ - --volume "$(pwd)/robots:/workspace/robots:ro" \ - --volume "$(pwd)/scenes:/workspace/scenes:ro" \ - --volume "$(pwd)/tests:/workspace/tests:ro" \ - openral:x86 \ - --config scenes/deploy/galaxea_a1_bench.yaml - -# Terminal 3 — observation gate: six named joints update; diagnostics are clean. -docker exec openral-galaxea-a1 bash -lc \ - 'source /opt/ros/jazzy/setup.bash && source /workspace/install/setup.bash && \ - ros2 topic echo /joint_states --once && ros2 topic echo /diagnostics --once' -``` - -An optional HAL-level HIL gate can run between Terminal 1 and the full deploy. -It opens one sidecar session, validates three fresh finite named-joint samples -plus cached motor health, and ends by verifying that downstream e-stop stops the -owned ROS 1 stack. Restart Terminal 1 afterwards: - -```bash -GALAXEA_A1_HIL=1 just hil galaxea_a1 -``` - -Only after that observation-only run passes, opt into a measured-current-pose -hold. Feedback within the tracked 0.01 rad endpoint tolerance is projected to -the exact command limit; any larger projection fails before publication. The -test also waits for the sidecar relay to report `ACTIVE`, proving the official -tracker has converged from its compiled `task.info` initial pose before any -host motor command is forwarded: - -```bash -GALAXEA_A1_HIL=1 GALAXEA_A1_ALLOW_HOLD=1 just hil galaxea_a1 -``` - -After the hold passes, a separate lab opt-in moves `arm_joint1` by +0.01 rad, -requires it to settle within 0.008 rad (covering the measured 0.007 rad -small-command residual), continuously bounds all six joint excursions, returns -to the measured start, and then performs the same downstream e-stop: - -```bash -GALAXEA_A1_HIL=1 GALAXEA_A1_ALLOW_NUDGE=1 just hil galaxea_a1 -``` - -The G2 gripper has its own opt-in. It uses the vendor example's 10 mm step, -mapped through the normalized `0..1` contract over the configured 104 mm -stroke, chooses the direction away from the nearest endpoint, verifies feedback -within the measured 2.5 mm steady-state tolerance, and returns to the measured -opening even when the outbound-leg assertion fails: - -```bash -GALAXEA_A1_HIL=1 GALAXEA_A1_ALLOW_GRIPPER=1 just hil galaxea_a1 -``` - -After the HAL-level gates pass, the full-graph HIL runs inside the deploy -container. It captures the current named-joint feedback itself, requires the -C++ kernel and real HAL to be active while the relay is still `LOCKED`, then -publishes only that measured hold through `candidate_action`. It verifies the -matching `safe_action`, exact staged/forwarded targets, zero kernel drops, and -less than one degree of drift. Its `finally` path publishes `/openral/estop` -three times and requires the HAL diagnostics to confirm the latch: - -```bash -docker exec \ - --env GALAXEA_A1_DEPLOY_HIL=1 \ - --env GALAXEA_A1_ALLOW_HOLD=1 \ - openral-galaxea-a1 \ - bash -lc 'source /opt/ros/jazzy/setup.bash && \ - source /workspace/install/setup.bash && \ - pytest -q /workspace/tests/hil/test_galaxea_a1_deploy.py' -``` - -After the current-pose full-graph gate passes, the same fixture has a separate -motion opt-in. It moves `arm_joint1` by +0.01 rad through -`candidate_action -> C++ safety kernel -> safe_action`, bounds all six joint -excursions, and returns to the measured start before the downstream e-stop: - -```bash -docker exec \ - --env GALAXEA_A1_DEPLOY_HIL=1 \ - --env GALAXEA_A1_ALLOW_HOLD=1 \ - --env GALAXEA_A1_ALLOW_NUDGE=1 \ - openral-galaxea-a1 \ - bash -lc 'source /opt/ros/jazzy/setup.bash && \ - source /workspace/install/setup.bash && \ - pytest -q /workspace/tests/hil/test_galaxea_a1_deploy.py' -``` - -This test intentionally ends the hardware session. Restart both the sidecar -and deploy container before any later motion test. - -Do not start a policy on the first pass. Stop both commands and investigate if -feedback/status becomes stale, a motor code other than the manifest's explicit -idle/gripper masks appears, joint order differs, the sidecar exits, or the arm -moves before an approved safe action. Motion validation then proceeds with a -current-pose hold and a single <=0.01 rad joint increment through OpenRAL's -standard candidate-action → C++ kernel → safe-action path, then return-to-start; -only afterwards run an A1-specific rSkill. - -#### LingBot-VA rSkill through the complete OpenRAL path - -`rskills/lingbot-va-galaxea-a1-fruit-placement/rskill.yaml` is the first -checkpoint-specific A1 rSkill. The dependency direction is deliberate: - -```text -A1 Camera Bridge -> OpenRAL WorldState -> LingBot-VA rSkill - -> A1 Runtime policy gateway (model contract + EEF/cache + IK) - -> OpenRAL candidate_action -> C++ safety kernel -> safe_action - -> GalaxeaA1HAL -> isolated ROS 1 sidecar -> official A1 driver -``` - -The A1 Runtime is a public capability provider, not a second controller: -start only its persistent camera owner, LingBot policy server, and OpenRAL -policy gateway. The gateway has no ROS imports or command publisher. Do not -start its LingBot ROS execution bridge or A1 joint runtime while OpenRAL owns -the deployment. The rSkill owns its `policy_extras.max_joint_substep_rad` replay -setting and reads the independent `max_target_step_rad` ceiling from the same -`RobotDescription` used to construct the HAL. Startup rejects a policy bound -that exceeds either the HAL's live target-step ceiling or locked-relay -alignment tolerance. The policy bound is 0.045 rad, below the 0.05 rad -locked-relay alignment threshold; the independent HAL/sidecar live limit is -0.08 rad. The gateway constructs Runtime's IK implementation with the active -OpenRAL `RobotDescription`'s ordered command limits after verifying they are no -wider than Runtime's envelope. Runtime calibration margins therefore cannot -widen the typed OpenRAL, safety-kernel, or official sidecar command envelope. - -The gateway emits one bounded target per 30 Hz control tick. When its IK -solution is farther than 0.045 rad from fresh feedback, it keeps advancing -toward that same solved target on subsequent ticks and only consumes the next -model action after the solved target has been dispatched. The FK of the actual -dispatched target is written into the KV cache, so the policy state reflects -what OpenRAL commanded rather than an unreachable ideal. The official tracker's -steady-state error cannot widen the command envelope or bypass the bounded -step. The A1 Runtime's 1.70 rad IK-solution validation remains an upstream -reachability check, not a motor-command step limit. - -```bash -# Terminal A — A1 Runtime capability providers only (no ROS command publisher). -cd /absolute/path/to/A1-Research -just cameras start -scripts/apps/lingbot/a1_lingbot_runtime.sh server -uv run galaxea-a1-openral-policy \ - --config configs/deployments/lingbot/fruit_placement_eef.toml \ - --repo-root . - -# Terminal B — official ROS 1 sidecar, as in the bring-up section above. -cd /absolute/path/to/OpenRAL -tools/run_galaxea_a1_sidecar.sh \ - --image openral/galaxea-a1-sidecar:noetic \ - --sdk-root /absolute/path/to/A1_SDK \ - --serial /dev/a1 - -# Terminal C — the complete OpenRAL real deployment. -cd /absolute/path/to/OpenRAL -uv run --group lingbot openral deploy run \ - --config scenes/deploy/galaxea_a1_bench.yaml -``` - -Submit the exact trained prompt (for example, `put the red mango into the blue -plate`) through the dashboard. Before allowing a task motion, first repeat the -observation, hold, joint-nudge, gripper, and full-graph gates above. Stop the -LingBot server afterwards with -`scripts/apps/lingbot/a1_lingbot_runtime.sh server-stop`. - -The A1 opts into hardware-downstream e-stop: `/openral/estop` stops the -sidecar-owned tracker and driver immediately. The generic -`/openral/estop_cleared` broadcast cannot re-arm this HAL; restart the lifecycle -and sidecar, re-read motor health, and repeat initial alignment instead. - ### `python/hal/src/openral_hal/h1.py` _MuJoCo digital twin for the Unitree H1 humanoid (Menagerie MJCF). Contract validator only — falls without an S0 cerebellum; gravity must be disabled in closed-loop tests (CLAUDE.md §6.2). Unlike the G1 / UR / Franka / SO-100 MJCFs, the H1 menagerie ships ``motor`` (torque) actuators, so this HAL runs a software PD position loop every physics step._ @@ -542,20 +291,20 @@ _SimTransport — in-memory simulated `ros2_control` transport._ ### `python/hal/src/openral_hal/lifecycle.py` _Generic ROS 2 managed lifecycle node wrapper for every HAL adapter — UR5e / UR10e / Franka / SO-100 / OpenArm / H1 / future HALs all share the same publish / subscribe / heartbeat / OTel-span wiring._ -- `class HALLifecycleNodeBase(LifecycleNode)` — Public base class. Owns the standard `/joint_states` + `~/joint_states` publishers, the `/openral/safe_action` + `/openral/estop` subscribers, the 1 Hz `DiagnosticsHeartbeat`, the per-tick `hal.read_state` + `hal.send_action` OTel spans, the estop latch, and the full configure → activate → deactivate → cleanup → shutdown transition wiring. The formerly used `~/command` (`trajectory_msgs/JointTrajectory`) subscriber + its `_on_command` callback + the `_subscriber` field were removed; `_send_action_traced` is now driven only by `_on_safe_action`. (L322) - - `_create_hal(self) -> HAL` — **Subclass hook (required)**: construct and return a HAL instance. Reads ROS-parameter-driven constructor args via `self.get_parameter(...)`. (L381) - - `_heartbeat_extra_fields(self) -> dict[str, str]` — Subclass hook (optional): extra key/values for the `/diagnostics` payload (e.g. `{"port": "/dev/ttyUSB0"}` for SO-100, `{"mjcf": "..."}` for OpenArm). Default: `{}`. (L393) - - `on_configure_post_hal(self) -> TransitionCallbackReturn` — Subclass hook (optional): robot-specific setup after the HAL connects (e.g. opening a camera renderer on OpenArm). Default: `SUCCESS`. (L445) - - `on_activate_post_subs(self) -> TransitionCallbackReturn` — Subclass hook (optional): robot-specific timers/publishers after the base wires its subs (e.g. the OpenArm camera-render timer). Default: `SUCCESS`. (L454) - - `on_deactivate_pre_teardown(self) -> None` — Subclass hook (optional): stop robot-specific timers before base teardown. Default: no-op. (L462) - - `on_cleanup_pre_disconnect(self) -> None` — Subclass hook (optional): tear down robot-specific resources (viewers, renderers) before HAL.disconnect(). Default: no-op. (L469) - - `_publish_joint_state(self) -> None` — Timer callback. Wraps `self._hal.read_state()` in a `hal.read_state` span (identity attrs + `producer.record_joint_state`) and publishes the standard `/joint_states` + `~/joint_states` messages. Subclasses may override + call `super()._publish_joint_state()` to extend (OpenArm does this for viewer-sync). (L775) - - `_on_safe_action(self, msg) -> None` — `/openral/safe_action` callback. Decodes the `openral_msgs/ActionChunk` into an `openral_core.Action` and forwards through `_send_action_traced(action, source="safe_action")`. (L846) - - `_send_action_traced(self, action, *, source) -> None` — Forward `action` to `self._hal.send_action` inside a `hal.send_action` span. The `source` attribute disambiguates the origin on the dashboard's Commands card (kept on the span so future subscriber additions can fan in without changing the span shape). (L866) -- `make_lifecycle_main(node_name, hal_factory) -> Callable[[], None]` — Build a `main()` entry point for a zero-parameter HAL adapter. Internally constructs a `_FactoryHALLifecycleNode(HALLifecycleNodeBase)` whose `_create_hal()` returns `hal_factory()`. Superseded for the standard arms by `make_lifecycle_main_from_manifest`; retained for bespoke nodes. (L194) -- `class ManifestHALLifecycleNode(HALLifecycleNodeBase)` — Public generic manifest-driven lifecycle node (promoted from the private `_ManifestHALLifecycleNode` under issue #191). Reads `robot_yaml` + `hal_mode` + sensor knobs as ROS params and builds its HAL via `openral_hal.build_hal`, so a robot's construction kwargs come from the manifest's `hal.parameters.defaults` — no bespoke `_create_hal` subclass. Attaches `SimSensorBridge` (cameras / depth / scan / viewer) in `on_activate_post_subs`. In `on_configure_post_hal`, **reflects** on the built HAL and opens `/openral//reset_to_pose` (`openral_msgs/srv/ResetToPose`) iff it exposes `reset_to_pose` — generalising the openarm-only service to every `MujocoArmHAL` sim arm (issue #191 Phase 2); HALs without the method (panda_mobile, scene-attached twins) get no service. In `_create_hal`, when a scene composition is declared (and not scene-attaching), calls the named composer and threads the composed MJCF in as the HAL's `mjcf_path` (issue #191 Phase 3b — openarm tabletop); the composition is read from the `scene_composition_json` ROS param (the DeployScene's own `composition`) which **takes precedence** over the robot manifest's `scene_defaults.composition` (back-compat fallback) — so the scene owns its arena, the robot manifest describes the robot. Bare-twin camera robots (so100/so101) need no composition: their cameras are spliced by the generic camera rig at HAL connect from `sensors[].sim_placement`. In `on_activate_post_subs`, when the manifest declares a planar base (`base_joints`), also attaches a `MobileBaseBridge` (`/odom` + `odom->base_link` TF + `/cmd_vel`→BODY_TWIST) — so panda_mobile runs on this node with no subclass (issue #191 Phase 3a). The per-robot lifecycle packages collapse into this node (issue #191 Phases 2-3). A back-compat alias `_ManifestHALLifecycleNode` is retained. (L1028) -- `make_lifecycle_main_from_manifest(node_name) -> Callable[[], None]` — Build a `main()` that spins up `ManifestHALLifecycleNode`. The node reads `robot_yaml` + `hal_mode` ("sim"|"real") ROS params and constructs its HAL via `openral_hal.build_hal(description, mode=hal_mode)` — one node class serves both modes for every robot. Used by franka / ur5e / ur10e / aloha / g1 / h1 / rizon4 / so100 / so101 (issue #191 Phase 2 migrated so100/so101 off their bespoke node); `openral deploy sim` injects `hal_mode="sim"`, `openral deploy run` injects `hal_mode="real"`. A robot lacking the requested mode raises `ROSCapabilityMismatch`. (L252) -- `decode_action_chunk(msg) -> Action | None` — Inverse of `ros_publishing_hal._flatten_action_payload`. Decodes the `ActionChunk` wire shape (`flat` + `n_dof` + `horizon` + `control_mode`) back into a typed `openral_core.Action` with the per-mode payload field populated (`cartesian_delta` / `gripper` / `body_twist` / `joint_*`). Returns `None` for degenerate chunks (`flat=[]`, `n_dof≤0`) and for modes the F1/F5 publisher doesn't encode (`CARTESIAN_POSE`, `FOOT_PLACEMENT`, `DEX_HAND_JOINT`). Used by `HALLifecycleNodeBase._on_safe_action`; lives at module scope so unit tests in `tests/unit/test_lifecycle_action_chunk_decoder.py` exercise it without a ROS 2 install. (L103) +- `class HALLifecycleNodeBase(LifecycleNode)` — Public base class. Owns the standard `/joint_states` + `~/joint_states` publishers, the `/openral/safe_action` + `/openral/estop` subscribers, the 1 Hz `DiagnosticsHeartbeat`, the per-tick `hal.read_state` + `hal.send_action` OTel spans, the estop latch, and the full configure → activate → deactivate → cleanup → shutdown transition wiring. The formerly used `~/command` (`trajectory_msgs/JointTrajectory`) subscriber + its `_on_command` callback + the `_subscriber` field were removed; `_send_action_traced` is now driven only by `_on_safe_action`. (L326) + - `_create_hal(self) -> HAL` — **Subclass hook (required)**: construct and return a HAL instance. Reads ROS-parameter-driven constructor args via `self.get_parameter(...)`. (L386) + - `_heartbeat_extra_fields(self) -> dict[str, str]` — Subclass hook (optional): extra key/values for the `/diagnostics` payload (e.g. `{"port": "/dev/ttyUSB0"}` for SO-100, `{"mjcf": "..."}` for OpenArm). Default: `{}`. (L398) + - `on_configure_post_hal(self) -> TransitionCallbackReturn` — Subclass hook (optional): robot-specific setup after the HAL connects (e.g. opening a camera renderer on OpenArm). Default: `SUCCESS`. (L450) + - `on_activate_post_subs(self) -> TransitionCallbackReturn` — Subclass hook (optional): robot-specific timers/publishers after the base wires its subs (e.g. the OpenArm camera-render timer). Default: `SUCCESS`. (L459) + - `on_deactivate_pre_teardown(self) -> None` — Subclass hook (optional): stop robot-specific timers before base teardown. Default: no-op. (L467) + - `on_cleanup_pre_disconnect(self) -> None` — Subclass hook (optional): tear down robot-specific resources (viewers, renderers) before HAL.disconnect(). Default: no-op. (L474) + - `_publish_joint_state(self) -> None` — Timer callback. Wraps `self._hal.read_state()` in a `hal.read_state` span (identity attrs + `producer.record_joint_state`) and publishes the standard `/joint_states` + `~/joint_states` messages. Subclasses may override + call `super()._publish_joint_state()` to extend (OpenArm does this for viewer-sync). (L796) + - `_on_safe_action(self, msg) -> None` — `/openral/safe_action` callback. Decodes the `openral_msgs/ActionChunk` into an `openral_core.Action` and forwards through `_send_action_traced(action, source="safe_action")`. (L875) + - `_send_action_traced(self, action, *, source) -> None` — Forward `action` to `self._hal.send_action` inside a `hal.send_action` span. The `source` attribute disambiguates the origin on the dashboard's Commands card (kept on the span so future subscriber additions can fan in without changing the span shape). (L895) +- `make_lifecycle_main(node_name, hal_factory) -> Callable[[], None]` — Build a `main()` entry point for a zero-parameter HAL adapter. Internally constructs a `_FactoryHALLifecycleNode(HALLifecycleNodeBase)` whose `_create_hal()` returns `hal_factory()`. Superseded for the standard arms by `make_lifecycle_main_from_manifest`; retained for bespoke nodes. (L198) +- `class ManifestHALLifecycleNode(HALLifecycleNodeBase)` — Public generic manifest-driven lifecycle node (promoted from the private `_ManifestHALLifecycleNode` under issue #191). Reads `robot_yaml` + `hal_mode` + sensor knobs as ROS params and builds its HAL via `openral_hal.build_hal`, so a robot's construction kwargs come from the manifest's `hal.parameters.defaults` — no bespoke `_create_hal` subclass. Attaches `SimSensorBridge` (cameras / depth / scan / viewer) in `on_activate_post_subs`. In `on_configure_post_hal`, **reflects** on the built HAL and opens `/openral//reset_to_pose` (`openral_msgs/srv/ResetToPose`) iff it exposes `reset_to_pose` — generalising the openarm-only service to every `MujocoArmHAL` sim arm (issue #191 Phase 2); HALs without the method (panda_mobile, scene-attached twins) get no service. In `_create_hal`, when a scene composition is declared (and not scene-attaching), calls the named composer and threads the composed MJCF in as the HAL's `mjcf_path` (issue #191 Phase 3b — openarm tabletop); the composition is read from the `scene_composition_json` ROS param (the DeployScene's own `composition`) which **takes precedence** over the robot manifest's `scene_defaults.composition` (back-compat fallback) — so the scene owns its arena, the robot manifest describes the robot. Bare-twin camera robots (so100/so101) need no composition: their cameras are spliced by the generic camera rig at HAL connect from `sensors[].sim_placement`. In `on_activate_post_subs`, when the manifest declares a planar base (`base_joints`), also attaches a `MobileBaseBridge` (`/odom` + `odom->base_link` TF + `/cmd_vel`→BODY_TWIST) — so panda_mobile runs on this node with no subclass (issue #191 Phase 3a). The per-robot lifecycle packages collapse into this node (issue #191 Phases 2-3). A back-compat alias `_ManifestHALLifecycleNode` is retained. (L1057) +- `make_lifecycle_main_from_manifest(node_name) -> Callable[[], None]` — Build a `main()` that spins up `ManifestHALLifecycleNode`. The node reads `robot_yaml` + `hal_mode` ("sim"|"real") ROS params and constructs its HAL via `openral_hal.build_hal(description, mode=hal_mode)` — one node class serves both modes for every robot. Used by franka / ur5e / ur10e / aloha / g1 / h1 / rizon4 / so100 / so101 (issue #191 Phase 2 migrated so100/so101 off their bespoke node); `openral deploy sim` injects `hal_mode="sim"`, `openral deploy run` injects `hal_mode="real"`. A robot lacking the requested mode raises `ROSCapabilityMismatch`. (L256) +- `decode_action_chunk(msg) -> Action | None` — Inverse of `ros_publishing_hal._flatten_action_payload`. Decodes the typed payload and preserves `ee_name`, `frame_id`, `confidence`, and the shared inference `tick_index` after the safety kernel. Returns `None` for degenerate or unwired modes. (L103) ### `python/hal/src/openral_hal/sim_bringup.py` _Resolve a `SimScene` or `BenchmarkScene` YAML path to a live `SimRollout`. Used by `build_hal`, which every manifest-driven node (incl. panda_mobile, issue #191 Phase 3) routes through._ @@ -571,23 +320,25 @@ _`SimAttachedHAL` — generic HAL Protocol adapter that wraps any in-process `Si - `pack_action_for_env(action: Action, description: RobotDescription, env_action_dim: int, prev: np.ndarray | None = None) -> np.ndarray` — Default `ActionPacker`. Translates `JOINT_POSITION` (arm-only or full base+arm), `BODY_TWIST` (vx/vy/wz → slots 0-2), `CARTESIAN_DELTA` (6-vec → arm slots `[base_dim:]`), and `GRIPPER_POSITION` (→ last slot) into the env's flat action vector. Raises `ROSConfigError` for unsupported modes or mismatched row widths. A single VLA policy step on a non-composite env (LIBERO OSC_POSE, SimplerEnv widowx — every `delta_ee_6d_plus_gripper` rSkill) splits into a CARTESIAN_DELTA then a GRIPPER_POSITION Action, each `env.step`-ed; `prev` (threaded from `SimAttachedHAL._last_env_action`) carries the last commanded gripper through the arm step and holds the arm on the gripper step, so the arm advances once per policy step with the gripper always commanded — mirroring `_pack_with_composite_split`. Without it each Action zeroed the other's slots and the arm barely moved. (L191) - `class SimAttachedHAL` — HAL Protocol adapter wrapping an in-process `SimRollout`. Reads live joint state via `normalized_joint_index` + `mj_name2id`; sends actions via `pack_action_for_env` (or a caller-supplied `ActionPacker`) into `env.step()`. Exposes `read_images()`, `mujoco_handles()`, `sim_time_ns()`, `base_pose`, `base_twist`, `base_pose_6dof()` for the ROS lifecycle node's camera publisher, viewer, sim-clock, and odom wiring. (L375) - `__init__(env: SimRollout, description: RobotDescription, *, action_packer: ActionPacker | None = None, env_reset_seed: int | None = None, env_action_dim: int | None = None, body_twist_dt_s: float = 0.05) -> None` (L404) - - `connect() -> None` — Reset the env at `env_reset_seed`; probe `env_action_dim` (via `_probe_env_action_dim`, which raises `ROSConfigError` naming the backend when no `action_dim` is introspectable and no override was given — never a silent fallback); invalidate joint-index cache. Idempotent. (L511) - - `disconnect() -> None` — Release env handle (idempotent). (L583) - - `read_state() -> JointState` — Walk `description.joints`, resolve each joint via `normalized_joint_index`, read live `qpos`/`qvel` from MJCF. (L587) - - `send_action(action: Action) -> None` — Pack action via composite-split or `ActionPacker`; call `env.step`. `BODY_TWIST` takes a direct-qpos Euler-integration path on a MuJoCo backend (`_apply_body_twist_to_qpos`, skips `env.step` so the arm doesn't churn); on a non-MuJoCo backend (Isaac kinematic base) it routes through `_apply_body_twist_via_env_step` instead — the scene integrates the base inside `env.step`. Stamps `last_action_ns` at the top (the single choke point both `_on_safe_action` and `_on_cmd_vel` reach) so the idle stepper yields to it. Routes the step through `_step_and_cache`, which auto-resets on episode termination — both the *returned* terminal (`StepResult.terminated/truncated` latched as `_episode_done`) and a *raised* terminal (raw-robosuite `ignore_done=False` backends throwing `is_terminated_episode_error`); the raised path resets once and re-steps so deploy-sim never freezes with the "env.step failed: executing action in terminated episode" spam. (L661) - - `idle_step() -> bool` — **Sim-only** free-running stepper (2026-06-04 amendment). Advances the wrapped `SimRollout` one tick with `np.zeros(env_action_dim)` (HOLD) so cameras keep rendering when no skill is executing — without it an idle deploy-sim scene freezes and the perception bus sees a dead scene. Returns `False` (suppressed) when not connected, estop-latched, `env_action_dim is None`, or no live MuJoCo handles; else steps and returns `True`. Mirrors `send_action`'s deferred-reset branch and `_last_obs` re-cache; does NOT touch `_last_env_action` / `_last_body_twist`. **Defined ONLY on `SimAttachedHAL`** — real HALs never define it; this method-only exclusion (not "zero is harmless") is the primary real-hardware guard, since a zero vector is a HOLD in sim but "drive to 0 rad" on a real position arm. (L892) - - `read_images() -> dict[str, Any]` — Return latest rendered camera frames keyed by camera name from the cached `_last_obs`. (L1452) + - `connect() -> None` — Reset the env at `env_reset_seed`; probe `env_action_dim` (via `_probe_env_action_dim`, which raises `ROSConfigError` naming the backend when no `action_dim` is introspectable and no override was given — never a silent fallback); invalidate joint-index cache. Idempotent. (L517) + - `disconnect() -> None` — Release env handle (idempotent). (L591) + - `read_state() -> JointState` — Walk `description.joints`, resolve each joint via `normalized_joint_index`, read live `qpos`/`qvel` from MJCF. (L595) + - `send_action(action: Action) -> None` — Pack action via composite-split or `ActionPacker`; call `env.step`. `BODY_TWIST` takes a direct-qpos Euler-integration path on a MuJoCo backend (`_apply_body_twist_to_qpos`, skips `env.step` so the arm doesn't churn); on a non-MuJoCo backend (Isaac kinematic base) it routes through `_apply_body_twist_via_env_step` instead — the scene integrates the base inside `env.step`. Stamps `last_action_ns` at the top (the single choke point both `_on_safe_action` and `_on_cmd_vel` reach) so the idle stepper yields to it. Routes the step through `_step_and_cache`, which auto-resets on episode termination — both the *returned* terminal (`StepResult.terminated/truncated` latched as `_episode_done`) and a *raised* terminal (raw-robosuite `ignore_done=False` backends throwing `is_terminated_episode_error`); the raised path resets once and re-steps so deploy-sim never freezes with the "env.step failed: executing action in terminated episode" spam. (L669) + - `_stage_action_group(action, group_step) -> None` — For backends exposing `action_group_size` + `step_action_group`, stage each safety-approved slot by `Action.tick_index` and commit exactly one simulator step only when the complete tick is present. Rejected/missing slots never actuate; incomplete prior ticks are visibly dropped. + - `idle_step() -> bool` — **Sim-only** free-running stepper (2026-06-04 amendment). Advances the wrapped `SimRollout` one tick with `np.zeros(env_action_dim)` (HOLD) so cameras keep rendering when no skill is executing — without it an idle deploy-sim scene freezes and the perception bus sees a dead scene. Returns `False` (suppressed) when not connected, estop-latched, `env_action_dim is None`, or no live MuJoCo handles; else steps and returns `True`. Mirrors `send_action`'s deferred-reset branch and `_last_obs` re-cache; does NOT touch `_last_env_action` / `_last_body_twist`. **Defined ONLY on `SimAttachedHAL`** — real HALs never define it; this method-only exclusion (not "zero is harmless") is the primary real-hardware guard, since a zero vector is a HOLD in sim but "drive to 0 rad" on a real position arm. (L959) + - `read_images() -> dict[str, Any]` — Return latest rendered camera frames keyed by camera name from the cached `_last_obs`. (L1541) + - `read_policy_state() -> list[float] | None` — Return cached simulator-native checkpoint proprioception for `/openral/policy_state`. - `read_depth_clouds() -> dict[str, NDArray]` — Per-depth-sensor `(N,3)` `base_link` point clouds from `_last_obs["depth_points"]` (a non-MuJoCo backend, e.g. the Isaac scene, deprojects via `Camera.get_pointcloud` so the HAL never re-derives geometry); `{}` when the backend renders no depth. `SimSensorBridge` publishes them as `PointCloud2` for octomap. - `read_scan() -> NDArray | None` — The 2-D LaserScan range fan (`base_link`, `angle_min=-π`→`+π`) from `_last_obs["scan"]` when a non-MuJoCo backend ray-casts a lidar (the Isaac scene); `None` when it renders no lidar. `SimSensorBridge._compute_scan_ranges` reads it for `/scan`. - - `mujoco_handles() -> tuple[Any, Any] | None` — Forward the env's `(model, data)` MJCF handles. (L1332) + - `mujoco_handles() -> tuple[Any, Any] | None` — Forward the env's `(model, data)` MJCF handles. (L1421) - `sim_time_ns() -> int | None` — Cross-reset-monotonic elapsed sim time in ns — the seam a sim `/clock` publisher reads. Returns the wrapped `SimRollout.sim_time_ns()` (per-episode) plus an accumulated offset: `connect` and the auto-resets fold each finished episode's elapsed sim-time into the offset (`_accumulate_sim_time_before_reset`) BEFORE the backend rewinds its clock, so the value is monotonic non-decreasing across `env.reset` (robocasa rewinds `MjData.time` to 0). `None` when the wrapped rollout has no sim clock (clock-less backend / sidecar) — the consumer then falls back to wall time. - `clock_authority() -> ClockAuthority` — Return the timestamp authority this HAL contributes to the graph: `ClockAuthority.simulation(, timestep_s=body_twist_dt_s)` when `sim_time_ns()` is live, otherwise `ClockAuthority.host_wall()` so launch keeps the graph on the host-wall authority. - - `estop() -> None` — Latch e-stop; subsequent `send_action` calls are dropped. (L1326) - - `base_pose -> tuple[float, float, float]` [@property] — Current `(x, y, yaw)`: from MJCF qpos on a MuJoCo backend, else from `obs["base_pose"]` the SimRollout surfaces (Isaac kinematic base); `(0,0,0)` when the backend reports neither. Feeds the `/odom` publisher. (L1516) - - `base_twist -> tuple[float, float, float, float, float, float]` [@property] — Last commanded body twist `(vx, vy, vz, wx, wy, wz)`. (L1569) + - `estop() -> None` — Latch e-stop; subsequent `send_action` calls are dropped. (L1413) + - `base_pose -> tuple[float, float, float]` [@property] — Current `(x, y, yaw)`: from MJCF qpos on a MuJoCo backend, else from `obs["base_pose"]` the SimRollout surfaces (Isaac kinematic base); `(0,0,0)` when the backend reports neither. Feeds the `/odom` publisher. (L1605) + - `base_twist -> tuple[float, float, float, float, float, float]` [@property] — Last commanded body twist `(vx, vy, vz, wx, wy, wz)`. (L1658) - `_apply_body_twist_via_env_step(row: list[float]) -> None` — Non-MuJoCo `BODY_TWIST`: validate the planar twist, latch it for `/odom`, pack `(vx, vy, wz)` into the FINAL three env-action slots (the manifest scene's `[arm…, gripper, base-twist]` layout), zero the arm/gripper so a pure base move holds the arm, and `_step_and_cache`. - - `base_pose_6dof() -> tuple[...] | None` — Full 6-DoF `(xyz, quat_xyzw)` from robocasa `raw_proprio`; falls back to `None` for non-robocasa backends. (L1579) - - `last_action_ns -> int` [@property] — Monotonic ns of the last real action through `send_action`; `0` until the first one. The idle stepper reads it (via `should_idle_step`) to yield to an active skill. (L1442) + - `base_pose_6dof() -> tuple[...] | None` — Full 6-DoF `(xyz, quat_xyzw)` from robocasa `raw_proprio`; falls back to `None` for non-robocasa backends. (L1668) + - `last_action_ns -> int` [@property] — Monotonic ns of the last real action through `send_action`; `0` until the first one. The idle stepper reads it (via `should_idle_step`) to yield to an active skill. (L1531) ### `python/hal/src/openral_hal/sim_sensor_bridge.py` _Shared sim-sensor + viewer bridge for scene-attached HAL lifecycle nodes. Republishes RGB camera frames and a live MuJoCo viewer for any manifest-driven node, and runs the sim-only idle stepper. Phase 2 adds `/scan` + depth `PointCloud2`. Depth comes from the MuJoCo ray-cast (`_publish_depth_clouds`) OR, for a non-MuJoCo backend that surfaces ready `base_link` clouds in obs (the Isaac scene), `_publish_depth_clouds_from_obs` — which wraps `hal.read_depth_clouds()` into a `base_link` `PointCloud2` (no ray-cast, no per-camera optical TF); `_setup_depth` creates the publishers when either source is present. rclpy imported lazily._ diff --git a/docs/methods/03-world-state.md b/docs/methods/03-world-state.md index 892f904..1b6f298 100644 --- a/docs/methods/03-world-state.md +++ b/docs/methods/03-world-state.md @@ -19,16 +19,17 @@ _WorldStateAggregator — tf2-aware, injectable snapshot producer._ - `class WorldStateAggregator` — Aggregates sensor data and produces `WorldState` snapshots. (L107) - `__init__(description, *, staleness_limit_s=DEFAULT_STALENESS_S, clock_fn=None)` (L160) - - `update_joint_state(state) -> None` — Record a fresh joint reading. (L235) - - `update_image(sensor_name, topic, stamp_ns) -> None` — Record image arrival. (L246) - - `update_ee_pose(ee_name, pose) -> None` — Record EE pose from tf2. (L302) - - `update_base_pose(pose, twist=None) -> None` — Record base pose (and optional twist). (L316) - - `update_battery(pct) -> None` — Record battery %. (L333) - - `set_error(component, status='error') -> None` — Latch a forced diagnostic. (L357) - - `clear_error(component) -> None` — Remove a forced diagnostic. (L373) - - `snapshot() -> WorldState` — Produce a typed snapshot (hot path, acquires lock). Emits a `world_state.snapshot` OTel span with `openral.world_state.components_stale` + `openral.world_state.has_latched_error` attributes, fires `openral.event.staleness_latched` / `openral.event.error_latched` events on first transition, records per-component `openral.world_state.staleness_ms` histogram + `openral.world_state.components_stale` up-down counter. (L384) - - `update_detected_objects(objects: list[DetectedObject]) -> None` — Replace the remembered detected-object set. Thread-safe; the next `snapshot()` call returns the new list. Called by the world-state lifecycle node's memory tick. (L342) - - `_emit_snapshot_telemetry(span, diag, ages_ms) -> None` — Internal: lift the snapshot diagnostics onto the OTel span + meter instruments. (L501) + - `update_joint_state(state) -> None` — Record a fresh joint reading. (L237) + - `update_policy_state(values) -> None` — Store a defensive copy of simulator-native checkpoint proprioception for `WorldState.policy_state`. + - `update_image(sensor_name, topic, stamp_ns) -> None` — Record image arrival. (L248) + - `update_ee_pose(ee_name, pose) -> None` — Record EE pose from tf2. (L315) + - `update_base_pose(pose, twist=None) -> None` — Record base pose (and optional twist). (L329) + - `update_battery(pct) -> None` — Record battery %. (L346) + - `set_error(component, status='error') -> None` — Latch a forced diagnostic. (L370) + - `clear_error(component) -> None` — Remove a forced diagnostic. (L386) + - `snapshot() -> WorldState` — Produce a typed snapshot (hot path, acquires lock). Emits a `world_state.snapshot` OTel span with `openral.world_state.components_stale` + `openral.world_state.has_latched_error` attributes, fires `openral.event.staleness_latched` / `openral.event.error_latched` events on first transition, records per-component `openral.world_state.staleness_ms` histogram + `openral.world_state.components_stale` up-down counter. (L397) + - `update_detected_objects(objects: list[DetectedObject]) -> None` — Replace the remembered detected-object set. Thread-safe; the next `snapshot()` call returns the new list. Called by the world-state lifecycle node's memory tick. (L355) + - `_emit_snapshot_telemetry(span, diag, ages_ms) -> None` — Internal: lift the snapshot diagnostics onto the OTel span + meter instruments. (L522) ### `python/world_state/src/openral_world_state/spatial_memory.py` _SpatialMemory — persistent object-centric scene-graph memory (advisory; never a safety input)._ diff --git a/docs/methods/07-eval-sim.md b/docs/methods/07-eval-sim.md index 67c0da2..fb0c475 100644 --- a/docs/methods/07-eval-sim.md +++ b/docs/methods/07-eval-sim.md @@ -76,13 +76,13 @@ _Benchmark runner — loops a bare `list[BenchmarkScene]` (loaded via `load_benc - `_task_matches(task_id, scene_id, declared) -> bool` — Whether a `declared` entry covers the scene: exact `task.id`, a `"/<…>"` family prefix (`"libero_spatial"` covers `libero_spatial/0..9`), or the bare `scene.id`. (L66) - `filter_scenes_for_skill(scenes, manifest) -> tuple[list[BenchmarkScene], list[BenchmarkScene]]` — Suite analogue of `check_benchmark_task_compatibility`: partitions a suite into `(kept, skipped)` by matching each scene against `manifest.evaluated_tasks` (via `_task_matches`). `None`/empty `evaluated_tasks` is permissive (keep all, mirrors the single-scene gate's legacy branch). Lets one `benchmark run` execute every task an rSkill supports and skip the rest, and closes the gap where the suite path never gated tasks (mismatched-but-same-embodiment ran to a silent 0). (L143) - `_manifest_for_filter(vla) -> RSkillManifest | None` — Loads the rSkill manifest for `filter_scenes_for_skill`, or `None` for unfilterable skills (built-in mock policies / raw `hf://` URIs — same guard as the single-scene gate). (L126) -- `run_benchmark(scenes, *, suite_id, vla, device=None, save_dir=None) -> tuple[RSkillEvalResult, list[EpisodeResult]]` — Auto-filters `scenes` to the rSkill's `evaluated_tasks` (`filter_scenes_for_skill`, logging a `benchmark_suite_task_filter` summary of skips and raising `ROSCapabilityMismatch` if nothing matches), then iterates the kept `scenes × range(seed, seed + n_episodes)`, drives each `(BenchmarkScene, seed)` tuple with a fresh `SimRunner`, and aggregates into a validated `RSkillEvalResult`. Per-scene `robot_id` / `task` / `max_steps` pulled from each `BenchmarkScene`; suite-level invariants pre-checked by `raise_on_invalid_suite` (the runner does not re-validate). All args after `scenes` are keyword-only — callers must name `suite_id` and `vla`. (L182) -- `_aggregate_results(scenes, *, suite_id, vla, per_task, episodes) -> RSkillEvalResult` — Roll per-task booleans into per-task / avg success rates. Suite-level `benchmark.name` / `benchmark.simulator` come from `scenes[0].metadata.display_name` / `.simulator` when present, else fall back to `suite_id` / `scenes[0].scene.id`. `benchmark.arxiv` auto-derived from `scenes[0].metadata.paper` when the URL contains `arxiv.org/`. `max_steps` in the protocol summary is `max(scene.task.max_steps for scene in scenes)` so the bound is the suite worst-case, not just `scenes[0]`. Pulled out for unit-test reuse. (L326) -- `run_benchmark_scene(scene, vla, *, device=None, save_dir=None, config_path=None, view=None, record_video=False) -> tuple[RSkillEvalResult, list[EpisodeResult]]` — Single-scene sibling of `run_benchmark`; backs `openral benchmark scene`. `record_video` (default False) captures per-step world frames into each `EpisodeResult.frames` so `benchmark scene --save-video` can write clean website MP4s. Iterates `range(scene.seed, scene.seed + scene.n_episodes)` against the one `(scene, task)` pair carried by a `BenchmarkScene` and emits the same `RSkillEvalResult` shape so `openral benchmark report` does not need to distinguish entrypoints. Raises `ROSConfigError` when `scene.robot_id is None`. `view` (tri-state, default `None`) is the opt-in viewer flag for parity with `sim run`: `None` keeps the historical headless behaviour (eval/CI unaffected), an explicit `True`/`False` is resolved through `cli._resolve_view` and passed to `SimRunner`. (L440) -- `_aggregate_scene_results(scene, vla, successes, episodes, config_path) -> RSkillEvalResult` — Single-scene counterpart of `_aggregate_results`; shares the output schema. PushT special-case mirrors the suite path. Embeds `config_path` into `reproduction_cli` for byte-identical reruns from disk. (L585) -- `default_output_path(weights_uri, benchmark_id) -> str` — Canonical mapping `rskills/` (or bare name) → `rskills//eval/.json`. (L676) -- `update_rskill_benchmarks(skill_dir, benchmark_id, score) -> Path` — Surgical rewrite of the `benchmarks:` block in `/rskill.yaml` that preserves every other comment + line; re-validates the merged manifest through `RSkillManifest` before writing. Closes the `openral benchmark run` → `rskill.yaml` loop so manifest headlines stay in sync with the eval JSONs. Raises `FileNotFoundError` if no manifest, `ROSConfigError` on unknown `benchmark_id` / out-of-range `score`. (L719) -- `update_rskill_benchmarks_from_uri(weights_uri, benchmark_id, score) -> Path` — Resolve the skill reference to a local dir and delegate to the manifest updater; mirrors `default_output_path` so the CLI passes the same ref it already holds. (L819) +- `run_benchmark(scenes, *, suite_id, vla, device=None, save_dir=None, video_dir=None) -> tuple[RSkillEvalResult, list[EpisodeResult]]` — Auto-filters `scenes` to the rSkill's `evaluated_tasks` (`filter_scenes_for_skill`, logging a `benchmark_suite_task_filter` summary of skips and raising `ROSCapabilityMismatch` if nothing matches), then iterates the kept `scenes × range(seed, seed + n_episodes)`, drives each `(BenchmarkScene, seed)` tuple with a fresh `SimRunner`, and aggregates into a validated `RSkillEvalResult`. `video_dir` (default None) enables per-step frame capture and writes one MP4 per episode via `_website_video.write_world_videos` (`[_seed]__.mp4` + `videos.json`), freeing each episode's frames after the write so large suites stay memory-flat. Per-scene `robot_id` / `task` / `max_steps` pulled from each `BenchmarkScene`; suite-level invariants pre-checked by `raise_on_invalid_suite` (the runner does not re-validate). All args after `scenes` are keyword-only — callers must name `suite_id` and `vla`. (L182) +- `_aggregate_results(scenes, *, suite_id, vla, per_task, episodes) -> RSkillEvalResult` — Roll per-task booleans into per-task / avg success rates. Suite-level `benchmark.name` / `benchmark.simulator` come from `scenes[0].metadata.display_name` / `.simulator` when present, else fall back to `suite_id` / `scenes[0].scene.id`. `benchmark.arxiv` auto-derived from `scenes[0].metadata.paper` when the URL contains `arxiv.org/`. `max_steps` in the protocol summary is `max(scene.task.max_steps for scene in scenes)` so the bound is the suite worst-case, not just `scenes[0]`. Pulled out for unit-test reuse. (L349) +- `run_benchmark_scene(scene, vla, *, device=None, save_dir=None, config_path=None, view=None, record_video=False) -> tuple[RSkillEvalResult, list[EpisodeResult]]` — Single-scene sibling of `run_benchmark`; backs `openral benchmark scene`. `record_video` (default False) captures per-step world frames into each `EpisodeResult.frames` so `benchmark scene --save-video` can write clean website MP4s. Iterates `range(scene.seed, scene.seed + scene.n_episodes)` against the one `(scene, task)` pair carried by a `BenchmarkScene` and emits the same `RSkillEvalResult` shape so `openral benchmark report` does not need to distinguish entrypoints. Raises `ROSConfigError` when `scene.robot_id is None`. `view` (tri-state, default `None`) is the opt-in viewer flag for parity with `sim run`: `None` keeps the historical headless behaviour (eval/CI unaffected), an explicit `True`/`False` is resolved through `cli._resolve_view` and passed to `SimRunner`. (L463) +- `_aggregate_scene_results(scene, vla, successes, episodes, config_path) -> RSkillEvalResult` — Single-scene counterpart of `_aggregate_results`; shares the output schema. PushT special-case mirrors the suite path. Embeds `config_path` into `reproduction_cli` for byte-identical reruns from disk. (L608) +- `default_output_path(weights_uri, benchmark_id) -> str` — Canonical mapping `rskills/` (or bare name) → `rskills//eval/.json`. (L699) +- `update_rskill_benchmarks(skill_dir, benchmark_id, score) -> Path` — Surgical rewrite of the `benchmarks:` block in `/rskill.yaml` that preserves every other comment + line; re-validates the merged manifest through `RSkillManifest` before writing. Closes the `openral benchmark run` → `rskill.yaml` loop so manifest headlines stay in sync with the eval JSONs. Raises `FileNotFoundError` if no manifest, `ROSConfigError` on unknown `benchmark_id` / out-of-range `score`. (L742) +- `update_rskill_benchmarks_from_uri(weights_uri, benchmark_id, score) -> Path` — Resolve the skill reference to a local dir and delegate to the manifest updater; mirrors `default_output_path` so the CLI passes the same ref it already holds. (L842) ### `python/sim/src/openral_sim/cli.py` - `sim_app: typer.Typer` — Public `openral sim` Typer group. Mounted into the top-level `openral` Typer tree by `openral_cli.main`. Hosts the `run` leaf (`--config / --robot / --scene / --task / --rskill / …`) and the `list` leaf (registry printer). @@ -377,7 +377,22 @@ _Auto-managed sidecar adapter for RLWRLD/RLDX-1 (Qwen3-VL-8B + Multi-Stream Acti - `_build_rldx(env_cfg) -> _RLDXSidecarAdapter` — `@POLICIES.register("rldx")` factory. Honours `OPENRAL_RLDX_HOST` / `OPENRAL_RLDX_PORT` / `OPENRAL_RLDX_AUTO_SPAWN` / `OPENRAL_RLDX_BOOT_TIMEOUT_S` / `OPENRAL_RLDX_QUANTIZATION` / `OPENRAL_RLDX_EMBODIMENT_TAG` / `OPENRAL_RLDX_MODEL_ID` / `OPENRAL_RLDX_SIDECAR_SCRIPT` env-var overrides; reads `replan_steps` / `image_size` / `timeout_ms` / `camera_keys` / `auto_spawn` / `boot_timeout_s` / `quantization` / `embodiment_tag` / `model_id` from `vla.extra`; dispatches the obs/action contract via `_resolve_state_layout` and the port via `_resolve_sidecar_port` (per-identity default when unpinned). #### `python/sim/src/openral_sim/policies/gr00t.py` -_NVIDIA Isaac GR00T N1.7 policy adapter — runs **in-process** under the workspace's Python 3.12 via lerobot 0.6.0's native `GrootPolicy` (`lerobot.policies.groot`), mirroring the smolvla adapter. The historical Python-3.10 ZMQ sidecar (`tools/gr00t_sidecar.py`) is deleted — N1.7's Cosmos-Reason2 / Qwen3-VL backbone is a stock `Qwen3VLForConditionalGeneration` in `transformers>=5.4`, so the py3.10 rationale disappears. NF4 (backbone-only) keeps the ~3 B model on an 8 GiB card (~5.2 GiB peak); live-validated LIBERO-spatial 5/5 at ~19 ms/step. **RLDX-1 (a GR00T-N1.5 finetune) still runs on its own ZMQ sidecar via the `rldx` adapter** — native lerobot rejects N1.5, so `tools/rldx_sidecar.py` and `_RLDXSidecarAdapter` are untouched._ +_NVIDIA Isaac GR00T N1.7 policy adapter — standard checkpoints run **in-process** under the workspace's Python 3.12 via lerobot 0.6.0's native `GrootPolicy` (`lerobot.policies.groot`), mirroring the smolvla adapter. NF4 (backbone-only) keeps the ~3 B model on an 8 GiB card (~5.2 GiB peak); live-validated LIBERO-spatial 5/5 at ~19 ms/step. A manifest with `policy_extras.implementation=behavior_b1k_sidecar` branches before HF/native checkpoint resolution into `behavior_groot.py`, because the official organizer checkpoint is pinned to the separate `wensi-ai/Isaac-GR00T` Python 3.10 runtime. **RLDX-1 (a GR00T-N1.5 finetune) still runs on its own ZMQ sidecar via the `rldx` adapter**._ + +#### `python/sim/src/openral_sim/policies/behavior_groot.py` +_Official BEHAVIOR-1K GR00T policy adapter. The organizer checkpoint depends on the pinned `wensi-ai/Isaac-GR00T` `behavior` branch rather than lerobot's native N1.7 loader, so it runs in an externally-provisioned Python 3.10 sidecar. The OpenRAL process carries only pyzmq/msgpack (`behavior-groot` dependency group)._ + +- `build_behavior_groot_policy(env_cfg, manifest, extra) -> _BehaviorGrootAdapter` — Resolve the local organizer checkpoint, sidecar endpoint, task/instruction, control mode, and quantization (`policy_extras.quantization`, default `nf4`, plus `nf4_min_params`); auto-spawn `tools/behavior_groot_sidecar.py` or connect to an operator/remote sidecar. +- `_BehaviorGrootAdapter` — `PolicyAdapter` implementation that preserves the official flattened evaluator observation when present, otherwise maps canonical `images.{head,left_wrist,right_wrist}` + 61-D `state` back to the R1Pro wire keys; validates a finite 23-D action. +- `_behavior_wire_observation(observation, *, instruction) -> dict[str, object]` — Pure official-wire assembler used by the adapter and unit tests. + +#### `python/sim/src/openral_sim/backends/behavior.py` +_BEHAVIOR-1K / OmniGibson scene adapter. Runs the official evaluator environment in its own sidecar and registers `scene.id=behavior`, fixed to `r1pro`._ + +- `class _BehaviorSidecar` — ZMQ-backed `SimRollout`; surfaces 61-D `state`/`policy_state`, manifest-order 22-joint positions/velocities, three RGB views, and sim time. +- `step_action_group(actions) -> StepResult` — Validate six equal-tick safety-approved slots and commit them as one official 23-D action. +- `_compose_action_group(actions) -> NDArray[float32]` — Compose base twist + torso/arm joint targets + dual grippers into the official action order. +- `_build_behavior_scene(env_cfg) -> _BehaviorSidecar` — Resolve the official BEHAVIOR Python and auto-spawn `tools/behavior_scene_sidecar.py`. - `_GrootAdapter(spec, device, _policy, _preprocessor, _postprocessor, _torch, ...)` — In-process `GrootPolicy` adapter. `libero_sim` uses native relative actions whose `GrootN17ActionDecodeStep` refuses per-step decoding, so it predicts the full chunk via `predict_action_chunk`, decodes it while the pack-step state is fresh, queues the first `_replan_steps` (GR00T's 8-of-16 `libero_sim` horizon) and pops one per `step` (chunk-replay, like the rldx adapter). `_build_batch` feeds float CHW `[0,1]` images to the embodiment's modality keys (`_image_input_keys` — `image`/`wrist_image` for LIBERO, `front`/`wrist` for SO-101) + a `_state_dim`-wide proprio state; honours rSkill `image_preprocessing.flip_180` / `flip_vertical`. Public contract: `reset/step/close/last_input_frame`. - `_build_groot_config(*, local_path, embodiment_tag, quantize, image_keys, state_dim, action_dim) -> GrootConfig` — Constructs a `GrootConfig` with `embodiment_tag` set at construction (the `libero_sim` gripper-flip / action-decode transform resolves in `__post_init__`; mutating it afterwards is too late) and explicit `input_features` / `output_features` pinning the head to the real 7-D LIBERO action instead of the 132-D padded default. `model_params_fp32=not quantize` (quantized params cannot be fp32-cast). - `_quantize_groot_nf4(policy, torch, *, scope) -> None` — NF4-rewrites the GR00T model via the accelerate-free `openral_sim._quantization.quantize_nf4_in_place`. `scope="backbone"` (default) packs only `_groot_model.backbone` (the ~2 B Qwen3-VL), leaving the diffusion head bf16 — enough for the 16-layer LIBERO head to fit 8 GB. `scope="model"` packs the whole `_groot_model` (backbone **and** the DiT head's large Linears), needed for heavier heads (the SO-101 fruit checkpoint's 32-layer DiT overshoots 8 GB otherwise). In both scopes the `>=4M`-param threshold spares the small `TimestepEncoder` MLP, so its params stay bf16 and the GR00T DiT uint8/`silu` bug cannot recur. @@ -455,4 +470,4 @@ _Boot helper + server for the InternVLA-N1 nav sidecar, companion to `openral_si - `_register_policies() -> None` — Side-effect imports of the policy-adapter modules so each registers its factory in `openral_sim.POLICIES` at import time. (L15) #### `python/sim/src/openral_sim/backends/__init__.py` -- `_register_backends() -> None` — Side-effect imports of the scene-backend modules so each registers its factory in `openral_sim.SCENES` at import time. (L39) +- `_register_backends() -> None` — Side-effect imports of the scene-backend modules so each registers its factory in `openral_sim.SCENES` at import time. (L42) diff --git a/docs/methods/08-cli.md b/docs/methods/08-cli.md index 2eef5de..1b40953 100644 --- a/docs/methods/08-cli.md +++ b/docs/methods/08-cli.md @@ -5,28 +5,28 @@ ### `python/cli/src/openral_cli/main.py` _openral CLI entry point — `openral` command (typer-based)._ -- `_root(ctx)` — Top-level callback. Picks the trace `sample_ratio` from `_SAMPLE_RATIO_BY_MODE` (hardware → 0.1, others → None → ALWAYS_ON), calls `configure_observability(service_name="ral", sample_ratio=...)`, and opens a `cli.command` root span via `cli_command_span(ctx.invoked_subcommand, mode=…)`, registered on the click `Context` so the span closes after the subcommand returns. Mode is derived from `_RUN_MODE_BY_SUBCOMMAND` (`sim` → sim, `benchmark` → benchmark, `deploy`/`connect` → hardware). (L387) -- `_RUN_MODE_BY_SUBCOMMAND: dict[str, str]` — Subcommand → `openral.run.mode` mapping consumed by `_root`. (L369) -- `_SAMPLE_RATIO_BY_MODE: dict[str, float]` — `openral.run.mode` → trace sample ratio. Hardware: 0.1 (2026-05-17 amendment). (L381) -- `class CheckResult(NamedTuple)` — One row in `openral doctor` output. (L418) +- `_root(ctx)` — Top-level callback. Picks the trace `sample_ratio` from `_SAMPLE_RATIO_BY_MODE` (hardware → 0.1, others → None → ALWAYS_ON), calls `configure_observability(service_name="ral", sample_ratio=...)`, and opens a `cli.command` root span via `cli_command_span(ctx.invoked_subcommand, mode=…)`, registered on the click `Context` so the span closes after the subcommand returns. Mode is derived from `_RUN_MODE_BY_SUBCOMMAND` (`sim` → sim, `benchmark` → benchmark, `deploy`/`connect` → hardware). (L389) +- `_RUN_MODE_BY_SUBCOMMAND: dict[str, str]` — Subcommand → `openral.run.mode` mapping consumed by `_root`. (L370) +- `_SAMPLE_RATIO_BY_MODE: dict[str, float]` — `openral.run.mode` → trace sample ratio. Hardware: 0.1 (2026-05-17 amendment). (L383) +- `class CheckResult(NamedTuple)` — One row in `openral doctor` output. (L420) fields: `check, status, details` -- `_check_python() -> CheckResult` (L435) -- `_check_platform() -> CheckResult` (L440) -- `_check_openral_core() -> CheckResult` (L444) -- `_check_ros2() -> list[CheckResult]` — ROS 2 binary, distro, RMW. (L452) -- `_check_colcon() -> CheckResult` (L493) -- `_check_gpu() -> list[CheckResult]` — One row per GPU. (L498) -- `_check_usb() -> list[CheckResult]` — Candidate robot USB serial devices. (L606) -- `_check_just() -> CheckResult` (L624) -- `_check_reasoner_llm() -> list[CheckResult]` — Model-first doctor dispatcher. Reads `OPENRAL_REASONER_MODEL`, resolves `REASONER_MODELS`, and reports the effective dialect / hosting / endpoint / key status; missing auth gets a follow-up row. An uncurated model fails until `OPENRAL_REASONER_ENDPOINT` + `OPENRAL_REASONER_DIALECT` are explicit, then reports `warn` (untested). The legacy provider contract still diagnoses through `_check_reasoner_llm_legacy` with a separate deprecation row. Never prints the API key value. (L815) -- `_check_reasoner_model(model_key) -> list[CheckResult]` — Resolve one curated/uncurated model into doctor rows; delegates loopback checks to `_reasoner_endpoint_probe_row`. (L738) -- `_reasoner_endpoint_probe_row(label, base_url, *, managed, autostart) -> CheckResult` — Generic loopback probe. Down managed-local + autostart is `info`; managed with autostart disabled and every BYO-local endpoint are `warn`. (L701) -- `_cosmos_autostart_enabled() -> bool` — mirrors the reasoner client's `OPENRAL_COSMOS3_AUTOSTART` parsing (falsy spellings `0/false/no/off`); kept local so doctor never imports the optionally-installed reasoner package. (L657) +- `_check_python() -> CheckResult` (L437) +- `_check_platform() -> CheckResult` (L442) +- `_check_openral_core() -> CheckResult` (L446) +- `_check_ros2() -> list[CheckResult]` — ROS 2 binary, distro, RMW. (L454) +- `_check_colcon() -> CheckResult` (L495) +- `_check_gpu() -> list[CheckResult]` — One row per GPU. (L500) +- `_check_usb() -> list[CheckResult]` — Candidate robot USB serial devices. (L608) +- `_check_just() -> CheckResult` (L626) +- `_check_reasoner_llm() -> list[CheckResult]` — Model-first doctor dispatcher. Reads `OPENRAL_REASONER_MODEL`, resolves `REASONER_MODELS`, and reports the effective dialect / hosting / endpoint / key status; missing auth gets a follow-up row. An uncurated model fails until `OPENRAL_REASONER_ENDPOINT` + `OPENRAL_REASONER_DIALECT` are explicit, then reports `warn` (untested). The legacy provider contract still diagnoses through `_check_reasoner_llm_legacy` with a separate deprecation row. Never prints the API key value. (L817) +- `_check_reasoner_model(model_key) -> list[CheckResult]` — Resolve one curated/uncurated model into doctor rows; delegates loopback checks to `_reasoner_endpoint_probe_row`. (L740) +- `_reasoner_endpoint_probe_row(label, base_url, *, managed, autostart) -> CheckResult` — Generic loopback probe. Down managed-local + autostart is `info`; managed with autostart disabled and every BYO-local endpoint are `warn`. (L703) +- `_cosmos_autostart_enabled() -> bool` — mirrors the reasoner client's `OPENRAL_COSMOS3_AUTOSTART` parsing (falsy spellings `0/false/no/off`); kept local so doctor never imports the optionally-installed reasoner package. (L659) - `_REASONER_PROVIDERS_REQUIRING_KEY` / `_REASONER_PROVIDER_DEFAULT_BASE_URL` — legacy provider-shim tables only; model-first doctor reads `openral_core.REASONER_MODELS`. -- `_is_local_base_url(url) -> bool` — True when host resolves to a loopback name. (L668) -- `_probe_tcp(host, port, *, timeout_s=0.2) -> bool` — Fast non-blocking TCP probe used to diagnose the Ollama daemon. (L674) -- `_gather_checks() -> list[CheckResult]` (L978) -- `doctor(--json)` — Diagnose host: Python, OS, ROS 2, GPU, USB. Delegates GPU enumeration to `probe_gpus`. (L1011) +- `_is_local_base_url(url) -> bool` — True when host resolves to a loopback name. (L670) +- `_probe_tcp(host, port, *, timeout_s=0.2) -> bool` — Fast non-blocking TCP probe used to diagnose the Ollama daemon. (L676) +- `_gather_checks() -> list[CheckResult]` (L980) +- `doctor(--json)` — Diagnose host: Python, OS, ROS 2, GPU, USB. Delegates GPU enumeration to `probe_gpus`. (L1013) - `detect(--output, --robot/--as, --report, --dds-timeout, --include, --no-write, --deployment, --yes)` — **Always-interactive** custom `robot.yaml` builder (replaces the old `ral init`); there is no `--interactive`/`-i` flag any more — probing + writing a manifest is unconditionally interactive, only `--no-write` short-circuits to non-interactive probe-only inspection (CI-safe: `enrich_cameras=True` reverse-looks-up detected cameras from the sensor catalog instead of prompting, then returns before any prompt). `--report ` is an orthogonal flag that dumps the raw `DetectionReport` JSON to disk and works with or without `--no-write`; used alone (without `--no-write`) it still falls through into the full interactive builder (name prompt → camera wizard → joint/safety gate → writes `robot.yaml`) — a headless/CI probe needs `--no-write` (optionally combined with `--report`), not `--report` by itself. `--robot/--as` forces the canonical base manifest (e.g. `so100`, since a bare Feetech plug-in defaults to the SO-101) over USB/DDS inference — the canonical manifest is a **template**, never the output. Flow: `_prompt_robot_name` asks for a custom rig name (default: canonical name), setting `RobotDescription.name` and — when `--output` is untouched — the default output path `robots//robot.yaml`; `_run_camera_binding_wizard` then walks every detected camera (V4L2 + RealSense + Orbbec) with a thumbnail grab and routes it into either the robot manifest's `sensors:` (canonical name reuse or a new robot sensor) or the DeployScene's workcell `sensors:` (`w:`) — everything but the sensor list is inherited verbatim from the canonical manifest (joints, URDF/MJCF, safety, capabilities, compute); `_maybe_customize_limits` then gates an opt-in per-joint position/velocity/effort + safety-scalar override (`N` default inherits canonical verbatim); `_relocate_file_assets` rewrites `file:` URDF/SRDF refs repo-root-relative when the output dir differs from the canonical rig's dir. `--deployment ` also scaffolds a `DeployScene` (robot_id + the wizard's workcell `sensors:`; safety unset → manifest envelope; no rSkill pinned; a `hal:` binding seeded from the robot manifest's `hal.parameters.defaults` + lerobot calibration placeholders (`id`/`calibration_dir`), with the `port` overridden by the USB probe's detected device when one matched, so the scaffold is a self-contained `deploy run` target once the operator commits the calibration). No effect without `--deployment`. - `_prompt_robot_name(default) -> str` — `typer.prompt` for the custom rig name; blank input keeps `default`. - `_run_camera_binding_wizard(canonical, detection) -> tuple[list[SensorSpec], list[SensorSpec]]` — `(robot_sensors, workcell_sensors)`; the routing prompt loop above over `_iter_wizard_cameras` (V4L2 + RealSense + Orbbec). Enter skips a device, dropping any canonical sensor never bound. @@ -46,16 +46,16 @@ _openral CLI entry point — `openral` command (typer-based)._ - `_render_rskill_search_results(rows, skipped, query) -> None` — Print the `rskill search` table or the no-results notice. - `skill_list(--json)` — List installed rSkills. - `skill_check(rskill_id?, --robot, --rskills-dir, --json)` — Two modes. With a positional id, resolves it via `load_rskill_manifest` and renders a per-section breakdown via `check_single_rskill`. Without an id, falls back to the legacy walk-all path (`check_installed_rskills`). `--rskills-dir` defaults to `rskills/` and is silently skipped when the directory does not exist. Exits 1 on any blocking failure. -- `rskill_new(ID, --out-dir, --owner, --license, --embodiment-tag, --family, --from-hf, --yes, --overwrite)` — Scaffold a new local rSkill from `rskills/template/` via `_rskill_scaffolder.scaffold_rskill`. Three modes: (1) `--from-hf ` introspects the Hub config to auto-fill policy_id / chunk_size / sensors / state_contract / aliases / weights_uri; (2) `--family ` overlays family-aware defaults; (3) interactive prompts for any missing flag (skipped under `--yes`). (L2363) +- `rskill_new(ID, --out-dir, --owner, --license, --embodiment-tag, --family, --from-hf, --yes, --overwrite)` — Scaffold a new local rSkill from `rskills/template/` via `_rskill_scaffolder.scaffold_rskill`. Three modes: (1) `--from-hf ` introspects the Hub config to auto-fill policy_id / chunk_size / sensors / state_contract / aliases / weights_uri; (2) `--family ` overlays family-aware defaults; (3) interactive prompts for any missing flag (skipped under `--yes`). (L2365) - `_resolve_or_prompt(value, *, prompt, default, skip_prompt) -> str` — Drives the owner / license / embodiment prompts only when the flag was not provided and `--yes` is off. - `_resolve_family_and_patch(*, family, from_hf, yes) -> tuple[RSkillFamily | None, RSkillPatch | None]` — Resolves `--family` / `--from-hf` into a family + manifest patch for `scaffold_rskill`. Prompts for family in interactive mode; bails non-zero with a clear message when `--from-hf` introspection fails or `--family` is unrecognized. - `_display_license_banner(name, license_value, version, con) -> None` -- `sensor_list(--vendor, --modality, --kind, --json)` — List entries in sensor catalog. (L2659) -- `sensor_show(SENSOR_ID, --name, --parent-frame, --json)` — Resolve catalog entry to a `SensorSpec`/`SensorBundle`. (L2750) -- `benchmark_report(--rskills-dir, --json)` — Aggregate `rskills/*/eval/*.json` benchmark blocks into a rich-table or JSON dump. Validates every JSON against `RSkillEvalResult`. `--rskills-dir` defaults to `rskills/`. (L3448) -- `benchmark_run(--suite, --rskill, --out, --device, --save-dir, --benchmarks-dir, --task, --n-episodes, --dry-run, --update-manifest/--no-update-manifest, --dashboard, --dashboard-port)` — Resolve `--suite` (built-in id or path) to a bare `list[BenchmarkScene]` via `openral_core.load_benchmark_suite` + `raise_on_invalid_suite`, parse the rSkill reference from `--rskill`, dispatch to `openral_sim.run_benchmark(scenes, vla, suite_id=)`, and write a validated `RSkillEvalResult` JSON. `--task ` runs a single explicit task from the suite (e.g. `libero_spatial/3`, `maniskill3/PushCube-v1`), erroring if it is not in the suite; without it `run_benchmark` auto-filters the suite to the rSkill's `evaluated_tasks` (run only what it supports, skip + log the rest). `--n-episodes` overrides every `BenchmarkScene.n_episodes` in the suite for smoke runs (mirrors `benchmark scene --n-episodes`). Default output path is `rskills//eval/.json`. With `--update-manifest` (default on), also writes `avg_success_rate` back into the manifest's `benchmarks.` field via `update_rskill_benchmarks`. (L2843) -- `benchmark_scene(--config, --rskill, --out, --device, --save-dir, --save-video, --video-size, --n-episodes, --view/--no-view, --dry-run, --update-manifest/--no-update-manifest, --dashboard, --dashboard-port)` — Single-scene sibling of `benchmark_run` (scene-hierarchy refactor). `--view/--no-view` (default unset = headless) mirrors `sim run --view` for parity — opens a live `mujoco.viewer` per episode; threaded into `run_benchmark_scene(view=…)`. `--save-video DIR` (with `--video-size`, default 1024) sets `run_benchmark_scene(record_video=True)` and writes a clean single-view world MP4 per episode (`__.mp4`) + a `videos.json` manifest via `openral_sim._website_video.write_world_videos` — for website hero clips; the task slug prevents benchmark scenes sharing a backend from overwriting each other; pair with `--n-episodes 1`. Strictly accepts a `BenchmarkScene` YAML via `load_scene_strict` (DeployScene/SimScene rejected with a redirect), optionally overrides `n_episodes` for smoke runs, dispatches to `openral_sim.run_benchmark_scene`, writes `rskills//eval/scene_.json`, and surgically updates the rSkill manifest's `benchmarks.` field. `--dry-run` short-circuits before rSkill resolution so it never touches the Hub. (L3152) -- `_default_benchmark_scene_out_path(vla_spec, scene) -> Path` — Mirrors `_default_benchmark_out_path` but for single-scene JSONs; the `scene_` prefix distinguishes per-scene outputs from multi-task suite outputs under the same rSkill directory. (L3431) +- `sensor_list(--vendor, --modality, --kind, --json)` — List entries in sensor catalog. (L2661) +- `sensor_show(SENSOR_ID, --name, --parent-frame, --json)` — Resolve catalog entry to a `SensorSpec`/`SensorBundle`. (L2752) +- `benchmark_report(--rskills-dir, --json)` — Aggregate `rskills/*/eval/*.json` benchmark blocks into a rich-table or JSON dump. Validates every JSON against `RSkillEvalResult`. `--rskills-dir` defaults to `rskills/`. (L3476) +- `benchmark_run(--suite, --rskill, --out, --device, --save-dir, --benchmarks-dir, --task, --n-episodes, --dry-run, --update-manifest/--no-update-manifest, --video/--no-video, --video-dir, --dashboard, --dashboard-port)` — Resolve `--suite` (built-in id or path) to a bare `list[BenchmarkScene]` via `openral_core.load_benchmark_suite` + `raise_on_invalid_suite`, parse the rSkill reference from `--rskill`, dispatch to `openral_sim.run_benchmark(scenes, vla, suite_id=)`, and write a validated `RSkillEvalResult` JSON. `--video` (default **on**) records one world MP4 per episode into `--video-dir` (default `/videos/`) through `run_benchmark(video_dir=…)`; `--no-video` restores allocation-light runs. `--task ` runs a single explicit task from the suite (e.g. `libero_spatial/3`, `maniskill3/PushCube-v1`), erroring if it is not in the suite; without it `run_benchmark` auto-filters the suite to the rSkill's `evaluated_tasks` (run only what it supports, skip + log the rest). `--n-episodes` overrides every `BenchmarkScene.n_episodes` in the suite for smoke runs (mirrors `benchmark scene --n-episodes`). Default output path is `rskills//eval/.json`. With `--update-manifest` (default on), also writes `avg_success_rate` back into the manifest's `benchmarks.` field via `update_rskill_benchmarks`. (L2845) +- `benchmark_scene(--config, --rskill, --out, --device, --save-dir, --save-video, --video-size, --n-episodes, --view/--no-view, --dry-run, --update-manifest/--no-update-manifest, --dashboard, --dashboard-port)` — Single-scene sibling of `benchmark_run` (scene-hierarchy refactor). `--view/--no-view` (default unset = headless) mirrors `sim run --view` for parity — opens a live `mujoco.viewer` per episode; threaded into `run_benchmark_scene(view=…)`. `--save-video DIR` (with `--video-size`, default 1024) sets `run_benchmark_scene(record_video=True)` and writes a clean single-view world MP4 per episode (`__.mp4`) + a `videos.json` manifest via `openral_sim._website_video.write_world_videos` — for website hero clips; the task slug prevents benchmark scenes sharing a backend from overwriting each other; pair with `--n-episodes 1`. Strictly accepts a `BenchmarkScene` YAML via `load_scene_strict` (DeployScene/SimScene rejected with a redirect), optionally overrides `n_episodes` for smoke runs, dispatches to `openral_sim.run_benchmark_scene`, writes `rskills//eval/scene_.json`, and surgically updates the rSkill manifest's `benchmarks.` field. `--dry-run` short-circuits before rSkill resolution so it never touches the Hub. (L3180) +- `_default_benchmark_scene_out_path(vla_spec, scene) -> Path` — Mirrors `_default_benchmark_out_path` but for single-scene JSONs; the `scene_` prefix distinguishes per-scene outputs from multi-task suite outputs under the same rSkill directory. (L3459) - `deploy sim(--config, --robot, --dashboard-port, --foxglove/--no-foxglove, --foxglove-port, --reset-to-pose-service, --hal, --memory-dir, --initial-task, --dry-run)` — Boot the full ROS graph (dashboard + C++ safety_kernel + reasoner + prompt_router + runtime + HAL) against a digital-twin HAL by sharding `ros2 launch openral_rskill_ros sim_e2e.launch.py` (one generic launch — no per-robot launch files). `--foxglove` (default off) also spawns the read-only `foxglove_bridge` live-scene surface on `ws://127.0.0.1:` (view-only — cannot actuate; see `packages/openral_foxglove_bringup`). `--initial-task` (optional string) is the single operator goal delivered to the reasoner at startup; when omitted no startup prompt is set and the reasoner idles until a manual `openral prompt` or dashboard prompt arrives. Loads a `DeployScene` YAML (mirrors `openral sim run --config`); picks the HAL package/executable/node-name from `_ROBOT_HAL_REGISTRY[robot_id]`; **asserts the registered HAL's `supported_robot_names` matches the manifest's `name` field** (mismatch fails loud at resolution time). **No envelope YAML is written or read**: the launch's `OpaqueFunction` loads `robot.yaml`, calls `openral_safety.envelope_loader.compute_intersection(robot, skill=None)`, and forwards each `EnvelopeIntersection` field as a ROS parameter on the kernel node (the C++ kernel grew a parameter-based loader alongside the legacy file path). `--robot` overrides the YAML's `robot_id`. `--hal key=value` (repeatable) overrides per-robot HAL defaults (JSON-parsed where possible). No `--rskill` flag: the reasoner picks the active rSkill dynamically from the in-tree `rskills/` palette at `on_configure`. `--memory-dir` points at a deploy memory bundle directory; `_memory_bundle_launch_args` derives `memory_md_path:=/MEMORY.md` (always — the reasoner creates it on the first `memory_write`) plus `spatial_memory_path:=/scene_graph.json` and `map_path:=/map.yaml` (each only when the file is present), so a deploy boots with the self-maintained memory + the 3D scene graph (`recall_object`) + the 2D occupancy grid (nav2 `map_server`). The dir must exist (raises `ROSConfigError` otherwise); `--memory-dir` overrides the `DeployScene.memory_dir` field. `--dry-run` prints the resolved argv without writing the HAL params temp file. Defined in `openral_cli.deploy_sim`. - `dashboard(--host, --port, --log-level, --inprocess)` — `openral dashboard` (closes #44). Boot a live debug pane that doubles as an OTLP/HTTP receiver on the same port; lazy-imports `openral_observability.dashboard.run_dashboard` so `openral --help` stays sub-second. `--inprocess` takes a single shell-quoted string (shlex-tokenised) and spawns it as a child workload with `OTEL_EXPORTER_OTLP_ENDPOINT` + `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` pre-set so a one-keystroke demo doesn't need a second shell. Defaults: bind `127.0.0.1:4318` (OTLP/HTTP standard; was `8000` until issue #132), uvicorn at `warning`. Works without Jaeger/Tempo (the dashboard is its own receiver). Inverse path: `openral sim run --dashboard` (sim spawns the dashboard). - `replay(BAG, --trace, --frame, --dataset-root, --dashboard, --out)` — ISSUE-109. Read a `.mcap` file or rosbag2 directory, join with OTel spans from `--dashboard` (`http://host:port`), and emit a chronological JSON timeline keyed by `trace_id`. `--frame //` (with `--dataset-root`) pivots from a written LeRobotDataset frame: it resolves that frame's `trace_id` via `openral_dataset.read_frame_trace` and uses it as the join key (mutually exclusive with `--trace`). `--out` writes to a file; otherwise prints to stdout. Bag-only when `--dashboard` is omitted. @@ -64,7 +64,8 @@ _openral CLI entry point — `openral` command (typer-based)._ - `profile session ACTION (start | stop | view) (--output, --name)` — Drive an LTTng session for the realtime hot path; surfaces `LttngSessionError` cleanly when `lttng` is missing on PATH. Set `OPENRAL_ROS2_TRACING=1` on the agent process to emit tracepoints; without the gate every tracepoint is a no-op. - `_resolve_benchmark_suite(suite: str, benchmarks_dir: Path) -> tuple[list[BenchmarkScene], str]` — Accept either a built-in id (looked up at `benchmarks/.yaml`) or a direct YAML path; raise `typer.BadParameter` listing catalogue entries on a typo. - `_parse_rskill_cli_arg(raw)` — Parse `--rskill ` into a `VLASpec`. Accepts bare names (`smolvla-libero`), paths (`rskills/smolvla-libero`), or HF repo ids; validates via `openral_rskill.loader._validate_skill_ref` so `VLASpec.weights_uri` rejects explicit URI schemes. The adapter id is read from the manifest's `model_family`. Raises `typer.BadParameter` on an invalid scheme or empty input. -- `_summarize_results(results: dict[str, object]) -> str` — Headline-line picker for free-form `results` blocks (`*_avg` → numeric → status fallback). (L3538) +- `behavior_serve(--rskill, --task, --instruction, --host, --port, --device, --state-dim, --action-dim)` — Serve one rSkill through the official BEHAVIOR Challenge WebSocket policy protocol. Defaults to the R1Pro observation/action contract; OmniGibson remains out-of-process and owns task loading, metrics, and videos. +- `_summarize_results(results: dict[str, object]) -> str` — Headline-line picker for free-form `results` blocks (`*_avg` → numeric → status fallback). (L3566) - `_path_completer(text: str, state: int) -> str | None` — Stdlib `readline`-shaped Tab completer wired into `_run_repl`. Globs `text*` (with `~` expansion), adds trailing `/` to directory matches, and rewrites a leading `$HOME` back to `~` so a user who typed `~/foo` keeps their literal tilde. Lets the REPL complete filesystem paths after `sim run --config `, `--rskill rskills/`, etc. Installed by `_run_repl` together with `readline.set_completer_delims(" \t\n=;|&><")` (shell-shaped delims so `/`, `.`, `-`, `~` are not word boundaries) and `parse_and_bind("tab: complete")` (or libedit's equivalent on macOS). Returns `None` past the last match per readline's state contract. - `render_banner(version_str: str, *, width: int | None = None) -> RenderableType` — Build the interactive-REPL welcome box as a rich `Panel` (Claude-Code style), content-sized (`expand=False`) and with the layout chosen to fit `width`: a white-bordered rounded box with `OPENRAL v` inline (left-aligned) in the top border. **Wide** (`>= _WIDE_MIN`): two columns split by a `MINIMAL`-box vertical divider — left holds the logo mark beside the OPENRAL wordmark (`_logo_wordmark`/`_identity`) over the tagline + capability strip; right holds the community links (`_LINKS`) above a `Rule` above the quick-start commands (`_COMMANDS`). **Narrow**: a single stacked column keeping every section, with the logo beside the wordmark while it fits (`>= _SIDE_BY_SIDE_MIN`) and stacked above it below that. The box is sized to its content rather than stretched to the terminal, so it stays compact on wide terminals (already-printed output cannot reflow if the window is later dragged narrower than the box). Returns a renderable (not a print) so it exports to plain text in tests independent of TTY/colour state. - `_logo_wordmark(*, stacked: bool) -> RenderableType` — Logo mark + OPENRAL wordmark side by side (vertical-middle grid) or stacked. diff --git a/docs/methods/10-tools.md b/docs/methods/10-tools.md index a1b8064..f6a1226 100644 --- a/docs/methods/10-tools.md +++ b/docs/methods/10-tools.md @@ -30,15 +30,15 @@ _Real GPU rollout audit for every YAML under `scenes/`. Operator-driven (not a p - `RunMode = Literal["sim", "benchmark", "deploy"]` (L61) — Tier selector on each `ConfigSpec` row; drives `_run_one` vs `_run_one_deploy` dispatch. - `@dataclass(frozen=True) class ConfigSpec(config, rskill, uv_group, run_mode)` (L65) — One row in the audit catalogue. `uv_group` is one of `libero / metaworld / robocasa / maniskill3 / simpler-env / sim`; `run_mode` is `"sim"` / `"benchmark"` / `"deploy"`; `rskill` is `""` for deploy rows (env-only — reasoner picks at runtime). Catalogue holds only pairs that actually exist in the tree — scenes without a matching in-tree rSkill are tracked in the scene YAML itself (and in `tests/unit/test_examples_sim_configs_load.py` for schema-load coverage), not as audit rows. - `CATALOGUE: tuple[ConfigSpec, ...]` (L99) — Explicit (YAML → rSkill → uv group → run_mode) mapping for every config currently in the tree: 13 sim + 7 benchmark + 4 deploy = 24 rows. -- `@dataclass class AuditRow(config, rskill, status, exit_code, wall_s, peak_vram_mib, tail)` (L188) — One result. `status` ∈ {`pass`, `pass-compat`, `fail-oom`, `fail-asset`, `fail-sidecar`, `fail-timeout`, `fail-other`, `fail-compat`, `skipped-opt-dep`, `skipped-host-setup`}. -- `_classify(returncode: int, tail: str) -> str` (L243) — Map subprocess result to a status by substring-matching stderr against `_OOM_PATTERNS` / `_ASSET_PATTERNS` / `_SIDECAR_PATTERNS` / `_OPT_DEP_PATTERNS` / `_HOST_SETUP_PATTERNS`. Exit 139 (MuJoCo/GL atexit SIGSEGV in gym-aloha) is treated as `pass` when no error patterns appear. -- `class _VramSampler` (L282) — Background `nvidia-smi --query-gpu=memory.used` poller, 200 ms cadence; `peak_mib` reported on `.stop()`. No-op without `nvidia-smi` on `$PATH`. -- `_check_compat(spec: ConfigSpec) -> AuditRow` (L327) — `--check-compatibility` gate: load scene via `openral_core.load_scene_strict`, validate rSkill manifest (sim/benchmark) or assert robot resolves in `openral_cli.deploy_sim._ROBOT_HAL_REGISTRY` (deploy). No subprocess, no GPU. Returns `pass-compat` / `fail-compat`. -- `_build_run_cmd(spec: ConfigSpec) -> list[str]` (L424) — Build the `uv run … openral …` argv for sim/benchmark rows. Refactored out of `_run_one` so the deploy path can stay focused on lifecycle teardown. -- `_run_one_deploy(spec, *, alive_grace_s, shutdown_grace_s, timeout_s) -> AuditRow` (L473) — Tier-2 deploy launch via `openral deploy sim --config --no-dashboard`: `Popen` in its own process group, wait `alive_grace_s`, send SIGINT to the group, wait `shutdown_grace_s`, escalate to SIGKILL on timeout. Pass criteria: banner seen in stdout AND returncode in `{0, -SIGINT, 130, -SIGTERM}`. -- `_classify_or_fallback(returncode, tail, spec, wall_s, peak_vram) -> AuditRow` (L658) — Deploy-mode wrapper around `_classify` that defaults to `fail-other` when no pattern matches (sim path defaults to `pass`). -- `_run_one(spec: ConfigSpec, timeout_s: int) -> AuditRow` (L703) — Tier-3 sim/benchmark rollout via `_build_run_cmd(spec)` with `MUJOCO_GL=egl` and `OPENRAL_SIM_SEQUENTIAL_INIT=1`. -- `main(argv) -> int` (L850) — CLI entry; flags `--timeout` / `--deploy-alive-grace` / `--deploy-shutdown-grace` / `--check-compatibility` / `--report`. Returns 0 on all-pass, 1 if any config failed, 2 on filter mismatch. +- `@dataclass class AuditRow(config, rskill, status, exit_code, wall_s, peak_vram_mib, tail)` (L195) — One result. `status` ∈ {`pass`, `pass-compat`, `fail-oom`, `fail-asset`, `fail-sidecar`, `fail-timeout`, `fail-other`, `fail-compat`, `skipped-opt-dep`, `skipped-host-setup`}. +- `_classify(returncode: int, tail: str) -> str` (L250) — Map subprocess result to a status by substring-matching stderr against `_OOM_PATTERNS` / `_ASSET_PATTERNS` / `_SIDECAR_PATTERNS` / `_OPT_DEP_PATTERNS` / `_HOST_SETUP_PATTERNS`. Exit 139 (MuJoCo/GL atexit SIGSEGV in gym-aloha) is treated as `pass` when no error patterns appear. +- `class _VramSampler` (L289) — Background `nvidia-smi --query-gpu=memory.used` poller, 200 ms cadence; `peak_mib` reported on `.stop()`. No-op without `nvidia-smi` on `$PATH`. +- `_check_compat(spec: ConfigSpec) -> AuditRow` (L334) — `--check-compatibility` gate: load scene via `openral_core.load_scene_strict`, validate rSkill manifest (sim/benchmark) or assert robot resolves in `openral_cli.deploy_sim._ROBOT_HAL_REGISTRY` (deploy). No subprocess, no GPU. Returns `pass-compat` / `fail-compat`. +- `_build_run_cmd(spec: ConfigSpec) -> list[str]` (L431) — Build the `uv run … openral …` argv for sim/benchmark rows. Refactored out of `_run_one` so the deploy path can stay focused on lifecycle teardown. +- `_run_one_deploy(spec, *, alive_grace_s, shutdown_grace_s, timeout_s) -> AuditRow` (L480) — Tier-2 deploy launch via `openral deploy sim --config --no-dashboard`: `Popen` in its own process group, wait `alive_grace_s`, send SIGINT to the group, wait `shutdown_grace_s`, escalate to SIGKILL on timeout. Pass criteria: banner seen in stdout AND returncode in `{0, -SIGINT, 130, -SIGTERM}`. +- `_classify_or_fallback(returncode, tail, spec, wall_s, peak_vram) -> AuditRow` (L665) — Deploy-mode wrapper around `_classify` that defaults to `fail-other` when no pattern matches (sim path defaults to `pass`). +- `_run_one(spec: ConfigSpec, timeout_s: int) -> AuditRow` (L710) — Tier-3 sim/benchmark rollout via `_build_run_cmd(spec)` with `MUJOCO_GL=egl` and `OPENRAL_SIM_SEQUENTIAL_INIT=1`. +- `main(argv) -> int` (L857) — CLI entry; flags `--timeout` / `--deploy-alive-grace` / `--deploy-shutdown-grace` / `--check-compatibility` / `--report`. Returns 0 on all-pass, 1 if any config failed, 2 on filter mismatch. ### `tools/select_tests.py` _Selective test execution — maps a git diff to the minimal pytest targets that can observe it. Backs `just test-changed` / the `test-selective` workflow. See [`docs/contributing/selective-testing.md`](../contributing/selective-testing.md)._ @@ -140,6 +140,19 @@ _Boot helper for the curated NVIDIA Cosmos 3 Edge reasoner model (`OPENRAL_REASO - `build_serve_argv(*, vllm_bin, model, host, port, tool_call_parser, max_model_len, gpu_memory_utilization, enforce_eager, served_model_name=None, kv_cache_dtype="auto") -> list[str]` — the `vllm serve` argv. `--max-model-len` (default 8192) holds the reasoner prompt + tools; `--enforce-eager` + `--gpu-memory-utilization` (default 0.90, `$OPENRAL_COSMOS3_GPU_MEM_UTIL`) + `--kv-cache-dtype fp8` are the 8 GB-fit knobs (fp8 KV verified required for the 8192 window with the native Edge impl on an 8 GB 4070). - `main() -> int` — argparse (`--model` default `nvidia/Cosmos3-Edge`, `--host`, `--port` default 8901, `--tool-call-parser`, `--max-model-len`, `--gpu-memory-utilization`, `--no-enforce-eager`, `--kv-cache-dtype`, `--home`, `--venv`); strips `PYTHONPATH`/`PYTHONHOME`, sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` (fragmentation OOM fix verified live), resolves the served view, and `os.execvpe`s into `vllm serve`. +### `tools/behavior_groot_sidecar.py` +_Python 3.10 sidecar for the official 2026 BEHAVIOR-1K GR00T N1.7 checkpoint. Imports the pinned `wensi-ai/Isaac-GR00T` behavior runtime, registers the official R1Pro modality slices, wraps `Gr00tPolicy` with `B1KPolicyWrapper`, and serves ZMQ `ping/reset/get_action/close`. Two 8 GB-host memory measures: whole-model NF4 (`--quantization nf4`, default) and dropping the unused Qwen3-VL `lm_head` (the wrapper reads only hidden states; ~840 MiB saved — 2.77 GiB inference peak, live-validated alongside OmniGibson)._ + +- `main(argv) -> int` — Parse checkpoint/task/instruction/control-mode/device/endpoint plus `--quantization {none,nf4,int8}` / `--nf4-min-params` (the min-params threshold applies to both quantizers), set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, load the real checkpoint (CPU-first under NF4/int8), and serve the 23-D action API. `OPENRAL_BEHAVIOR_GROOT_DUMP_OBS=` pickles the first 32 observations for offline quantization A/Bs. +- `_drop_backbone_lm_head(model) -> None` — Replace the Qwen3-VL `lm_head` with `Identity`; the B1K wrapper consumes only `hidden_states[-1]`, so the full-vocab logits projection is dead weight and was the largest single inference allocation. +- `_quantize_nf4(model, *, device, min_params) -> None` — Whole-model `bitsandbytes` `Linear4bit` rewrite of large `nn.Linear`s (manifest pins `nf4_min_params: 1000000`), keeping small layers floating; patches the model `dtype` property to the first floating parameter dtype, then moves to CUDA. Offline A/B/C on captured obs: 2531 MiB peak, action MAE 0.0029 vs bf16 — behaviourally lossless for this checkpoint. +- `_quantize_int8(model, *, device, min_params) -> None` — Same rewrite with `bitsandbytes` `Linear8bitLt` (LLM.int8, threshold 6.0). 3678 MiB peak / MAE 0.0017 vs bf16 on the same captured obs — too big next to OmniGibson on 8 GB, so its role is offline quality comparison, not co-resident serving. + +### `tools/behavior_scene_sidecar.py` +_Official OmniGibson evaluator environment sidecar for `scene.id=behavior`. Resolves the configured task instance, exposes the challenge wrapper observation, applies external 23-D actions, and returns success/metrics/sim time over ZMQ._ + +- `main(argv) -> int` — Parse task/instance/mode/wrapper/max-steps/endpoint, load the real BEHAVIOR environment, and serve `ping/reset/step/close`. + ### `tools/_robometer_scorer.py` _In-process stateless scorer for the Robometer-4B reward monitor, companion to `openral_runner.backends.reward.robometer_reward.RobometerInProcessReward`._ `reward_monitor_node` imports `_robometer_scorer.py::_Scorer` directly; there is no separate Robometer ZMQ process or dedicated venv. As of lerobot 0.6.0 the reward model is lerobot's in-tree `lerobot.rewards.robometer.RobometerRewardModel` — a vanilla `AutoModelForImageTextToText` (Qwen3-VL-4B) loaded with plain `transformers`. There is **no** pinned `robometer` git package and **no** `transformers==4.57.1` force-pin. The scorer keeps OpenRAL's NF4 pre-quantized checkpoint (`OpenRAL/rskill-robometer_4b-any-general-nf4`, ~3.3 GB resident), meta-builds the native `RobometerRewardModel` skeleton and drops the packed 4-bit weights (remapped into the native module) in directly — no bf16 spike, no Qwen weight download. Validated live: 3.33 GB NF4, progress ramps to 0.88 + success 0.90 at task completion. diff --git a/docs/methods/11-ros2-nodes.md b/docs/methods/11-ros2-nodes.md index 2de2910..6b26e66 100644 --- a/docs/methods/11-ros2-nodes.md +++ b/docs/methods/11-ros2-nodes.md @@ -10,7 +10,7 @@ Thin wrappers around the Python-layer adapters; each exposes a single - `packages/openral_hal_franka/openral_hal_franka/lifecycle_node.py` — Franka Panda HAL. - `packages/openral_hal_ur5e/openral_hal_ur5e/lifecycle_node.py` — UR5e HAL. - `packages/openral_hal_ur10e/openral_hal_ur10e/lifecycle_node.py` — UR10e HAL. -- `packages/world_state/openral_world_state_ros/lifecycle_node.py` — `main() -> None` — `WorldStateAggregator` lifecycle node. Publishes typed `openral_msgs/WorldStateStamped` on `/openral/world_state_fast` (`publish_rate_hz_fast`, default 30 Hz) and `/openral/world_state_slow` (`publish_rate_hz_slow`, default 5 Hz) from one in-memory snapshot per fast tick. JSON `/world_state` topic removed. Module exposes `TOPIC_FAST`, `TOPIC_SLOW`, and `build_world_state_stamped_msg(node, world_state) -> WorldStateStamped` (L849) — pure-Python translation of `openral_core.WorldState` to the typed message (sorted parallel arrays for determinism; `trace_id` set from current OTel context via `openral_observability.propagation.current_traceparent`). `build_world_state_stamped_msg` now also serialises the spatial memory onto the `detected_object_*` parallel arrays (labels / confidences / `geometry_msgs/Point` positions / `int32` track-ids (`-1` = unset) / frame) via the private helper `_fill_detected_objects(msg, world_state) -> None` (L999), so separate-process consumers of `/openral/world_state_slow` now see `WorldState.detected_objects` on the wire. `__init__` accepts an optional pre-constructed `WorldStateAggregator` so the compose factory can share one instance with `rskill_runner_node` (single shared aggregator). +- `packages/world_state/openral_world_state_ros/lifecycle_node.py` — `main() -> None` — `WorldStateAggregator` lifecycle node. Subscribes `/joint_states`, `/openral/policy_state`, camera/perception streams; publishes typed fast/slow world-state summaries. The composed runtime shares the same aggregator with `rskill_runner_node`, so simulator-native policy state reaches the active rSkill without an untyped side channel. - `world_state_from_idl(msg) -> WorldState` (L717) — Symmetric inverse of `build_world_state_stamped_msg`: reconstructs an `openral_core.WorldState` from a `WorldStateStamped` msg (joint/base/EE/diagnostic state plus the `detected_object_*` parallel arrays → `list[DetectedObject]`). Used by the reasoner_node to feed real state into its `ContextRenderer`. `image_frames` is always `None` (the IDL carries image topic refs, not inline pixels). - `packages/openral_perception_ros/openral_perception_ros/ros_image_detector_node.py` — `main(args=None) -> None` (L51) — **Standalone ROS-Image object detector (no GStreamer).** Defines the node class lazily inside `main()` behind ROS imports: `RosImageObjectDetectorNode(LifecycleNode)` — subscribes a camera `sensor_msgs/Image` on `image_topic`, runs an `openral_runner` detector backend, and publishes `openral_msgs/PromptStamped` (carrying the detector's `ObjectsMetadata` as `metadata_json`) on `output_topic` (default `/openral/perception/objects`), `header.frame_id` stamped with `sensor_id`. **Backend selection** (`_build_detector`, 2026-06-09 amendment): with a `manifest_path` param set, builds via `openral_runner...detector_factory.build_manifest_detector` (RT-DETR ONNX for `runtime: onnx`, or the open-vocab `LocateAnythingDetector` / `VLM_SIDECAR` for `runtime: pytorch`); without it, the legacy `ObjectsDetector` (RT-DETR ONNX) from `onnx_path` + `labels`. **Detector mode** — `_resolve_wiring` reads the manifest's `detector.mode` at `on_configure` (via `detector_node_wiring`): a `continuous` detector runs the primary camera's detect+publish leg and does NOT expose `locate_in_view` / subscribe `detector_query`; an `on_demand` detector exposes the service + query topic and does NOT publish continuously (frames still cached). Legacy ONNX path (no manifest) = continuous. **Dynamic query** (`_on_query`, on_demand only): an initial `query` param is applied and the node subscribes a `std_msgs/String` `query_topic` (default `/openral/perception/detector_query`) to retarget the persistent query live. **Camera-agnostic**: a `cameras` param maps logical ids → image topics (falling back to a single `image_topic` under `primary_camera`); every camera's latest frame is cached (`_cache_frame`), so no camera name is baked in. **locate_in_view service (on_demand only)**: serves `/openral/perception/locate_in_view` (`openral_msgs/srv/LocateInView`) — a read-only one-shot detect of a requested query on a requested camera's cached frame (`_on_locate_in_view`, uses `detect_with_query` so the persistent query is untouched); offered only for `on_demand` detectors with the IDL built. Params: `cameras`, `primary_camera`, `image_topic`, `output_topic`, `sensor_id`, `onnx_path`, `manifest_path`, `model_id`, `score_threshold`, `input_size`, `max_rate_hz`, `labels`, `query`, `query_topic`. **Lifecycle** — `RosImageObjectDetectorNode(LifecycleNode)`: the GPU detector backend is built on `on_activate` and released on `on_deactivate`/`on_cleanup` (`_release_detector`) so the reasoner can free its VRAM before a co-resident grab policy loads. **Continuous-leg observability (issue #12)** — `_detect_and_publish` never swallows a per-frame outcome silently: `classify_continuous_tick(*, error, detection_count) -> tuple[str, str]` (L63, module-level, pure) maps a tick to a `(log_level, message)` — a `detect()` exception → `warning` (a crashing/OOM detector must be visible, not hidden, since the "publish nothing when nothing is seen" contract makes a silent crash look identical to a quiet scene), an empty result → `info` liveness heartbeat, a non-empty result → `debug`; the node logs it via `_log_throttled` (`throttle_duration_sec=5.0`) without changing what lands on the bus. **DEBUG-on-demand** — `_apply_env_log_level` (on_configure) honours the `DETECTOR_LOG_LEVEL_ENV` (`OPENRAL_DETECTOR_LOG_LEVEL`) env var via `normalize_log_level(value) -> str | None` (L83, pure: case-insensitive, `warning`→`WARN`, unknown→`None`) + `rclpy.logging.set_logger_level`, so `OPENRAL_DETECTOR_LOG_LEVEL=debug openral deploy sim …` surfaces the per-publish DEBUG line the default INFO console hides (the wrapped launch can't easily inject `--ros-args --log-level`). Best-effort producer. Launched via `openral deploy sim --enable-object-detector` (`--object-detector-manifest` selects the VLM); the reasoner offers `locate_in_view` when `detector_available` is set. - `packages/openral_perception_ros/openral_perception_ros/scene_vlm_node.py` — `main(args=None) -> None` — **Scene-VLM query service node.** Defines `SceneVlmNode(Node)` lazily inside `main()` behind ROS imports: subscribes one or more camera `sensor_msgs/Image` streams, caches each camera's latest BGR frame (`_make_cache_cb`), and builds a `QwenSceneVlm` backend from a `kind:"vlm"` `manifest_path` (`_build_vlm` → `build_scene_vlm`). **query_scene service**: serves `/openral/perception/query_scene` (`openral_msgs/srv/QueryScene`) — read-only, on-demand "answer this question about camera Y's current frame" (`_on_query_scene` → `QwenSceneVlm.query`, returns free text); offered only if the IDL is built. The scene-reasoning counterpart of `ros_image_detector_node` (which serves `locate_in_view`) — separate node because a scene VLM is a reasoning aid, not a continuous detector. Params: `cameras`, `primary_camera`, `image_topic`, `manifest_path` (required), `sidecar_host`, `sidecar_port`. The reasoner offers `query_scene` when `scene_query_available` is set. diff --git a/docs/reference/robots.md b/docs/reference/robots.md index 6a1da65..6dfa82c 100644 --- a/docs/reference/robots.md +++ b/docs/reference/robots.md @@ -22,6 +22,7 @@ Every embodiment is a typed `RobotDescription` manifest under `robots/ | Rethink Sawyer | [`robots/sawyer/`](https://github.com/OpenRAL/openral/tree/master/robots/sawyer/) | eval-only · real-HW HAL planned (#57) | ✓ sim | | Fourier GR1 | [`robots/gr1/`](https://github.com/OpenRAL/openral/tree/master/robots/gr1/) | eval-only (RoboCasa GR1 fork + RLDX-1) | ✓ sim | | Panda mobile (RoboCasa kitchen) | [`robots/panda_mobile/`](https://github.com/OpenRAL/openral/tree/master/robots/panda_mobile/) | eval-only (drives RoboCasa kitchen via robosuite) | ✓ sim | +| Galaxea R1 Pro | [`robots/r1pro/`](https://github.com/OpenRAL/openral/tree/master/robots/r1pro/) | generic scene-attached HAL + official BEHAVIOR/OmniGibson sidecar | ✓ sim | | Google Robot (SimplerEnv) | [`robots/google_robot/`](https://github.com/OpenRAL/openral/tree/master/robots/google_robot/) | eval-only (SimplerEnv `fractal20220817_data` bridge env) | ✓ sim | | WidowX (SimplerEnv) | [`robots/widowx/`](https://github.com/OpenRAL/openral/tree/master/robots/widowx/) | eval-only (SimplerEnv `bridge_orig` env) | ✓ sim | | PushT 2-D (gym-pusht) | [`robots/pusht_2d/`](https://github.com/OpenRAL/openral/tree/master/robots/pusht_2d/) | eval-only (`pymunk` 2-D rigid-body) | ✓ sim | diff --git a/docs/reference/rskills.md b/docs/reference/rskills.md index 6c98971..8e18533 100644 --- a/docs/reference/rskills.md +++ b/docs/reference/rskills.md @@ -61,6 +61,7 @@ All entries are published under `OpenRAL/rskill-*` on HuggingFace Hub and exerci | [`openvla-oft-simpler-widowx-nf4`](https://github.com/OpenRAL/openral/tree/master/rskills/openvla-oft-simpler-widowx-nf4/) | OpenVLA-OFT (RLinf PPO ManiSkill3 PutOnPlateInScene25; NF4) | `widowx` | MIT — transformers custom-code; validated 2/5 on SimplerEnv carrot | | [`gr00t-n17-libero`](https://github.com/OpenRAL/openral/tree/master/rskills/gr00t-n17-libero/) | NVIDIA Isaac GR00T N1.7 (3B, Cosmos-Reason2-2B VLM backbone) | `franka_panda` | NVIDIA Open Model License (commercial OK) — in-process lerobot 0.6.0 `GrootPolicy`, backbone-only NF4 | | [`gr00t-n17-so101-fruit`](https://github.com/OpenRAL/openral/tree/master/rskills/gr00t-n17-so101-fruit/) | GR00T N1.7 (3B) SO-101 fruit pick-and-place (`new_embodiment`, 6-D) | `so101_follower` | NVIDIA Open Model License (commercial OK) — in-process `GrootPolicy` whole-model NF4 (`quantize_scope: model`); GPU-verified 5.8 GiB peak on 8 GB | +| [`gr00t-n17-b1k-turning-on-radio`](https://github.com/OpenRAL/openral/tree/master/rskills/gr00t-n17-b1k-turning-on-radio/) | Official 2026 BEHAVIOR-1K GR00T N1.7 `turning_on_radio` checkpoint | `r1pro` | License unknown for the organizer Drive artifact; evaluator bridge + deploy-sim, pinned Isaac-GR00T Python 3.10 sidecar, 61-D state / atomic 23-D mixed action | | [`smolvla-so101-pen`](https://github.com/OpenRAL/openral/tree/master/rskills/smolvla-so101-pen/) | SmolVLA SO-101 pen checkpoint | `so101_follower` | Apache-2.0 | | [`smolvla-so101-pick-place-pen`](https://github.com/OpenRAL/openral/tree/master/rskills/smolvla-so101-pick-place-pen/) | SmolVLA SO-101 pick/place pen checkpoint; optional split ONNX/TensorRT fast path | `so101_follower` | Apache-2.0 | | [`smolvla-robotwin`](https://github.com/OpenRAL/openral/tree/master/rskills/smolvla-robotwin/) | SmolVLA finetuned on RoboTwin 2.0 (50 bimanual SAPIEN tasks) | `aloha_agilex` | Apache-2.0 — py3.10 SAPIEN sidecar | diff --git a/docs/reference/schemas/Action.json b/docs/reference/schemas/Action.json index f8f6116..f806041 100644 --- a/docs/reference/schemas/Action.json +++ b/docs/reference/schemas/Action.json @@ -74,7 +74,7 @@ "type": "object" } }, - "description": "A single action step or chunk produced by a Skill.\n\nAttributes:\n control_mode: Target action space.\n horizon: Number of steps (1 = single step, H = chunk).\n joint_targets: Joint position targets, shape ``(H, N)``.\n joint_velocities: Joint velocity targets, shape ``(H, N)``.\n joint_torques: Joint torque targets, shape ``(H, N)``.\n cartesian_pose: EE pose targets.\n cartesian_delta: EE pose deltas.\n cartesian_twist: EE velocity targets.\n body_twist: Base twist targets.\n foot_placements: Discrete footstep targets.\n gripper: Gripper commands in [0, 1].\n dex_hand_joints: Dexterous hand joint targets.\n confidence: rSkill confidence in [0, 1].\n stamp_ns: Action timestamp in nanoseconds.\n ee_name: Target end-effector name.\n frame_id: Reference frame for Cartesian actions.\n safety_overrides: Operator-approved safety override tokens.", + "description": "A single action step or chunk produced by a Skill.\n\nAttributes:\n control_mode: Target action space.\n horizon: Number of steps (1 = single step, H = chunk).\n joint_targets: Joint position targets, shape ``(H, N)``.\n joint_velocities: Joint velocity targets, shape ``(H, N)``.\n joint_torques: Joint torque targets, shape ``(H, N)``.\n cartesian_pose: EE pose targets.\n cartesian_delta: EE pose deltas.\n cartesian_twist: EE velocity targets.\n body_twist: Base twist targets.\n foot_placements: Discrete footstep targets.\n gripper: Gripper commands in [0, 1].\n dex_hand_joints: Dexterous hand joint targets.\n confidence: rSkill confidence in [0, 1].\n stamp_ns: Action timestamp in nanoseconds.\n ee_name: Target end-effector name.\n frame_id: Reference frame for Cartesian actions.\n tick_index: Shared 1-based inference tick for every slot emitted by one\n policy step. Preserved across the ROS safety wire so a simulation HAL\n can commit a multi-surface action atomically after every slot passes.\n safety_overrides: Operator-approved safety override tokens.", "properties": { "control_mode": { "$ref": "#/$defs/ControlMode" @@ -324,6 +324,12 @@ "default": null, "title": "Frame Id" }, + "tick_index": { + "default": 0, + "minimum": 0, + "title": "Tick Index", + "type": "integer" + }, "safety_overrides": { "additionalProperties": true, "title": "Safety Overrides", diff --git a/docs/reference/schemas/RSkillManifest.json b/docs/reference/schemas/RSkillManifest.json index f612516..1a50a76 100644 --- a/docs/reference/schemas/RSkillManifest.json +++ b/docs/reference/schemas/RSkillManifest.json @@ -1343,6 +1343,7 @@ "panda_mobile", "pusht", "rizon4", + "r1pro", "sawyer", "so100_follower", "so101_follower", @@ -1468,6 +1469,7 @@ "aloha", "aloha_insertion", "aloha_transfer_cube", + "behavior", "gr1_tabletop", "libero_10", "libero_goal", diff --git a/docs/reference/schemas/WorldState.json b/docs/reference/schemas/WorldState.json index d05f740..55c456f 100644 --- a/docs/reference/schemas/WorldState.json +++ b/docs/reference/schemas/WorldState.json @@ -444,7 +444,7 @@ "type": "object" } }, - "description": "Snapshot consumed by Reasoner and Skills.\n\nAttributes:\n stamp_ns: Snapshot timestamp in nanoseconds.\n joint_state: Current joint state.\n base_pose: Base link pose (mobile robots).\n base_twist: Base link twist (vx, vy, vz, wx, wy, wz).\n ee_poses: End-effector poses keyed by EE name.\n contact_forces: Contact forces keyed by contact name.\n images: Sensor name \u2192 ROS 2 topic reference (not the raw image).\n image_frames: Optional per-sensor :class:`SensorFrame` snapshot for\n no-ROS / in-process deployments. When ``None`` the\n consumer reads frames via :attr:`images` topic refs as before.\n point_clouds: Sensor name \u2192 ROS 2 topic reference.\n tactile: Sensor name \u2192 ROS 2 topic reference.\n detected_objects: List of detected objects.\n battery_pct: Battery percentage in [0, 100].\n diagnostics: Per-component diagnostic status.\n collision_primitives: Bounded set of placed convex obstacle volumes\n the kernel checks robot links against (world-collision). Empty\n until a perception / SLAM source populates it.\n occupancy_grid: Optional 2D occupancy grid reference for mobile-base\n footprint checks. ``None`` until populated; an absent or stale\n grid is treated as unavailable (fail-closed).", + "description": "Snapshot consumed by Reasoner and Skills.\n\nAttributes:\n stamp_ns: Snapshot timestamp in nanoseconds.\n joint_state: Current joint state.\n base_pose: Base link pose (mobile robots).\n base_twist: Base link twist (vx, vy, vz, wx, wy, wz).\n policy_state: Optional simulator-native proprioception vector for a\n checkpoint whose state includes more than joint positions/velocities.\n Populated through a typed HAL -> WorldState topic; never inferred.\n ee_poses: End-effector poses keyed by EE name.\n contact_forces: Contact forces keyed by contact name.\n images: Sensor name \u2192 ROS 2 topic reference (not the raw image).\n image_frames: Optional per-sensor :class:`SensorFrame` snapshot for\n no-ROS / in-process deployments. When ``None`` the\n consumer reads frames via :attr:`images` topic refs as before.\n point_clouds: Sensor name \u2192 ROS 2 topic reference.\n tactile: Sensor name \u2192 ROS 2 topic reference.\n detected_objects: List of detected objects.\n battery_pct: Battery percentage in [0, 100].\n diagnostics: Per-component diagnostic status.\n collision_primitives: Bounded set of placed convex obstacle volumes\n the kernel checks robot links against (world-collision). Empty\n until a perception / SLAM source populates it.\n occupancy_grid: Optional 2D occupancy grid reference for mobile-base\n footprint checks. ``None`` until populated; an absent or stale\n grid is treated as unavailable (fail-closed).", "properties": { "stamp_ns": { "title": "Stamp Ns", @@ -498,6 +498,21 @@ "default": null, "title": "Base Twist" }, + "policy_state": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Policy State" + }, "ee_poses": { "additionalProperties": { "$ref": "#/$defs/Pose6D" diff --git a/docs/reference/schemas/all.json b/docs/reference/schemas/all.json index 3ebfda4..5c70714 100644 --- a/docs/reference/schemas/all.json +++ b/docs/reference/schemas/all.json @@ -6217,6 +6217,7 @@ "panda_mobile", "pusht", "rizon4", + "r1pro", "sawyer", "so100_follower", "so101_follower", @@ -6342,6 +6343,7 @@ "aloha", "aloha_insertion", "aloha_transfer_cube", + "behavior", "gr1_tabletop", "libero_10", "libero_goal", @@ -7397,7 +7399,7 @@ "type": "object" } }, - "description": "Snapshot consumed by Reasoner and Skills.\n\nAttributes:\n stamp_ns: Snapshot timestamp in nanoseconds.\n joint_state: Current joint state.\n base_pose: Base link pose (mobile robots).\n base_twist: Base link twist (vx, vy, vz, wx, wy, wz).\n ee_poses: End-effector poses keyed by EE name.\n contact_forces: Contact forces keyed by contact name.\n images: Sensor name \u2192 ROS 2 topic reference (not the raw image).\n image_frames: Optional per-sensor :class:`SensorFrame` snapshot for\n no-ROS / in-process deployments. When ``None`` the\n consumer reads frames via :attr:`images` topic refs as before.\n point_clouds: Sensor name \u2192 ROS 2 topic reference.\n tactile: Sensor name \u2192 ROS 2 topic reference.\n detected_objects: List of detected objects.\n battery_pct: Battery percentage in [0, 100].\n diagnostics: Per-component diagnostic status.\n collision_primitives: Bounded set of placed convex obstacle volumes\n the kernel checks robot links against (world-collision). Empty\n until a perception / SLAM source populates it.\n occupancy_grid: Optional 2D occupancy grid reference for mobile-base\n footprint checks. ``None`` until populated; an absent or stale\n grid is treated as unavailable (fail-closed).", + "description": "Snapshot consumed by Reasoner and Skills.\n\nAttributes:\n stamp_ns: Snapshot timestamp in nanoseconds.\n joint_state: Current joint state.\n base_pose: Base link pose (mobile robots).\n base_twist: Base link twist (vx, vy, vz, wx, wy, wz).\n policy_state: Optional simulator-native proprioception vector for a\n checkpoint whose state includes more than joint positions/velocities.\n Populated through a typed HAL -> WorldState topic; never inferred.\n ee_poses: End-effector poses keyed by EE name.\n contact_forces: Contact forces keyed by contact name.\n images: Sensor name \u2192 ROS 2 topic reference (not the raw image).\n image_frames: Optional per-sensor :class:`SensorFrame` snapshot for\n no-ROS / in-process deployments. When ``None`` the\n consumer reads frames via :attr:`images` topic refs as before.\n point_clouds: Sensor name \u2192 ROS 2 topic reference.\n tactile: Sensor name \u2192 ROS 2 topic reference.\n detected_objects: List of detected objects.\n battery_pct: Battery percentage in [0, 100].\n diagnostics: Per-component diagnostic status.\n collision_primitives: Bounded set of placed convex obstacle volumes\n the kernel checks robot links against (world-collision). Empty\n until a perception / SLAM source populates it.\n occupancy_grid: Optional 2D occupancy grid reference for mobile-base\n footprint checks. ``None`` until populated; an absent or stale\n grid is treated as unavailable (fail-closed).", "properties": { "stamp_ns": { "title": "Stamp Ns", @@ -7451,6 +7453,21 @@ "default": null, "title": "Base Twist" }, + "policy_state": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Policy State" + }, "ee_poses": { "additionalProperties": { "$ref": "#/$defs/Pose6D" @@ -7639,7 +7656,7 @@ "type": "object" } }, - "description": "A single action step or chunk produced by a Skill.\n\nAttributes:\n control_mode: Target action space.\n horizon: Number of steps (1 = single step, H = chunk).\n joint_targets: Joint position targets, shape ``(H, N)``.\n joint_velocities: Joint velocity targets, shape ``(H, N)``.\n joint_torques: Joint torque targets, shape ``(H, N)``.\n cartesian_pose: EE pose targets.\n cartesian_delta: EE pose deltas.\n cartesian_twist: EE velocity targets.\n body_twist: Base twist targets.\n foot_placements: Discrete footstep targets.\n gripper: Gripper commands in [0, 1].\n dex_hand_joints: Dexterous hand joint targets.\n confidence: rSkill confidence in [0, 1].\n stamp_ns: Action timestamp in nanoseconds.\n ee_name: Target end-effector name.\n frame_id: Reference frame for Cartesian actions.\n safety_overrides: Operator-approved safety override tokens.", + "description": "A single action step or chunk produced by a Skill.\n\nAttributes:\n control_mode: Target action space.\n horizon: Number of steps (1 = single step, H = chunk).\n joint_targets: Joint position targets, shape ``(H, N)``.\n joint_velocities: Joint velocity targets, shape ``(H, N)``.\n joint_torques: Joint torque targets, shape ``(H, N)``.\n cartesian_pose: EE pose targets.\n cartesian_delta: EE pose deltas.\n cartesian_twist: EE velocity targets.\n body_twist: Base twist targets.\n foot_placements: Discrete footstep targets.\n gripper: Gripper commands in [0, 1].\n dex_hand_joints: Dexterous hand joint targets.\n confidence: rSkill confidence in [0, 1].\n stamp_ns: Action timestamp in nanoseconds.\n ee_name: Target end-effector name.\n frame_id: Reference frame for Cartesian actions.\n tick_index: Shared 1-based inference tick for every slot emitted by one\n policy step. Preserved across the ROS safety wire so a simulation HAL\n can commit a multi-surface action atomically after every slot passes.\n safety_overrides: Operator-approved safety override tokens.", "properties": { "control_mode": { "$ref": "#/$defs/ControlMode" @@ -7889,6 +7906,12 @@ "default": null, "title": "Frame Id" }, + "tick_index": { + "default": 0, + "minimum": 0, + "title": "Tick Index", + "type": "integer" + }, "safety_overrides": { "additionalProperties": true, "title": "Safety Overrides", diff --git a/docs/reference/sim-environments.md b/docs/reference/sim-environments.md index 0ad5be0..c2d76ce 100644 --- a/docs/reference/sim-environments.md +++ b/docs/reference/sim-environments.md @@ -55,6 +55,7 @@ reasoner picks the rSkill. Consumed by `openral deploy sim`. | [`libero_pnp.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/deploy/libero_pnp.yaml) | `franka_panda` *(scene-fixed)* | `libero_spatial` | LIBERO (robosuite + MuJoCo) | Boot LIBERO in deploy mode so a reasoner can issue arbitrary pick-and-place commands | | [`openarm_tabletop.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/deploy/openarm_tabletop.yaml) | `openarm` *(free-axis)* | `openarm_tabletop_pnp` | Custom MJCF | OpenArm bimanual tabletop sandbox; default top camera matches the mddoai dataset POV | | [`robocasa_pnp.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/deploy/robocasa_pnp.yaml) | `panda_mobile` *(scene-fixed)* | `robocasa/PickPlaceCounterToCabinet` | RoboCasa (MuJoCo) | Mobile-base kitchen pick-and-place sandbox; reasoner-driven | +| [`behavior_r1pro.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/deploy/behavior_r1pro.yaml) | `r1pro` *(scene-fixed)* | `behavior` | BEHAVIOR-1K / OmniGibson (Isaac Sim sidecar) | Full deploy graph on public `turning_on_radio` instance 0 | | [`so101_box.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/deploy/so101_box.yaml) | `so101_follower` *(scene-fixed)* | `so101_box` | Custom MJCF | 100×61.5×75 cm box arena + OAK-D Pro overhead + wrist camera; deploy sandbox | | [`so101_bench.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/deploy/so101_bench.yaml) | `so101_follower` *(scene-fixed)* | `so101_bench` | Custom MJCF | SO-101 bench-arena deploy sandbox | | [`libero_object.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/deploy/libero_object.yaml) | `franka_panda` *(scene-fixed)* | `libero_object` | LIBERO (robosuite + MuJoCo) | Boot LIBERO-Object in deploy mode; reasoner-driven pick-and-place | @@ -86,6 +87,7 @@ Consumed by `openral sim run`. | [`franka_tabletop_push.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/sim/franka_tabletop_push.yaml) | `franka_panda` *(scene-fixed)* | `tabletop_push` | `tabletop_push/push_to_goal` | Franka variant of the robot-agnostic cube push-to-goal | | [`pusht.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/sim/pusht.yaml) | `pusht_2d` *(scene-fixed; 2-D pymunk)* | `pusht` | `pusht/0` | gym-pusht 2-D push smoke | | [`isaac_franka_bowl_plate.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/sim/isaac_franka_bowl_plate.yaml) | `franka_panda` *(scene-fixed)* | `isaac_sim` | `isaac_sim/put_the_bowl_on_the_plate` | Isaac Sim bowl-on-plate; two-camera layout used by `gr00t-n17-libero` | +| [`behavior_turning_on_radio.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/sim/behavior_turning_on_radio.yaml) | `r1pro` *(scene-fixed)* | `behavior` | `behavior/turning_on_radio` | Official public-test instance 0 through the OmniGibson evaluator sidecar | ## BenchmarkScene catalogue (`scenes/benchmark/`) @@ -107,6 +109,7 @@ YAML root) under | [`metaworld_push.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/benchmark/metaworld_push.yaml) | `sawyer` *(scene-fixed)* | `metaworld` | `metaworld/push-v3` | 50 | [MetaWorld MT10/MT50](https://arxiv.org/abs/1910.10897) | | [`metaworld_pick_place.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/benchmark/metaworld_pick_place.yaml) | `sawyer` *(scene-fixed)* | `metaworld` | `metaworld/pick-place-v3` | 50 | [MetaWorld MT10/MT50](https://arxiv.org/abs/1910.10897) | | [`metaworld_button_press.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/benchmark/metaworld_button_press.yaml) | `sawyer` *(scene-fixed)* | `metaworld` | `metaworld/button-press-v3` | 50 | [MetaWorld MT10/MT50](https://arxiv.org/abs/1910.10897) | +| [`behavior_turning_on_radio.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/benchmark/behavior_turning_on_radio.yaml) | `r1pro` *(scene-fixed)* | `behavior` | `behavior/turning_on_radio` | 1 | [BEHAVIOR-1K](https://arxiv.org/abs/2403.09227) | | [`metaworld_door_open.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/benchmark/metaworld_door_open.yaml) | `sawyer` *(scene-fixed)* | `metaworld` | `metaworld/door-open-v3` | 50 | [MetaWorld MT10/MT50](https://arxiv.org/abs/1910.10897) | | [`metaworld_drawer_open.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/benchmark/metaworld_drawer_open.yaml) | `sawyer` *(scene-fixed)* | `metaworld` | `metaworld/drawer-open-v3` | 50 | [MetaWorld MT10/MT50](https://arxiv.org/abs/1910.10897) | | [`pusht.yaml`](https://github.com/OpenRAL/openral/blob/master/scenes/benchmark/pusht.yaml) | `pusht_2d` *(scene-fixed; 2-D pymunk)* | `pusht` | `pusht/0` | 200 | [Diffusion Policy](https://arxiv.org/abs/2303.04137) | diff --git a/docs/reference/vla_compatibility.md b/docs/reference/vla_compatibility.md index 6d20322..9089b9d 100644 --- a/docs/reference/vla_compatibility.md +++ b/docs/reference/vla_compatibility.md @@ -38,6 +38,7 @@ Embodiment tags are short strings that appear in `rskill.yaml` under `embodiment | `libero` | Franka Panda on LIBERO benchmark | 7 + gripper | LIBERO (Yuke Zhu et al., NeurIPS 2023) | Simulation-only tag for LIBERO benchmark training | | `franka_panda` | Franka Panda (real + sim) | 7 + gripper | Standard industry robot; widespread in BridgeData / Open X | Broader tag; use `libero` when targeting LIBERO-specific checkpoints | | `widowx` | WidowX 250s | 6 | [BridgeData V2](https://rail-berkeley.github.io/bridgedata/) | Low-cost research arm; common in Open X-Embodiment | +| `r1pro` | Galaxea R1 Pro (BEHAVIOR simulation) | 22 articulated joints + holonomic base | [BEHAVIOR-1K 2026](https://behavior.stanford.edu/challenge/) | Sim-only OpenRAL manifest; 61-D policy state and mixed 23-D action | | `gr1` | Unitree GR1 humanoid | 23 | [NVIDIA Arena dataset](https://huggingface.co/nvidia) | Full humanoid; requires S0 cerebellar layer | | `aloha` | Aloha bimanual teleoperation setup | 2 × 7 | [ACT paper](https://arxiv.org/abs/2304.13705) (Stanford / Toyota) | Bimanual; two Viperx arms with overhead + wrist cameras | | `aloha_agilex` | ALOHA-AgileX dual-arm (RoboTwin 2.0) | 2 × 7 | [RoboTwin 2.0](https://arxiv.org/abs/2506.18088) | Bimanual SAPIEN benchmark embodiment; targeted by `smolvla-robotwin` | @@ -160,6 +161,12 @@ Columns: | ACT (transfer cube) | gym-aloha | `aloha_bimanual` | 2 × 7 | overhead + wrist | Yes (in ckpt) | `rskills/act-aloha/` | MIT | Bimanual cube-transfer; `scenes/benchmark/aloha_transfer_cube.yaml` | | ACT (insertion) | gym-aloha | `aloha_bimanual` | 2 × 7 | overhead + wrist | Yes (in ckpt) | `rskills/act-aloha-insertion/` | MIT | Custom-example insertion checkpoint; `scenes/benchmark/aloha_insertion.yaml` | +### 3.11 BEHAVIOR-1K 2026 Challenge (Galaxea R1 Pro) + +| VLA / source | Sim env | Robot tag | State dim | Cameras | Action | rSkill | License | Notes | +|---|---|---|---|---|---|---|---|---| +| Organizer GR00T N1.7 `turning_on_radio` checkpoint ([baseline](https://behavior.stanford.edu/challenge/baselines.html)) | OmniGibson / Isaac Sim | `r1pro` | **61-D** official R1Pro proprio order | 224² RGB under the official `DefaultWrapper` (`RGBDFullResWrapper` crashes at boot on the pinned OmniGibson build — it reads joint state before the physics views exist) | **23-D** base velocity (3) + torso (4) + arms (7+7) + symmetric grippers (1+1) | `rskills/gr00t-n17-b1k-turning-on-radio` | **Unknown** for the organizer Drive artifact | Runs through `openral behavior serve`, `openral sim run`, `openral benchmark run --suite behavior`, or the full `openral deploy sim` graph. Deploy preserves the 61-D state and commits all six safety-approved typed slots as one simulator step. | + --- ## 4. Sim Environment Reference @@ -174,6 +181,7 @@ Columns: | SO-101 Box (`so101_box`) | MuJoCo (raw, `python/sim/src/openral_sim/backends/so101_box/`) | `uv sync --group sim` | SO-101 | tube-insertion (geometric success: tube vertical + lower tip ≥ 10 mm below the slotted-block hole top) — both block and tube spawn at random (x, y, yaw) on the floor each `reset()` | OAK-D Pro overhead (RGB + depth, default 640×480) + wrist RGB parented to the gripper body | | SimplerEnv WidowX | ManiSkill3/SAPIEN via `simpler_env` | `uv sync --group simpler-env` + `uv pip install "simpler-env @ git+https://github.com/simpler-env/SimplerEnv.git@maniskill3"` | WidowX 250s | carrot-on-plate (`simpler_env/widowx_carrot_on_plate`) | 3rd-view RGB surfaced as `top` | | NVIDIA Arena | Isaac Sim | Requires NVIDIA Isaac Sim license | GR1 | microwave | TBD | +| BEHAVIOR-1K 2026 | OmniGibson / Isaac Sim | Official `behavior` environment plus `just sync --group behavior-groot` for the OpenRAL-side wire | Galaxea R1 Pro | 100 household tasks; packaged starter is `turning_on_radio` | head + dual wrist RGB; official full wrapper also exposes depth | | ManiSkill3 | SAPIEN via `mani_skill` | `uv sync --group maniskill3` | Franka Panda | `PickCube-v1` (+ more) | single RGB `camera1` | | RoboTwin 2.0 | SAPIEN (py3.10 sidecar) | py3.10 SAPIEN sidecar (auto-provisioned) | ALOHA-AgileX | 50 dual-arm tasks (`robotwin/*`) | dual-arm camera set | | VLABench | MuJoCo | `uv sync --group vlabench` | Franka Panda | 97 tasks (`vlabench/*`) | TBD | diff --git a/packages/openral_rskill_ros/openral_rskill_ros/rskill_runner_node.py b/packages/openral_rskill_ros/openral_rskill_ros/rskill_runner_node.py index 314ad3f..93ef507 100644 --- a/packages/openral_rskill_ros/openral_rskill_ros/rskill_runner_node.py +++ b/packages/openral_rskill_ros/openral_rskill_ros/rskill_runner_node.py @@ -1826,7 +1826,7 @@ def _build_runtime_skill_from_manifest( from openral_core.exceptions import ROSConfigError, ROSRuntimeError from openral_sim.factory import make_policy from openral_sim.policy_deps import ( - model_family_install_hint, + manifest_install_hint, purge_partial_imports, ) @@ -1898,7 +1898,7 @@ def _build_runtime_skill_from_manifest( # is removed from ``sys.modules`` mid-process. purge_partial_imports(("lerobot", "transformers")) family = manifest.model_family - install_hint = model_family_install_hint(family) + install_hint = manifest_install_hint(manifest) raise ROSRuntimeError( f"failed to build {family!r} policy for rSkill " f"{manifest.name!r}: {type(exc).__name__}: {exc}. " @@ -2262,8 +2262,19 @@ def _make_policy_adapter_skill( assembles ``obs["state"]`` via that layout's assembler instead of the raw joint-state slice. None preserves the joint-space path (every VLA shipped before the state-contract bindings design). + + ``obs["state"]`` substitution paths in ``_step_impl`` (either sets + ``state_assembled`` and skips the joint-permutation + rad/deg conversion; + if a manifest somehow declares both, the layout assembler runs second and + wins): + + 1. ``policy_extras.use_world_state_policy_state`` — the manifest opts in + to the simulator-native ``WorldState.policy_state`` vector (BEHAVIOR-1K + R1Pro 61-D contract), staleness-gated via ``ROSPerceptionStale``. + 2. ``tf_lookup`` + ``state_contract.layout`` — see above. """ import numpy as np + from openral_core.exceptions import ROSConfigError, ROSPerceptionStale, ROSRuntimeError from openral_core.schemas import Action, ControlMode from openral_rskill.base import rSkillBase @@ -2293,7 +2304,6 @@ def _make_policy_adapter_skill( str(getattr(_declared_units, "value", _declared_units)) == "degrees" ) else: - from openral_core.exceptions import ROSConfigError from openral_core.schemas import ActionRepresentation if ( @@ -2483,7 +2493,9 @@ def _dump_obs_to_disk( flush=True, ) - def _step_impl(self, world_state: Any) -> Action | list[Action]: + def _step_impl( # noqa: PLR0912, PLR0915 # reason: linear policy observation/action boundary with one branch per supported state/action contract + self, world_state: Any + ) -> Action | list[Action]: obs: dict[str, object] = {"task": self._prompt} js = world_state.joint_state robot_state = np.asarray(list(js.position), dtype=np.float32) @@ -2498,6 +2510,29 @@ def _step_impl(self, world_state: Any) -> Action | list[Action]: sc = getattr(self.manifest, "state_contract", None) layout = getattr(sc, "layout", None) if sc is not None else None bindings = getattr(sc, "bindings", None) if sc is not None else None + use_policy_state = bool( + getattr(self.manifest, "policy_extras", {}).get("use_world_state_policy_state") + ) + if use_policy_state: + raw_policy_state = getattr(world_state, "policy_state", None) + if raw_policy_state is None: + raise ROSRuntimeError( + f"rSkill {self.manifest.name!r} requires WorldState.policy_state, " + "but no /openral/policy_state sample has arrived." + ) + if getattr(world_state, "diagnostics", {}).get("policy_state") != "ok": + raise ROSPerceptionStale( + f"rSkill {self.manifest.name!r} requires fresh policy_state." + ) + policy_state = np.asarray(raw_policy_state, dtype=np.float32).reshape(-1) + expected_dim = getattr(sc, "dim", None) if sc is not None else None + if expected_dim is not None and policy_state.shape != (int(expected_dim),): + raise ROSRuntimeError( + f"rSkill {self.manifest.name!r} requires policy_state dim " + f"{expected_dim}, got {policy_state.shape[0]}." + ) + obs["state"] = policy_state + state_assembled = True if self._tf_lookup is not None and layout is not None and bindings is not None: # Deferred import keeps the runner module load light # (openral_state_adapter pulls numpy + the layout diff --git a/packages/world_state/openral_world_state_ros/lifecycle_node.py b/packages/world_state/openral_world_state_ros/lifecycle_node.py index 5ff41ec..7f9112d 100644 --- a/packages/world_state/openral_world_state_ros/lifecycle_node.py +++ b/packages/world_state/openral_world_state_ros/lifecycle_node.py @@ -208,6 +208,7 @@ def __init__(self, aggregator: object | None = None) -> None: self._pub_fast = None self._pub_slow = None self._joint_sub = None + self._policy_state_sub = None self._camera_subs: dict[str, object] = {} self._slow_divider = 1 self._tick_count = 0 @@ -240,7 +241,9 @@ def _init_lift_state(self) -> None: self._voxel_staleness_ns = 0 @log_lifecycle_errors - def on_configure(self, state: object) -> TransitionCallbackReturn: + def on_configure( # noqa: PLR0915 # reason: linear ROS subscription/publisher wiring + self, state: object + ) -> TransitionCallbackReturn: """Initialise the aggregator (if owned) and topic plumbing.""" from openral_msgs.msg import ( # type: ignore[import-untyped] WorldStateStamped, @@ -306,6 +309,14 @@ def on_configure(self, state: object) -> TransitionCallbackReturn: self._on_joint_state, sensor_qos, ) + from std_msgs.msg import Float32MultiArray + + self._policy_state_sub = self.create_subscription( + Float32MultiArray, + "/openral/policy_state", + self._on_policy_state, + sensor_qos, + ) # Per-camera image subscriptions on `//image`. # BEST_EFFORT per CLAUDE.md §2 (sensor streams): a BEST_EFFORT @@ -503,6 +514,9 @@ def on_cleanup(self, state: object) -> TransitionCallbackReturn: if self._joint_sub is not None: self.destroy_subscription(self._joint_sub) self._joint_sub = None + if self._policy_state_sub is not None: + self.destroy_subscription(self._policy_state_sub) + self._policy_state_sub = None for sub in self._camera_subs.values(): self.destroy_subscription(sub) # type: ignore[arg-type] self._camera_subs.clear() @@ -682,6 +696,14 @@ def _on_joint_state(self, msg: object) -> None: ) self._aggregator.update_joint_state(js) + def _on_policy_state(self, msg: object) -> None: + """Store a simulator-native policy state vector.""" + if self._aggregator is None: + return + self._aggregator.update_policy_state( + [float(value) for value in (getattr(msg, "data", []) or [])] + ) + def _on_voxels(self, msg: object) -> None: # OccupancyVoxels """Store the latest occupancy voxel grid (best-effort; cheap).""" self._latest_voxels = msg diff --git a/pyproject.toml b/pyproject.toml index 78fc5b8..c5887a7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -296,6 +296,12 @@ gr00t = [ # in-process solves LIBERO-spatial 5/5 at ~19 ms/step on an 8 GiB card. "lerobot[groot]", ] +behavior-groot = [ + # OpenRAL-side wire for the official BEHAVIOR-1K Isaac-GR00T runtime. + # The pinned wensi-ai/Isaac-GR00T behavior checkout + checkpoint stay in + # their own externally-provisioned Python 3.10 venv. + { include-group = "sidecar-wire" }, +] isaacsim = [ # Opt-in NVIDIA Isaac Sim scene backend. # diff --git a/python/cli/README.md b/python/cli/README.md index a125a36..aac8c40 100644 --- a/python/cli/README.md +++ b/python/cli/README.md @@ -13,3 +13,30 @@ eight-layer model in the project docs. > All OpenRAL workspace packages move in lockstep at `0.1.x` until the first > public release. + +## BEHAVIOR Challenge + +Serve an R1Pro-compatible rSkill through the official BEHAVIOR WebSocket +protocol, then run OmniGibson's evaluator in its own environment: + +```bash +openral behavior serve \ + --rskill rskills/gr00t-n17-b1k-turning-on-radio \ + --task turning_on_radio + +conda run -n behavior python -m omnigibson.eval.eval \ + --task-name turning_on_radio \ + --host 127.0.0.1 --port 8000 \ + --instance-indices 0 --num-rollouts 1 \ + --output-dir outputs/openral --write-video +``` + +The bridge defaults to the official R1Pro contract: three RGB cameras, 61-D +proprioception, and a 23-D action. + +The same contract can run through the full deploy graph: + +```bash +openral deploy sim --config scenes/deploy/behavior_r1pro.yaml \ + --initial-task "turn on the radio" +``` diff --git a/python/cli/pyproject.toml b/python/cli/pyproject.toml index 22d26db..84306d3 100644 --- a/python/cli/pyproject.toml +++ b/python/cli/pyproject.toml @@ -16,6 +16,9 @@ dependencies = [ "rich>=13.7", # reason: terminal formatting "pyudev>=0.24.4", "opencv-python-headless>=4.9,<5", # reason: camera thumbnails in `openral detect` sensor wizard; <5 keeps numpy<2.5 (opencv 5 pulls numpy 2.5, breaking the workspace pin) + "fastapi>=0.110", # reason: BEHAVIOR evaluator WebSocket + /healthz policy endpoint + "uvicorn[standard]>=0.27", # reason: serve the BEHAVIOR policy ASGI app + "msgpack>=1", # reason: official BEHAVIOR ndarray wire format ] [project.scripts] diff --git a/python/cli/src/openral_cli/behavior.py b/python/cli/src/openral_cli/behavior.py new file mode 100644 index 0000000..5b35525 --- /dev/null +++ b/python/cli/src/openral_cli/behavior.py @@ -0,0 +1,241 @@ +"""BEHAVIOR Challenge WebSocket policy server.""" + +from __future__ import annotations + +import asyncio +import functools +from typing import TYPE_CHECKING + +import msgpack +import numpy as np +import uvicorn +from fastapi import FastAPI, WebSocket, WebSocketDisconnect +from numpy.typing import NDArray +from openral_core.exceptions import ROSRuntimeError + +if TYPE_CHECKING: + from openral_core import VLASpec + from openral_sim.policy import PolicyAdapter + from openral_sim.rollout import Observation + +_R1PRO_CAMERA_SENSORS = { + "head": "robot_r1::robot_r1:zed_link:Camera:0", + "left_wrist": "robot_r1::robot_r1:left_realsense_link:Camera:0", + "right_wrist": "robot_r1::robot_r1:right_realsense_link:Camera:0", +} +_R1PRO_STATE_KEY = "robot_r1::proprio" +_R1PRO_STATE_DIM = 61 +_R1PRO_ACTION_DIM = 23 +_RGB_RANK = 3 +_RGB_CHANNELS = 3 + + +def _pack_data(value: object) -> object: + """Encode NumPy values with BEHAVIOR's msgpack wire format.""" + if isinstance(value, np.ndarray): + return { + b"__ndarray__": True, + b"data": value.tobytes(), + b"dtype": value.dtype.str, + b"shape": value.shape, + } + if isinstance(value, np.generic): + return { + b"__npgeneric__": True, + b"data": value.item(), + b"dtype": value.dtype.str, + } + return value + + +def _unpack_data(value: dict[object, object]) -> object: + """Decode NumPy values from BEHAVIOR's msgpack wire format.""" + if b"__ndarray__" in value: + data = value.get(b"data") + dtype = value.get(b"dtype") + shape = value.get(b"shape") + if ( + isinstance(data, bytes) + and isinstance(dtype, str) + and isinstance(shape, (list, tuple)) + and all(isinstance(size, int) for size in shape) + ): + return np.frombuffer(data, dtype=np.dtype(dtype)).reshape(tuple(shape)) + if b"__npgeneric__" in value: + data = value.get(b"data") + dtype = value.get(b"dtype") + if isinstance(dtype, str): + return np.asarray(data, dtype=np.dtype(dtype)).reshape(()).item() + return value + + +_packb = functools.partial(msgpack.packb, default=_pack_data) +_unpackb = functools.partial(msgpack.unpackb, object_hook=_unpack_data) + + +def _as_message(payload: object) -> dict[str, object]: + if not isinstance(payload, dict) or not all(isinstance(key, str) for key in payload): + raise ROSRuntimeError("BEHAVIOR policy request must be a string-keyed mapping.") + return payload + + +def _normalize_behavior_observation( + raw: dict[str, object], + *, + instruction: str, + state_dim: int, +) -> Observation: + """Map the official R1Pro wrapper output into OpenRAL's policy observation.""" + state_raw = raw.get(_R1PRO_STATE_KEY) + if state_raw is None: + raise ROSRuntimeError( + f"BEHAVIOR observation is missing R1Pro proprioception key {_R1PRO_STATE_KEY!r}." + ) + state = np.asarray(state_raw, dtype=np.float32).reshape(-1) + if state.shape != (state_dim,): + raise ROSRuntimeError( + f"BEHAVIOR R1Pro proprioception must have {state_dim} values, got {state.shape[0]}." + ) + + images: dict[str, NDArray[np.uint8]] = {} + depths: dict[str, NDArray[np.float32]] = {} + for role, sensor_key in _R1PRO_CAMERA_SENSORS.items(): + rgb = raw.get(f"{sensor_key}::rgb") + if rgb is not None: + image = np.asarray(rgb, dtype=np.uint8) + if image.ndim != _RGB_RANK or image.shape[-1] < _RGB_CHANNELS: + raise ROSRuntimeError( + f"BEHAVIOR camera {role!r} RGB must be HWC with at least 3 channels, " + f"got {image.shape}." + ) + images[role] = image[..., :_RGB_CHANNELS] + + depth = raw.get(f"{sensor_key}::depth_linear") + if depth is not None: + depths[role] = np.asarray(depth, dtype=np.float32) + + if not images: + raise ROSRuntimeError("BEHAVIOR observation contains no configured R1Pro RGB cameras.") + + return { + "images": images, + "depths": depths, + "state": state, + "task": instruction, + "behavior_raw": raw, + } + + +def _policy_action( + policy: PolicyAdapter, + observation: Observation, + instruction: str, + action_dim: int, +) -> NDArray[np.float32]: + action = np.asarray(policy.step(observation, instruction), dtype=np.float32).reshape(-1) + if action.shape != (action_dim,): + raise ROSRuntimeError( + f"BEHAVIOR R1Pro action must have {action_dim} values, got {action.shape[0]}." + ) + if not np.isfinite(action).all(): + raise ROSRuntimeError("BEHAVIOR policy emitted a non-finite action.") + return action + + +def _create_behavior_app( + policy: PolicyAdapter, + *, + task: str, + instruction: str, + state_dim: int, + action_dim: int, +) -> FastAPI: + app = FastAPI(title="OpenRAL BEHAVIOR policy server", docs_url=None, redoc_url=None) + + @app.get("/healthz") + async def _healthz() -> dict[str, str]: + return {"status": "ok"} + + @app.websocket("/") + async def _policy_socket(websocket: WebSocket) -> None: + await websocket.accept() + await websocket.send_bytes( + _packb( + { + "policy": policy.spec.weights_uri, + "task": task, + "action_dim": action_dim, + } + ) + ) + try: + while True: + message = _as_message(_unpackb(await websocket.receive_bytes())) + if message.get("reset") is True: + await asyncio.to_thread(policy.reset) + continue + observation = _normalize_behavior_observation( + message, + instruction=instruction, + state_dim=state_dim, + ) + action = await asyncio.to_thread( + _policy_action, + policy, + observation, + instruction, + action_dim, + ) + await websocket.send_bytes(_packb({"action": action})) + except WebSocketDisconnect: + return + + return app + + +def _serve_behavior_policy( + vla_spec: VLASpec, + *, + task: str, + instruction: str, + host: str, + port: int, + state_dim: int = _R1PRO_STATE_DIM, + action_dim: int = _R1PRO_ACTION_DIM, +) -> None: + """Serve an OpenRAL rSkill through the official BEHAVIOR policy protocol.""" + from openral_core import ( # noqa: PLC0415 # reason: keep `openral --help` light + PhysicsBackend, + SceneSpec, + SimEnvironment, + TaskSpec, + ) + from openral_sim import make_policy # noqa: PLC0415 # reason: load registry on serve only + + env = SimEnvironment( + robot_id="r1pro", + scene=SceneSpec( + id="behavior_challenge", + backend=PhysicsBackend.ISAACSIM, + cameras=list(_R1PRO_CAMERA_SENSORS), + ), + task=TaskSpec( + id=f"behavior/{task}", + scene_id="behavior_challenge", + instruction=instruction, + ), + vla=vla_spec, + ) + policy = make_policy(env) + app = _create_behavior_app( + policy, + task=task, + instruction=instruction, + state_dim=state_dim, + action_dim=action_dim, + ) + print(f"OpenRAL BEHAVIOR policy: ws://{host}:{port} task={task}", flush=True) + try: + uvicorn.run(app, host=host, port=port, log_level="warning") + finally: + policy.close() diff --git a/python/cli/src/openral_cli/deploy_sim.py b/python/cli/src/openral_cli/deploy_sim.py index e4442a0..2e44790 100644 --- a/python/cli/src/openral_cli/deploy_sim.py +++ b/python/cli/src/openral_cli/deploy_sim.py @@ -268,6 +268,15 @@ class _HalSpec: manifest_driven=True, supports_sim_env_yaml=True, ), + "r1pro": _HalSpec( + package="openral_hal_scene_attached", + executable="lifecycle_node.py", + node_name="openral_hal_scene_attached", + supported_robot_names=frozenset({"r1pro"}), + default_params={}, + manifest_driven=True, + supports_sim_env_yaml=True, + ), "g1": _HalSpec( package="openral_hal_g1", executable="lifecycle_node.py", @@ -2015,9 +2024,9 @@ def _preflight_palette_deps( # noqa: PLR0912, PLR0915 # reason: linear flow from openral_core import RobotDescription, RSkillManifest from openral_reasoner.palette import build_tool_palette from openral_sim.policy_deps import ( - can_import_policy_family, - model_family_install_groups, - model_family_install_hint, + can_import_policy_manifest, + manifest_install_groups, + manifest_install_hint, ) rskills_dir = repo_root / "rskills" @@ -2056,11 +2065,11 @@ def _preflight_palette_deps( # noqa: PLR0912, PLR0915 # reason: linear flow if m.name not in matching_ids: continue family = getattr(m, "model_family", None) or "" - ok, _ = can_import_policy_family(family) + ok, _ = can_import_policy_manifest(m) if ok: continue blocked.append((m.name, family)) - install_groups.update(model_family_install_groups(family)) + install_groups.update(manifest_install_groups(m)) if not blocked: return @@ -2073,7 +2082,8 @@ def _preflight_palette_deps( # noqa: PLR0912, PLR0915 # reason: linear flow ) for name, family in blocked: _console.print(f" • {name} (model_family={family!r})") - _console.print(f" {model_family_install_hint(family)}") + manifest = next(item for item in manifests if item.name == name) + _console.print(f" {manifest_install_hint(manifest)}") install_cmd: list[str] | None = None if install_groups: @@ -2151,7 +2161,13 @@ def _preflight_palette_deps( # noqa: PLR0912, PLR0915 # reason: linear flow # process runs from; new files on disk need an invalidate to # show up via PathFinder), then recompute the blocked set. importlib.invalidate_caches() - blocked = [(n, f) for n, f in blocked if not can_import_policy_family(f)[0]] + blocked = [ + (name, family) + for name, family in blocked + if not can_import_policy_manifest( + next(manifest for manifest in manifests if manifest.name == name) + )[0] + ] if not blocked: _console.print("[green]preflight:[/green] extras installed; continuing launch.") return diff --git a/python/cli/src/openral_cli/install.py b/python/cli/src/openral_cli/install.py index d769735..bfd5747 100644 --- a/python/cli/src/openral_cli/install.py +++ b/python/cli/src/openral_cli/install.py @@ -116,6 +116,10 @@ "pyzmq>=25", "msgpack>=1", ], + "behavior-groot": [ + "pyzmq>=25", + "msgpack>=1", + ], } # Mutually exclusive groups — mirrors ``[tool.uv].conflicts`` in the workspace diff --git a/python/cli/src/openral_cli/main.py b/python/cli/src/openral_cli/main.py index 0512f34..bcd655d 100644 --- a/python/cli/src/openral_cli/main.py +++ b/python/cli/src/openral_cli/main.py @@ -14,6 +14,7 @@ doctor Diagnose the host environment (Python, OS, ROS 2, GPU, USB). detect Probe hardware and write a full RobotDescription robot.yaml. connect Open a HAL connection to a robot and verify it responds. +behavior serve Serve an rSkill to the official BEHAVIOR evaluator. calibrate camera Calibrate a camera sensor using ros2 camera_calibration. install Install opt-in dependency groups (sim, ros, libero, …). rskill search Find installable rSkills on the OpenRAL HF Hub org. @@ -368,6 +369,7 @@ def _run_repl() -> None: _RUN_MODE_BY_SUBCOMMAND: dict[str, str] = { "sim": semconv.RUN_MODE_SIM, + "behavior": semconv.RUN_MODE_SIM, "benchmark": semconv.RUN_MODE_BENCHMARK, "deploy": semconv.RUN_MODE_HARDWARE, "connect": semconv.RUN_MODE_HARDWARE, @@ -2924,6 +2926,24 @@ def benchmark_run( "Disable for read-only paper-number runs." ), ), + video: bool = typer.Option( + True, + "--video/--no-video", + help=( + "Record per-step world frames and write one MP4 per episode " + "(named [_seed]__.mp4 plus a " + "videos.json manifest). On by default; frames are written and " + "freed per episode. Disable for allocation-light CI runs." + ), + ), + video_dir: Path | None = typer.Option( + None, + "--video-dir", + help=( + "Destination directory for --video MP4s. Defaults to " + "/videos/ next to the RSkillEvalResult." + ), + ), dashboard: bool = typer.Option( False, "--dashboard", @@ -3001,6 +3021,11 @@ class was removed so the suite is a bare list of scenes whose id is out_path = out if out is not None else _default_benchmark_out_path(vla_spec, suite_id) out_path.parent.mkdir(parents=True, exist_ok=True) + effective_video_dir: Path | None = None + if video: + effective_video_dir = ( + video_dir if video_dir is not None else (out_path.parent / "videos" / suite_id) + ) from openral_observability.dashboard import attached_dashboard from openral_sim.benchmark import run_benchmark @@ -3014,7 +3039,10 @@ class was removed so the suite is a bare list of scenes whose id is vla=vla_spec, device=device, save_dir=str(save_dir) if save_dir is not None else None, + video_dir=str(effective_video_dir) if effective_video_dir is not None else None, ) + if effective_video_dir is not None: + console.print(f"[cyan]videos[/cyan] {effective_video_dir}") out_path.write_text(result.model_dump_json(indent=2)) avg = result.results.get("avg_success_rate", 0.0) @@ -3574,6 +3602,79 @@ def _summarize_results(results: dict[str, object]) -> str: # `tests/unit/test_cli_eval.py::test_bh_cli_import_is_light` guards this. app.add_typer(sim_app, name="sim") +# `openral behavior serve` — expose an rSkill through the official BEHAVIOR +# Challenge WebSocket policy protocol. OmniGibson remains in its own conda +# environment and owns task loading, metrics, and video recording. +behavior_app = typer.Typer( + name="behavior", + help="BEHAVIOR Challenge integration — serve OpenRAL rSkills to OmniGibson.", + no_args_is_help=True, +) +app.add_typer(behavior_app, name="behavior") + + +@behavior_app.command("serve") +def behavior_serve( + rskill: str = typer.Option( + ..., + "--rskill", + help="rSkill reference: local name/path or Hugging Face repo id.", + ), + task: str = typer.Option( + ..., + "--task", + help="BEHAVIOR task name, e.g. turning_on_radio.", + ), + instruction: str | None = typer.Option( + None, + "--instruction", + help="Policy language instruction. Defaults to the task name with underscores replaced.", + ), + host: str = typer.Option( + "127.0.0.1", + "--host", + help="WebSocket bind address.", + ), + port: int = typer.Option( + 8000, + "--port", + min=1, + max=65535, + help="WebSocket port used by omnigibson.eval.eval.", + ), + device: str = typer.Option( + "auto", + "--device", + help="Policy device override, e.g. auto, cuda:0, or cpu.", + ), + state_dim: int = typer.Option( + 61, + "--state-dim", + min=1, + help="Expected R1Pro proprioception width.", + ), + action_dim: int = typer.Option( + 23, + "--action-dim", + min=1, + help="Expected R1Pro action width.", + ), +) -> None: + """Serve one OpenRAL rSkill to BEHAVIOR's official evaluator.""" + from openral_cli.behavior import _serve_behavior_policy + + vla_spec = _parse_rskill_cli_arg(rskill).model_copy(update={"device": device}) + _serve_behavior_policy( + vla_spec, + task=task, + instruction=instruction or task.replace("_", " "), + host=host, + port=port, + state_dim=state_dim, + action_dim=action_dim, + ) + + # `openral install ` — post-install escape hatch for the # Tier-0 curl-bash installer (`scripts/install.sh`). The base install puts # `openral` on $PATH with the CLI's own thin runtime; sim physics, LIBERO, diff --git a/python/core/src/openral_core/schemas.py b/python/core/src/openral_core/schemas.py index 95bc481..ea46bf9 100644 --- a/python/core/src/openral_core/schemas.py +++ b/python/core/src/openral_core/schemas.py @@ -2201,6 +2201,9 @@ class WorldState(BaseModel): joint_state: Current joint state. base_pose: Base link pose (mobile robots). base_twist: Base link twist (vx, vy, vz, wx, wy, wz). + policy_state: Optional simulator-native proprioception vector for a + checkpoint whose state includes more than joint positions/velocities. + Populated through a typed HAL -> WorldState topic; never inferred. ee_poses: End-effector poses keyed by EE name. contact_forces: Contact forces keyed by contact name. images: Sensor name → ROS 2 topic reference (not the raw image). @@ -2224,6 +2227,7 @@ class WorldState(BaseModel): joint_state: JointState base_pose: Pose6D | None = None base_twist: tuple[float, float, float, float, float, float] | None = None + policy_state: list[float] | None = None ee_poses: dict[str, Pose6D] = Field(default_factory=dict) contact_forces: dict[str, tuple[float, ...]] = Field(default_factory=dict) images: dict[str, str] = Field(default_factory=dict) @@ -2551,6 +2555,9 @@ class Action(BaseModel): stamp_ns: Action timestamp in nanoseconds. ee_name: Target end-effector name. frame_id: Reference frame for Cartesian actions. + tick_index: Shared 1-based inference tick for every slot emitted by one + policy step. Preserved across the ROS safety wire so a simulation HAL + can commit a multi-surface action atomically after every slot passes. safety_overrides: Operator-approved safety override tokens. """ @@ -2575,6 +2582,7 @@ class Action(BaseModel): stamp_ns: int = 0 ee_name: str | None = None frame_id: str | None = None + tick_index: int = Field(default=0, ge=0) safety_overrides: dict[str, object] = Field(default_factory=dict) @@ -4016,6 +4024,19 @@ def scene_family(task_id: str) -> str: action_dim=8, runs_via_default_packers=False, ), + # BEHAVIOR-1K OmniGibson R1 Pro mixed 23-D: base body_twist(3) + torso+arms + # joint_position(4+7+7) + grippers(1+1), committed atomically as one + # official evaluator step. + "behavior": SceneTaskSpace( + modes=frozenset( + { + ControlMode.BODY_TWIST, + ControlMode.JOINT_POSITION, + ControlMode.GRIPPER_POSITION, + } + ), + action_dim=23, + ), } @@ -4319,6 +4340,7 @@ class EmbodimentExtra(BaseModel): "panda_mobile", "pusht", "rizon4", + "r1pro", "sawyer", "so100_follower", "so101_follower", @@ -4362,6 +4384,7 @@ class EmbodimentExtra(BaseModel): "aloha", "aloha_insertion", "aloha_transfer_cube", + "behavior", "gr1_tabletop", "libero_10", "libero_goal", diff --git a/python/hal/src/openral_hal/lifecycle.py b/python/hal/src/openral_hal/lifecycle.py index 09d4dd1..c6ed150 100644 --- a/python/hal/src/openral_hal/lifecycle.py +++ b/python/hal/src/openral_hal/lifecycle.py @@ -133,6 +133,10 @@ def decode_action_chunk(msg: object) -> object | None: rows: list[list[float]] = [flat[s * n_dof : (s + 1) * n_dof] for s in range(horizon)] kwargs: dict[str, Any] = {"control_mode": mode, "horizon": horizon} + kwargs["ee_name"] = str(getattr(msg, "ee_name", "") or "") or None + kwargs["frame_id"] = str(getattr(msg, "frame_id", "") or "") or None + kwargs["confidence"] = float(getattr(msg, "confidence", 1.0) or 1.0) + kwargs["tick_index"] = int(getattr(msg, "tick_index", 0) or 0) if mode in (ControlMode.JOINT_POSITION, ControlMode.JOINT_TRAJECTORY): kwargs["joint_targets"] = rows elif mode is ControlMode.JOINT_VELOCITY: @@ -339,6 +343,7 @@ def __init__(self, node_name: str) -> None: self._timer: Any = None self._publisher: Any = None self._joint_state_pub: Any = None + self._policy_state_pub: Any = None self._safe_action_sub: Any = None self._estop_sub: Any = None self._estop_reset_sub: Any = None @@ -515,7 +520,7 @@ def on_activate(self, state: object) -> TransitionCallbackReturn: QoSReliabilityPolicy, ) from sensor_msgs.msg import JointState as RosJointState - from std_msgs.msg import Empty + from std_msgs.msg import Empty, Float32MultiArray control_qos = QoSProfile( reliability=QoSReliabilityPolicy.RELIABLE, @@ -531,6 +536,11 @@ def on_activate(self, state: object) -> TransitionCallbackReturn: RosJointState, "/joint_states", control_qos ) self._publisher = self.create_publisher(RosJointState, "~/joint_states", control_qos) + self._policy_state_pub = self.create_publisher( + Float32MultiArray, + "/openral/policy_state", + control_qos, + ) # Sim-attached HALs (those exposing ``idle_step``) read # MjData; publish odom/joint_state off a dedicated thread (below) # from a plain-data snapshot, so they aren't starved by env.step. @@ -664,6 +674,9 @@ def on_deactivate(self, state: object) -> TransitionCallbackReturn: if self._joint_state_pub is not None: self.destroy_publisher(self._joint_state_pub) self._joint_state_pub = None + if self._policy_state_pub is not None: + self.destroy_publisher(self._policy_state_pub) + self._policy_state_pub = None return TransitionCallbackReturn.SUCCESS def on_cleanup(self, state: object) -> TransitionCallbackReturn: @@ -762,6 +775,13 @@ def _capture_proprio(self) -> None: # env.step. ``None`` for clock-less / sidecar HALs → no /clock. sim_time_getter = getattr(self._hal, "sim_time_ns", None) sim_time_ns = sim_time_getter() if sim_time_getter is not None else None + policy_state_getter = getattr(self._hal, "read_policy_state", None) + policy_state_raw = policy_state_getter() if policy_state_getter is not None else None + policy_state = ( + tuple(float(value) for value in policy_state_raw) + if policy_state_raw is not None + else None + ) self._proprio.set( ProprioFrame( state=state, @@ -769,6 +789,7 @@ def _capture_proprio(self) -> None: base_pose_6dof=pose_6dof, base_twist=tuple(float(v) for v in twist), sim_time_ns=sim_time_ns, + policy_state=policy_state, ) ) @@ -842,6 +863,14 @@ def _publish_joint_state(self) -> None: self._publisher.publish(msg) if self._joint_state_pub is not None: self._joint_state_pub.publish(msg) + if self._policy_state_pub is not None: + frame = self._proprio.latest() if self._proprio is not None else None + if frame is not None and frame.policy_state is not None: + from std_msgs.msg import Float32MultiArray + + policy_msg = Float32MultiArray() + policy_msg.data = list(frame.policy_state) + self._policy_state_pub.publish(policy_msg) def _on_safe_action(self, msg: object) -> None: """``/openral/safe_action`` callback. diff --git a/python/hal/src/openral_hal/proprio_snapshot.py b/python/hal/src/openral_hal/proprio_snapshot.py index 661754a..2827204 100644 --- a/python/hal/src/openral_hal/proprio_snapshot.py +++ b/python/hal/src/openral_hal/proprio_snapshot.py @@ -52,6 +52,7 @@ class ProprioFrame: step (`SimAttachedHAL.sim_time_ns`), or ``None`` for a clock-less / wall-clock HAL. Captured here so the publisher thread can emit ``/clock`` without touching the simulator. + policy_state: Optional simulator-native checkpoint state vector. """ state: JointState @@ -59,6 +60,7 @@ class ProprioFrame: base_pose_6dof: Pose6Dof | None base_twist: tuple[float, ...] sim_time_ns: int | None = None + policy_state: tuple[float, ...] | None = None class ProprioSnapshot: diff --git a/python/hal/src/openral_hal/sim_attached.py b/python/hal/src/openral_hal/sim_attached.py index 19daad2..16af033 100644 --- a/python/hal/src/openral_hal/sim_attached.py +++ b/python/hal/src/openral_hal/sim_attached.py @@ -473,6 +473,12 @@ def __init__( # arm's OSC-delta slot is RE-ZEROED right before each step so # the policy's per-step delta is applied once, not accumulated. self._last_env_action: NDArray[np.float32] | None = None + # Optional atomic multi-surface action staging. Backends that expose + # ``step_action_group(actions)`` receive every safety-approved slot from + # one ActionChunk.tick_index and step exactly once when their declared + # ``action_group_size`` is complete. + self._pending_action_tick: int | None = None + self._pending_actions: list[Action] = [] # Latched when the last env.step reported terminal # (terminated/truncated). The next send_action resets the env # before stepping so episodic backends (LIBERO) never step a @@ -529,6 +535,8 @@ def connect(self) -> None: self._last_state_ns = time.time_ns() self._connected = True self._episode_done = False # fresh episode after (re)connect + self._pending_action_tick = None + self._pending_actions.clear() self._joint_index = None # rebuilt on next read_state (model identity stable per env) if self._env_action_dim is None: self._env_action_dim = self._probe_env_action_dim() @@ -687,6 +695,10 @@ def send_action(self, action: Action) -> None: "SimAttachedHAL.send_action: env_action_dim resolved to None; " "re-connect or pass it explicitly to the constructor." ) + group_step = getattr(self._env, "step_action_group", None) + if callable(group_step): + self._stage_action_group(action, group_step) + return # BODY_TWIST direct-qpos path. The default ``pack_action_for_env`` # packs ``[vx, vy, wz]`` into slots 0-2 of robocasa's composite- # controller action vector, but robocasa's BASIC controller @@ -772,6 +784,57 @@ def send_action(self, action: Action) -> None: self._last_env_action = None # stale: belongs to the prior episode self._step_and_cache(env_action, source="send_action") + def _stage_action_group( + self, + action: Action, + group_step: Callable[[list[Action]], Any], + ) -> None: + """Commit one simulator step after every slot in an inference tick is safe.""" + group_size = int(getattr(self._env, "action_group_size", 0) or 0) + if group_size <= 0: + raise ROSConfigError( + "SimAttachedHAL: backend exposes step_action_group() without a positive " + "action_group_size." + ) + tick = int(action.tick_index) + if tick <= 0: + raise ROSConfigError( + "SimAttachedHAL: atomic action-group backend requires Action.tick_index > 0." + ) + if self._pending_action_tick is not None and tick != self._pending_action_tick: + print( + f"[sim_attached.send_action] dropping incomplete safe action group " + f"tick={self._pending_action_tick} " + f"slots={len(self._pending_actions)}/{group_size}; " + f"starting tick={tick}", + flush=True, + ) + self._pending_actions.clear() + self._pending_action_tick = None + if self._pending_action_tick is None: + self._pending_action_tick = tick + self._pending_actions.append(action) + if len(self._pending_actions) < group_size: + return + if len(self._pending_actions) > group_size: + self._pending_actions.clear() + self._pending_action_tick = None + raise ROSRuntimeError( + f"SimAttachedHAL: action group tick={tick} exceeded {group_size} slots." + ) + if self._episode_done: + self._reset_terminated_episode("send_action_group", trigger="returned-terminal") + actions = list(self._pending_actions) + self._pending_actions.clear() + self._pending_action_tick = None + try: + step_result = group_step(actions) + except Exception as exc: + raise ROSRuntimeError( + f"SimAttachedHAL.send_action_group: env step failed: {exc}" + ) from exc + self._cache_step_result(step_result) + def _step_and_cache(self, env_action: NDArray[np.float32], *, source: str) -> bool: """Deferred-reset → ``env.step`` → re-cache ``_last_obs`` → re-latch terminal. @@ -837,6 +900,11 @@ def _step_and_cache(self, env_action: NDArray[np.float32], *, source: str) -> bo # re-stepping the simulator. A dict-shaped Protocol fallback # ``getattr(..., 'observation', None)`` is enough because every # in-tree backend returns a ``StepResult`` with this attribute. + self._cache_step_result(step_result) + return True + + def _cache_step_result(self, step_result: object) -> None: + """Cache one backend transition and latch its terminal state.""" obs = getattr(step_result, "observation", None) if isinstance(obs, dict): self._last_obs = dict(obs) @@ -844,7 +912,6 @@ def _step_and_cache(self, env_action: NDArray[np.float32], *, source: str) -> bo self._episode_done = bool( getattr(step_result, "terminated", False) or getattr(step_result, "truncated", False) ) - return True def _reset_terminated_episode(self, source: str, *, trigger: str) -> None: """Reset the env after an episode terminal and clear the terminal latch. @@ -919,7 +986,10 @@ def idle_step(self) -> bool: targets) a zero vector commands the joints *toward 0 rad* rather than holding the current pose. That is acceptable here — the goal is to keep the scene physically live so cameras render, not to freeze the arm in - place — but it is NOT a literal hold for those backends. + place — but it is NOT a literal hold for those backends. A backend can + opt out of the zero vector entirely by exposing ``idle_action()`` + returning its own hold vector (the BEHAVIOR-1K backend returns the + current joint targets so idle ticks genuinely hold pose). Action-dim note: ``_env_action_dim`` is the env's authoritative width, resolved by :meth:`_probe_env_action_dim` from the backend's own @@ -943,21 +1013,38 @@ def idle_step(self) -> bool: return False if self._env_action_dim is None: return False + if self._pending_actions: + return False # No MuJoCo-handle gate: idle-stepping is valid for ANY wrapped # SimRollout — a zero action is a HOLD for the sim's velocity / OSC-delta # controllers, and the method-only-on-SimAttachedHAL exclusion (real HALs # never define idle_step) is the real safety guarantee. Non-MuJoCo # backends (Isaac Sim sidecar, ManiSkill3) step via env.step(zeros) the # same as MJCF ones. - # Zero-action step — the same env.step(zeros) idiom as the - # deferred-reset path / send_action's tail (NOT robocasa.refresh_obs, - # which re-renders WITHOUT stepping). The shared ``_step_and_cache`` - # does the deferred reset → step → obs re-cache → terminal re-latch so - # this path cannot drift from send_action. Never build a non-zero - # vector here. ``_step_and_cache`` deliberately leaves - # ``_last_env_action`` / ``_last_body_twist`` untouched. - zero_action = np.zeros(self._env_action_dim, dtype=np.float32) - return self._step_and_cache(zero_action, source="idle_step") + # Hold-action step — the same env.step idiom as the deferred-reset + # path / send_action's tail (NOT robocasa.refresh_obs, which + # re-renders WITHOUT stepping). The shared ``_step_and_cache`` does + # the deferred reset → step → obs re-cache → terminal re-latch so + # this path cannot drift from send_action. The hold vector is zeros + # (a true HOLD for velocity / OSC-delta controllers) UNLESS the + # backend exposes ``idle_action()`` — position-controlled backends + # (BEHAVIOR-1K joint targets) return their current hold pose there, + # because for them a zero vector is a "drive to origin" command, not + # a hold. Never build any other vector here. ``_step_and_cache`` + # deliberately leaves ``_last_env_action`` / ``_last_body_twist`` + # untouched. + idle_action_getter = getattr(self._env, "idle_action", None) + idle_action = ( + np.asarray(idle_action_getter(), dtype=np.float32).reshape(-1) + if callable(idle_action_getter) + else np.zeros(self._env_action_dim, dtype=np.float32) + ) + if idle_action.shape != (self._env_action_dim,): + raise ROSRuntimeError( + f"SimAttachedHAL.idle_step: backend idle_action width " + f"{idle_action.shape[0]} != env_action_dim {self._env_action_dim}." + ) + return self._step_and_cache(idle_action, source="idle_step") # ── Per-mode → composite-controller slot mapping ────────── def _composite_controller(self) -> Any: # noqa: ANN401 # reason: robosuite composite controller is an untyped third-party object @@ -1326,6 +1413,8 @@ def _merge_refreshed_obs(self, refreshed: Any) -> None: # noqa: ANN401 # reaso def estop(self) -> None: """Latch e-stop. Subsequent send_action calls are dropped.""" self._estop_latched = True + self._pending_action_tick = None + self._pending_actions.clear() # ── Helpers exposed to the lifecycle node ────────────────────────── @@ -1645,3 +1734,19 @@ def base_pose_6dof( def reset_estop(self) -> None: """Clear the estop latch. Caller asserts the cause has been resolved.""" self._estop_latched = False + + def read_policy_state(self) -> list[float] | None: + """Return the simulator-native policy state, when the backend exposes one. + + Only an explicit ``obs["policy_state"]`` counts — the generic + ``obs["state"]`` every sim backend populates is NOT a policy state, and + falling back to it would silently publish `/openral/policy_state` for + robots whose manifests never opted in (WorldState.policy_state is + documented as never inferred). + """ + if self._last_obs is None: + return None + raw = self._last_obs.get("policy_state") + if raw is None: + return None + return [float(value) for value in np.asarray(raw, dtype=np.float32).reshape(-1)] diff --git a/python/sim/src/openral_sim/_deps.py b/python/sim/src/openral_sim/_deps.py index c17b80f..7309d02 100644 --- a/python/sim/src/openral_sim/_deps.py +++ b/python/sim/src/openral_sim/_deps.py @@ -1142,6 +1142,34 @@ def _has_isaac_client() -> bool: return _has_module("zmq") and _has_module("msgpack") +def _behavior_groot_client_plan() -> BackendInstallPlan: + uv = _uv() + return BackendInstallPlan( + backend_id="behavior_groot_client", + display_name="BEHAVIOR GR00T adapter wire (pyzmq + msgpack)", + license_note=( + "Pulls pyzmq (LGPL+ZeroMQ exception) and msgpack (Apache-2.0). " + "The official Isaac-GR00T behavior runtime and checkpoint remain in " + "an externally-provisioned Python 3.10 sidecar." + ), + probe=_has_isaac_client, + steps=( + InstallStep( + description="uv sync --group behavior-groot --inexact", + argv=[ + uv, + "sync", + "--all-packages", + "--group", + "behavior-groot", + "--inexact", + ], + ), + ), + manual_hint="just sync --all-packages --group behavior-groot --inexact", + ) + + def _isaac_client_plan() -> BackendInstallPlan: uv = _uv() return BackendInstallPlan( @@ -1746,6 +1774,7 @@ def _openarm_robosuite_plan() -> BackendInstallPlan: "openarm_robosuite": _openarm_robosuite_plan, "rldx_client": _rldx_client_plan, "rldx_sidecar_setup": _rldx_sidecar_setup_plan, + "behavior_groot_client": _behavior_groot_client_plan, "isaac_client": _isaac_client_plan, "rlbench_client": _rlbench_client_plan, "robotwin_client": _robotwin_client_plan, diff --git a/python/sim/src/openral_sim/backends/__init__.py b/python/sim/src/openral_sim/backends/__init__.py index a04fb47..b05f32f 100644 --- a/python/sim/src/openral_sim/backends/__init__.py +++ b/python/sim/src/openral_sim/backends/__init__.py @@ -30,7 +30,10 @@ box/tube task is coupled to the so_arm101 MJCF schema), ``rlbench`` (franka_panda — CoppeliaSim/PyRep tasks behind an out-of-process py3.10 sidecar), ``robotwin`` (aloha_agilex — the RoboTwin 2.0 - SAPIEN dual-arm benchmark behind a py3.10 sidecar). + SAPIEN dual-arm benchmark behind a py3.10 sidecar), ``vlabench`` + (franka_panda — native in-process on lerobot's VLABenchEnv), ``behavior`` + (r1pro — the BEHAVIOR-1K 2026 OmniGibson evaluator behind an + out-of-process sidecar in the official BEHAVIOR Python environment). """ from __future__ import annotations @@ -45,6 +48,7 @@ def _register_backends() -> None: """ from openral_sim.backends import ( aloha, + behavior, isaac_sim, libero, maniskill3, diff --git a/python/sim/src/openral_sim/backends/behavior.py b/python/sim/src/openral_sim/backends/behavior.py new file mode 100644 index 0000000..09c6c41 --- /dev/null +++ b/python/sim/src/openral_sim/backends/behavior.py @@ -0,0 +1,347 @@ +"""BEHAVIOR-1K / OmniGibson scene adapter via the official evaluator.""" + +from __future__ import annotations + +import contextlib +import hashlib +import os +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any + +import numpy as np +from numpy.typing import NDArray +from openral_core.exceptions import ROSConfigError, ROSRuntimeError +from openral_core.schemas import Action, ControlMode + +from openral_sim.registry import SCENES +from openral_sim.rollout import StepResult +from openral_sim.sidecar import SidecarClient + +if TYPE_CHECKING: + from openral_core import SceneSpec, SimEnvironment, TaskSpec + + from openral_sim.rollout import Observation + +_SCENE_ID = "behavior" +_ROBOT_ID = "r1pro" +_SIDECAR_PYTHON_ENV = "OPENRAL_BEHAVIOR_SIDECAR_PYTHON" +_SIDECAR_SCRIPT_ENV = "OPENRAL_BEHAVIOR_SIDECAR_SCRIPT" +_AUTO_SPAWN_ENV = "OPENRAL_BEHAVIOR_AUTO_SPAWN" +_HOST_ENV = "OPENRAL_BEHAVIOR_HOST" +_PORT_ENV = "OPENRAL_BEHAVIOR_PORT" + +_DEFAULT_HOST = "127.0.0.1" +_PORT_MIN = 23_000 +_PORT_MAX = 23_999 +_DEFAULT_TIMEOUT_MS = 120_000 +_DEFAULT_BOOT_TIMEOUT_S = 1_200.0 +_ACTION_DIM = 23 +_ACTION_GROUP_SIZE = 6 +_JOINT_DIM = 22 + +_STATE_KEY = "robot_r1::proprio" +_CAMERA_KEYS = { + "head": "robot_r1::robot_r1:zed_link:Camera:0::rgb", + "left_wrist": "robot_r1::robot_r1:left_realsense_link:Camera:0::rgb", + "right_wrist": "robot_r1::robot_r1:right_realsense_link:Camera:0::rgb", +} + + +def _scene_default_port(task: str, instance_index: int, mode: str) -> int: + key = f"behavior|{task}|{instance_index}|{mode}".encode() + digest = int.from_bytes(hashlib.sha256(key).digest()[:4], "big") + return _PORT_MIN + (digest % (_PORT_MAX - _PORT_MIN)) + + +def _opt_int(value: object, default: int) -> int: + if isinstance(value, bool) or not isinstance(value, (int, float, str)): + return default + try: + return int(value) + except (TypeError, ValueError): + return default + + +def _opt_float(value: object, default: float) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float, str)): + return default + try: + return float(value) + except (TypeError, ValueError): + return default + + +def _sidecar_python() -> Path: + override = os.environ.get(_SIDECAR_PYTHON_ENV) + candidates = ( + [Path(override).expanduser()] + if override + else [ + Path.home() / ".cache" / "openral" / "behavior" / ".venv" / "bin" / "python", + Path.home() / "miniconda3" / "envs" / "behavior" / "bin" / "python", + Path.home() / "anaconda3" / "envs" / "behavior" / "bin" / "python", + ] + ) + for path in candidates: + if path.is_file(): + return path + raise ROSConfigError( + "BEHAVIOR sidecar Python not found. Install BEHAVIOR-1K v3.9.0 with " + "`./setup.sh --new-env --omnigibson --bddl --joylo --dataset --eval`, " + f"then set {_SIDECAR_PYTHON_ENV} to that environment's python." + ) + + +def _locate_sidecar_script() -> Path: + override = os.environ.get(_SIDECAR_SCRIPT_ENV) + if override: + path = Path(override).expanduser().resolve() + if not path.is_file(): + raise ROSConfigError(f"{_SIDECAR_SCRIPT_ENV}={override!r} is not a file.") + return path + here = Path(__file__).resolve() + for parent in here.parents: + candidate = parent / "tools" / "behavior_scene_sidecar.py" + if candidate.is_file(): + return candidate + raise ROSConfigError( + f"Could not locate tools/behavior_scene_sidecar.py upwards from {here}. " + f"Set {_SIDECAR_SCRIPT_ENV} to its absolute path." + ) + + +def _joint_state_from_policy_state( + state: NDArray[np.float32], +) -> tuple[NDArray[np.float32], NDArray[np.float32]]: + positions = np.concatenate( + [ + state[53:57], + state[3:10], + state[24:26], + state[28:35], + state[49:51], + ] + ).astype(np.float32) + velocities = np.concatenate( + [ + state[57:61], + state[10:17], + state[26:28], + state[35:42], + state[51:53], + ] + ).astype(np.float32) + return positions, velocities + + +def _compose_action_group(actions: list[Action]) -> NDArray[np.float32]: + if len(actions) != _ACTION_GROUP_SIZE: + raise ROSRuntimeError( + f"BEHAVIOR action group requires {_ACTION_GROUP_SIZE} slots, got {len(actions)}." + ) + expected = ( + ControlMode.BODY_TWIST, + ControlMode.JOINT_POSITION, + ControlMode.JOINT_POSITION, + ControlMode.GRIPPER_POSITION, + ControlMode.JOINT_POSITION, + ControlMode.GRIPPER_POSITION, + ) + modes = tuple(action.control_mode for action in actions) + if modes != expected: + raise ROSRuntimeError( + f"BEHAVIOR action slot order mismatch: got {[mode.value for mode in modes]}." + ) + + base, torso, left_arm, left_gripper, right_arm, right_gripper = actions + if not base.body_twist: + raise ROSRuntimeError("BEHAVIOR base slot has no body_twist payload.") + vx, vy, _vz, _wx, _wy, wz = base.body_twist[0] + + def _joint_row(action: Action) -> NDArray[np.float32]: + if not action.joint_targets: + raise ROSRuntimeError("BEHAVIOR joint slot has no joint_targets payload.") + row = np.asarray(action.joint_targets[0], dtype=np.float32).reshape(-1) + if row.shape != (_JOINT_DIM,): + raise ROSRuntimeError( + f"BEHAVIOR joint slot must be padded to {_JOINT_DIM}, got {row.shape[0]}." + ) + return row + + torso_row = _joint_row(torso) + left_row = _joint_row(left_arm) + right_row = _joint_row(right_arm) + if not left_gripper.gripper or not right_gripper.gripper: + raise ROSRuntimeError("BEHAVIOR gripper slots must carry scalar gripper payloads.") + + return np.asarray( + [ + vx, + vy, + wz, + *torso_row[0:4], + *left_row[4:11], + float(left_gripper.gripper[0]), + *right_row[13:20], + float(right_gripper.gripper[0]), + ], + dtype=np.float32, + ) + + +@dataclass +class _BehaviorSidecar: + scene: SceneSpec + task: TaskSpec + _client: SidecarClient + _last_image: NDArray[np.uint8] | None = None + _last_state: NDArray[np.float32] | None = None + _sim_time_ns: int | None = None + + @property + def action_dim(self) -> int: + return _ACTION_DIM + + @property + def action_group_size(self) -> int: + return _ACTION_GROUP_SIZE + + def reset(self, seed: int | None = None) -> Observation: + reply = self._client.call("reset", {"seed": seed}) + self._sim_time_ns = int(reply.get("sim_time_ns", 0)) + return self._wrap_obs(self._client.require(reply, "observation")) + + def step(self, action: NDArray[np.float32]) -> StepResult: + return self._step_vector(np.asarray(action, dtype=np.float32).reshape(-1)) + + def idle_action(self) -> NDArray[np.float32]: + """Hold the current articulated pose while advancing scene physics.""" + if self._last_state is None: + return np.zeros(_ACTION_DIM, dtype=np.float32) + state = self._last_state + left_gripper = 2.0 * float(np.mean(state[24:26]) / 0.05) - 1.0 + right_gripper = 2.0 * float(np.mean(state[49:51]) / 0.05) - 1.0 + return np.asarray( + [ + 0.0, + 0.0, + 0.0, + *state[53:57], + *state[3:10], + np.clip(left_gripper, -1.0, 1.0), + *state[28:35], + np.clip(right_gripper, -1.0, 1.0), + ], + dtype=np.float32, + ) + + def step_action_group(self, actions: list[Action]) -> StepResult: + ticks = {action.tick_index for action in actions} + if len(ticks) != 1 or 0 in ticks: + raise ROSRuntimeError( + f"BEHAVIOR action group has invalid tick indices: {sorted(ticks)}" + ) + return self._step_vector(_compose_action_group(actions)) + + def _step_vector(self, action: NDArray[np.float32]) -> StepResult: + if action.shape != (_ACTION_DIM,): + raise ROSRuntimeError( + f"BEHAVIOR environment expects {_ACTION_DIM} actions, got {action.shape[0]}." + ) + reply = self._client.call("step", {"action": action}) + self._sim_time_ns = int(reply.get("sim_time_ns", 0)) + return StepResult( + observation=self._wrap_obs(self._client.require(reply, "observation")), + reward=float(self._client.require(reply, "reward")), + terminated=bool(self._client.require(reply, "terminated")), + truncated=bool(self._client.require(reply, "truncated")), + info=dict(reply.get("info", {})), + ) + + def render(self) -> NDArray[np.uint8] | None: + return None if self._last_image is None else self._last_image.copy() + + def sim_time_ns(self) -> int | None: + return self._sim_time_ns + + def close(self) -> None: + with contextlib.suppress(Exception): + self._client.call("close") + self._client.close() + + def _wrap_obs(self, raw: dict[str, Any]) -> Observation: + state = np.asarray(raw.get(_STATE_KEY, []), dtype=np.float32).reshape(-1) + if state.shape != (61,): + raise ROSRuntimeError( + f"BEHAVIOR sidecar observation state must have 61 values, got {state.shape[0]}." + ) + self._last_state = state.copy() + images: dict[str, NDArray[np.uint8]] = {} + for role, raw_key in _CAMERA_KEYS.items(): + image = raw.get(raw_key) + if image is not None: + images[role] = np.asarray(image, dtype=np.uint8) + if images: + self._last_image = images.get("head", next(iter(images.values()))) + positions, velocities = _joint_state_from_policy_state(state) + return { + "images": images, + "state": state, + "policy_state": state, + "joint_positions": positions, + "joint_velocities": velocities, + "task": self.task.instruction, + "behavior_raw": raw, + } + + +@SCENES.register(_SCENE_ID, fixed_robot=_ROBOT_ID) +def _build_behavior_scene(env_cfg: SimEnvironment) -> _BehaviorSidecar: + from openral_sim._deps import ensure_backend_deps + + ensure_backend_deps("behavior_groot_client") + opts = env_cfg.scene.backend_options + task = str(opts.get("task") or env_cfg.task.id.removeprefix("behavior/")) + instance_index = _opt_int(opts.get("instance_index"), 0) + mode = str(opts.get("mode", "public_test")) + env_wrapper = str(opts.get("env_wrapper", "omnigibson.eval.wrappers.DefaultWrapper")) + host = os.environ.get(_HOST_ENV, str(opts.get("host", _DEFAULT_HOST))) + default_port = _scene_default_port(task, instance_index, mode) + port = _opt_int(os.environ.get(_PORT_ENV, opts.get("port")), default_port) + auto_spawn = os.environ.get(_AUTO_SPAWN_ENV, "1") != "0" + + launch_argv: list[str] = [] + if auto_spawn: + launch_argv = [ + str(_sidecar_python()), + str(_locate_sidecar_script()), + "--task", + task, + "--instance-index", + str(instance_index), + "--mode", + mode, + "--env-wrapper", + env_wrapper, + "--max-steps", + str(env_cfg.task.max_steps or 500), + "--host", + host, + "--port", + str(port), + "--headless", + ] + + client = SidecarClient( + name="behavior", + host=host, + port=port, + timeout_ms=_opt_int(opts.get("timeout_ms"), _DEFAULT_TIMEOUT_MS), + boot_timeout_s=_opt_float(opts.get("boot_timeout_s"), _DEFAULT_BOOT_TIMEOUT_S), + launch_argv=launch_argv, + auto_spawn=auto_spawn, + expected_identity={"scene": "behavior", "task": task, "instance_index": instance_index}, + ) + client.connect() + return _BehaviorSidecar(scene=env_cfg.scene, task=env_cfg.task, _client=client) diff --git a/python/sim/src/openral_sim/benchmark.py b/python/sim/src/openral_sim/benchmark.py index 3c083f9..7c371ba 100644 --- a/python/sim/src/openral_sim/benchmark.py +++ b/python/sim/src/openral_sim/benchmark.py @@ -186,6 +186,7 @@ def run_benchmark( vla: VLASpec, device: str | None = None, save_dir: str | None = None, + video_dir: str | None = None, ) -> tuple[RSkillEvalResult, list[EpisodeResult]]: """Run a benchmark suite end-to-end against one rSkill. @@ -213,6 +214,13 @@ def run_benchmark( for adapter-side artefacts (videos, traces). Unrelated to where the final :class:`RSkillEvalResult` JSON lives — that path is chosen by the caller (see :func:`default_output_path`). + video_dir: When set, record per-step world frames and write one MP4 + per episode into this directory via + :func:`openral_sim._website_video.write_world_videos` + (named ``[_seed]__.mp4`` plus a + merged ``videos.json``). Frames are written and freed after each + episode so multi-hundred-episode suites don't accumulate + gigabytes of RGB in memory. ``None`` disables frame capture. Returns: A pair ``(result, episodes)`` where ``result`` is a validated @@ -293,6 +301,7 @@ def run_benchmark( seed=seed, n_episodes=1, save_dir=save_dir, + record_video=video_dir is not None, ) runner = SimRunner(env_cfg, view=False, strict_view=False) try: @@ -302,6 +311,20 @@ def run_benchmark( finally: runner.deactivate() episode = episodes[0] + if video_dir is not None: + from openral_sim._website_video import write_world_videos + + # Suffix the seed so multi-episode sweeps of one task don't + # overwrite each other's __.mp4. + video_slug = task_id if len(seeds) == 1 else f"{task_id}_seed{seed}" + write_world_videos( + [episode], + Path(video_dir), + scene=video_slug, + rskill=Path(str(vla.weights_uri)).name, + section="benchmark", + ) + episode.frames.clear() per_task[task_id].append(episode.success) all_episodes.append(episode) _log.info( diff --git a/python/sim/src/openral_sim/policies/behavior_groot.py b/python/sim/src/openral_sim/policies/behavior_groot.py new file mode 100644 index 0000000..c8801a1 --- /dev/null +++ b/python/sim/src/openral_sim/policies/behavior_groot.py @@ -0,0 +1,272 @@ +"""Official BEHAVIOR-1K GR00T policy adapter via an isolated sidecar.""" + +from __future__ import annotations + +import contextlib +import hashlib +import os +from dataclasses import dataclass, field +from pathlib import Path +from typing import TYPE_CHECKING, Protocol + +import numpy as np +from numpy.typing import NDArray +from openral_core.exceptions import ROSConfigError, ROSRuntimeError + +from openral_sim.sidecar import SidecarClient + +if TYPE_CHECKING: + from openral_core import RSkillManifest, VLASpec + + from openral_sim.rollout import Observation + +_SIDECAR_PYTHON_ENV = "OPENRAL_BEHAVIOR_GROOT_SIDECAR_PYTHON" +_SIDECAR_SCRIPT_ENV = "OPENRAL_BEHAVIOR_GROOT_SIDECAR_SCRIPT" +_CHECKPOINT_ENV = "OPENRAL_BEHAVIOR_GROOT_CHECKPOINT" +_HOST_ENV = "OPENRAL_BEHAVIOR_GROOT_HOST" +_PORT_ENV = "OPENRAL_BEHAVIOR_GROOT_PORT" +_AUTO_SPAWN_ENV = "OPENRAL_BEHAVIOR_GROOT_AUTO_SPAWN" + +_DEFAULT_SIDECAR_PYTHON = ( + Path.home() / ".cache" / "openral" / "behavior-groot" / ".venv" / "bin" / "python" +) +_DEFAULT_CHECKPOINT = Path("checkpoints/behavior-groot-turning-on-radio") +_DEFAULT_HOST = "127.0.0.1" +_PORT_MIN = 22_000 +_PORT_MAX = 22_999 +_DEFAULT_TIMEOUT_MS = 120_000 +_DEFAULT_BOOT_TIMEOUT_S = 600.0 +_IMPLEMENTATION = "behavior_b1k_sidecar" + +_STATE_KEY = "robot_r1::proprio" +_CAMERA_KEYS = { + "head": "robot_r1::robot_r1:zed_link:Camera:0::rgb", + "left_wrist": "robot_r1::robot_r1:left_realsense_link:Camera:0::rgb", + "right_wrist": "robot_r1::robot_r1:right_realsense_link:Camera:0::rgb", +} +_STATE_DIM = 61 +_ACTION_DIM = 23 + + +class _EnvCfg(Protocol): + vla: VLASpec + + +def _policy_default_port(task: str, checkpoint: str) -> int: + key = f"behavior-groot|{task}|{checkpoint}".encode() + digest = int.from_bytes(hashlib.sha256(key).digest()[:4], "big") + return _PORT_MIN + (digest % (_PORT_MAX - _PORT_MIN)) + + +def _opt_int(value: object, default: int) -> int: + if isinstance(value, bool) or not isinstance(value, (int, float, str)): + return default + try: + return int(value) + except (TypeError, ValueError): + return default + + +def _opt_float(value: object, default: float) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float, str)): + return default + try: + return float(value) + except (TypeError, ValueError): + return default + + +def _checkpoint_path(manifest: RSkillManifest) -> Path: + override = os.environ.get(_CHECKPOINT_ENV) + if override: + return Path(override).expanduser() + weights_uri = str(manifest.weights_uri or "") + if weights_uri.startswith("local://"): + return Path(weights_uri.removeprefix("local://")).expanduser() + return _DEFAULT_CHECKPOINT + + +def _sidecar_python() -> Path: + override = os.environ.get(_SIDECAR_PYTHON_ENV) + path = Path(override).expanduser() if override else _DEFAULT_SIDECAR_PYTHON + if path.is_file(): + return path + raise ROSConfigError( + "BEHAVIOR GR00T sidecar venv not found. Provision the pinned " + "wensi-ai/Isaac-GR00T behavior branch under Python 3.10, then set " + f"{_SIDECAR_PYTHON_ENV} to its .venv/bin/python. See " + "rskills/gr00t-n17-b1k-turning-on-radio/README.md." + ) + + +def _locate_sidecar_script() -> Path: + override = os.environ.get(_SIDECAR_SCRIPT_ENV) + if override: + path = Path(override).expanduser().resolve() + if not path.is_file(): + raise ROSConfigError(f"{_SIDECAR_SCRIPT_ENV}={override!r} is not a file.") + return path + here = Path(__file__).resolve() + for parent in here.parents: + candidate = parent / "tools" / "behavior_groot_sidecar.py" + if candidate.is_file(): + return candidate + raise ROSConfigError( + f"Could not locate tools/behavior_groot_sidecar.py upwards from {here}. " + f"Set {_SIDECAR_SCRIPT_ENV} to its absolute path." + ) + + +def _behavior_wire_observation( + observation: Observation, + *, + instruction: str, +) -> dict[str, object]: + raw = observation.get("behavior_raw") + if isinstance(raw, dict) and all(isinstance(key, str) for key in raw): + return {**raw, "openral_instruction": instruction} + + state_raw = observation.get("state") + if state_raw is None: + raise ROSRuntimeError( + "BEHAVIOR GR00T requires either observation['behavior_raw'] from the " + "official evaluator or a 61-D observation['state'] vector." + ) + state = np.asarray(state_raw, dtype=np.float32).reshape(-1) + if state.shape != (_STATE_DIM,): + raise ROSRuntimeError( + f"BEHAVIOR GR00T state must have {_STATE_DIM} values, got {state.shape[0]}." + ) + + images = observation.get("images", {}) + wire: dict[str, object] = { + _STATE_KEY: state, + "openral_instruction": instruction, + } + for role, wire_key in _CAMERA_KEYS.items(): + image = images.get(role) + if image is None: + raise ROSRuntimeError(f"BEHAVIOR GR00T observation is missing camera {role!r}.") + wire[wire_key] = np.asarray(image, dtype=np.uint8) + return wire + + +@dataclass +class _BehaviorGrootAdapter: + """PolicyAdapter proxying the official B1K GR00T runtime.""" + + spec: VLASpec + device: str + _client: SidecarClient + _last_input: NDArray[np.uint8] | None = field(default=None) + + def reset(self) -> None: + self._client.call("reset") + + def step(self, observation: Observation, instruction: str) -> NDArray[np.float32]: + images = observation.get("images", {}) + head = images.get("head") + if head is not None: + self._last_input = np.asarray(head, dtype=np.uint8) + reply = self._client.call( + "get_action", + {"observation": _behavior_wire_observation(observation, instruction=instruction)}, + ) + action = np.asarray(self._client.require(reply, "action"), dtype=np.float32).reshape(-1) + if action.shape != (_ACTION_DIM,): + raise ROSRuntimeError( + f"BEHAVIOR GR00T sidecar returned {action.shape[0]} actions; " + f"expected {_ACTION_DIM}." + ) + if not np.isfinite(action).all(): + raise ROSRuntimeError("BEHAVIOR GR00T sidecar returned a non-finite action.") + return action + + def last_input_frame(self) -> NDArray[np.uint8] | None: + return None if self._last_input is None else self._last_input.copy() + + def close(self) -> None: + with contextlib.suppress(Exception): + self._client.call("close") + self._client.close() + + +def build_behavior_groot_policy( + env_cfg: _EnvCfg, + manifest: RSkillManifest, + extra: dict[str, object], +) -> _BehaviorGrootAdapter: + """Build the official B1K GR00T adapter behind its Python 3.10 sidecar.""" + from openral_sim._deps import ensure_backend_deps + + ensure_backend_deps("behavior_groot_client") + try: + import msgpack # type: ignore[import-not-found,import-untyped,unused-ignore] # noqa: F401 reason: opt-in sidecar wire + import zmq # type: ignore[import-not-found,import-untyped,unused-ignore] # noqa: F401 reason: opt-in sidecar wire + except ImportError as exc: # pragma: no cover - runtime install error + raise ROSConfigError( + "BEHAVIOR GR00T needs pyzmq + msgpack on the OpenRAL venv: " + "just sync --all-packages --group behavior-groot" + ) from exc + + spec = env_cfg.vla + task = str(extra.get("task", "turning_on_radio")) + instruction = str(extra.get("instruction", task.replace("_", " "))) + control_mode = str(extra.get("control_mode", "temporal_ensemble")) + quantization = str(extra.get("quantization", "nf4")) + nf4_min_params = _opt_int(extra.get("nf4_min_params"), 4_000_000) + host = os.environ.get(_HOST_ENV, str(extra.get("host", _DEFAULT_HOST))) + checkpoint = _checkpoint_path(manifest) + default_port = _policy_default_port(task, str(checkpoint)) + port = _opt_int(os.environ.get(_PORT_ENV, extra.get("port")), default_port) + auto_spawn = os.environ.get(_AUTO_SPAWN_ENV, "1") != "0" + + launch_argv: list[str] = [] + if auto_spawn: + if not checkpoint.is_dir(): + raise ROSConfigError( + f"BEHAVIOR GR00T checkpoint not found at {checkpoint}. Download the official " + "turning_on_radio checkpoint and set " + f"{_CHECKPOINT_ENV}, or place it at {_DEFAULT_CHECKPOINT}." + ) + device = "cuda" if spec.device == "auto" else str(spec.device) + launch_argv = [ + "env", + "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", + str(_sidecar_python()), + str(_locate_sidecar_script()), + "--checkpoint", + str(checkpoint), + "--task", + task, + "--instruction", + instruction, + "--control-mode", + control_mode, + "--device", + device, + "--quantization", + quantization, + "--nf4-min-params", + str(nf4_min_params), + "--host", + host, + "--port", + str(port), + ] + + client = SidecarClient( + name="behavior-groot", + host=host, + port=port, + timeout_ms=_DEFAULT_TIMEOUT_MS, + boot_timeout_s=_opt_float(extra.get("boot_timeout_s"), _DEFAULT_BOOT_TIMEOUT_S), + launch_argv=launch_argv, + auto_spawn=auto_spawn, + expected_identity={"model": "behavior_groot", "task": task}, + ) + client.connect() + return _BehaviorGrootAdapter(spec=spec, device="sidecar", _client=client) + + +__all__ = ["build_behavior_groot_policy"] diff --git a/python/sim/src/openral_sim/policies/gr00t.py b/python/sim/src/openral_sim/policies/gr00t.py index a829cfd..2db9a87 100644 --- a/python/sim/src/openral_sim/policies/gr00t.py +++ b/python/sim/src/openral_sim/policies/gr00t.py @@ -27,6 +27,12 @@ diffusion action head stays in bf16 — this preserves action quality and side-steps the GR00T DiT ``TimestepEncoder`` uint8 bug (bug (b) below). +The official 2026 BEHAVIOR-1K checkpoint is a deliberate exception to this +native path: its organizer runtime is pinned to ``wensi-ai/Isaac-GR00T`` under +Python 3.10, so a manifest with +``policy_extras.implementation=behavior_b1k_sidecar`` dispatches to +:mod:`openral_sim.policies.behavior_groot` before the lerobot loader runs. + Embodiment mapping ------------------ The rSkill manifest declares its OpenRAL robot tag (e.g. ``franka_panda`` / @@ -73,6 +79,7 @@ if TYPE_CHECKING: from openral_core import VLASpec + from openral_sim.policy import PolicyAdapter from openral_sim.rollout import Observation # The gr00t-n17-libero rSkill targets nvidia/GR00T-N1.7-LIBERO. GR00T's own @@ -402,7 +409,7 @@ def _build_groot_config( @POLICIES.register("gr00t") -def _build_gr00t(env_cfg: Any) -> _GrootAdapter: +def _build_gr00t(env_cfg: Any) -> PolicyAdapter: """Load the in-process lerobot ``GrootPolicy`` (GR00T N1.7) backend. YAML knobs (via ``vla.extra``): ``device``, ``embodiment_tag``, @@ -420,14 +427,19 @@ def _build_gr00t(env_cfg: Any) -> _GrootAdapter: "gr00t adapter requires a bare rSkill reference as weights_uri so the " "loader can resolve the GR00T checkpoint + its embodiment contract." ) - repo_id, revision = resolve_rskill_repo_revision(spec.weights_uri, adapter_name="GR00T") - # Merge the manifest's policy_extras under any scene/CLI spec.extra overrides # (spec.extra wins) so a checkpoint's embodiment_tag / image_modality_keys # travel in its rSkill without every scene having to repeat them. policy_extras = dict(getattr(manifest, "policy_extras", {}) or {}) extra = {**policy_extras, **extra} + if extra.get("implementation") == "behavior_b1k_sidecar": + from openral_sim.policies.behavior_groot import ( + build_behavior_groot_policy, + ) + + return build_behavior_groot_policy(env_cfg, manifest, extra) + repo_id, revision = resolve_rskill_repo_revision(spec.weights_uri, adapter_name="GR00T") embodiment_tag = str( os.environ.get("OPENRAL_GR00T_EMBODIMENT_TAG") or extra.get("embodiment_tag", _GR00T_DEFAULT_EMBODIMENT_TAG) diff --git a/python/sim/src/openral_sim/policy_deps.py b/python/sim/src/openral_sim/policy_deps.py index a0f5a3c..2f9b6b0 100644 --- a/python/sim/src/openral_sim/policy_deps.py +++ b/python/sim/src/openral_sim/policy_deps.py @@ -51,7 +51,10 @@ __all__ = [ "can_import_policy_family", + "can_import_policy_manifest", "filter_importable_manifests", + "manifest_install_groups", + "manifest_install_hint", "model_family_install_groups", "model_family_install_hint", "model_family_required_imports", @@ -82,7 +85,9 @@ "gr00t": ( "Install the gr00t extras: `just sync --all-packages --group gr00t` " "(adds lerobot[groot]). GR00T-N1.7 now runs in-process on lerobot 0.6.0 " - "under this repo's Python 3.12 — no sidecar." + "under this repo's Python 3.12. The official BEHAVIOR-1K organizer " + "checkpoint is a manifest-selected exception and uses the " + "`behavior-groot` sidecar-wire group." ), "diffuser_actor": ( "Install the rlbench extras: `just sync --all-packages --group rlbench` " @@ -218,7 +223,11 @@ def can_import_policy_family(family: str) -> tuple[bool, str | None]: primary error, not a cascade) and returns ``(False, reason)`` where ``reason`` carries the leaf import error. """ - required = model_family_required_imports(family) + return _can_import_modules(model_family_required_imports(family)) + + +def _can_import_modules(required: tuple[str, ...]) -> tuple[bool, str | None]: + """Probe an explicit import set with the same partial-import cleanup.""" for mod in required: try: importlib.import_module(mod) @@ -236,6 +245,38 @@ def can_import_policy_family(family: str) -> tuple[bool, str | None]: return True, None +def _is_behavior_groot_manifest(manifest: Any) -> bool: + extras = getattr(manifest, "policy_extras", {}) or {} + return ( + getattr(manifest, "model_family", None) == "gr00t" + and extras.get("implementation") == "behavior_b1k_sidecar" + ) + + +def can_import_policy_manifest(manifest: Any) -> tuple[bool, str | None]: + """Probe the manifest-selected runtime, including GR00T sidecar variants.""" + if _is_behavior_groot_manifest(manifest): + return _can_import_modules(("zmq", "msgpack")) + return can_import_policy_family(getattr(manifest, "model_family", None) or "") + + +def manifest_install_groups(manifest: Any) -> tuple[str, ...]: + """Return dependency groups for the manifest-selected policy runtime.""" + if _is_behavior_groot_manifest(manifest): + return ("behavior-groot",) + return model_family_install_groups(getattr(manifest, "model_family", None) or "") + + +def manifest_install_hint(manifest: Any) -> str: + """Return the install hint for the manifest-selected policy runtime.""" + if _is_behavior_groot_manifest(manifest): + return ( + "Install the BEHAVIOR GR00T wire extras: " + "`just sync --all-packages --group behavior-groot`." + ) + return model_family_install_hint(getattr(manifest, "model_family", None) or "") + + def filter_importable_manifests( manifests: Iterable[Any], *, @@ -257,13 +298,13 @@ def filter_importable_manifests( kept: list[Any] = [] for manifest in manifests: family = getattr(manifest, "model_family", None) or "" - ok, reason = can_import_policy_family(family) + ok, reason = can_import_policy_manifest(manifest) if ok: kept.append(manifest) continue if log_fn is not None: name = getattr(manifest, "name", "") - hint = model_family_install_hint(family) + hint = manifest_install_hint(manifest) log_fn(f"palette: dropping rSkill {name!r} (model_family={family!r}): {reason}. {hint}") return kept diff --git a/python/world_state/src/openral_world_state/aggregator.py b/python/world_state/src/openral_world_state/aggregator.py index 1b10ac1..3b3c769 100644 --- a/python/world_state/src/openral_world_state/aggregator.py +++ b/python/world_state/src/openral_world_state/aggregator.py @@ -188,6 +188,8 @@ def __init__( self._base_pose: Pose6D | None = None self._base_pose_stamp_ns: int = 0 self._base_twist: tuple[float, float, float, float, float, float] | None = None + self._policy_state: list[float] | None = None + self._policy_state_stamp_ns: int = 0 # battery self._battery_pct: float | None = None # latest object-memory snapshot (already deduped/evicted by @@ -255,6 +257,17 @@ def update_image(self, sensor_name: str, topic: str, stamp_ns: int) -> None: self._images[sensor_name] = (topic, self._clock_fn()) log.debug("world_state.image.updated", sensor=sensor_name) + def update_policy_state(self, values: list[float]) -> None: + """Record the simulator-native policy proprioception vector.""" + with self._lock: + self._policy_state = [float(value) for value in values] + self._policy_state_stamp_ns = self._clock_fn() + log.debug( + "world_state.policy_state.updated", + robot=self.description.name, + dim=len(values), + ) + def update_image_frame(self, sensor_name: str, frame: SensorFrame) -> None: """Record an inline pixel payload for a named sensor. @@ -431,6 +444,13 @@ def snapshot(self) -> WorldState: diag["joint_state"] = "ok" if age_ns <= self._staleness_limit_ns else "stale" ages_ms["joint_state"] = age_ns / 1e6 + if self._policy_state is not None: + policy_age_ns = now_ns - self._policy_state_stamp_ns + diag["policy_state"] = ( + "ok" if policy_age_ns <= self._staleness_limit_ns else "stale" + ) + ages_ms["policy_state"] = policy_age_ns / 1e6 + # Images — topic refs from last received frames images: dict[str, str] = {} for sensor_name in self._sensor_names: @@ -490,6 +510,7 @@ def snapshot(self) -> WorldState: joint_state=js, base_pose=self._base_pose, base_twist=self._base_twist, + policy_state=list(self._policy_state) if self._policy_state is not None else None, ee_poses=ee_poses, images=images, image_frames=image_frames, diff --git a/robots/r1pro/README.md b/robots/r1pro/README.md new file mode 100644 index 0000000..9d56a1d --- /dev/null +++ b/robots/r1pro/README.md @@ -0,0 +1,19 @@ +# Galaxea R1 Pro (BEHAVIOR-1K simulation) + +This manifest describes the simulated R1 Pro used by the 2026 BEHAVIOR-1K +challenge. Joint position, velocity, and effort limits come from the upstream +`behavior-1k/omnigibson-robot-assets` `r1pro.urdf`. + +The robot is **simulation-only** in OpenRAL. `hal.real` remains null; no physical +R1 Pro safety or transport claim is made. + +```bash +openral deploy sim --config scenes/deploy/behavior_r1pro.yaml \ + --initial-task "turn on the radio" +``` + +The scene sidecar requires the official BEHAVIOR environment via +`OPENRAL_BEHAVIOR_SIDECAR_PYTHON`. The GR00T policy sidecar requires the pinned +Isaac-GR00T environment and organizer checkpoint documented in +`rskills/gr00t-n17-b1k-turning-on-radio/README.md`. + diff --git a/robots/r1pro/robot.yaml b/robots/r1pro/robot.yaml new file mode 100644 index 0000000..fec7559 --- /dev/null +++ b/robots/r1pro/robot.yaml @@ -0,0 +1,281 @@ +# Simulated Galaxea R1 Pro used by the 2026 BEHAVIOR-1K challenge. +# Joint limits come from behavior-1k/omnigibson-robot-assets r1pro.urdf. + +name: "r1pro" +embodiment_kind: "mobile_manipulator" +base_frame: "base_link" +odom_frame: "odom" + +joints: + - name: "torso_joint1" + joint_type: "revolute" + parent_link: "base_link" + child_link: "torso_link1" + position_limits: [-1.1345, 1.8326] + velocity_limit: 2.5 + effort_limit: 100.0 + actuator_kind: "bldc" + - name: "torso_joint2" + joint_type: "revolute" + parent_link: "torso_link1" + child_link: "torso_link2" + position_limits: [-2.7925, 2.5307] + velocity_limit: 2.5 + effort_limit: 100.0 + actuator_kind: "bldc" + - name: "torso_joint3" + joint_type: "revolute" + parent_link: "torso_link2" + child_link: "torso_link3" + position_limits: [-1.8326, 1.5708] + velocity_limit: 2.5 + effort_limit: 100.0 + actuator_kind: "bldc" + - name: "torso_joint4" + joint_type: "revolute" + parent_link: "torso_link3" + child_link: "torso_link4" + position_limits: [-3.0543, 3.0543] + velocity_limit: 2.5 + effort_limit: 100.0 + actuator_kind: "bldc" + + - name: "left_arm_joint1" + joint_type: "revolute" + parent_link: "left_arm_base_link" + child_link: "left_arm_link1" + position_limits: [-4.4506, 1.3090] + velocity_limit: 7.1209 + effort_limit: 55.0 + actuator_kind: "bldc" + - name: "left_arm_joint2" + joint_type: "revolute" + parent_link: "left_arm_link1" + child_link: "left_arm_link2" + position_limits: [-0.1745, 3.1416] + velocity_limit: 7.1209 + effort_limit: 55.0 + actuator_kind: "bldc" + - name: "left_arm_joint3" + joint_type: "revolute" + parent_link: "left_arm_link2" + child_link: "left_arm_link3" + position_limits: [-2.356196, 2.356196] + velocity_limit: 8.3776 + effort_limit: 25.0 + actuator_kind: "bldc" + - name: "left_arm_joint4" + joint_type: "revolute" + parent_link: "left_arm_link3" + child_link: "left_arm_link4" + position_limits: [-2.0944, 0.3491] + velocity_limit: 8.3776 + effort_limit: 25.0 + actuator_kind: "bldc" + - name: "left_arm_joint5" + joint_type: "revolute" + parent_link: "left_arm_link4" + child_link: "left_arm_link5" + position_limits: [-2.356196, 2.356196] + velocity_limit: 10.472 + effort_limit: 18.0 + actuator_kind: "bldc" + - name: "left_arm_joint6" + joint_type: "revolute" + parent_link: "left_arm_link5" + child_link: "left_arm_link6" + position_limits: [-1.047198, 1.047198] + velocity_limit: 10.472 + effort_limit: 18.0 + actuator_kind: "bldc" + - name: "left_arm_joint7" + joint_type: "revolute" + parent_link: "left_arm_link6" + child_link: "left_arm_link7" + position_limits: [-1.5708, 1.5708] + velocity_limit: 10.472 + effort_limit: 18.0 + actuator_kind: "bldc" + - name: "left_gripper_finger_joint1" + joint_type: "prismatic" + parent_link: "left_gripper_link" + child_link: "left_gripper_finger_link1" + position_limits: [0.0, 0.05] + velocity_limit: 0.25 + effort_limit: 100.0 + actuator_kind: "servo" + role: "gripper" + - name: "left_gripper_finger_joint2" + joint_type: "prismatic" + parent_link: "left_gripper_link" + child_link: "left_gripper_finger_link2" + position_limits: [0.0, 0.05] + velocity_limit: 0.25 + effort_limit: 100.0 + actuator_kind: "servo" + role: "gripper" + + - name: "right_arm_joint1" + joint_type: "revolute" + parent_link: "right_arm_base_link" + child_link: "right_arm_link1" + position_limits: [-4.4506, 1.3090] + velocity_limit: 7.1209 + effort_limit: 55.0 + actuator_kind: "bldc" + - name: "right_arm_joint2" + joint_type: "revolute" + parent_link: "right_arm_link1" + child_link: "right_arm_link2" + position_limits: [-3.1416, 0.1745] + velocity_limit: 7.1209 + effort_limit: 55.0 + actuator_kind: "bldc" + - name: "right_arm_joint3" + joint_type: "revolute" + parent_link: "right_arm_link2" + child_link: "right_arm_link3" + position_limits: [-2.356196, 2.356196] + velocity_limit: 8.3776 + effort_limit: 25.0 + actuator_kind: "bldc" + - name: "right_arm_joint4" + joint_type: "revolute" + parent_link: "right_arm_link3" + child_link: "right_arm_link4" + position_limits: [-2.0944, 0.3491] + velocity_limit: 8.3776 + effort_limit: 25.0 + actuator_kind: "bldc" + - name: "right_arm_joint5" + joint_type: "revolute" + parent_link: "right_arm_link4" + child_link: "right_arm_link5" + position_limits: [-2.356196, 2.356196] + velocity_limit: 10.472 + effort_limit: 18.0 + actuator_kind: "bldc" + - name: "right_arm_joint6" + joint_type: "revolute" + parent_link: "right_arm_link5" + child_link: "right_arm_link6" + position_limits: [-1.047198, 1.047198] + velocity_limit: 10.472 + effort_limit: 18.0 + actuator_kind: "bldc" + - name: "right_arm_joint7" + joint_type: "revolute" + parent_link: "right_arm_link6" + child_link: "right_arm_link7" + position_limits: [-1.5708, 1.5708] + velocity_limit: 10.472 + effort_limit: 18.0 + actuator_kind: "bldc" + - name: "right_gripper_finger_joint1" + joint_type: "prismatic" + parent_link: "right_gripper_link" + child_link: "right_gripper_finger_link1" + position_limits: [0.0, 0.05] + velocity_limit: 0.25 + effort_limit: 100.0 + actuator_kind: "servo" + role: "gripper" + - name: "right_gripper_finger_joint2" + joint_type: "prismatic" + parent_link: "right_gripper_link" + child_link: "right_gripper_finger_link2" + position_limits: [0.0, 0.05] + velocity_limit: 0.25 + effort_limit: 100.0 + actuator_kind: "servo" + role: "gripper" + +end_effectors: + - name: "left_gripper" + kind: "parallel_gripper" + parent_link: "left_gripper_link" + actuated: true + - name: "right_gripper" + kind: "parallel_gripper" + parent_link: "right_gripper_link" + actuated: true + +capabilities: + locomotion: ["wheeled"] + can_lift_kg: 3.5 + has_dexterous_hands: false + has_tactile: false + has_force_control: true + has_vision: true + has_lidar: false + bimanual: true + supported_control_modes: + - "body_twist" + - "joint_position" + - "gripper_position" + embodiment_tags: + - "r1pro" + - "mobile_base" + +safety: + max_ee_speed_m_s: 0.5 + max_joint_speed_factor: 0.5 + max_force_n: 50.0 + max_torque_nm: 10.0 + deadman_required: true + max_base_linear_speed_m_s: 0.7 + max_base_angular_speed_rad_s: 0.3 + +sensors: + - name: "head" + modality: "rgb" + frame_id: "zed_link" + rate_hz: 30.0 + intrinsics: + width: 720 + height: 720 + fx: 306.0 + fy: 306.0 + cx: 360.0 + cy: 360.0 + encoding: "rgb8" + vla_feature_key: "observation.images.head" + vendor: "sim" + model: "zed" + - name: "left_wrist" + modality: "rgb" + frame_id: "left_realsense_link" + rate_hz: 30.0 + intrinsics: + width: 480 + height: 480 + fx: 388.6639 + fy: 388.6639 + cx: 240.0 + cy: 240.0 + encoding: "rgb8" + vla_feature_key: "observation.images.left_wrist" + vendor: "sim" + model: "realsense" + - name: "right_wrist" + modality: "rgb" + frame_id: "right_realsense_link" + rate_hz: 30.0 + intrinsics: + width: 480 + height: 480 + fx: 388.6639 + fy: 388.6639 + cx: 240.0 + cy: 240.0 + encoding: "rgb8" + vla_feature_key: "observation.images.right_wrist" + vendor: "sim" + model: "realsense" + +sdk_kind: "open" +hal: + sim: null + real: null +schema_version: "0.1" + diff --git a/rskills/gr00t-n17-b1k-turning-on-radio/README.md b/rskills/gr00t-n17-b1k-turning-on-radio/README.md new file mode 100644 index 0000000..f1cf37d --- /dev/null +++ b/rskills/gr00t-n17-b1k-turning-on-radio/README.md @@ -0,0 +1,186 @@ +--- +language: +- en +license: other +pipeline_tag: robotics +tags: +- OpenRAL +- rskill +- gr00t +- vision-language-action +- r1pro +- behavior-1k +datasets: +- behavior-1k/2026-challenge-demos +inference: false +base_model: +- nvidia/GR00T-N1.7-3B +base_model_relation: finetune +--- + +# rskill-gr00t-n17-b1k-turning-on-radio + +> **OpenRAL rSkill** — the official 2026 BEHAVIOR-1K GR00T N1.7 +> `turning_on_radio` checkpoint for the simulated Galaxea R1 Pro. + +This package contains the OpenRAL manifest and adapter configuration only. The +organizer checkpoint is downloaded separately from the +[BEHAVIOR baseline page](https://behavior.stanford.edu/challenge/baselines.html); +weights are not copied into this repository. + +## Preview + +The official task demonstration is available in the +[BEHAVIOR challenge gallery](https://behavior.stanford.edu/challenge/tasks/). +No local rollout image is shipped until the checkpoint has been reproduced on +this host. + +## What this skill does + +The policy navigates the R1 Pro to a household radio and manipulates its controls +to turn it on. It is task-specific to `turning_on_radio`. + +| Field | Value | +| --- | --- | +| Actions | rotate, push | +| Objects | radio, dial, button | +| Scenes | household living room | +| Embodiment | BEHAVIOR R1 Pro (`r1pro`) | + +## How it works + +The adapter runs the pinned `wensi-ai/Isaac-GR00T` `behavior` branch in an +isolated Python 3.10 sidecar. The sidecar uses the upstream `Gr00tPolicy` and +`B1KPolicyWrapper` unchanged, including temporal ensembling and the official +R1Pro modality split. Two 8 GB-host memory measures are applied at load time: +whole-model NF4 quantization of large linears, and replacing the Qwen3-VL +`lm_head` with `Identity` (the wrapper consumes only hidden states, so the +full-vocab logits projection is dead weight). + +### Observation -> action contract + +| Direction | Key | Shape | Notes | +| --- | --- | --- | --- | +| in | head RGB | HWC uint8 | Official ZED head camera | +| in | left wrist RGB | HWC uint8 | Left RealSense | +| in | right wrist RGB | HWC uint8 | Right RealSense | +| in | proprioception | `(61,)` float32 | Official `PROPRIOCEPTION_INDICES["R1Pro"]` order | +| out | action | `(23,)` float32 | Base velocity 3 + torso 4 + arms 7+7 + grippers 1+1 | + +The organizer wrapper converts the 61-D vector into base, torso, arm, and +gripper state groups. It emits a 16-step GR00T action horizon and returns one +temporally-ensembled 23-D action per evaluator step. + +## Upstream model / training + +| Field | Value | +| --- | --- | +| Source code | [`wensi-ai/Isaac-GR00T@ace36d9`](https://github.com/wensi-ai/Isaac-GR00T/tree/ace36d935b376fbf25cd56371e23877b95407c40) | +| Checkpoint | Organizer-provided `turning_on_radio` Google Drive checkpoint | +| Base model | [`nvidia/GR00T-N1.7-3B`](https://huggingface.co/nvidia/GR00T-N1.7-3B) | +| Dataset | [`behavior-1k/2026-challenge-demos`](https://huggingface.co/datasets/behavior-1k/2026-challenge-demos) | +| Paper | [arXiv:2503.14734](https://arxiv.org/abs/2503.14734) | +| Parameters | approximately 3.1 B | +| Weights license | Unknown: the Drive checkpoint ships without a separately published license file | + +## Supported robots + +| Robot | Embodiment tag | Status | Notes | +| --- | --- | --- | --- | +| Simulated Galaxea R1 Pro | `r1pro` | evaluator + deploy sim | Official BEHAVIOR observation/action contract | + +`openral deploy sim` uses `robots/r1pro/robot.yaml`, publishes the simulator's +native 61-D policy state through WorldState, validates every typed action slot +through the safety kernel, then atomically commits all six slots as one 23-D +OmniGibson step. + +## Sensors required + +| Key | Modality | Min resolution | Format | +| --- | --- | --- | --- | +| `observation.images.head` | RGB | 224 x 224 | HWC uint8 | +| `observation.images.left_wrist` | RGB | 224 x 224 | HWC uint8 | +| `observation.images.right_wrist` | RGB | 224 x 224 | HWC uint8 | +| `observation.state` | proprioception | `(61,)` | float32 | + +## Manifest summary + +| Field | Value | +| --- | --- | +| `name` | `OpenRAL/rskill-gr00t-n17-b1k-turning-on-radio` | +| `version` | `0.1.0` | +| `license` | `unknown` | +| `role` | `s1` | +| `model_family` | `gr00t` | +| `runtime` | external Python 3.10 Isaac-GR00T sidecar, whole-model NF4 | +| `weights_uri` | `local://checkpoints/behavior-groot-turning-on-radio` | +| `chunk_size` | 16 | +| `state_contract.dim` / `action_contract.dim` | 61 / 23 | +| `latency_budget.per_chunk_ms` | 1500 | + +## Quick start + +```bash +git clone https://github.com/wensi-ai/Isaac-GR00T \ + ~/.cache/openral/behavior-groot/source +git -C ~/.cache/openral/behavior-groot/source checkout \ + ace36d935b376fbf25cd56371e23877b95407c40 +cd ~/.cache/openral/behavior-groot/source +uv sync --frozen --python 3.10 + +export OPENRAL_BEHAVIOR_GROOT_SIDECAR_PYTHON="$PWD/.venv/bin/python" +export OPENRAL_BEHAVIOR_GROOT_CHECKPOINT=/absolute/path/to/checkpoint + +cd /path/to/openral +just sync --group behavior-groot +openral behavior serve \ + --rskill rskills/gr00t-n17-b1k-turning-on-radio \ + --task turning_on_radio +``` + +Full deploy graph: + +```bash +openral deploy sim \ + --config scenes/deploy/behavior_r1pro.yaml \ + --initial-task "turn on the radio" +``` + +In the BEHAVIOR environment: + +```bash +python -m omnigibson.eval.eval \ + --task-name turning_on_radio \ + --host 127.0.0.1 --port 8000 \ + --instance-indices 0 --num-rollouts 1 \ + --output-dir outputs/openral --write-video +``` + +## Reproduction + +The command above is the canonical one-rollout reproduction. Use public +instances `0-9` for challenge reporting and keep the evaluator-generated JSON +and videos unmodified. + +## Evaluation + +No OpenRAL-generated score is shipped. The full official-evaluator loop has +been reproduced locally on an 8 GB RTX 4070 Laptop GPU (NF4 sidecar 2.77 GiB +inference peak alongside OmniGibson, ~0.8-1.3 steps/s): public instance 0 +(ID 301), 3225 steps to timeout, `success=false`, `q_score 0.0`. Whether the +zero q_score reflects NF4 degradation or the checkpoint's zero-shot behavior +on this instance has not been isolated; no success-rate claim is made. + +## License + +The OpenRAL adapter, manifest, and documentation are Apache-2.0. The +organizer-provided fine-tuned checkpoint is marked `license: unknown` because +the Google Drive artifact has no separately published license file. The base +GR00T N1.7 model uses the NVIDIA Open Model License, but this package does not +silently assume that the fine-tune inherits identical terms. + +## See also + +- [BEHAVIOR evaluation rules](https://behavior.stanford.edu/challenge/evaluation.html) +- [`docs/reference/vla_compatibility.md`](../../docs/reference/vla_compatibility.md) +- [`python/cli/src/openral_cli/behavior.py`](../../python/cli/src/openral_cli/behavior.py) diff --git a/rskills/gr00t-n17-b1k-turning-on-radio/SKILL.md b/rskills/gr00t-n17-b1k-turning-on-radio/SKILL.md new file mode 100644 index 0000000..cee09dd --- /dev/null +++ b/rskills/gr00t-n17-b1k-turning-on-radio/SKILL.md @@ -0,0 +1,73 @@ +--- +name: gr00t-n17-b1k-turning-on-radio +description: >- + S1 Vision-Language-Action policy. Capabilities: rotate, push on radio, dial, button. GR00T N1.7 finetuned for the 2026 BEHAVIOR-1K turning_on_radio task on the simulated Galaxea R1 Pro. Consumes head plus dual-wrist RGB and 61-D proprioception; emits the official mixed 23-D mobile-bimanual action. Discovery view of an OpenRAL rSkill — NOT directly runnable by an agent harness; it runs via rSkill.from_pretrained + the robot HAL. +metadata: + openral_rskill: true # generated discovery view of an rSkill + schema_version: 0.1 + rskill_id: OpenRAL/rskill-gr00t-n17-b1k-turning-on-radio + manifest: ./rskill.yaml + role: s1 + kind: vla + model_family: gr00t + embodiment_tags: [r1pro] + actions: [rotate, push] + objects: [radio, dial, button] + scenes: [household, living_room] + sensors_required: ['rgb:observation.images.head', 'rgb:observation.images.left_wrist', 'rgb:observation.images.right_wrist'] + state_dim: 61 + action_dim: 23 + runtime: pytorch + quantization: bf16/pytorch + chunk_size: 16 + n_action_steps: 1 + latency_budget: {per_chunk_ms: 1500.0} + license_code: Apache-2.0 + license_weights: unknown # NOT permissive — see License section + weights_uri: local://checkpoints/behavior-groot-turning-on-radio + paper_url: https://arxiv.org/abs/2503.14734 +--- + +# gr00t-n17-b1k-turning-on-radio — rSkill discovery view + +> **Generated view, not a hand-written skill.** This `SKILL.md` is a discovery-only +> mirror of [`rskill.yaml`](./rskill.yaml), produced by `tools/generate_rskill_skillmd.py`. +> It lets tools that read the standard agent-skill format find and reason about this +> OpenRAL rSkill. The `rskill.yaml` manifest is the single source of truth +> (CLAUDE.md §1.3). Do not edit by hand — edit the manifest and regenerate. + +## What it is + +An OpenRAL **Vision-Language-Action policy** (`role: s1`, `kind: vla`). GR00T N1.7 finetuned for the 2026 BEHAVIOR-1K turning_on_radio task on the simulated Galaxea R1 Pro. Consumes head plus dual-wrist RGB and 61-D proprioception; emits the official mixed 23-D mobile-bimanual action. + +## Capabilities + +- **Verbs:** rotate · push +- **Objects:** radio · dial · button +- **Scenes:** household · living_room +- **Embodiments:** r1pro + +## Why this is discovery-only + +An agent skill is natural-language instructions loaded into an LLM's context. An rSkill +is an executable artifact: it carries a typed capability/embodiment contract, model weights, +a runtime, and a license/provenance gate — none of which fit in freeform markdown. So an +agent can use this view to *select* the right skill, but cannot *execute* it by loading +this file. Execution always goes through the OpenRAL loader and the robot HAL. + +## License + +- **Code:** Apache-2.0. +- **Weights:** `unknown` — **NOT** fully permissive. The loader surfaces this posture and enforces the non-commercial guard (`OPENRAL_ALLOW_NONCOMMERCIAL=1`) where applicable. Commercial use may require a separate upstream agreement. This is third-party weight lineage; OpenRAL's own code is Apache-2.0. + +## How to actually run it (not via an agent harness) + +```python +from openral_rskill import rSkill + +skill = rSkill.from_pretrained("OpenRAL/rskill-gr00t-n17-b1k-turning-on-radio") +# the loader validates embodiment / sensors / runtime / quantization against the target +# RobotDescription and enforces the weight-license gate before any weights load. +``` + +See [`rskill.yaml`](./rskill.yaml) for the authoritative, validated manifest. diff --git a/rskills/gr00t-n17-b1k-turning-on-radio/eval/behavior.json b/rskills/gr00t-n17-b1k-turning-on-radio/eval/behavior.json new file mode 100644 index 0000000..e0aaa59 --- /dev/null +++ b/rskills/gr00t-n17-b1k-turning-on-radio/eval/behavior.json @@ -0,0 +1,41 @@ +{ + "schema_version": "0.1", + "source": { + "paper": "https://arxiv.org/abs/2403.09227", + "arxiv": "https://arxiv.org/abs/2403.09227", + "model_variant": "gr00t", + "evaluated_by": "OpenRAL:openral benchmark run", + "reproduced_locally": true, + "reproduction_planned": null, + "reproduction_cli": "openral benchmark run --suite behavior --rskill rskills/gr00t-n17-b1k-turning-on-radio", + "table": null, + "status": "reproduced" + }, + "benchmark": { + "name": "BEHAVIOR-1K 2026 challenge (100 tasks)", + "dataset": null, + "protocol": "1 episodes per task, success_key=is_success, max_steps=3224", + "robot": "r1pro", + "simulator": "OmniGibson / Isaac Sim (sidecar)" + }, + "eval_config": { + "n_episodes": 1, + "seeds": [ + 0 + ], + "success_key": "is_success", + "max_steps": 3224, + "vla_id": "gr00t", + "weights_uri": "rskills/gr00t-n17-b1k-turning-on-radio" + }, + "results": { + "behavior/turning_on_radio_success_rate": 0.0, + "avg_success_rate": 0.0, + "n_tasks": 1, + "n_episodes_per_task": 1, + "n_episodes_total": 1, + "mean_step_latency_ms_avg": 436.5139267810073 + }, + "baselines": {}, + "trace_id": null +} \ No newline at end of file diff --git a/rskills/gr00t-n17-b1k-turning-on-radio/rskill.yaml b/rskills/gr00t-n17-b1k-turning-on-radio/rskill.yaml new file mode 100644 index 0000000..6fd94a1 --- /dev/null +++ b/rskills/gr00t-n17-b1k-turning-on-radio/rskill.yaml @@ -0,0 +1,151 @@ +schema_version: "0.1" +name: "OpenRAL/rskill-gr00t-n17-b1k-turning-on-radio" +version: "0.1.0" +license: "unknown" +role: "s1" +kind: "vla" +model_family: "gr00t" + +embodiment_tags: + - "r1pro" + +sensors_required: + - modality: "rgb" + vla_feature_key: "observation.images.head" + min_width: 224 + min_height: 224 + - modality: "rgb" + vla_feature_key: "observation.images.left_wrist" + min_width: 224 + min_height: 224 + - modality: "rgb" + vla_feature_key: "observation.images.right_wrist" + min_width: 224 + min_height: 224 + +actuators_required: + - kind: "body_twist" + n_dof: 3 + vla_action_key: "action.base" + control_mode_semantics: + mode: "absolute" + - kind: "joint_position" + n_dof: 18 + vla_action_key: "action.joints" + control_mode_semantics: + mode: "absolute" + joint_order: + - "torso_joint1" + - "torso_joint2" + - "torso_joint3" + - "torso_joint4" + - "left_arm_joint1" + - "left_arm_joint2" + - "left_arm_joint3" + - "left_arm_joint4" + - "left_arm_joint5" + - "left_arm_joint6" + - "left_arm_joint7" + - "right_arm_joint1" + - "right_arm_joint2" + - "right_arm_joint3" + - "right_arm_joint4" + - "right_arm_joint5" + - "right_arm_joint6" + - "right_arm_joint7" + - kind: "gripper_position" + n_dof: 2 + vla_action_key: "action.grippers" + control_mode_semantics: + mode: "absolute" + gripper_convention: "normalized_open_symmetric" + +runtime: "pytorch" +quantization: + dtype: "bf16" + backend: "pytorch" +weights_uri: "local://checkpoints/behavior-groot-turning-on-radio" + +policy_extras: + implementation: "behavior_b1k_sidecar" + task: "turning_on_radio" + instruction: "turn on the radio" + control_mode: "temporal_ensemble" + quantization: "nf4" + quantize_scope: "model" + nf4_min_params: 1000000 + source_revision: "ace36d935b376fbf25cd56371e23877b95407c40" + use_world_state_policy_state: true + +state_contract: + dim: 61 + +action_contract: + dim: 23 + slots: + - range: [0, 2] + control_mode: "body_twist" + frame: "base_link" + - range: [3, 6] + control_mode: "joint_position" + joint_names: + - "torso_joint1" + - "torso_joint2" + - "torso_joint3" + - "torso_joint4" + - range: [7, 13] + control_mode: "joint_position" + joint_names: + - "left_arm_joint1" + - "left_arm_joint2" + - "left_arm_joint3" + - "left_arm_joint4" + - "left_arm_joint5" + - "left_arm_joint6" + - "left_arm_joint7" + - range: [14, 14] + control_mode: "gripper_position" + ee: "left_gripper" + - range: [15, 21] + control_mode: "joint_position" + joint_names: + - "right_arm_joint1" + - "right_arm_joint2" + - "right_arm_joint3" + - "right_arm_joint4" + - "right_arm_joint5" + - "right_arm_joint6" + - "right_arm_joint7" + - range: [22, 22] + control_mode: "gripper_position" + ee: "right_gripper" + +chunk_size: 16 +n_action_steps: 1 +latency_budget: + per_chunk_ms: 1500.0 + +evaluated_tasks: + - "behavior/turning_on_radio" + +paper_url: "https://arxiv.org/abs/2503.14734" +dataset_uri: "hf://behavior-1k/2026-challenge-demos" + +description: > + GR00T N1.7 finetuned for the 2026 BEHAVIOR-1K turning_on_radio task on the + simulated Galaxea R1 Pro. Consumes head plus dual-wrist RGB and 61-D + proprioception; emits the official mixed 23-D mobile-bimanual action. + +actions: + - "rotate" + - "push" +objects: + - "radio" + - "dial" + - "button" +scenes: + - "household" + - "living_room" + +benchmarks: + behavior: 0.0 diff --git a/scenes/README.md b/scenes/README.md index 4a1305e..448515b 100644 --- a/scenes/README.md +++ b/scenes/README.md @@ -46,6 +46,10 @@ sibling is **not** valid for paper claims; the loader-strictness gate # DeployScene — env-only playground (reasoner picks the rSkill at runtime). openral deploy sim --config scenes/deploy/openarm_tabletop.yaml +# BEHAVIOR-1K R1 Pro — official OmniGibson evaluator environment. +openral deploy sim --config scenes/deploy/behavior_r1pro.yaml \ + --initial-task "turn on the radio" + # SimScene — single rollout with an explicit rSkill (and any override flag). MUJOCO_GL=egl uv run --group libero \ openral sim run --config scenes/sim/libero_spatial.yaml \ @@ -137,6 +141,7 @@ supplied at the CLI via `--rskill `. | gym-aloha | `aloha_transfer_cube`, `aloha_insertion` | `python/sim/.../backends/aloha.py` | | gym-pusht | `pusht` (2-D pymunk) | `python/sim/.../backends/pusht.py` | | RLBench (CoppeliaSim/PyRep) | `rlbench` (scene-fixed Franka Panda; task selected by `backend_options.rlbench_task`) | `python/sim/.../backends/rlbench.py` | +| BEHAVIOR-1K (OmniGibson / Isaac Sim) | `behavior` (scene-fixed R1 Pro; task/instance selected by `backend_options`) | `python/sim/.../backends/behavior.py` | | RoboCasa (MuJoCo) | `robocasa` (procedural) + ~19 curated kitchen tasks (e.g. `robocasa/PickPlaceCounterToCabinet`) + 24 GR1 tabletop tasks (e.g. `robocasa/gr1/PnPCupToDrawerClose`) | `python/sim/.../backends/robocasa.py` | | ManiSkill3 (SAPIEN) | `maniskill3` (free-axis; passes `` to `gym.make`) | `python/sim/.../backends/maniskill3.py` | | SimplerEnv (SAPIEN) | `simpler_env` (Bridge V2 digital twin: 4 WidowX tasks on MS3 v3.0.x) | `python/sim/.../backends/simpler_env.py` | @@ -161,6 +166,7 @@ Some scenes hard-wire the physics robot via `@SCENES.register(..., fixed_robot=. | `so101_box` | `so101_follower` | | `robocasa/*` (kitchen) | `panda_mobile` | | `robocasa/gr1/*` (humanoid tabletop) | `gr1` | +| `behavior` | `r1pro` | Passing `--robot` (or authoring `robot_id:` in a YAML) with a value that disagrees with the scene's `fixed_robot` raises `ROSConfigError` at diff --git a/scenes/benchmark/behavior_turning_on_radio.yaml b/scenes/benchmark/behavior_turning_on_radio.yaml new file mode 100644 index 0000000..189444c --- /dev/null +++ b/scenes/benchmark/behavior_turning_on_radio.yaml @@ -0,0 +1,44 @@ +# BEHAVIOR-1K 2026 — turning_on_radio (public instance 0), OmniGibson, R1 Pro. +# Run: openral benchmark scene --config scenes/benchmark/behavior_turning_on_radio.yaml \ +# --rskill rskills/gr00t-n17-b1k-turning-on-radio +# Setup: official BEHAVIOR-1K OmniGibson environment in its own py3.10 venv +# (OPENRAL_BEHAVIOR_SIDECAR_PYTHON); `just sync --group behavior-groot` +# for the openral-side wire. + +robot_id: r1pro + +scene: + id: behavior + backend: isaacsim + observation_height: 224 + observation_width: 224 + cameras: ["head", "left_wrist", "right_wrist"] + backend_options: + task: turning_on_radio + instance_index: 0 + mode: public_test + env_wrapper: "omnigibson.eval.wrappers.DefaultWrapper" + boot_timeout_s: 1200 + timeout_ms: 120000 + +task: + id: behavior/turning_on_radio + scene_id: behavior + instruction: "turn on the radio" + max_steps: 3224 # organizer cap: int(1.5 x mean human-demo length) for this task + success_key: is_success + +n_episodes: 1 # ~1-2 h wall-clock per episode on the 8 GB reference host +seed: 0 + +metadata: + paper: "https://arxiv.org/abs/2403.09227" + display_name: "BEHAVIOR-1K 2026 — turning_on_radio" + simulator: "OmniGibson / Isaac Sim (py3.10 sidecar)" + honest_scope: > + Single public instance (index 0) of turning_on_radio, the one 2026 + challenge task with a released organizer baseline checkpoint. The official + leaderboard metric is q_score partial credit averaged over all public + instances; this scene records the evaluator's binary is_success. The + organizer baseline itself is flaky on this task (Isaac-GR00T issue #515: + q_score 0.0 in 9/10 episodes vs the claimed 0.70). diff --git a/scenes/deploy/behavior_r1pro.yaml b/scenes/deploy/behavior_r1pro.yaml new file mode 100644 index 0000000..4ff5fe9 --- /dev/null +++ b/scenes/deploy/behavior_r1pro.yaml @@ -0,0 +1,17 @@ +# BEHAVIOR-1K R1 Pro deploy-sim environment. +# OmniGibson owns the scene/task physics in its official conda environment. + +scene: + id: "behavior" + backend: "isaacsim" + observation_height: 224 + observation_width: 224 + cameras: ["head", "left_wrist", "right_wrist"] + backend_options: + task: "turning_on_radio" + instance_index: 0 + mode: "public_test" + env_wrapper: "omnigibson.eval.wrappers.DefaultWrapper" + boot_timeout_s: 1200 + timeout_ms: 120000 + diff --git a/scenes/sim/behavior_turning_on_radio.yaml b/scenes/sim/behavior_turning_on_radio.yaml new file mode 100644 index 0000000..cbf2ad5 --- /dev/null +++ b/scenes/sim/behavior_turning_on_radio.yaml @@ -0,0 +1,24 @@ +scene: + id: "behavior" + backend: "isaacsim" + observation_height: 224 + observation_width: 224 + cameras: ["head", "left_wrist", "right_wrist"] + backend_options: + task: "turning_on_radio" + instance_index: 0 + mode: "public_test" + env_wrapper: "omnigibson.eval.wrappers.DefaultWrapper" + boot_timeout_s: 1200 + timeout_ms: 120000 + +task: + id: "behavior/turning_on_radio" + scene_id: "behavior" + instruction: "turn on the radio" + max_steps: 1000 + success_key: "is_success" + +seed: 0 +n_episodes: 1 +record_video: false diff --git a/scripts/install.sh b/scripts/install.sh index 719a198..e028d47 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -192,6 +192,7 @@ Opt-in extras (installed into the same managed venv): openral install simpler-env # real-to-sim correlator openral install robocasa # RoboCasa kitchens (excludes libero — conflicting robosuite pins) openral install rldx # RLDX-1 sidecar client + openral install behavior-groot # BEHAVIOR-1K GR00T + OmniGibson sidecar wire openral install list # show every known group System bootstrap (needs sudo + clones the openral repo): diff --git a/tests/unit/test_all_manifests_validate.py b/tests/unit/test_all_manifests_validate.py index 05f7002..d0198f3 100644 --- a/tests/unit/test_all_manifests_validate.py +++ b/tests/unit/test_all_manifests_validate.py @@ -54,6 +54,7 @@ "panda_mobile", "panda_mobile_vslam", "pusht_2d", + "r1pro", "rizon4", "sawyer", "so100_follower", @@ -66,6 +67,7 @@ _BENCHMARK_STEMS: list[str] = [ "aloha_insertion", "aloha_transfer_cube", + "behavior_turning_on_radio", "libero_10", "libero_goal", "libero_object", @@ -92,6 +94,7 @@ _SIM_STEMS: list[str] = [ "aloha_transfer_cube", + "behavior_turning_on_radio", "franka_tabletop_push", "isaac_franka_bowl_plate", "libero_spatial", @@ -106,6 +109,7 @@ ] _DEPLOY_STEMS: list[str] = [ + "behavior_r1pro", "g1_vln", "galaxea_a1_bench", "isaac_franka", diff --git a/tests/unit/test_behavior_groot_adapter.py b/tests/unit/test_behavior_groot_adapter.py new file mode 100644 index 0000000..da7f847 --- /dev/null +++ b/tests/unit/test_behavior_groot_adapter.py @@ -0,0 +1,136 @@ +"""BEHAVIOR-1K GR00T rSkill and adapter contract.""" + +from __future__ import annotations + +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import pytest +from openral_core import VLASpec +from openral_core.exceptions import ROSConfigError +from openral_rskill.loader import load_rskill_manifest +from openral_sim.policies import behavior_groot + +_REPO_ROOT = Path(__file__).resolve().parents[2] +_RSKILL = _REPO_ROOT / "rskills" / "gr00t-n17-b1k-turning-on-radio" + + +def test_behavior_groot_rskill_manifest_loads() -> None: + manifest = load_rskill_manifest(str(_RSKILL)) + assert manifest.model_family == "gr00t" + assert manifest.license == "unknown" + assert manifest.state_contract is not None + assert manifest.state_contract.dim == 61 + assert manifest.action_contract is not None + assert manifest.action_contract.dim == 23 + assert manifest.policy_extras["implementation"] == "behavior_b1k_sidecar" + assert manifest.policy_extras["quantization"] == "nf4" + assert manifest.policy_extras["nf4_min_params"] == 1_000_000 + + +def test_behavior_wire_observation_preserves_official_payload() -> None: + raw = { + "robot_r1::proprio": np.zeros(61, dtype=np.float32), + "robot_r1::robot_r1:zed_link:Camera:0::rgb": np.zeros((8, 8, 3), dtype=np.uint8), + } + wire = behavior_groot._behavior_wire_observation( + {"behavior_raw": raw}, + instruction="turn on the radio", + ) + assert wire["robot_r1::proprio"] is raw["robot_r1::proprio"] + assert wire["openral_instruction"] == "turn on the radio" + + +def test_behavior_wire_observation_builds_from_canonical_fields() -> None: + images = { + "head": np.zeros((8, 8, 3), dtype=np.uint8), + "left_wrist": np.zeros((8, 8, 3), dtype=np.uint8), + "right_wrist": np.zeros((8, 8, 3), dtype=np.uint8), + } + wire = behavior_groot._behavior_wire_observation( + {"images": images, "state": np.zeros(61, dtype=np.float32)}, + instruction="turn on the radio", + ) + assert np.asarray(wire["robot_r1::proprio"]).shape == (61,) + for key in behavior_groot._CAMERA_KEYS.values(): + assert np.asarray(wire[key]).shape == (8, 8, 3) + + +def test_behavior_sidecar_python_error_has_setup_hint( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + missing = tmp_path / "missing" / "python" + monkeypatch.setenv(behavior_groot._SIDECAR_PYTHON_ENV, str(missing)) + with pytest.raises(ROSConfigError, match=r"Python 3\.10"): + behavior_groot._sidecar_python() + + +def test_behavior_checkpoint_env_override( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + manifest = load_rskill_manifest(str(_RSKILL)) + checkpoint = tmp_path / "checkpoint" + monkeypatch.setenv(behavior_groot._CHECKPOINT_ENV, str(checkpoint)) + assert behavior_groot._checkpoint_path(manifest) == checkpoint + + +def test_behavior_policy_ports_are_stable_and_task_specific() -> None: + a = behavior_groot._policy_default_port("turning_on_radio", "/checkpoint") + b = behavior_groot._policy_default_port("turning_on_radio", "/checkpoint") + other = behavior_groot._policy_default_port("cleaning_a_table", "/checkpoint") + assert a == b + assert behavior_groot._PORT_MIN <= a < behavior_groot._PORT_MAX + assert a != other + + +def test_gr00t_factory_dispatches_behavior_sidecar( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from openral_sim.policies import gr00t + + sentinel = object() + monkeypatch.setattr( + behavior_groot, + "build_behavior_groot_policy", + lambda _env, _manifest, _extra: sentinel, + ) + env = SimpleNamespace( + vla=VLASpec(id="gr00t", weights_uri=str(_RSKILL)), + scene=SimpleNamespace(cameras=["head", "left_wrist", "right_wrist"]), + ) + assert gr00t._build_gr00t(env) is sentinel + + +def test_gr00t_factory_reports_missing_organizer_checkpoint( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + from openral_sim.policies import gr00t + + monkeypatch.setenv(behavior_groot._CHECKPOINT_ENV, str(tmp_path / "missing-checkpoint")) + monkeypatch.setenv(behavior_groot._AUTO_SPAWN_ENV, "1") + env = SimpleNamespace( + vla=VLASpec(id="gr00t", weights_uri=str(_RSKILL)), + scene=SimpleNamespace(cameras=["head", "left_wrist", "right_wrist"]), + ) + with pytest.raises(ROSConfigError, match="checkpoint not found"): + gr00t._build_gr00t(env) + + +def test_sidecar_parses_int8_quantization() -> None: + # The sidecar CLI is the adapter's argv contract: `quantization` from + # policy_extras is forwarded verbatim, so every advertised choice must parse. + import importlib.util + + spec = importlib.util.spec_from_file_location( + "behavior_groot_sidecar", _REPO_ROOT / "tools" / "behavior_groot_sidecar.py" + ) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + args = module._parse_args(["--checkpoint", str(_RSKILL), "--quantization", "int8"]) + assert args.quantization == "int8" + assert args.nf4_min_params == 4_000_000 diff --git a/tests/unit/test_behavior_scene_backend.py b/tests/unit/test_behavior_scene_backend.py new file mode 100644 index 0000000..3c92dc7 --- /dev/null +++ b/tests/unit/test_behavior_scene_backend.py @@ -0,0 +1,169 @@ +"""BEHAVIOR scene adapter, R1Pro manifest, and atomic action mapping.""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +from openral_core import Action, ControlMode, JointState, RobotDescription, VLASpec, WorldState +from openral_core.loaders import load_scene_strict +from openral_core.schemas import DeployScene, SimScene +from openral_rskill.loader import load_rskill_manifest +from openral_sim.backends.behavior import ( + _ACTION_DIM, + _CAMERA_KEYS, + _BehaviorSidecar, + _compose_action_group, + _joint_state_from_policy_state, +) + +_ROOT = Path(__file__).resolve().parents[2] + + +def _joint_action(values: list[float], tick: int) -> Action: + return Action( + control_mode=ControlMode.JOINT_POSITION, + joint_targets=[values], + tick_index=tick, + ) + + +def test_r1pro_manifest_and_behavior_scenes_load() -> None: + robot = RobotDescription.from_yaml(str(_ROOT / "robots" / "r1pro" / "robot.yaml")) + deploy = load_scene_strict( + str(_ROOT / "scenes" / "deploy" / "behavior_r1pro.yaml"), + DeployScene, + ) + sim = load_scene_strict( + str(_ROOT / "scenes" / "sim" / "behavior_turning_on_radio.yaml"), + SimScene, + ) + assert robot.name == "r1pro" + assert len(robot.joints) == 22 + assert deploy.scene.id == "behavior" + assert sim.task.id == "behavior/turning_on_radio" + assert list(_CAMERA_KEYS) == ["head", "left_wrist", "right_wrist"] + + +def test_policy_state_extracts_manifest_order_joint_state() -> None: + state = np.arange(61, dtype=np.float32) + positions, velocities = _joint_state_from_policy_state(state) + expected_positions = np.concatenate( + [state[53:57], state[3:10], state[24:26], state[28:35], state[49:51]] + ) + expected_velocities = np.concatenate( + [state[57:61], state[10:17], state[26:28], state[35:42], state[51:53]] + ) + np.testing.assert_array_equal(positions, expected_positions) + np.testing.assert_array_equal(velocities, expected_velocities) + + +def test_behavior_idle_action_holds_current_pose() -> None: + state = np.zeros(61, dtype=np.float32) + state[53:57] = [0.5, -0.5, 0.25, 0.0] + state[3:10] = np.arange(7, dtype=np.float32) + state[28:35] = -np.arange(7, dtype=np.float32) + state[24:26] = 0.05 + state[49:51] = 0.0 + rollout = object.__new__(_BehaviorSidecar) + rollout._last_state = state + action = rollout.idle_action() + assert action.shape == (23,) + np.testing.assert_array_equal(action[:3], np.zeros(3)) + np.testing.assert_array_equal(action[3:7], state[53:57]) + assert action[14] == 1.0 + assert action[22] == -1.0 + + +def test_six_safe_slots_compose_one_official_action_vector() -> None: + tick = 9 + torso = np.zeros(22, dtype=np.float32) + torso[0:4] = [3, 4, 5, 6] + left = np.zeros(22, dtype=np.float32) + left[4:11] = np.arange(7, 14) + right = np.zeros(22, dtype=np.float32) + right[13:20] = np.arange(15, 22) + actions = [ + Action( + control_mode=ControlMode.BODY_TWIST, + body_twist=[(0.1, 0.2, 0.0, 0.0, 0.0, 0.3)], + frame_id="base_link", + tick_index=tick, + ), + _joint_action(torso.tolist(), tick), + _joint_action(left.tolist(), tick), + Action( + control_mode=ControlMode.GRIPPER_POSITION, + gripper=[0.5], + ee_name="left_gripper", + tick_index=tick, + ), + _joint_action(right.tolist(), tick), + Action( + control_mode=ControlMode.GRIPPER_POSITION, + gripper=[-0.5], + ee_name="right_gripper", + tick_index=tick, + ), + ] + vector = _compose_action_group(actions) + assert vector.shape == (_ACTION_DIM,) + np.testing.assert_array_equal( + vector, + np.asarray( + [0.1, 0.2, 0.3, *range(3, 14), 0.5, *range(15, 22), -0.5], + dtype=np.float32, + ), + ) + + +def test_runtime_skill_consumes_world_state_policy_state() -> None: + from openral_rskill_ros.rskill_runner_node import _make_policy_adapter_skill + + robot = RobotDescription.from_yaml(str(_ROOT / "robots" / "r1pro" / "robot.yaml")) + manifest = load_rskill_manifest(str(_ROOT / "rskills" / "gr00t-n17-b1k-turning-on-radio")) + + class Adapter: + spec = VLASpec( + id="gr00t", weights_uri=str(_ROOT / "rskills" / "gr00t-n17-b1k-turning-on-radio") + ) + device = "cpu" + + def __init__(self) -> None: + self.state: np.ndarray | None = None + + def reset(self) -> None: + return None + + def step(self, observation: dict[str, object], instruction: str) -> np.ndarray: + del instruction + self.state = np.asarray(observation["state"], dtype=np.float32) + return np.zeros(23, dtype=np.float32) + + def close(self) -> None: + return None + + adapter = Adapter() + skill = _make_policy_adapter_skill( + manifest=manifest, + adapter=adapter, + prompt="turn on the radio", + description=robot, + ) + state = np.arange(61, dtype=np.float32) + actions = skill.step( + WorldState( + stamp_ns=1, + joint_state=JointState( + name=[joint.name for joint in robot.joints], + position=[0.0] * len(robot.joints), + velocity=[0.0] * len(robot.joints), + stamp_ns=1, + ), + policy_state=state.tolist(), + diagnostics={"policy_state": "ok"}, + ) + ) + assert isinstance(actions, list) + assert len(actions) == 6 + np.testing.assert_array_equal(adapter.state, state) diff --git a/tests/unit/test_benchmark_schemas.py b/tests/unit/test_benchmark_schemas.py index db33dfb..012f96e 100644 --- a/tests/unit/test_benchmark_schemas.py +++ b/tests/unit/test_benchmark_schemas.py @@ -481,6 +481,16 @@ def test_load_benchmark_suite_rejects_non_mapping_entry(tmp_path: Path) -> None: "RoboTwin 2.0 — dual-arm (aloha-agilex)", "RoboTwin 2.0 (SAPIEN), aloha-agilex", ), + ( + "behavior", + "r1pro", + "behavior", + 100, + 1, + "is_success", + "BEHAVIOR-1K 2026 challenge (100 tasks)", + "OmniGibson / Isaac Sim (sidecar)", + ), ] diff --git a/tests/unit/test_cli_behavior.py b/tests/unit/test_cli_behavior.py new file mode 100644 index 0000000..4ed0ad5 --- /dev/null +++ b/tests/unit/test_cli_behavior.py @@ -0,0 +1,103 @@ +"""BEHAVIOR policy-server protocol and CLI coverage.""" + +from __future__ import annotations + +import msgpack +import numpy as np +from fastapi.testclient import TestClient +from openral_cli.behavior import _create_behavior_app +from openral_cli.main import app +from openral_core import PhysicsBackend, SceneSpec, SimEnvironment, TaskSpec, VLASpec +from openral_sim import make_policy +from openral_sim.policy import PolicyAdapter +from typer.testing import CliRunner + +_CAMERAS = { + "robot_r1::robot_r1:zed_link:Camera:0::rgb", + "robot_r1::robot_r1:left_realsense_link:Camera:0::rgb", + "robot_r1::robot_r1:right_realsense_link:Camera:0::rgb", +} + + +def _zero_policy() -> PolicyAdapter: + env = SimEnvironment( + robot_id="r1pro", + scene=SceneSpec( + id="behavior_challenge", + backend=PhysicsBackend.ISAACSIM, + cameras=["head", "left_wrist", "right_wrist"], + ), + task=TaskSpec( + id="behavior/turning_on_radio", + scene_id="behavior_challenge", + instruction="turn on the radio", + ), + vla=VLASpec(id="zero", weights_uri="mock://behavior", extra={"action_dim": 23}), + ) + return make_policy(env) + + +def _pack(value: object) -> bytes: + def _encode(item: object) -> object: + if isinstance(item, np.ndarray): + return { + b"__ndarray__": True, + b"data": item.tobytes(), + b"dtype": item.dtype.str, + b"shape": item.shape, + } + return item + + return msgpack.packb(value, default=_encode) + + +def _unpack(value: bytes) -> object: + def _decode(item: dict[object, object]) -> object: + if b"__ndarray__" not in item: + return item + return np.ndarray( + buffer=item[b"data"], + dtype=np.dtype(item[b"dtype"]), + shape=item[b"shape"], + ) + + return msgpack.unpackb(value, object_hook=_decode) + + +def test_behavior_serve_help() -> None: + result = CliRunner().invoke(app, ["behavior", "serve", "--help"]) + assert result.exit_code == 0, result.output + for flag in ("--rskill", "--task", "--host", "--port", "--state-dim", "--action-dim"): + assert flag in result.output + + +def test_behavior_websocket_protocol_round_trip() -> None: + policy = _zero_policy() + behavior_app = _create_behavior_app( + policy, + task="turning_on_radio", + instruction="turn on the radio", + state_dim=61, + action_dim=23, + ) + with TestClient(behavior_app) as client: + assert client.get("/healthz").json() == {"status": "ok"} + with client.websocket_connect("/") as websocket: + metadata = _unpack(websocket.receive_bytes()) + assert isinstance(metadata, dict) + assert metadata["task"] == "turning_on_radio" + assert metadata["action_dim"] == 23 + + websocket.send_bytes(_pack({"reset": True})) + observation: dict[str, object] = { + "robot_r1::proprio": np.zeros(61, dtype=np.float32), + } + for key in _CAMERAS: + observation[key] = np.zeros((8, 8, 4), dtype=np.uint8) + websocket.send_bytes(_pack(observation)) + + reply = _unpack(websocket.receive_bytes()) + assert isinstance(reply, dict) + np.testing.assert_array_equal(reply["action"], np.zeros(23, dtype=np.float32)) + + policy.close() diff --git a/tests/unit/test_cli_deploy_sim.py b/tests/unit/test_cli_deploy_sim.py index 15a4c74..c387d9a 100644 --- a/tests/unit/test_cli_deploy_sim.py +++ b/tests/unit/test_cli_deploy_sim.py @@ -44,6 +44,7 @@ _PANDA_MOBILE_CONFIG = _REPO_ROOT / "scenes" / "deploy" / "robocasa_pnp.yaml" _SO101_CONFIG = _REPO_ROOT / "scenes" / "deploy" / "so101_box.yaml" _LIBERO_PNP_CONFIG = _REPO_ROOT / "scenes" / "deploy" / "libero_pnp.yaml" +_BEHAVIOR_CONFIG = _REPO_ROOT / "scenes" / "deploy" / "behavior_r1pro.yaml" def test_bh_deploy_sim_help_renders() -> None: @@ -242,6 +243,20 @@ def test_deploy_sim_scene_attached_sapien_registry_uses_generic_hal(tmp_path: Pa assert "clock_origin:=simulation" in joined +def test_deploy_sim_behavior_r1pro_uses_generic_scene_hal() -> None: + invocation = resolve_launch_invocation( + config=_BEHAVIOR_CONFIG, + robot_override=None, + dashboard_port=4318, + reset_to_pose_service=None, + hal_param_overrides=None, + ) + assert invocation.robot_id == "r1pro" + assert invocation.hal.package == "openral_hal_scene_attached" + assert invocation.hal_params["sim_env_yaml"] == str(_BEHAVIOR_CONFIG.resolve()) + assert invocation.clock_origin == "simulation" + + def test_deploy_sim_real_mode_uses_host_wall_clock_origin() -> None: """Real deployments never publish OpenRAL /clock; they use the graph wall clock.""" invocation = resolve_launch_invocation( diff --git a/tests/unit/test_intree_manifests_have_processors.py b/tests/unit/test_intree_manifests_have_processors.py index d54016e..1d2157c 100644 --- a/tests/unit/test_intree_manifests_have_processors.py +++ b/tests/unit/test_intree_manifests_have_processors.py @@ -33,6 +33,7 @@ { "act-aloha", "act-aloha-insertion", + "gr00t-n17-b1k-turning-on-radio", "gr00t-n17-libero", "gr00t-n17-so101-fruit", # Non-lerobot VLAs: 3D Diffuser Actor (RLBench — own point-cloud/pose diff --git a/tests/unit/test_lifecycle_action_chunk_decoder.py b/tests/unit/test_lifecycle_action_chunk_decoder.py index 74c5d17..22032ac 100644 --- a/tests/unit/test_lifecycle_action_chunk_decoder.py +++ b/tests/unit/test_lifecycle_action_chunk_decoder.py @@ -38,6 +38,10 @@ class itself is just a struct with no behaviour. n_dof: int = 0 horizon: int = 1 control_mode: int = 0 + ee_name: str = "" + frame_id: str = "" + confidence: float = 1.0 + tick_index: int = 0 class TestDecodeActionChunk: @@ -93,6 +97,23 @@ def test_body_twist_populates_typed_field(self) -> None: assert action.control_mode is ControlMode.BODY_TWIST assert action.body_twist == [(0.2, 0.0, 0.0, 0.0, 0.0, 0.3)] + def test_slot_metadata_survives_safety_wire(self) -> None: + chunk = FakeChunk( + flat=[0.25], + n_dof=1, + control_mode=CONTROL_MODE_TO_UINT8[ControlMode.GRIPPER_POSITION], + ee_name="left_gripper", + frame_id="base_link", + confidence=0.75, + tick_index=42, + ) + action = decode_action_chunk(chunk) + assert isinstance(action, Action) + assert action.ee_name == "left_gripper" + assert action.frame_id == "base_link" + assert action.confidence == pytest.approx(0.75) + assert action.tick_index == 42 + def test_cartesian_twist_populates_typed_field(self) -> None: chunk = FakeChunk( flat=[0.1, 0.0, 0.0, 0.0, 0.0, 0.0], diff --git a/tests/unit/test_policy_deps_filter.py b/tests/unit/test_policy_deps_filter.py index 1b112ad..8e68f82 100644 --- a/tests/unit/test_policy_deps_filter.py +++ b/tests/unit/test_policy_deps_filter.py @@ -23,7 +23,10 @@ _FAMILY_INSTALL_HINTS, _FAMILY_REQUIRED_IMPORTS, can_import_policy_family, + can_import_policy_manifest, filter_importable_manifests, + manifest_install_groups, + manifest_install_hint, model_family_install_groups, model_family_required_imports, purge_partial_imports, @@ -36,6 +39,7 @@ class _StubManifest: name: str model_family: str + policy_extras: dict[str, object] | None = None def test_install_hints_and_required_imports_cover_the_same_families() -> None: @@ -167,6 +171,19 @@ def test_filter_importable_manifests_keeps_unknown_families() -> None: assert len(kept) == 1 +def test_behavior_groot_manifest_uses_sidecar_wire_profile() -> None: + manifest = _StubManifest( + name="behavior", + model_family="gr00t", + policy_extras={"implementation": "behavior_b1k_sidecar"}, + ) + ok, reason = can_import_policy_manifest(manifest) + assert ok is True + assert reason is None + assert manifest_install_groups(manifest) == ("behavior-groot",) + assert "behavior-groot" in manifest_install_hint(manifest) + + def test_purge_partial_imports_drops_only_matching_prefixes() -> None: """Belt-and-braces: ``can_import_policy_family`` purges on failure.""" sys.modules["__pd_test_target__"] = object() # type: ignore[assignment] diff --git a/tests/unit/test_sim_attached_non_mujoco.py b/tests/unit/test_sim_attached_non_mujoco.py index 6d91ae3..1a6a791 100644 --- a/tests/unit/test_sim_attached_non_mujoco.py +++ b/tests/unit/test_sim_attached_non_mujoco.py @@ -18,7 +18,7 @@ import numpy as np import pytest -from openral_core import RobotDescription +from openral_core import Action, ControlMode, RobotDescription from openral_hal.sim_attached import SimAttachedHAL from openral_sim.rollout import StepResult @@ -73,6 +73,24 @@ def close(self) -> None: return None +class _GroupedSim(_FakeSim): + action_group_size = 2 + + def __init__(self, joint_positions: list[float]) -> None: + super().__init__(joint_positions) + self.groups: list[list[Action]] = [] + + def step(self, action: np.ndarray) -> StepResult: + raise AssertionError(f"flat step must not run for grouped backend: {action}") + + def step_action_group(self, actions: list[Action]) -> StepResult: + self.groups.append(actions) + self.steps += 1 + obs = self._obs() + obs["policy_state"] = np.arange(8, dtype=np.float32) + return StepResult(obs, 0.0, False, False, {}) + + def test_read_state_uses_obs_joint_positions() -> None: description = _franka_description() jp = [0.1 * i for i in range(len(description.joints))] @@ -150,3 +168,45 @@ def test_read_state_tolerates_length_mismatch(jp_len: int) -> None: assert len(state.position) == len(description.joints) finally: hal.disconnect() + + +def test_read_policy_state_requires_explicit_key() -> None: + # A backend with only the generic obs["state"] exposes NO policy state: + # WorldState.policy_state is never inferred, so the HAL must not fall + # back to obs["state"] and silently publish /openral/policy_state. + description = _franka_description() + hal = SimAttachedHAL(_FakeSim([0.0] * len(description.joints)), description) + hal.connect() + try: + assert hal.idle_step() is True + assert hal.read_policy_state() is None + finally: + hal.disconnect() + + +def test_atomic_action_group_steps_once_after_every_safe_slot() -> None: + description = _franka_description() + env = _GroupedSim([0.0] * len(description.joints)) + hal = SimAttachedHAL(env, description) + hal.connect() + first = Action( + control_mode=ControlMode.BODY_TWIST, + body_twist=[(0.1, 0.0, 0.0, 0.0, 0.0, 0.0)], + tick_index=7, + ) + second = Action( + control_mode=ControlMode.GRIPPER_POSITION, + gripper=[1.0], + ee_name="panda_hand", + tick_index=7, + ) + + hal.send_action(first) + assert env.steps == 0 + assert hal.idle_step() is False + hal.send_action(second) + + assert env.steps == 1 + assert len(env.groups) == 1 + assert [action.tick_index for action in env.groups[0]] == [7, 7] + assert hal.read_policy_state() == list(np.arange(8, dtype=np.float32)) diff --git a/tests/unit/test_task_space_sweep.py b/tests/unit/test_task_space_sweep.py index d4d3677..f076403 100644 --- a/tests/unit/test_task_space_sweep.py +++ b/tests/unit/test_task_space_sweep.py @@ -176,14 +176,17 @@ def test_rc365_sim_executable_after_fix() -> None: assert match.ok is True, match.reasons -def test_every_actuating_skill_has_a_matching_robot() -> None: - """No actuating rSkill ships pointing at an embodiment no robot provides.""" +def test_every_actuating_skill_has_a_matching_robot_or_custom_contract() -> None: + """Every actuating rSkill names a robot or carries the explicit custom contract.""" robots = _robots() orphans = [] for path in RSKILL_YAMLS: skill = RSkillManifest.from_yaml(path) if skill.action_contract is None: continue + if "custom" in skill.embodiment_tags: + assert skill.embodiment_extra is not None + continue if not _matching_robot_names(skill, robots): orphans.append((_name(path), skill.embodiment_tags)) assert not orphans, f"actuating skills with no matching robot: {orphans}" diff --git a/tests/unit/test_world_state.py b/tests/unit/test_world_state.py index 3e6160b..8424bcb 100644 --- a/tests/unit/test_world_state.py +++ b/tests/unit/test_world_state.py @@ -223,6 +223,13 @@ def test_sequential_updates_keep_latest(self) -> None: ws = agg.snapshot() assert ws.joint_state.position[0] == pytest.approx(2.0) + def test_policy_state_round_trips(self) -> None: + agg = _make_agg() + agg.update_policy_state([1.0, 2.0, 3.0]) + snapshot = agg.snapshot() + assert snapshot.policy_state == [1.0, 2.0, 3.0] + assert snapshot.diagnostics["policy_state"] == "ok" + def test_stale_joint_state_detected(self, monkeypatch: pytest.MonkeyPatch) -> None: """Advance the clock past staleness_limit_s and verify 'stale'.""" now_ns = time.time_ns() diff --git a/tools/audit_sim_configs.py b/tools/audit_sim_configs.py index d42d0a3..b4a20d6 100644 --- a/tools/audit_sim_configs.py +++ b/tools/audit_sim_configs.py @@ -127,6 +127,12 @@ class ConfigSpec: "robocasa", "sim", ), + ConfigSpec( + "scenes/sim/behavior_turning_on_radio.yaml", + "rskills/gr00t-n17-b1k-turning-on-radio", + "behavior-groot", + "sim", + ), # ---- BenchmarkScene tier (openral benchmark scene --no-update-manifest --n-episodes 1) ---- # Each row is a demo-grade run, not a paper-comparable claim. ConfigSpec( @@ -181,6 +187,7 @@ class ConfigSpec: ConfigSpec("scenes/deploy/so101_box.yaml", "", "sim", "deploy"), ConfigSpec("scenes/deploy/robocasa_pnp.yaml", "", "robocasa", "deploy"), ConfigSpec("scenes/deploy/libero_pnp.yaml", "", "libero", "deploy"), + ConfigSpec("scenes/deploy/behavior_r1pro.yaml", "", "behavior-groot", "deploy"), ) diff --git a/tools/behavior_groot_sidecar.py b/tools/behavior_groot_sidecar.py new file mode 100644 index 0000000..c0dc18f --- /dev/null +++ b/tools/behavior_groot_sidecar.py @@ -0,0 +1,367 @@ +#!/usr/bin/env python +"""Official Isaac-GR00T BEHAVIOR-1K policy sidecar.""" + +from __future__ import annotations + +import argparse +import importlib +import io +import os +import sys +from typing import Any + +import numpy as np +from numpy.typing import NDArray + + +def _encode_ndarray(obj: Any) -> Any: + if isinstance(obj, np.ndarray): + buf = io.BytesIO() + np.save(buf, obj, allow_pickle=False) + return {"__ndarray__": True, "npy": buf.getvalue()} + return obj + + +def _decode_ndarray(obj: dict[str, Any]) -> Any: + if "__ndarray__" in obj: + return np.load(io.BytesIO(obj["npy"]), allow_pickle=False) + return obj + + +def _parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser(description="OpenRAL BEHAVIOR GR00T policy sidecar") + parser.add_argument("--checkpoint", required=True) + parser.add_argument("--task", default="turning_on_radio") + parser.add_argument("--instruction", default="turn on the radio") + parser.add_argument( + "--control-mode", + choices=("temporal_ensemble", "receeding_temporal", "receeding_horizon"), + default="temporal_ensemble", + ) + parser.add_argument("--device", default="cuda") + parser.add_argument("--quantization", choices=("none", "nf4", "int8"), default="nf4") + parser.add_argument("--nf4-min-params", type=int, default=4_000_000) + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=22000) + return parser.parse_args(argv) + + +def _register_r1pro_modality() -> dict[str, object]: + configs = importlib.import_module("gr00t.configs.data.embodiment_configs") + tags = importlib.import_module("gr00t.data.embodiment_tags") + types = importlib.import_module("gr00t.data.types") + register_modality_config = configs.register_modality_config + EmbodimentTag = tags.EmbodimentTag + ActionConfig = types.ActionConfig + ActionFormat = types.ActionFormat + ActionRepresentation = types.ActionRepresentation + ActionType = types.ActionType + ModalityConfig = types.ModalityConfig + + config = { + "name": "robot_r1", + "observation": { + "head": "robot_r1::robot_r1:zed_link:Camera:0::rgb", + "left_wrist": "robot_r1::robot_r1:left_realsense_link:Camera:0::rgb", + "right_wrist": "robot_r1::robot_r1:right_realsense_link:Camera:0::rgb", + }, + "video": ModalityConfig( + delta_indices=[0], + modality_keys=["head", "left_wrist", "right_wrist"], + ), + "state": ModalityConfig( + delta_indices=[0], + modality_keys=[ + "base_qvel", + "torso", + "left_arm", + "left_gripper", + "right_arm", + "right_gripper", + ], + ), + "action": ModalityConfig( + delta_indices=list(range(16)), + modality_keys=[ + "base", + "torso", + "left_arm", + "left_gripper", + "right_arm", + "right_gripper", + ], + action_configs=[ + ActionConfig( + rep=ActionRepresentation.ABSOLUTE, + type=ActionType.NON_EEF, + format=ActionFormat.DEFAULT, + ), + ActionConfig( + rep=ActionRepresentation.RELATIVE, + type=ActionType.NON_EEF, + format=ActionFormat.DEFAULT, + state_key="torso", + ), + ActionConfig( + rep=ActionRepresentation.RELATIVE, + type=ActionType.NON_EEF, + format=ActionFormat.DEFAULT, + state_key="left_arm", + ), + ActionConfig( + rep=ActionRepresentation.ABSOLUTE, + type=ActionType.NON_EEF, + format=ActionFormat.DEFAULT, + is_gripper=True, + ), + ActionConfig( + rep=ActionRepresentation.RELATIVE, + type=ActionType.NON_EEF, + format=ActionFormat.DEFAULT, + state_key="right_arm", + ), + ActionConfig( + rep=ActionRepresentation.ABSOLUTE, + type=ActionType.NON_EEF, + format=ActionFormat.DEFAULT, + is_gripper=True, + ), + ], + ), + "language": ModalityConfig( + delta_indices=[0], + modality_keys=["annotation.human.task_description"], + ), + } + register_modality_config(config, embodiment_tag=EmbodimentTag.NEW_EMBODIMENT) + return { + "state": { + "base_qvel": {"start": 0, "end": 3}, + "torso": {"start": 53, "end": 57}, + "left_arm": {"start": 3, "end": 10}, + "left_gripper": {"start": 24, "end": 26}, + "right_arm": {"start": 28, "end": 35}, + "right_gripper": {"start": 49, "end": 51}, + }, + "action": { + "base": {"start": 0, "end": 3}, + "torso": {"start": 3, "end": 7}, + "left_arm": {"start": 7, "end": 14}, + "left_gripper": {"start": 14, "end": 15}, + "right_arm": {"start": 15, "end": 22}, + "right_gripper": {"start": 22, "end": 23}, + }, + } + + +class _BehaviorGrootPolicy: + def __init__(self, args: argparse.Namespace) -> None: + tags = importlib.import_module("gr00t.data.embodiment_tags") + wrapper = importlib.import_module("gr00t.eval.eval_b1k_wrapper") + policy_module = importlib.import_module("gr00t.policy.gr00t_policy") + EmbodimentTag = tags.EmbodimentTag + B1KPolicyWrapper = wrapper.B1KPolicyWrapper + Gr00tPolicy = policy_module.Gr00tPolicy + + modality = _register_r1pro_modality() + load_device = "cpu" if args.quantization in ("nf4", "int8") else args.device + policy = Gr00tPolicy( + embodiment_tag=EmbodimentTag.NEW_EMBODIMENT, + model_path=args.checkpoint, + device=load_device, + strict=True, + ) + _drop_backbone_lm_head(policy.model) + if args.quantization == "nf4": + _quantize_nf4( + policy.model, + device=args.device, + min_params=args.nf4_min_params, + ) + elif args.quantization == "int8": + _quantize_int8( + policy.model, + device=args.device, + min_params=args.nf4_min_params, + ) + self._policy = B1KPolicyWrapper( + policy=policy, + embodiment_tag=EmbodimentTag.NEW_EMBODIMENT, + modality_config=modality, + text_prompt=args.instruction, + control_mode=args.control_mode, + ) + + def reset(self) -> None: + self._policy.reset() + + def get_action(self, observation: dict[str, object]) -> NDArray[np.float32]: + instruction = observation.pop("openral_instruction", None) + if isinstance(instruction, str): + self._policy.text_prompt = instruction + dump_dir = os.environ.get("OPENRAL_BEHAVIOR_GROOT_DUMP_OBS") + if dump_dir: + self._dump_count = getattr(self, "_dump_count", 0) + if self._dump_count < 32: + import pickle + from pathlib import Path + + Path(dump_dir).mkdir(parents=True, exist_ok=True) + with open(f"{dump_dir}/obs_{self._dump_count:03d}.pkl", "wb") as fh: + pickle.dump(observation, fh) + self._dump_count += 1 + action = self._policy.act(observation) + return np.asarray(action.detach().cpu().numpy(), dtype=np.float32).reshape(-1) + + +def _drop_backbone_lm_head(model: Any) -> None: + """Replace the Qwen3-VL lm_head with Identity. + + The B1K backbone consumes only ``hidden_states[-1]``; the full-vocab logits + projection (151k x seq) is dead weight and its output was the single largest + inference allocation (~594 MiB) on 8 GB hosts. + """ + torch = importlib.import_module("torch") + for module in model.modules(): + if hasattr(module, "lm_head") and hasattr(module, "language_model"): + module.lm_head = torch.nn.Identity() + + +def _quantize_nf4(model: Any, *, device: str, min_params: int = 4_000_000) -> None: + """Whole-model NF4 rewrite matching OpenRAL's native GR00T N1.7 path.""" + if not device.startswith("cuda"): + raise ValueError("NF4 quantization requires a CUDA device.") + torch = importlib.import_module("torch") + bnb = importlib.import_module("bitsandbytes") + + def _replace(module: Any) -> None: + for name, child in list(module.named_children()): + if isinstance(child, torch.nn.Linear) and child.weight.numel() >= min_params: + quantized = bnb.nn.Linear4bit( + child.in_features, + child.out_features, + bias=child.bias is not None, + compute_dtype=torch.bfloat16, + quant_type="nf4", + ) + quantized.weight = bnb.nn.Params4bit( + child.weight.data.clone(), + requires_grad=False, + quant_type="nf4", + ) + if child.bias is not None: + quantized.bias = torch.nn.Parameter( + child.bias.data.clone().to(torch.bfloat16), + requires_grad=False, + ) + setattr(module, name, quantized) + else: + _replace(child) + + def _first_float_dtype(self: Any) -> Any: + for parameter in self.parameters(): + if parameter.is_floating_point(): + return parameter.dtype + return torch.bfloat16 + + model.__class__.dtype = property(_first_float_dtype) + _replace(model) + model.to(device) + model.eval() + + +def _quantize_int8(model: Any, *, device: str, min_params: int = 4_000_000) -> None: + """Whole-model LLM.int8 rewrite; ~2x NF4 size but far less lossy on 3B policies.""" + if not device.startswith("cuda"): + raise ValueError("int8 quantization requires a CUDA device.") + torch = importlib.import_module("torch") + bnb = importlib.import_module("bitsandbytes") + + def _replace(module: Any) -> None: + for name, child in list(module.named_children()): + if isinstance(child, torch.nn.Linear) and child.weight.numel() >= min_params: + quantized = bnb.nn.Linear8bitLt( + child.in_features, + child.out_features, + bias=child.bias is not None, + has_fp16_weights=False, + threshold=6.0, + ) + quantized.weight = bnb.nn.Int8Params( + child.weight.data.clone(), + requires_grad=False, + has_fp16_weights=False, + ) + if child.bias is not None: + quantized.bias = torch.nn.Parameter( + child.bias.data.clone().to(torch.float16), + requires_grad=False, + ) + setattr(module, name, quantized) + else: + _replace(child) + + def _first_float_dtype(self: Any) -> Any: + for parameter in self.parameters(): + if parameter.is_floating_point(): + return parameter.dtype + return torch.bfloat16 + + model.__class__.dtype = property(_first_float_dtype) + _replace(model) + model.to(device) + model.eval() + + +def _serve(policy: _BehaviorGrootPolicy, *, task: str, host: str, port: int) -> int: + import msgpack + import zmq + + ctx = zmq.Context() + socket = ctx.socket(zmq.REP) + socket.bind(f"tcp://{host}:{port}") + print(f"[behavior_groot_sidecar] serving on tcp://{host}:{port}", flush=True) + running = True + while running: + request = msgpack.unpackb(socket.recv(), object_hook=_decode_ndarray, raw=False) + endpoint = request.get("endpoint") + data = request.get("data", {}) or {} + try: + if endpoint == "ping": + reply: dict[str, object] = { + "ok": True, + "model": "behavior_groot", + "task": task, + "action_dim": 23, + } + elif endpoint == "reset": + policy.reset() + reply = {"ok": True} + elif endpoint == "get_action": + reply = {"action": policy.get_action(dict(data["observation"]))} + elif endpoint == "close": + reply = {"ok": True} + running = False + else: + reply = {"error": f"unknown endpoint {endpoint!r}"} + except Exception as exc: + reply = {"error": f"{type(exc).__name__}: {exc}"} + socket.send(msgpack.packb(reply, default=_encode_ndarray, use_bin_type=True)) + socket.close(linger=0) + ctx.term() + return 0 + + +def main(argv: list[str]) -> int: + os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") + args = _parse_args(argv) + return _serve( + _BehaviorGrootPolicy(args), + task=args.task, + host=args.host, + port=args.port, + ) + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/tools/behavior_scene_sidecar.py b/tools/behavior_scene_sidecar.py new file mode 100644 index 0000000..84c9457 --- /dev/null +++ b/tools/behavior_scene_sidecar.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python +"""BEHAVIOR-1K OmniGibson environment sidecar.""" + +from __future__ import annotations + +import argparse +import importlib +import io +import sys +from typing import Any + +import numpy as np +from numpy.typing import NDArray + + +def _encode_ndarray(obj: Any) -> Any: + if isinstance(obj, np.ndarray): + buf = io.BytesIO() + np.save(buf, obj, allow_pickle=False) + return {"__ndarray__": True, "npy": buf.getvalue()} + return obj + + +def _decode_ndarray(obj: dict[str, Any]) -> Any: + if "__ndarray__" in obj: + return np.load(io.BytesIO(obj["npy"]), allow_pickle=False) + return obj + + +def _parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser(description="OpenRAL BEHAVIOR scene sidecar") + parser.add_argument("--task", required=True) + parser.add_argument("--instance-index", type=int, default=0) + parser.add_argument( + "--mode", choices=("train", "public_test", "hidden_test"), default="public_test" + ) + parser.add_argument( + "--env-wrapper", + # DefaultWrapper (rgb-only 224x224) is the only wrapper that boots on the + # pinned OmniGibson build: RGBDFullResWrapper reloads the full observation + # space in __init__, reading joint state before og.sim.update_handles(). + default="omnigibson.eval.wrappers.DefaultWrapper", + ) + parser.add_argument("--max-steps", type=int, default=500) + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=23000) + parser.add_argument("--headless", action="store_true") + return parser.parse_args(argv) + + +def _to_numpy(value: object) -> object: + if isinstance(value, dict): + return {str(key): _to_numpy(item) for key, item in value.items()} + if isinstance(value, list): + return [_to_numpy(item) for item in value] + if isinstance(value, tuple): + return tuple(_to_numpy(item) for item in value) + detach = getattr(value, "detach", None) + if callable(detach): + return detach().cpu().numpy() + return value + + +def _to_numpy_dict(value: object) -> dict[str, object]: + converted = _to_numpy(value) + if not isinstance(converted, dict) or not all(isinstance(key, str) for key in converted): + raise TypeError( + f"expected string-keyed observation mapping, got {type(converted).__name__}" + ) + return converted + + +class _BehaviorEnv: + def __init__(self, args: argparse.Namespace) -> None: + omega = importlib.import_module("omegaconf") + evaluator_module = importlib.import_module("omnigibson.eval.evaluator") + self._torch = importlib.import_module("torch") + self._task = args.task + self._instance_index = args.instance_index + self._mode = args.mode + instance_id = evaluator_module.resolve_instance_ids( + args.task, + [args.instance_index], + mode=args.mode, + )[0] + cfg = omega.OmegaConf.create( + { + "env_wrapper": {"_target_": args.env_wrapper}, + "policy_name": "local", + "model": { + "_target_": "omnigibson.eval.policies.LocalPolicy", + "action_dim": None, + }, + "headless": args.headless, + "partial_scene_load": True, + "max_steps": args.max_steps, + "write_video": False, + "mode": args.mode, + "seed": 0, + "task": {"name": args.task}, + "robot": None, + } + ) + self._evaluator = evaluator_module.Evaluator(cfg) + self._evaluator.reset() + self._evaluator.load_task_instance(instance_id) + self._steps = 0 + self.reset() + + @property + def action_dim(self) -> int: + return int(self._evaluator.robot.action_dim) + + def reset(self, seed: int | None = None) -> dict[str, object]: + del seed + self._evaluator.reset() + self._steps = 0 + return _to_numpy_dict(self._evaluator.obs) + + def step(self, action: NDArray[np.float32]) -> dict[str, object]: + evaluator = self._evaluator + action_tensor = self._torch.from_numpy(np.asarray(action, dtype=np.float32)) + evaluator.robot_action = action_tensor + obs, reward, terminated, truncated, info = evaluator.env.step( + action_tensor, + n_render_iterations=1, + ) + obs = evaluator._sync_lights_and_get_obs(obs) + evaluator.obs = evaluator._preprocess_obs(obs) + for metric in evaluator.metrics: + metric.step(evaluator.env, action_tensor, obs, reward, terminated, truncated, info) + self._steps += 1 + metrics: dict[str, object] = {} + if terminated or truncated: + for metric in evaluator.metrics: + metrics.update(metric.aggregate(evaluator.env)) + success = bool(info.get("done", {}).get("success", False)) + metric_info = _to_numpy_dict(metrics) + return { + "observation": _to_numpy_dict(evaluator.obs), + "reward": float(reward), + "terminated": bool(terminated), + "truncated": bool(truncated), + "info": {"is_success": success, **metric_info}, + "sim_time_ns": round(self._steps * (1.0 / 30.0) * 1e9), + } + + def close(self) -> None: + self._evaluator.__exit__(None, None, None) + + +def _serve(env: _BehaviorEnv, *, args: argparse.Namespace) -> int: + msgpack = importlib.import_module("msgpack") + zmq = importlib.import_module("zmq") + ctx = zmq.Context() + socket = ctx.socket(zmq.REP) + socket.bind(f"tcp://{args.host}:{args.port}") + print(f"[behavior_scene_sidecar] serving on tcp://{args.host}:{args.port}", flush=True) + running = True + while running: + request = msgpack.unpackb(socket.recv(), object_hook=_decode_ndarray, raw=False) + endpoint = request.get("endpoint") + data = request.get("data", {}) or {} + try: + if endpoint == "ping": + reply: dict[str, object] = { + "ok": True, + "scene": "behavior", + "task": args.task, + "instance_index": args.instance_index, + "action_dim": env.action_dim, + } + elif endpoint == "reset": + reply = { + "observation": env.reset(seed=data.get("seed")), + "sim_time_ns": 0, + } + elif endpoint == "step": + reply = env.step(np.asarray(data["action"], dtype=np.float32)) + elif endpoint == "close": + env.close() + reply = {"ok": True} + running = False + else: + reply = {"error": f"unknown endpoint {endpoint!r}"} + except Exception as exc: + reply = {"error": f"{type(exc).__name__}: {exc}"} + socket.send(msgpack.packb(reply, default=_encode_ndarray, use_bin_type=True)) + socket.close(linger=0) + ctx.term() + return 0 + + +def main(argv: list[str]) -> int: + args = _parse_args(argv) + return _serve(_BehaviorEnv(args), args=args) + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/uv.lock b/uv.lock index aaa6780..f85deb7 100644 --- a/uv.lock +++ b/uv.lock @@ -2628,6 +2628,8 @@ name = "openral-cli" version = "0.1.0" source = { editable = "python/cli" } dependencies = [ + { name = "fastapi" }, + { name = "msgpack" }, { name = "opencv-python-headless" }, { name = "openral-core" }, { name = "openral-detect" }, @@ -2638,10 +2640,13 @@ dependencies = [ { name = "pyudev" }, { name = "rich" }, { name = "typer" }, + { name = "uvicorn", extra = ["standard"] }, ] [package.metadata] requires-dist = [ + { name = "fastapi", specifier = ">=0.110" }, + { name = "msgpack", specifier = ">=1" }, { name = "opencv-python-headless", specifier = ">=4.9,<5" }, { name = "openral-core", editable = "python/core" }, { name = "openral-detect", editable = "python/detect" }, @@ -2652,6 +2657,7 @@ requires-dist = [ { name = "pyudev", specifier = ">=0.24.4" }, { name = "rich", specifier = ">=13.7" }, { name = "typer", specifier = ">=0.12" }, + { name = "uvicorn", extras = ["standard"], specifier = ">=0.27" }, ] [[package]] @@ -2996,6 +3002,10 @@ dependencies = [ ] [package.dev-dependencies] +behavior-groot = [ + { name = "msgpack" }, + { name = "pyzmq" }, +] clip = [ { name = "open-clip-torch" }, ] @@ -3144,6 +3154,10 @@ requires-dist = [ ] [package.metadata.requires-dev] +behavior-groot = [ + { name = "msgpack", specifier = ">=1" }, + { name = "pyzmq", specifier = ">=25" }, +] clip = [{ name = "open-clip-torch", specifier = ">=2.24" }] dataset = [{ name = "openral-dataset", extras = ["lerobot"], editable = "python/dataset" }] dev = [