feat(sim): add BEHAVIOR-1K challenge support - #25
Conversation
f70f2e1 to
7829d12
Compare
Code reviewFound 3 issues:
openral/python/core/src/openral_core/schemas.py Lines 2205 to 2207 in c521377 openral/python/hal/src/openral_hal/sim_attached.py Lines 1733 to 1735 in c521377
openral/packages/openral_rskill_ros/openral_rskill_ros/rskill_runner_node.py Lines 2490 to 2492 in c521377
openral/packages/openral_rskill_ros/openral_rskill_ros/rskill_runner_node.py Lines 1894 to 1896 in c521377 🤖 Generated with Claude Code - If this code review was useful, please react with 👍. Otherwise, react with 👎. |
Three findings from the multi-agent review of PR #25 (all confirmed >=80 confidence), plus one contradicted comment: - SimAttachedHAL.read_policy_state no longer falls back to the generic obs["state"]: WorldState.policy_state is documented "never inferred", and the fallback silently published /openral/policy_state for every sim backend (LIBERO, RoboCasa, ...) that never opted in. The BEHAVIOR backend always emits an explicit obs["policy_state"]. New unit test covers the state-only backend returning None. - rskill_runner_node's ImportError safety net now uses the manifest-aware manifest_install_hint(manifest) so a failed behavior_b1k_sidecar import suggests `just sync --group behavior-groot` instead of the wrong in-process gr00t hint. - _make_policy_adapter_skill docstring documents both obs["state"] substitution paths (use_world_state_policy_state and tf_lookup + state_contract.layout). - idle_step comment/docstring rewritten: they claimed "never build a non-zero vector" directly above the new idle_action() hold-pose override; both now describe the position-controlled-backend escape hatch instead of contradicting it. Review: #25 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
|
The 3 review findings above are addressed in 29a2b7ae4f4a4574f652659e5e0f1a55665da9f9 (fallback dropped + test, manifest-aware hint, both docstrings). |
Three findings from the multi-agent review of PR #25 (all confirmed >=80 confidence), plus one contradicted comment: - SimAttachedHAL.read_policy_state no longer falls back to the generic obs["state"]: WorldState.policy_state is documented "never inferred", and the fallback silently published /openral/policy_state for every sim backend (LIBERO, RoboCasa, ...) that never opted in. The BEHAVIOR backend always emits an explicit obs["policy_state"]. New unit test covers the state-only backend returning None. - rskill_runner_node's ImportError safety net now uses the manifest-aware manifest_install_hint(manifest) so a failed behavior_b1k_sidecar import suggests `just sync --group behavior-groot` instead of the wrong in-process gr00t hint. - _make_policy_adapter_skill docstring documents both obs["state"] substitution paths (use_world_state_policy_state and tf_lookup + state_contract.layout). - idle_step comment/docstring rewritten: they claimed "never build a non-zero vector" directly above the new idle_action() hold-pose override; both now describe the position-controlled-backend escape hatch instead of contradicting it. Review: #25 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
b3830d5 to
c85f9f0
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
The official BEHAVIOR-1K turning_on_radio checkpoint could not run
alongside OmniGibson on an 8 GB host: BF16 needed ~6.13 GiB and the
first NF4 attempt still peaked at 3.6 GiB because the Qwen3-VL backbone
computed full-vocab logits (151k x seq, ~594 MiB) that B1KPolicyWrapper
never reads.
- tools/behavior_groot_sidecar.py: add --quantization {none,nf4} +
--nf4-min-params whole-model bitsandbytes Linear4bit rewrite
(CPU-first load, dtype property patch), and drop the unused Qwen3-VL
lm_head (Identity) - inference peak 3.6 -> 2.77 GiB.
- behavior_groot adapter forwards quantization/nf4_min_params from
policy_extras; rskill.yaml pins nf4 with min_params 1000000.
- openral behavior serve: accept RGBA evaluator frames and slice to
RGB (official wrappers emit 224x224x4).
- tests: RGBA protocol round-trip, NF4 manifest assertions.
Live-validated on RTX 4070 Laptop 8 GB against the official evaluator:
full rollout instance 301, 3225 steps to timeout, MP4 + metrics JSON
written (success=false, q_score 0.0 - no success-rate claim).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Three findings from the multi-agent review of PR #25 (all confirmed >=80 confidence), plus one contradicted comment: - SimAttachedHAL.read_policy_state no longer falls back to the generic obs["state"]: WorldState.policy_state is documented "never inferred", and the fallback silently published /openral/policy_state for every sim backend (LIBERO, RoboCasa, ...) that never opted in. The BEHAVIOR backend always emits an explicit obs["policy_state"]. New unit test covers the state-only backend returning None. - rskill_runner_node's ImportError safety net now uses the manifest-aware manifest_install_hint(manifest) so a failed behavior_b1k_sidecar import suggests `just sync --group behavior-groot` instead of the wrong in-process gr00t hint. - _make_policy_adapter_skill docstring documents both obs["state"] substitution paths (use_world_state_policy_state and tf_lookup + state_contract.layout). - idle_step comment/docstring rewritten: they claimed "never build a non-zero vector" directly above the new idle_action() hold-pose override; both now describe the position-controlled-backend escape hatch instead of contradicting it. Review: #25 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
The BEHAVIOR GR00T sidecar gains --quantization int8 (bitsandbytes Linear8bitLt, same min-params threshold as NF4) and an env-gated observation dump (OPENRAL_BEHAVIOR_GROOT_DUMP_OBS) that pickles the first 32 wire observations. Together they enable the offline bf16/NF4/int8 action comparison that exonerated NF4 for this checkpoint (MAE 0.0029 vs bf16; int8 0.0017 but 3.7 GiB peak cannot co-reside with OmniGibson on 8 GB, so NF4 stays the serving default). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
tick_index (Action), policy_state (WorldState) and the rSkill manifest state/action contracts were added earlier on this branch without re-running `just schema-export`; the quality gate's drift check caught it. Backward-compatible additions only — no schema_version bump needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
c85f9f0 to
5e77967
Compare
RGBDFullResWrapper can never boot on the pinned OmniGibson build: its __init__ reloads the full observation space, which reads joint positions before Evaluator.__init__ calls og.sim.update_handles(), so the articulation view is still None and the sidecar segfaults. Every previously verified rollout actually ran the official evaluator's DefaultWrapper (rgb-only 224x224 - also GR00T's native input); the RGBDFullResWrapper default in the scene sidecar, the sim backend, and the sim/deploy scene YAMLs was latent-broken and first exercised by the new 'openral benchmark run --suite behavior' path. Caught live on the 8 GB reference host; wrapper stays overridable via backend_options. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
benchmarks/behavior.yaml carries all 100 public tasks of the 2026 challenge (BEHAVIOR-1K defines 1,000 activities; the challenge evaluator exposes this 100-task subset), each with the official per-task step cap (int(1.5 x mean human-demo length), 3224-39090) read from the installed challenge human-demo stats. Like maniskill3_panda, --rskill auto-filters to the policy's evaluated_tasks - today that runs turning_on_radio (the one released organizer checkpoint) and skips 99 with a logged summary. - scenes/benchmark/behavior_turning_on_radio.yaml for 'benchmark scene' - BenchmarkName literal gains 'behavior' (the manifest write-back refused the new suite key); exported JSON schemas regenerated - SCENE_FAMILY_TASK_SPACE declares the behavior family's mixed 23-D body_twist+joint+gripper interface (task-space sweep) - rSkill manifest: evaluated_tasks + benchmarks.behavior from a real run; eval/behavior.json is a full 3224-step episode on the 8 GB host (success=false - the organizer checkpoint's known flakiness, Isaac-GR00T#515; NF4-vs-bf16 action MAE is 0.003) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
'openral benchmark run' now records one world MP4 per episode by default (--no-video restores allocation-light runs), landing in <eval JSON dir>/videos/<suite_id> (override with --video-dir) as <task>[_seed<n>]_<rskill>_<success|fail>.mp4 plus a videos.json manifest - the same write_world_videos pipeline 'benchmark scene --save-video' already used. run_benchmark(video_dir=...) writes and frees each episode's frames as it goes, so multi-hundred-episode suites stay memory-flat. Verified live: the behavior suite run wrote an 88 MB 3224-step rollout MP4. Videos are gitignored next to the checked-in eval JSON. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
What changed
turning_on_radioB1K checkpoint as an R1Pro rSkillsim run, and fulldeploy simconfigsWhy
OpenRAL needs a reproducible path to evaluate and deploy its policies on the 2026 BEHAVIOR-1K household tasks while retaining the normal safety-kernel and observability graph.
How tested
just lintjust test— 4,000 passed, 46 skippedopenral sim run --config scenes/sim/behavior_turning_on_radio.yaml --rskill rskills/gr00t-n17-b1k-turning-on-radio --dry-runopenral deploy sim --config scenes/deploy/behavior_r1pro.yaml --initial-task "turn on the radio" --no-object-detector --dry-runLive validation (RTX 4070 Laptop, 8 GB)
Commit
c521377adds whole-model NF4 in the GR00T sidecar (plus dropping the unused Qwen3-VLlm_head, whose full-vocab logits were the largest single inference allocation) and RGBA->RGB slicing in the WebSocket bridge. With that, the full official-evaluator loop runs on one 8 GB card:success=false,q_score 0.0— infrastructure reproduction only; whether the zero score is NF4 degradation or checkpoint zero-shot behavior is not yet isolated, so no success-rate claim is maderuff+mypy --strictclean