Skip to content

feat(sim): add BEHAVIOR-1K challenge support - #25

Draft
AdrianLlopart wants to merge 12 commits into
masterfrom
feat/behavior_challenge_support
Draft

feat(sim): add BEHAVIOR-1K challenge support#25
AdrianLlopart wants to merge 12 commits into
masterfrom
feat/behavior_challenge_support

Conversation

@AdrianLlopart

@AdrianLlopart AdrianLlopart commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add the official BEHAVIOR WebSocket evaluation bridge and pinned GR00T N1.7 sidecar adapter
  • package the turning_on_radio B1K checkpoint as an R1Pro rSkill
  • add the R1Pro manifest, OmniGibson scene sidecar, sim run, and full deploy sim configs
  • preserve the native 61-D policy state through WorldState and atomically commit six safety-approved slots as one 23-D OmniGibson action

Why

OpenRAL needs a reproducible path to evaluate and deploy its policies on the 2026 BEHAVIOR-1K household tasks while retaining the normal safety-kernel and observability graph.

How tested

  • just lint
  • just test — 4,000 passed, 46 skipped
  • openral sim run --config scenes/sim/behavior_turning_on_radio.yaml --rskill rskills/gr00t-n17-b1k-turning-on-radio --dry-run
  • openral deploy sim --config scenes/deploy/behavior_r1pro.yaml --initial-task "turn on the radio" --no-object-detector --dry-run
  • rSkill publisher dry-run validation

Live validation (RTX 4070 Laptop, 8 GB)

Commit c521377 adds whole-model NF4 in the GR00T sidecar (plus dropping the unused Qwen3-VL lm_head, whose full-vocab logits were the largest single inference allocation) and RGBA->RGB slicing in the WebSocket bridge. With that, the full official-evaluator loop runs on one 8 GB card:

  • policy sidecar: 2.70 GiB loaded, 2.77 GiB inference peak (BF16 needed ~6.13 GiB and OOM'd next to OmniGibson); ~320-540 ms per inference
  • full rollout, public instance 0 (ID 301): 3225 steps to timeout, evaluator MP4 + metrics JSON written; total GPU ~7.0/8.2 GiB, no OOM
  • result: success=false, q_score 0.0 — infrastructure reproduction only; whether the zero score is NF4 degradation or checkpoint zero-shot behavior is not yet isolated, so no success-rate claim is made
  • revalidated after the change: unit suite 4057 passed / 49 skipped, ruff + mypy --strict clean

@AdrianLlopart
AdrianLlopart force-pushed the feat/behavior_challenge_support branch from f70f2e1 to 7829d12 Compare July 22, 2026 12:35
@AdrianLlopart
AdrianLlopart changed the base branch from feat/cuVSLAM to master July 22, 2026 12:35
@AdrianLlopart

Copy link
Copy Markdown
Contributor Author

Code review

Found 3 issues:

  1. WorldState.policy_state's docstring claims it is "never inferred", but SimAttachedHAL.read_policy_state() silently falls back to the generic obs["state"] key when a backend exposes no policy_state — and since HALLifecycleNodeBase._capture_proprio publishes it unconditionally, every sim backend that populates obs["state"] (LIBERO, RoboCasa, …) now feeds /openral/policy_state, not just r1pro/BEHAVIOR (CLAUDE.md says "Explicit beats implicit. No hidden retries, fallbacks, or magic globals"). The fallback branch is also untested. Either drop the fallback or make the docstring/tests match the real behavior.

checkpoint whose state includes more than joint positions/velocities.
Populated through a typed HAL -> WorldState topic; never inferred.
ee_poses: End-effector poses keyed by EE name.

return None
raw = self._last_obs.get("policy_state", self._last_obs.get("state"))
if raw is None:

  1. _make_policy_adapter_skill's docstring documents exactly one obs["state"] substitution path (tf_lookup + state_contract.layout), but the PR adds a second, earlier path (policy_extras.use_world_state_policy_stateworld_state.policy_state) that also sets state_assembled=True and skips the same downstream conversion (CLAUDE.md says "Docs travel with the code. Every PR updates all affected docs in the same commit range").

use_policy_state = bool(
getattr(self.manifest, "policy_extras", {}).get("use_world_state_policy_state")
)

  1. The ImportError belt-and-braces handler still calls the family-only model_family_install_hint(family), so a BEHAVIOR-1K manifest (policy_extras.implementation == "behavior_b1k_sidecar") that fails to import gets the generic gr00t hint — which this PR itself rewrote to say GR00T "runs in-process … no sidecar" — instead of just sync --group behavior-groot. The manifest-aware manifest_install_hint(manifest) added in policy_deps.py (and already used by deploy_sim.py's preflight) is the correct call here (bug due to install_hint = model_family_install_hint(family) in rskill_runner_node.py).

family = manifest.model_family
install_hint = model_family_install_hint(family)
raise ROSRuntimeError(

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

AdrianLlopart added a commit that referenced this pull request Jul 24, 2026
Three findings from the multi-agent review of PR #25 (all confirmed
>=80 confidence), plus one contradicted comment:

- SimAttachedHAL.read_policy_state no longer falls back to the generic
  obs["state"]: WorldState.policy_state is documented "never inferred",
  and the fallback silently published /openral/policy_state for every
  sim backend (LIBERO, RoboCasa, ...) that never opted in. The BEHAVIOR
  backend always emits an explicit obs["policy_state"]. New unit test
  covers the state-only backend returning None.
- rskill_runner_node's ImportError safety net now uses the
  manifest-aware manifest_install_hint(manifest) so a failed
  behavior_b1k_sidecar import suggests `just sync --group
  behavior-groot` instead of the wrong in-process gr00t hint.
- _make_policy_adapter_skill docstring documents both obs["state"]
  substitution paths (use_world_state_policy_state and
  tf_lookup + state_contract.layout).
- idle_step comment/docstring rewritten: they claimed "never build a
  non-zero vector" directly above the new idle_action() hold-pose
  override; both now describe the position-controlled-backend escape
  hatch instead of contradicting it.

Review: #25 (comment)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
@AdrianLlopart

Copy link
Copy Markdown
Contributor Author

The 3 review findings above are addressed in 29a2b7ae4f4a4574f652659e5e0f1a55665da9f9 (fallback dropped + test, manifest-aware hint, both docstrings).

AdrianLlopart added a commit that referenced this pull request Aug 1, 2026
Three findings from the multi-agent review of PR #25 (all confirmed
>=80 confidence), plus one contradicted comment:

- SimAttachedHAL.read_policy_state no longer falls back to the generic
  obs["state"]: WorldState.policy_state is documented "never inferred",
  and the fallback silently published /openral/policy_state for every
  sim backend (LIBERO, RoboCasa, ...) that never opted in. The BEHAVIOR
  backend always emits an explicit obs["policy_state"]. New unit test
  covers the state-only backend returning None.
- rskill_runner_node's ImportError safety net now uses the
  manifest-aware manifest_install_hint(manifest) so a failed
  behavior_b1k_sidecar import suggests `just sync --group
  behavior-groot` instead of the wrong in-process gr00t hint.
- _make_policy_adapter_skill docstring documents both obs["state"]
  substitution paths (use_world_state_policy_state and
  tf_lookup + state_contract.layout).
- idle_step comment/docstring rewritten: they claimed "never build a
  non-zero vector" directly above the new idle_action() hold-pose
  override; both now describe the position-controlled-backend escape
  hatch instead of contradicting it.

Review: #25 (comment)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
@AdrianLlopart
AdrianLlopart force-pushed the feat/behavior_challenge_support branch from b3830d5 to c85f9f0 Compare August 1, 2026 12:08
AdrianLlopart and others added 9 commits August 1, 2026 14:39
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
The official BEHAVIOR-1K turning_on_radio checkpoint could not run
alongside OmniGibson on an 8 GB host: BF16 needed ~6.13 GiB and the
first NF4 attempt still peaked at 3.6 GiB because the Qwen3-VL backbone
computed full-vocab logits (151k x seq, ~594 MiB) that B1KPolicyWrapper
never reads.

- tools/behavior_groot_sidecar.py: add --quantization {none,nf4} +
  --nf4-min-params whole-model bitsandbytes Linear4bit rewrite
  (CPU-first load, dtype property patch), and drop the unused Qwen3-VL
  lm_head (Identity) - inference peak 3.6 -> 2.77 GiB.
- behavior_groot adapter forwards quantization/nf4_min_params from
  policy_extras; rskill.yaml pins nf4 with min_params 1000000.
- openral behavior serve: accept RGBA evaluator frames and slice to
  RGB (official wrappers emit 224x224x4).
- tests: RGBA protocol round-trip, NF4 manifest assertions.

Live-validated on RTX 4070 Laptop 8 GB against the official evaluator:
full rollout instance 301, 3225 steps to timeout, MP4 + metrics JSON
written (success=false, q_score 0.0 - no success-rate claim).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Three findings from the multi-agent review of PR #25 (all confirmed
>=80 confidence), plus one contradicted comment:

- SimAttachedHAL.read_policy_state no longer falls back to the generic
  obs["state"]: WorldState.policy_state is documented "never inferred",
  and the fallback silently published /openral/policy_state for every
  sim backend (LIBERO, RoboCasa, ...) that never opted in. The BEHAVIOR
  backend always emits an explicit obs["policy_state"]. New unit test
  covers the state-only backend returning None.
- rskill_runner_node's ImportError safety net now uses the
  manifest-aware manifest_install_hint(manifest) so a failed
  behavior_b1k_sidecar import suggests `just sync --group
  behavior-groot` instead of the wrong in-process gr00t hint.
- _make_policy_adapter_skill docstring documents both obs["state"]
  substitution paths (use_world_state_policy_state and
  tf_lookup + state_contract.layout).
- idle_step comment/docstring rewritten: they claimed "never build a
  non-zero vector" directly above the new idle_action() hold-pose
  override; both now describe the position-controlled-backend escape
  hatch instead of contradicting it.

Review: #25 (comment)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
The BEHAVIOR GR00T sidecar gains --quantization int8 (bitsandbytes
Linear8bitLt, same min-params threshold as NF4) and an env-gated
observation dump (OPENRAL_BEHAVIOR_GROOT_DUMP_OBS) that pickles the
first 32 wire observations. Together they enable the offline
bf16/NF4/int8 action comparison that exonerated NF4 for this
checkpoint (MAE 0.0029 vs bf16; int8 0.0017 but 3.7 GiB peak cannot
co-reside with OmniGibson on 8 GB, so NF4 stays the serving default).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
tick_index (Action), policy_state (WorldState) and the rSkill manifest
state/action contracts were added earlier on this branch without
re-running `just schema-export`; the quality gate's drift check caught
it. Backward-compatible additions only — no schema_version bump needed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
@AdrianLlopart
AdrianLlopart force-pushed the feat/behavior_challenge_support branch from c85f9f0 to 5e77967 Compare August 1, 2026 12:43
AdrianLlopart and others added 3 commits August 1, 2026 16:55
RGBDFullResWrapper can never boot on the pinned OmniGibson build: its
__init__ reloads the full observation space, which reads joint positions
before Evaluator.__init__ calls og.sim.update_handles(), so the
articulation view is still None and the sidecar segfaults. Every
previously verified rollout actually ran the official evaluator's
DefaultWrapper (rgb-only 224x224 - also GR00T's native input); the
RGBDFullResWrapper default in the scene sidecar, the sim backend, and
the sim/deploy scene YAMLs was latent-broken and first exercised by the
new 'openral benchmark run --suite behavior' path. Caught live on the
8 GB reference host; wrapper stays overridable via backend_options.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
benchmarks/behavior.yaml carries all 100 public tasks of the 2026
challenge (BEHAVIOR-1K defines 1,000 activities; the challenge
evaluator exposes this 100-task subset), each with the official
per-task step cap (int(1.5 x mean human-demo length), 3224-39090)
read from the installed challenge human-demo stats. Like
maniskill3_panda, --rskill auto-filters to the policy's
evaluated_tasks - today that runs turning_on_radio (the one released
organizer checkpoint) and skips 99 with a logged summary.

- scenes/benchmark/behavior_turning_on_radio.yaml for 'benchmark scene'
- BenchmarkName literal gains 'behavior' (the manifest write-back
  refused the new suite key); exported JSON schemas regenerated
- SCENE_FAMILY_TASK_SPACE declares the behavior family's mixed 23-D
  body_twist+joint+gripper interface (task-space sweep)
- rSkill manifest: evaluated_tasks + benchmarks.behavior from a real
  run; eval/behavior.json is a full 3224-step episode on the 8 GB host
  (success=false - the organizer checkpoint's known flakiness,
  Isaac-GR00T#515; NF4-vs-bf16 action MAE is 0.003)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
'openral benchmark run' now records one world MP4 per episode by
default (--no-video restores allocation-light runs), landing in
<eval JSON dir>/videos/<suite_id> (override with --video-dir) as
<task>[_seed<n>]_<rskill>_<success|fail>.mp4 plus a videos.json
manifest - the same write_world_videos pipeline 'benchmark scene
--save-video' already used. run_benchmark(video_dir=...) writes and
frees each episode's frames as it goes, so multi-hundred-episode
suites stay memory-flat. Verified live: the behavior suite run wrote
an 88 MB 3224-step rollout MP4. Videos are gitignored next to the
checked-in eval JSON.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVtL7tuJTB1iUcBprmUviF
Signed-off-by: Adrian Llopart <adrianllopart@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant