Headless Apple-Silicon (arm64/CPU) sim port: VNC GUIs, SLAM, autonomous nav2, and a ROS debug MCP server#5
Open
correll wants to merge 19 commits into
Open
Headless Apple-Silicon (arm64/CPU) sim port: VNC GUIs, SLAM, autonomous nav2, and a ROS debug MCP server#5correll wants to merge 19 commits into
correll wants to merge 19 commits into
Conversation
Standalone Colima-based bringup of ROS + MuJoCo/RoboCasa on Apple Silicon,
headless and CPU-only (Isaac omitted). Verified end-to-end: both containers
boot and exchange data (/clock ~86 Hz, /joint_states ~18 Hz, sim receives
/lowcmd), with the Pink IK frame_task_server and safety_node running.
New:
- docker/docker-compose.mac.yml: standalone compose (no NVIDIA/X11/dri/input);
robocasa + ros on network_mode:host, RMW=rmw_fastrtps_cpp.
- docker/{Base,Robocasa}Dockerfile.arm64, docker/RosDockerfile.slim.arm64:
arm64 images; slim ROS image drops the heavy x86 ML stack.
- docker/scripts/launch_{robocasa,ros}_mac.sh: osmesa+headless sim launcher;
slim 5-package colcon build + minimal bringup.
- core_ws/.../h1_sim_bringup_mac.launch.py: core nodes only (state pubs, Pink
IK, safety, camera TF) — no vision/Nav2/rviz.
Two cross-container fixes needed on arm64/Colima:
- ipc:host on both services. FastDDS prefers shared-memory transport for
same-host peers; with private IPC namespaces (separate /dev/shm) discovery
works over UDP but data is silently dropped (sim logs "Command timeout!
Releasing motors"). x86 uses CycloneDDS/UDP and avoids this.
- colcon --build-base/--install-base on named volumes (native ext4), not the
virtiofs bind-mount, where --symlink-install stalls for minutes at 0% CPU.
h12_ros2_model derives CL_Assets from its build dir, so CL_Assets is also
mounted at /opt/CL_Assets for the ros build.
Also: gitignore process core dumps; drop a stray blank line in h12_mujoco.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… to the Mac arm64 port MuJoCo's and RViz's OpenGL windows cannot forward to macOS XQuartz (broken indirect GLX on Apple Silicon), so render them in-container with software GL (llvmpipe) into an Xvfb and stream pixels out over VNC/noVNC. - RobocasaDockerfile.arm64 / RosDockerfile.slim.arm64: add xvfb, x11vnc, novnc, websockify, fluxbox, libgl1-mesa-dri (+ ros-humble-rviz2 for the ros image), appended last so they don't bust the heavy asset/colcon layers. - launch_robocasa_mac.sh: HAMS_DISPLAY=vnc renders the MuJoCo viewer to Xvfb :99, served on noVNC :6080. Default headless (OSMesa) is unchanged. - launch_ros_mac.sh: HAMS_RVIZ=vnc renders RViz to Xvfb :100, served on :6081. Distinct display number is required — both containers share one network namespace (network_mode: host), so two Xvfb on :99 collide on the abstract X socket. - docker-compose.mac.yml: pass HAMS_DISPLAY (robocasa) and HAMS_RVIZ (ros). - mac_vnc_tunnel.sh: SSH-tunnel the noVNC/VNC ports to the Mac (Colima does not forward host-network container ports); forwards both viewers. - h1_sim.rviz: minimal RViz config (RobotModel + TF, fixed frame pelvis). - robot_cli.sh: rob_pose (named-config postures) + rob_grip helpers to drive the H1. rob_reach/FrameTask is stubbed out — it currently crashes frame_task_server. - README.md: add a self-contained macOS (Apple Silicon) section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ros2 launch h1_bringup <TAB> failed: the package was never colcon-built (colcon-ros refused it — its in-workspace exec_depends estop/livox_ros_driver2/ fast_lio/model_server aren't built on this CPU-only image), and interactive shells sourced the stale host install instead of the real build. - launch_ros_mac.sh: add h1_bringup to the build list, dropping empty package.sh stubs for the 4 unbuilt deps first so colcon installs h1_bringup's launch/ config/rviz files and registers it (stubs carry no ament marker, so they never appear in `ros2 pkg list`). Only h1_sim_bringup_mac.launch.py runs here. - RosDockerfile.slim.arm64: source /opt/core_ws/install (the real INSTALL_BASE, a named volume) in .bashrc instead of the virtiofs /home/code/core_ws/install, so `docker exec … bash` sees the same packages as the running bringup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The MuJoCo bridge already publishes /livox/pointcloud (sensor_msgs/PointCloud2), but its frame_id 'lidar_link' had no TF, so RViz couldn't place it. - h1_sim_bringup_mac.launch.py: static TF livox_link -> lidar_link (identity; they're co-located) so the cloud transforms into the robot/TF tree. - h1_sim.rviz: add a PointCloud2 display on /livox/pointcloud (best-effort QoS, AxisColor by Z). Verified: ~20k pts/scan render around the robot. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…popup Lower-body controller (HAMS_LOWERBODY, default off): - HAMS_LOWERBODY=fame runs the RMA fame_node — it releases the elastic-band tether (after the IK inits) and BALANCES the robot standing unsupported (verified: no fall). It stands/squats but does not locomote. HAMS_LOWERBODY=walk runs the TorchScript walk policy, which currently falls ~4s after release in the RoboCasa sim (kept for completeness). True forward walking needs a sim-tuned policy — future work. - launch_ros_mac.sh builds unitree_hg + h12_lowerbody_controller; the bringup launches fame_node/walking_node gated on HAMS_LOWERBODY; compose threads it. - RosDockerfile: add ros-humble-rosidl-generator-dds-idl (unitree_hg needs it to generate DDS IDL). torch is already in the image. Wallpaper popup: - fluxbox's default style calls fbsetbg, which pops an xmessage "wallpaper cannot be set" error (no wallpaper setter installed). Both images now replace /usr/bin/fbsetbg with a shim that sets a solid root via fbsetroot (bundled). The fluxbox rootCommand init override does NOT stop it — fbsetbg is invoked from the style, so replacing the binary is what works. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pping
Makes the H1 walk and build a live occupancy map on the Mac port.
- Switchable lower-body controller: HAMS_LOWERBODY=switch runs
lowerbody_controller_node (band-held idle -> /lowerbody/start_{fame,walk} with a
gated handover). FAME stands/manipulates; walk locomotes. From a stable FAME
stance the walk policy stays UP and translates (0 falls), vs. the raw walk node
that toppled in ~4s.
- Sim ground-truth odometry: mujoco_ros_bridge publishes odom -> pelvis TF +
/odom from the pelvis free-joint world pose. This is the SLAM odom source, so no
FAST-LIO / Livox-SDK build is needed.
- 2D SLAM (HAMS_SLAM=1 in the bringup): pointcloud_to_laserscan flattens the Livox
hemisphere cloud into /converted_scan (tuned per h1_navigation), and slam_toolbox
builds /map against /odom. Verified: ~126 scan returns/frame, occupancy /map
updating in RViz as the robot walks. Packages baked into the ros image.
- RViz: add Map + LaserScan displays, fixed frame odom, top-down view following
the pelvis.
- robot_cli.sh: rob_stand / rob_walk / rob_go / rob_stop / rob_odom helpers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three fixes to make the walk+map demo actually build a map: - HAMS_SLAM was never threaded through compose to the ros container, so the bringup never launched pointcloud_to_laserscan / slam_toolbox (earlier "working" runs were manual). Add the passthrough. - pointcloud_to_laserscan's h1_navigation tuning (range_min 0.6, band -0.90..0.55) filtered every return out for close-quarters kitchen spawns (drops the counter the robot is against, keeps only the floor which is below the band). Relax to range_min 0.4, band -0.85..1.2, range_max 8 -> ~128 returns; verified the occupancy /map builds in RViz as the robot walks/turns. - HAMS_CAMERAS=0 drops the 3 RGBD camera renders (the heaviest per-step CPU cost); ~doubles sim RTF (0.1 -> 0.2x) for locomotion/SLAM. Lidar + odom unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The walk policy is blind (tracks /cmd_vel, no obstacle awareness). nav2 adds the
avoidance layer: it plans a collision-free path on the SLAM occupancy map + a
lidar obstacle costmap and drives /cmd_vel, which the walk policy follows.
- nav2_config_mac.yaml: nav2_config.yaml with the odom frame swapped from
FAST-LIO's camera_init to the sim's odom, and inflation_radius trimmed to 0.2.
RegulatedPurePursuit controller + Smac hybrid planner, costmap from /map +
/converted_scan, robot_base_frame pelvis.
- Bringup: HAMS_NAV2=1 includes nav2_bringup/navigation_launch.py (implies SLAM).
Verified: the full stack (controller/planner/bt_navigator/costmaps/velocity_
smoother) comes up ACTIVE, and its velocity_smoother publishes /cmd_vel -> the
walk policy consumes it directly. Send goals via RViz "2D Nav Goal" or
/navigate_to_pose.
- ros image: bake ros-humble-nav2-bringup.
Note: nav2 rejects a goal when the robot's start cell is occupied ("start in
lethal space") -- the robot must be standing in open floor, not pressed against a
counter, for planning to succeed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
place_robot_collision_free gains extra_backoff: after clearing the spawn collision, keep backing the robot off the counter (its -x) up to N metres, stopping before a new collision behind, so it stands in open floor with room in front. HAMS_SPAWN_BACKOFF (metres, default 0 = at the counter for manipulation) drives it from h12_mujoco. Verified: "backed robot 1.00 m further into open floor". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The torso lidar returns the robot's own legs/feet at ~0.4m; mapping them puts a phantom obstacle under the robot so nav2 rejects the start. 0.65 drops most of it (the arms/thighs in gait still reach ~0.65m -- a proper self-filter is the full fix). Pairs with HAMS_SPAWN_BACKOFF (robot spawns off the counter, so 0.65 keeps the real walls). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The torso-mounted lidar looks down and its rays hit the robot's own legs/arms (mj_multiRay's bodyexclude only drops the one torso body). Those self-returns get mapped as a phantom obstacle under the robot, so nav2 rejects the start as "lethal space". Flag every geom belonging to a robot body (name prefix robot0_/gripper0_) and drop rays that hit them in _publish_lidar_scan, using the per-geom mask so real obstacles are kept even when a limb is at the same range. Default on for this sim (HAMS_LIDAR_SELF_FILTER=0 restores the limbs as occluders). Verified: the closest scan return jumps from ~0.4 m (legs) to ~0.7 m (real structure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the green planned-path display and the SetGoal tool (-> /goal_pose) so you can click navigation goals directly in RViz. Completes the autonomous-nav demo view: occupancy map + laser scan + robot + planned path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
frontier_explore.py: reads the slam /map, finds frontiers (free cells touching unknown, not against walls), clusters them (scipy.ndimage), and sends the nearest sizeable one as a nav2 goal; loops until no frontiers remain. Nearest-frontier policy with a failure blacklist and per-goal timeout. robot_cli.sh: rob_explore helper (start_walk + run it). Needs HAMS_SLAM=1 HAMS_NAV2=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The global costmap's obstacle_layer had its params nested under an extra `ros__parameters:` level, so nav2 silently ignored them and gave the layer zero observation sources. The planner was steering on the static SLAM /map alone -- a thin 2D slice that misses appliances -- so it routed the robot straight into them (looked like "MuJoCo collision is broken"; it wasn't: the robot correctly could not pass through, and nav2's progress-checker aborted while the frontier explorer blacklisted and moved on). Fixes: - Flatten the global static/obstacle/inflation layer params to sit directly under the plugin key (matching the working local_costmap). The obstacle layer now subscribes to laser_scan_sensor + pointcloud_sensor, and the inflation layer uses its configured radius instead of defaults. - Add the full 3D /livox/pointcloud as a marking source to BOTH costmaps, height band 0.15-1.8 m (drops the floor at z~0, keeps counter/appliance height; pelvis stands at ~0.97 m). Self-returns are already stripped by the bridge's lidar_self_filter. - Fix the local voxel layer's z_voxels 60 -> 16 (nav2 hard-caps at 16, so it was silently clipped to 0.8 m tall, below counter height) with z_resolution 0.125 -> 2.0 m tall. Verified: global_costmap now "Subscribed to Topics: laser_scan_sensor pointcloud_sensor" (was empty), voxel_grid 16-z-cap error gone. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The unitree_interface motor-command watchdog zeroed all motors if no
rt/lowcmd arrived within 0.1s of WALL time. On the headless CPU sim running
~0.2x real-time, that 0.1s wall window is only ~20ms of sim time, so the
compute-starved control loop (policy inference sharing cores with MuJoCo +
nav2 + slam) kept overshooting it -- the watchdog released the motors and
the robot dropped ~0.7s after the elastic band let go ("Command timeout!
Releasing motors" right before each fall).
Measure the watchdog in SIM time (data.time) instead, so the threshold is
independent of how fast the sim actually runs, and default it to 0.5s sim
(override via HAMS_CMD_TIMEOUT). A paused sim no longer trips it either.
Verified: robot now survives the band release and stands freely to sim
t=48s with no watchdog fire (was: fell at sim t~20s, 0.67s after release).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`docker restart` reuses the container filesystem, so a leftover /tmp/.X99-lock (and its abstract socket) from the previous run made Xvfb abort with "Server is already active for display 99" -- the entrypoint then exit(1)'d and the whole sim container died on restart (no MuJoCo, no /clock, no VNC). Kill any stale Xvfb and rm the lock/socket before starting the VNC stack, in both the robocasa (:99) and ros (:100) launch scripts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add docs/NAVIGATION_DEMO.md (SLAM + nav2 + frontier exploration walkthrough: run command, what to watch, the env knobs, pipeline diagram, and nav-specific gotchas), link it from the README, and add a "Navigation demo" subsection. Also correct docs that the fixes made stale: - the walk policy stays upright now (via the switchable controller + the sim-time motor watchdog) -- was documented as "falls in this sim"; - HAMS_LOWERBODY=switch is the stand<->walk mode; - HAMS_SPAWN_BACKOFF is baked at container-create (recreate, don't restart) and 1.5-2.0 keeps the robot off the counters; - restart the two containers coherently (the sim owns /clock). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Give Claude Code first-class tools to inspect and drive the sim H1 instead of shelling in and parsing ros2 CLI output. The server runs INSIDE the ros container (where the FastDDS graph is reachable) and exposes one HTTP MCP port; reach it from the Mac over the same SSH tunnel as the noVNC viewers (Colima forwards no ports, so running an MCP server on the Mac + DDS-across-the-boundary is the trap this avoids). A warm rclpy node subscribes once to the hot topics (/odom, global costmap, /plan, /converted_scan, /clock) and caches the latest message, so status calls answer instantly instead of paying node-startup + DDS-discovery per query. Tools: robot_status, costmap_summary, nav_status, scan_status, set_lowerbody, drive, wait_for, plus generic list/echo/hz/call_service/node_list passthroughs. - docker/scripts/ros_mcp_server.py — the server (FastMCP streamable-http, port 6082). Sensor topics (/odom, /converted_scan) are best-effort publishers, so subscribe best-effort or a reliable reader silently gets nothing. - launch_ros_mac.sh — HAMS_ROS_MCP=1 starts it (pip installs `mcp` on first run). - docker-compose.mac.yml — HAMS_ROS_MCP / HAMS_ROS_MCP_PORT passthrough. - mac_vnc_tunnel.sh — forward 6082 alongside the viewer ports. - docs/ROS_MCP_DEBUG.md — setup, `claude mcp add` registration, tool list; linked from the README. Verified end-to-end: launcher auto-starts the server; MCP round-trip returns live robot_status (pose/uprightness/posture), scan_status, wait_for, costmap_summary; tunnel reaches http://localhost:6082/mcp from the Mac. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a headless, CPU-only Apple-Silicon (arm64) port of the HAMS simulation — ROS 2 (Humble) + MuJoCo/RoboCasa — that runs entirely under Colima/Docker on a Mac with no Isaac, no NVIDIA, no GPU. The change is almost entirely additive (
+2618 / −13across 21 files): new arm64 Docker images, a standalonedocker/docker-compose.mac.yml, and helper scripts. The existing Linux/Isaac path is untouched.From a cold
docker compose up, this brings the H1-2 up in a RoboCasa kitchen and can autonomously map and navigate it — verified end to end (robot walked ~3 m avoiding furniture, upright throughout, 0 falls).What's included
Base sim port
BaseDockerfile.arm64,RobocasaDockerfile.arm64,RosDockerfile.slim.arm64— arm64 images.docker-compose.mac.yml— both containers onnetwork_mode: host+ipc: host, FastDDS (rmw_fastrtps_cpp), named volumes for the colcon build/install (virtiofs stalls symlink-install).GUIs over VNC/noVNC (XQuartz can't do indirect GLX on Apple Silicon)
HAMS_DISPLAY=vnc(MuJoCo viewer, :6080) andHAMS_RVIZ=vnc(RViz, :6081) are independent.mac_vnc_tunnel.sh— Colima forwards no container ports, so an SSH tunnel over Colima's own SSH exposes the noVNC (and MCP) ports to the Mac.Robot control
robot_cli.sh—rob_pose/rob_grip/rob_stand/rob_walk/rob_go/rob_explorehelpers.HAMS_LOWERBODY=fame(RMA policy stands unsupported) orswitch(switchable stand↔walk with a gated FAME→walk handover).Perception + 2D SLAM (
HAMS_SLAM=1)/odom+odom→pelvisTF from the pelvis free-joint (skips a heavy FAST-LIO build).pointcloud_to_laserscan(Livox cloud →/converted_scan) →slam_toolbox→/map.Autonomous navigation (
HAMS_NAV2=1)frontier_explore.py— nearest-frontier exploration with a failure blacklist.HAMS_SPAWN_BACKOFF— spawn the robot in open floor so nav2 has room to plan.Stability & robustness fixes (this is where most of the recent debugging went)
ros__parameters:key, so the planner had zero sensor sources and drove into furniture. Flattened, and added the 3D cloud so it sees obstacles the flat scan slice misses.rt/lowcmdwatchdog used a 100 ms wall-clock timeout, far too tight on a ~0.2× sim, so it zeroed the motors and dropped the robot ~0.7 s after the tether released. Made it sim-time based (HAMS_CMD_TIMEOUT).docker restartleft a stale/tmp/.X99-lockthat crashed Xvfb and took the whole sim container down. Cleared before start, in both launch scripts.ROS debugging MCP server (
HAMS_ROS_MCP=1)ros_mcp_server.pyruns inside the ros container (where DDS is reachable) and exposes an HTTP MCP endpoint, tunneled to the Mac like the viewers. Warm rclpy subscribers back tools likerobot_status,wait_for,costmap_summary,drive— so tooling can inspect/drive the robot without the slowros2CLI dance.Docs — a macOS section in
README.md, plusdocs/NAVIGATION_DEMO.mdanddocs/ROS_MCP_DEBUG.md.Key gotchas (baked into the code + docs)
ipc: host, or FastDDS SHM data silently fails across the container boundary (discovery works, data doesn't).:99/:100) — the containers share one network namespace./clock, so restarting only one desyncs sim time (TF extrapolation errors, a frozen0×0SLAM map).HAMS_SPAWN_BACKOFFandHAMS_ROS_MCPare baked at container-create — apply withcompose up --force-recreate, not a baredocker restart.Testing
robot_status/scan_status/wait_for/costmap_summaryover an MCP round-trip; reachable from the Mac through the tunnel.Known limitations
HAMS_LOWERBODY=walkpolicy just marches in place; FAME can occasionally topple.CL_Assetssubmodule is required at runtime; the MuJoCo:99viewer and the RoboCasa fall-logger aren't reachable from the ros-side MCP server (a second endpoint could add those).🤖 Generated with Claude Code