feat(Robotics): add WarehouseRobotRouting benchmark

[oushihsabiy]

Summary

• Add Robotics/WarehouseRobotRouting, a structured-solution benchmark for joint warehouse order assignment and collision-free multi-robot routing.
• Require complete discrete-time paths and pickup/delivery actions while independently verifying graph moves, vertex collisions, head-on edge swaps, action ordering, payload capacity, and completion of every order.
• Minimize verified total robot movement with the per-instance score log2((D_baseline + 1) / (D_candidate + 1)), averaged across the evaluation set.
• Include bilingual task documentation, deterministic instance generation, weak/baseline/reference solvers, provenance, an immutable Docker runtime, and the nine Frontier unified metadata files.

Distinction from existing Robotics tasks

DynamicObstacleAvoidanceNavigation evaluates one differential-drive robot in continuous 2D scenes with time-indexed controls, moving obstacles, and arrival-time scoring. This benchmark instead evaluates multiple robots on a discrete warehouse graph and couples path planning with pickup-and-delivery assignment, payload capacity, vertex/edge conflict avoidance, and aggregate fleet-distance optimization.

Evidence and scope

The public MAPD reference is Ma et al., Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks (AAMAS 2017): https://arxiv.org/abs/1705.10868.

The benchmark intentionally models deterministic unit-time graph movement. It does not claim to model continuous dynamics, acceleration, localization error, charging, communication latency, online order arrivals, or hardware failure; these limitations are documented in the benchmark README.

Validation

• Benchgen publish profile: 30 passed, 0 failed, 0 skipped.
• Deterministic calibration means:
  • random: -0.004339488391815513
  • baseline: 0.0
  • starter: 0.0
  • reference: 0.2729877727407376
• Pinned runtime image: python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7.
• Frontier unified zero-iteration run passed in both the detached staging worktree and the live branch:

python -m frontier_eval \
  task=unified \
  task.benchmark=Robotics/WarehouseRobotRouting \
  task.runtime.isolation_mode=process \
  algorithm=openevolve \
  algorithm.iterations=0

Result: benchmark_returncode=0, instances_valid=10/10, Best score: 0.0.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new benchmark task titled "Warehouse Robot Routing" under the Robotics domain. It is a Multi-Agent Path Finding (MAPF) problem with pickup-and-delivery (MAPD) constraints, requiring solvers to plan collision-free paths for multiple robots while respecting load capacities.
• Modified File Structure & Modifications:
  • benchmarks/Robotics/README.md & README_zh-CN.md: Updated the domain index to include the new task and added Chinese translations.
  • benchmarks/Robotics/WarehouseRobotRouting/README.md & README_zh-CN.md: Provided high-level task overviews, objective definitions, and execution instructions.
  • benchmarks/Robotics/WarehouseRobotRouting/Task.md & Task_zh-CN.md: Detailed technical specifications, including hard constraints (collision avoidance, capacity limits) and JSON schemas for input/output.
  • benchmarks/Robotics/WarehouseRobotRouting/benchmark.yaml: Configuration file defining the benchmark metadata, resource limits (Docker isolation, 240s timeout), and evaluation metrics.
  • benchmarks/Robotics/WarehouseRobotRouting/baseline/: Contains heuristic.py and weak.py providing baseline and random solver implementations.
  • benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/: Added integration files (agent_files.txt, constraints.txt, etc.) for the frontier_eval framework.
  • benchmarks/Robotics/WarehouseRobotRouting/data/seeds.json: Defined public and evaluation seeds for deterministic instance generation.

2. AI Content Analysis

• Estimated AI Component: 15%
• Reasoning & Evidence: The PR follows a highly structured template (Benchgen). While the JSON schemas and the boilerplate in benchmark.yaml might have been assisted by AI or generated via a tool, the domain-specific logic (MAPD constraints, specific scoring functions like log2(C_baseline / C)) and the integration with the frontier_eval framework show significant human engineering. The documentation is technically precise and lacks the "fluff" or over-commenting typical of pure AI generation.

3. Engineering & Economic Assessment

• Engineering Reality Check: This addresses a production-grade engineering problem. Multi-agent coordination in constrained environments is a core challenge in automated logistics (e.g., Amazon/Alibaba warehouses). The task handles realistic edge cases such as vertex conflicts, edge swaps, and dynamic payload changes. It explicitly acknowledges the "Reality Gap" (e.g., lack of continuous dynamics), which is a sign of professional engineering maturity.
• Economic Value: High. Optimizing warehouse robot routing directly impacts fulfillment throughput and reduces operational costs (energy, wear and tear). Providing a standardized benchmark for this problem helps in evaluating and improving algorithms that have direct industrial applications.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: WarehouseRobotRouting
  • Execution & Dependencies: The README.md and Task.md files clearly document the execution commands for both local (--local) and Docker-based evaluation. It specifies the exact Docker image hash for reproducibility.
• Documentation Quality: Excellent. The documentation is provided in both English and Chinese. It includes clear objective definitions, detailed constraint lists, and strict input/output schemas. No significant grammatical errors or redundancies were found.
• Organizational Structure: The structure is logical and modular. It separates the problem definition (Task.md), configuration (benchmark.yaml), baseline implementations (baseline/), and evaluation logic (frontier_eval/).

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific files (.vscode/) were detected.
• Absolute Paths: None detected. The Python scripts (e.g., in baseline/) use relative path resolution via Path(__file__).resolve().parents[1].

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 在 Robotics 领域下引入了一个名为 “仓储机器人路径规划 (Warehouse Robot Routing)” 的新基准任务。这是一个带有取送货约束的多智能体路径规划 (MAPD) 问题，要求求解器在遵守载重限制的同时，为多台机器人规划无碰撞路径。
• 修改的文件结构与变更摘要:
  • benchmarks/Robotics/README.md & README_zh-CN.md: 更新了领域索引以包含新任务，并添加了中文翻译。
  • benchmarks/Robotics/WarehouseRobotRouting/README.md & README_zh-CN.md: 提供了任务概览、目标定义和运行指南。
  • benchmarks/Robotics/WarehouseRobotRouting/Task.md & Task_zh-CN.md: 详细的术规范，包括硬约束（避障、载重限制）以及输入/输出的 JSON Schema。
  • benchmarks/Robotics/WarehouseRobotRouting/benchmark.yaml: 配置文件，定义了基准元数据、资源限制（Docker 隔离，240秒超时）和评估指标。
  • benchmarks/Robotics/WarehouseRobotRouting/baseline/: 包含 heuristic.py 和 weak.py，提供基线和随机求解器实现。
  • benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/: 添加了用于 frontier_eval 框架的集成文件（agent_files.txt、constraints.txt 等）。
  • benchmarks/Robotics/WarehouseRobotRouting/data/seeds.json: 定义了用于确定性实例生成的公开和评测种子。

2. AI 成分分析

• 预估 AI 含量: 15%
• 判断依据与证据: 该 PR 遵循高度结构化的模板 (Benchgen)。虽然 JSON Schema 和 benchmark.yaml 中的模板内容可能由 AI 辅助或工具生成，但特定领域的逻辑（MAPD 约束、特定的评分函数如 log2(C_baseline / C)）以及与 frontier_eval 框架的集成显示了显著的人工工程迹象。文档技术精确，没有纯 AI 生成常见的冗余或过度注释。

3. 工程与经济评估

• 工程现实检验: 该任务解决的是 生产级别的工程问题。受限环境下的多智能体协作是自动化物流（如亚马逊/阿里巴巴仓库）的核心挑战。任务处理了现实中的边缘情况，如节点冲突、边交换和动态载荷变化。它明确承认了“现实差距”（如缺乏连续动力学模型），这是工程成熟度的体现。
• 经济价值: 高。优化仓储机器人路径规划直接影响履约吞吐量并降低运营成本（能耗、设备磨损）。为此问题提供标准化的基准有助于评估和改进具有直接工业应用价值的算法。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: WarehouseRobotRouting
  • 运行与依赖: README.md 和 Task.md 文件清晰地记录了本地 (--local) 和基于 Docker 的评估运行命令。它指定了用于复现的精确 Docker 镜像哈希值。
• 文档质量: 优秀。文档提供了中英双语版本。包括清晰的目标定义、详细的约束列表和严格的输入/输出 Schema。未发现明显的语法错误或信息冗余。
• 组织结构: 文件组织逻辑清晰且模块化。将问题定义 (Task.md)、配置 (benchmark.yaml)、基线实现 (baseline/) 和评估逻辑 (frontier_eval/) 进行了有效分离。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未检测到 .env、API 密钥或 IDE 配置文件 (.vscode/)。
• 绝对路径: 未检测到。Python 脚本（如 baseline/ 中的脚本）通过 Path(__file__).resolve().parents[1] 使用相对路径解析。

[y-ji24]

PR #97 Review: feat(Robotics) - add WarehouseRobotRouting benchmark

Thank you for contributing WarehouseRobotRouting. The overall benchmark is well structured, with the task definition, baseline, reference solver, verification pipeline, and evaluator all in place. The bilingual documentation is also appreciated. Overall, the implementation covers the basic workflow of multi-robot warehouse routing, and the task constraints are clearly defined.

Before merging, I would recommend addressing the following issues related to benchmark consistency and evaluation reliability.

1. Data generation and evaluation are not fully consistent

Currently, generate_instances() in verification/problem.py generates four instances for each seed:

def generate_instances(seed):
    instances = []
    for index in range(4):
        instance = _make_instance(seed, index)
        instances.append(instance)
    return instances

However, the evaluator only uses the first two instances:

INSTANCES_PER_SEED = 2

for index, instance in enumerate(instances[:INSTANCES_PER_SEED]):

As a result, the last two generated instances never participate in evaluation. This creates an inconsistency between data generation and the actual evaluation protocol, and may also introduce unnecessary maintenance overhead.

It would be better to keep these two components consistent, for example by generating only the evaluated instances or by evaluating all generated instances.

2. Consider reducing the risk of overfitting to the evaluation set

The evaluation instances are generated from a fixed set of random seeds stored in data/seeds.json, which is available to the agent together with the benchmark.

Since the instance generation process is entirely deterministic given these seeds, an agent could reconstruct all evaluation instances in advance and optimize specifically for this fixed evaluation set rather than solving the underlying routing problem more generally.

It would be worth considering separating evaluation seeds from the publicly accessible benchmark resources. For example, the evaluation seeds could be removed from the agent-accessible files, or hidden seeds could be used during evaluation. This would improve the benchmark's ability to evaluate generalization rather than optimization for a fixed test set.

3. Consider documenting the evaluation design

The current benchmark adopts an offline batch-routing setting, uses total travel distance as the optimization objective, and reports scores through the Frontier log2_baseline_ratio metric.

The cited work by Ma et al. (AAMAS 2017), however, studies online MAPD, where the primary objective is throughput rather than offline distance minimization. Both settings are meaningful, but they correspond to different problem formulations and optimization goals.

It would be helpful to briefly explain the design choices in the README, for example:

• why the benchmark focuses on offline batch planning;
• why total travel distance is used as the primary optimization objective;
• what aspect of warehouse planning or coordination this benchmark is intended to evaluate.

Providing this context would make the benchmark design easier to understand and clarify its relationship to existing MAPD literature.

There are also a few non-blocking improvements that could be considered in future updates:

• scripts/init.py currently duplicates the baseline implementation and could instead reuse the existing implementation to reduce maintenance effort.
• The Input/Output schema is currently maintained in multiple places. Using a single source of truth would make future updates easier.
• The thread synchronization logic in process_runner.py could be further improved to make log collection more robust when processes terminate unexpectedly.

Overall, the benchmark is in good shape. Addressing the data generation inconsistency, improving evaluation set isolation, and documenting the evaluation design would further strengthen the benchmark's reliability and long-term maintainability.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new benchmark task titled "WarehouseRobotRouting" under the Robotics domain. It is a Multi-Agent Pickup and Delivery (MAPD) problem requiring solvers to coordinate multiple robots in a warehouse grid while satisfying collision, capacity, and time-horizon constraints.
• Modified File Structure & Modifications:
  • benchmarks/Robotics/README.md & README_zh-CN.md: Updated the task index to include the new WarehouseRobotRouting task.
  • benchmarks/Robotics/WarehouseRobotRouting/README.md & README_zh-CN.md: Provided high-level documentation, objective definitions (distance minimization), and evaluation design.
  • benchmarks/Robotics/WarehouseRobotRouting/Task.md & Task_zh-CN.md: Detailed technical specifications, including input/output JSON schemas and hard constraints (e.g., vertex/edge conflict rules).
  • benchmarks/Robotics/WarehouseRobotRouting/benchmark.yaml: Configuration file defining the benchmark metadata, runtime environment (Docker image), and resource limits.
  • benchmarks/Robotics/WarehouseRobotRouting/baseline/: Contains heuristic.py and weak.py providing baseline and random solver implementations.
  • benchmarks/Robotics/WarehouseRobotRouting/scripts/init.py: The entry point for candidate solutions.
  • benchmarks/Robotics/WarehouseRobotRouting/verification/: Contains evaluator.py and problem.py (logic for instance generation, reference solvers, and strict validation).

2. AI Content Analysis

• Estimated AI Component: 25%
• Reasoning & Evidence: The documentation structure (READMEs and Task descriptions) follows a highly standardized template seen in other Benchgen tasks, suggesting the use of AI for boilerplate generation. However, the domain-specific nuances—such as the specific mention of "opposite-direction edge swaps" (a classic MAPF constraint) and the citation of Ma et al. (2017)—indicate significant human expert oversight and technical tailoring. The JSON schemas are precise and standard, likely AI-assisted but manually verified.

3. Engineering & Economic Assessment

• Engineering Reality Check: This is a production-grade engineering problem. Unlike "toy" pathfinding examples, it incorporates:
  • Multi-agent coordination: Handling vertex and edge collisions.
  • Resource constraints: Robot load capacities and order weights.
  • Temporal constraints: A fixed horizon $H$.
  • Validation: A robust verifier that recomputes costs rather than trusting agent claims.
• Economic Value: High. Warehouse automation is a multi-billion dollar industry. Algorithms that reduce total move distance directly translate to lower energy consumption, reduced equipment wear, and higher operational efficiency.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: Robotics/WarehouseRobotRouting
  • Execution & Dependencies: The .md files clearly document the execution commands for both local and Docker environments. It specifies a pinned Docker image (python:3.12.11-slim-bookworm) with a specific SHA256 hash for reproducibility.
• Documentation Quality: Excellent. The PR provides comprehensive bilingual (English/Chinese) documentation. The inclusion of "Scope and Reality Gap" sections demonstrates high professional maturity by acknowledging what the model does not cover (e.g., continuous dynamics).
• Organizational Structure: The structure is logical and scalable, separating the solver interface (scripts/), the evaluation logic (verification/), and the baseline implementations.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations (.vscode/) were detected in the diff.
• Absolute Paths: None detected. The code uses Path(__file__).resolve() and relative imports, which is the correct practice for portable benchmarks.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 在 Robotics 领域引入了一个名为 “WarehouseRobotRouting”（仓储机器人路径规划） 的新基准任务。这是一个多智能体取送货（MAPD）问题，要求求解器在满足碰撞、载重和规划时域约束的情况下，协调仓库网格中的多台机器人。
• 修改的文件结构与变更摘要:
  • benchmarks/Robotics/README.md & README_zh-CN.md: 更新了任务索引，包含了新的仓储机器人路径规划任务。
  • benchmarks/Robotics/WarehouseRobotRouting/README.md & README_zh-CN.md: 提供了高层级文档、优化目标定义（移动距离最小化）和评测设计说明。
  • benchmarks/Robotics/WarehouseRobotRouting/Task.md & Task_zh-CN.md: 详细的技术规范，包括输入/输出 JSON Schema 和硬约束（如节点/边冲突规则）。
  • benchmarks/Robotics/WarehouseRobotRouting/benchmark.yaml: 配置文件，定义了基准元数据、运行时环境（Docker 镜像）和资源限制。
  • benchmarks/Robotics/WarehouseRobotRouting/baseline/: 包含 heuristic.py 和 weak.py，提供基线和随机求解器的实现。
  • benchmarks/Robotics/WarehouseRobotRouting/scripts/init.py: 候选方案的入口文件。
  • benchmarks/Robotics/WarehouseRobotRouting/verification/: 包含 evaluator.py 和 problem.py（用于实例生成、参考求解器和严格验证的逻辑）。

2. AI 成分分析

• 预估 AI 含量: 25%
• 判断依据与证据: 文档结构（README 和任务描述）遵循了 Benchgen 任务的高度标准化模板，表明使用了 AI 生成模板代码。然而，领域特定的细节——例如明确提到“反向边交换”（经典的 MAPF 约束）以及引用 Ma 等人 (2017) 的论文——表明了显著的人工专家监督和技术定制。JSON Schema 精确且标准，很可能是 AI 辅助生成后经过人工校验的。

3. 工程与经济评估

• 工程现实检验: 这是一个生产级别的工程问题。与“玩具级”路径规划示例不同，它结合了：
  • 多智能体协同: 处理节点和边碰撞。
  • 资源约束: 机器人载重上限和订单重量。
  • 时间约束: 固定规划时域 $H$。
  • 验证机制: 一个鲁棒的验证器，重新计算成本而非信任 Agent 声明。
• 经济价值: 高。仓库自动化是一个价值数十亿美元的行业。能够减少总移动距离的算法可以直接转化为更低的能耗、更少的设备磨损和更高的运营效率。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: Robotics/WarehouseRobotRouting
  • 运行与依赖: .md 文件清晰地记录了本地和 Docker 环境的运行命令。它指定了带有特定 SHA256 哈希的 Docker 镜像（python:3.12.11-slim-bookworm），以确保可复现性。
• 文档质量: 优秀。该 PR 提供了全面的中英双语文档。包含“适用范围与现实差距”章节体现了高度的专业成熟度，明确告知了模型未涵盖的内容（如连续动力学）。
• 组织结构: 文件组织逻辑清晰且具备可扩展性，将求解器接口 (scripts/)、评测逻辑 (verification/) 和基线实现进行了有效分离。

5. 安全与隐私检查

• 敏感文件: 未发现异常。Diff 中未检测到 .env、API 密钥或 IDE 特定配置（如 .vscode/）。
• 绝对路径: 未检测到。代码使用了 Path(__file__).resolve() 和相对导入，这是开发可移植基准任务的正确做法。

[oushihsabiy]

@y-ji24 Thank you for the detailed review. I addressed the three main points in commit 5c9e32b.

1. Instance generation/evaluation consistency

   • generate_instances(seed) now returns exactly two instances.
   • The evaluator no longer accepts extra instances and slices them. Both smoke and evaluation preparation now require len(instances) == instances_per_seed, so future drift fails validation instead of being silently ignored.
   • The same exact-count contract and a regression test were added to the generator template.

2. Evaluation-set isolation

   • data/seeds.json no longer publishes evaluation seed values. It contains only the public seeds, evaluation-set count, and instances_per_seed.
   • The README/Task now document the actual isolation boundary: benchmark.yaml, data/, verification/, baseline/, and reference/ are excluded from Frontier agent_files, while an untrusted candidate container mounts only the candidate source and receives the current instance via stdin.
   • Evaluation seeds remain in the frozen benchmark/verifier configuration for reproducibility. Because the repository is open source, this prevents routine runtime access rather than providing cryptographic secrecy against a person who inspects the repository before submission. Moving them to platform-private evaluator assets would require Frontier support, and the limitation is now stated explicitly.

3. Evaluation design documentation

   • Added English and Chinese sections explaining that this is deliberately an offline batch MAPD variant.
   • Documented that the task isolates assignment, sequencing, capacity management, and collision-free scheduling.
   • Documented why verified total movement is used as a deterministic proxy for energy, wear, and aisle occupancy, and explicitly distinguished it from the online throughput objective in Ma et al.

Regarding the non-blocking points:

• scripts/init.py remains self-contained intentionally. Frontier evolves a single candidate file, and the untrusted candidate container does not mount baseline/; importing the trusted baseline would break that isolation boundary.
• The schemas in the generated artifacts are derived from the benchmark specification by Benchgen and checked for consistency, rather than maintained as independent handwritten definitions.
• I left the process-runner synchronization improvement for a separate shared-runner change because it affects every generated benchmark rather than this task alone.

Revalidation after the changes:

• full generator test suite: passed;
• Ruff and strict mypy: passed;
• Benchgen publish profile: 30 passed, 0 failed, 0 skipped;
• Frontier live zero-run: benchmark_returncode=0, instances_valid=10/10, Best score: 0.0.

[y-ji24]

Thanks for the update. This revision is now substantially more complete. The issues raised in the previous review have been addressed: the evaluation design and isolation rationale are now clearly documented, the inconsistency between problem generation and the evaluator has been resolved, and seeds.json no longer exposes evaluation seed values.

I took another pass over the changes and did not identify any remaining blocking issues.

Thanks for the contribution!