Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions benchmarks/Robotics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,9 @@ This domain contains robotics control and planning tasks for unified evaluation.
- `QuadrupedGaitOptimization`: `.venvs/frontier-eval-driver/bin/python -m frontier_eval task=unified task.benchmark=Robotics/QuadrupedGaitOptimization task.runtime.env_name=frontier-v1-main algorithm.iterations=0`
- `RobotArmCycleTimeOptimization`: `.venvs/frontier-eval-driver/bin/python -m frontier_eval task=unified task.benchmark=Robotics/RobotArmCycleTimeOptimization task.runtime.env_name=frontier-v1-main algorithm.iterations=0`
- `UAVInspectionCoverageWithWind`: `python -m frontier_eval task=unified task.benchmark=Robotics/UAVInspectionCoverageWithWind algorithm.iterations=0`

## Benchgen Tasks

<!-- BENCHGEN-TASK-INDEX-START -->
- [WarehouseRobotRouting](WarehouseRobotRouting/README.md): 带碰撞与载重约束的多仓储机器人路径规划 - 给定离散仓库通行图、机器人初始位置、载重上限、规划时域,以及包含取货点、送货点和货物重量的订单,参赛系统输出每台机器人的逐时刻路径与取送货动作。实例覆盖不同仓库布局、机器人密度、订单规模和拥堵程度,并以全部订单完成后的机器人总移动距离衡量方案质量。
<!-- BENCHGEN-TASK-INDEX-END -->
9 changes: 9 additions & 0 deletions benchmarks/Robotics/README_zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Robotics

该领域的工程 benchmark。

## Benchgen Tasks

<!-- BENCHGEN-TASK-INDEX-START -->
- [WarehouseRobotRouting](WarehouseRobotRouting/README.md): 带碰撞与载重约束的多仓储机器人路径规划 - 给定离散仓库通行图、机器人初始位置、载重上限、规划时域,以及包含取货点、送货点和货物重量的订单,参赛系统输出每台机器人的逐时刻路径与取送货动作。实例覆盖不同仓库布局、机器人密度、订单规模和拥堵程度,并以全部订单完成后的机器人总移动距离衡量方案质量。
<!-- BENCHGEN-TASK-INDEX-END -->
80 changes: 80 additions & 0 deletions benchmarks/Robotics/WarehouseRobotRouting/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Warehouse Robot Routing

This benchmark asks a solver to assign pickup-and-delivery orders to warehouse robots and produce complete discrete-time paths without collisions or capacity violations.

## Objective

For a valid solution, the verifier independently counts every edge traversal by every robot:

- `D`: verified total move distance; waiting does not contribute.
- Raw metric: `C = D + 1`, minimized.
- Baseline-normalized score: `log2(C_baseline / C)`.
- Dataset score: arithmetic mean across instances.
- Invalid solutions receive `-1e18`.

A baseline-equivalent solution scores `0`; lower distance gives a positive score.

## Instances

`generate_instances(seed)` deterministically returns exactly two instances, matching the evaluator's `instances_per_seed` contract. Public seeds are `101`, `103`, `107`, `109`, and `113`.

Each instance contains an undirected warehouse graph, robot start nodes and capacities, weighted pickup-and-delivery orders, and a finite horizon `H`. Generated layouts use aisle-like grids with narrow cross-aisles and private robot parking nodes.

## Evaluation Design

This benchmark deliberately uses offline batch planning: all orders in an instance are known before `solve(instance)` runs. It evaluates the coupled combinatorial core of warehouse planning - order assignment, service sequencing, capacity management, and collision-free fleet scheduling - without also requiring an online arrival process or a long-running dispatch policy.

Total verified move distance is the primary objective because it is deterministic and independently recomputable from complete submitted paths. Within the fixed horizon and mandatory-delivery constraints, reducing movement is a useful proxy for fleet energy use, equipment wear, and aisle traffic. The benchmark does not claim that distance is a replacement for throughput in an online warehouse system.

Ma et al. provide the MAPD task structure and warehouse coordination motivation. Their lifelong online formulation optimizes throughput, whereas this benchmark isolates an offline distance-minimization formulation so candidate quality can be compared with a compact, reproducible verifier.

## Evaluation-Set Isolation

`data/seeds.json` publishes the public seeds and the evaluation-set size, but not the evaluation seed values. Evaluation seeds remain in the frozen benchmark/verifier configuration for reproducibility. Frontier's `agent_files.txt` excludes `benchmark.yaml`, `data/`, `verification/`, `baseline/`, and `reference/`; an untrusted candidate container mounts only the candidate program and receives one current instance through standard input. This prevents routine agent or candidate access to the held-out seed list during an evaluation run.

Because the benchmark implementation is open source, this is an execution-time isolation boundary rather than cryptographic secrecy against a person who inspects the repository before submitting hand-written code. Moving the seed material to platform-owned private evaluator assets would require corresponding Frontier support and can be done later without changing the candidate schema.

## Solver contract

The candidate entry point is `solve(instance)` in `scripts/init.py`. It returns one record for every robot. Each path must contain exactly `H + 1` node identifiers, representing times `0` through `H` inclusive. Actions refer to the robot position at their stated time.

The supplied candidate is a deterministic feasible baseline. It greedily inserts orders into per-robot service sequences, routes one order at a time, reserves non-overlapping robot execution windows, and pads all paths to the horizon.

`verification/problem.py` provides deterministic instance generation, random and baseline solvers, a stronger reference solver, strict validation, and evaluation. The reference uses subset dynamic programming to optimize assignment and macro-order sequencing, and safely avoids the baseline's unnecessary final return for the last active robot. Benchgen's smoke and calibration gates independently check baseline feasibility, determinism, and reference improvement.

## Verification

The verifier does not trust declared costs, loads, or feasibility. It checks the complete output structure, identifiers, starts, graph moves, vertex conflicts, opposite-direction edge swaps, action locations, action uniqueness and ordering, robot capacities, and completion of every order. It then recomputes distance from the submitted paths.

## Running Modes

Docker isolation is the publish and evaluation default. Run the normal Benchgen command without `--local` for publish-quality results. The immutable runtime image is declared in `benchmark.yaml`; networking is disabled.

Trusted local development may append `--local` to the same Benchgen command. Local mode executes candidate code directly on the host and must only be used with code you trust. Do not publish results produced only in local mode.

## Scope and Reality Gap

The benchmark models deterministic unit-time graph movement, vertex conflicts, head-on edge conflicts, and payload changes. It does not model continuous dynamics, acceleration, localization error, temporary obstacles, charging, communication latency, or hardware failure. Deployment requires motion control, safety margins, and online replanning beyond this benchmark.

Reducing valid route distance can lower energy use and equipment wear while supporting fulfillment efficiency. The benchmark cites Ma et al., *Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks* (AAMAS 2017, arXiv:1705.10868), as its public MAPD evidence source; the online-throughput and offline-distance formulations are intentionally distinguished above.

<!-- BENCHGEN-PUBLIC-CONTRACT-START -->
## Evaluation Contract

The verifier recomputes `verified_total_move_distance_plus_one` and candidates must minimize it.
Each valid case is scored by `log2` improvement over the baseline and the final score is the
mean across cases. Invalid solutions receive `-1e18`.

Local execution is only for reviewed code:

```bash
python verification/evaluator.py scripts/init.py --local
```

Publish evaluation requires Docker and the pinned runtime image:

```bash
docker pull python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7
python verification/evaluator.py scripts/init.py
```
<!-- BENCHGEN-PUBLIC-CONTRACT-END -->
80 changes: 80 additions & 0 deletions benchmarks/Robotics/WarehouseRobotRouting/README_zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# 仓储机器人路径规划

本基准要求求解器为仓储机器人分配取送货订单,并输出完整的离散时间路径,同时满足碰撞与载重约束。

## 优化目标

对于合法方案,验证器独立统计所有机器人的通行边移动次数:

- `D`:验证后的总移动距离;原地等待不计入距离。
- 原始指标:`C = D + 1`,越小越好。
- 相对基线分数:`log2(C_baseline / C)`。
- 数据集分数:各实例分数的算术平均值。
- 非法方案得分为 `-1e18`。

与基线距离相同的方案得分为 `0`;距离更短时得分为正。

## 实例

`generate_instances(seed)` 确定性地返回恰好两个实例,与评测器的 `instances_per_seed` 契约一致。公开种子为 `101`、`103`、`107`、`109` 和 `113`。

每个实例包含无向仓库图、机器人的初始节点和载重上限、带重量的取送货订单,以及有限规划时域 `H`。生成布局采用带狭窄横向通道的巷道网格,并为每台机器人设置独立停车节点。

## 评测设计

本 benchmark 有意采用离线批量规划:在调用 `solve(instance)` 前,实例中的全部订单均已给出。它集中评测仓储规划中的组合优化核心,即订单分配、服务顺序、载重管理和无碰撞的多机器人调度,而不同时引入在线订单到达过程或长期调度策略。

总移动距离是主要目标,因为验证器可以根据完整路径确定性地独立重算该指标。在固定规划时域和强制完成全部订单的约束下,减少移动是车队能耗、设备磨损和通道占用的实用代理指标。本 benchmark 并不主张离线距离可以替代在线仓储系统中的吞吐量目标。

Ma 等人的工作为本 benchmark 提供 MAPD 任务结构和仓储协同动机;其 lifelong online 设定优化吞吐量,而本 benchmark 有意隔离离线距离最小化问题,以便使用紧凑且可复现的验证器比较候选方案质量。

## 评测集隔离

`data/seeds.json` 只公开公共种子和评测集规模,不公开评测种子的具体数值。为保证复现性,评测种子仍保存在冻结的 benchmark/验证器配置中。Frontier 的 `agent_files.txt` 不包含 `benchmark.yaml`、`data/`、`verification/`、`baseline/` 和 `reference/`;不可信候选容器只挂载候选程序,并通过标准输入接收当前单个实例。这可以防止 agent 或候选程序在评测运行期间按常规方式读取保留种子列表。

由于 benchmark 实现是开源的,这一机制属于运行时隔离边界,而不是针对提交前人工检查仓库并编写硬编码代码的密码学保密。未来若 Frontier 支持平台私有的验证器资产,可以在不改变候选接口的前提下进一步迁移种子材料。

## 求解器接口

候选入口为 `scripts/init.py` 中的 `solve(instance)`。返回结果必须恰好包含每台机器人一条记录。每条路径必须包含 `H + 1` 个节点标识,依次对应时刻 `0` 到 `H`。动作发生位置由该机器人在动作时刻的路径节点确定。

随附候选实现是确定且可行的基线。它将订单贪心插入各机器人的服务序列,每次只运输一个订单,以互不重叠的执行窗口规避碰撞,并将所有路径补齐到规划时域。

`verification/problem.py` 提供确定性的实例生成器、随机求解器、基线求解器、更强的参考求解器、严格验证和评估逻辑。参考求解器使用子集动态规划优化订单分配与宏观服务顺序,并安全地省略最后一台活动机器人不必要的返航。Benchgen 的 smoke 与 calibration 门禁会独立检查基线可行性、确定性和参考解提升。

## 验证

验证器不信任提交方声明的成本、载荷或可行性。它会检查完整输出结构、标识符、起点、图上移动、节点冲突、反向边交换、动作位置、动作唯一性与先后关系、机器人载重,以及所有订单是否完成,然后根据路径重新计算距离。

## 运行模式

Docker 隔离是发布和正式评测的默认模式。发布结果时,应使用不带 `--local` 的常规 Benchgen 命令。不可变运行镜像在 `benchmark.yaml` 中声明,并禁用网络。

仅在可信的本地开发中,才可向同一 Benchgen 命令追加 `--local`。本地模式会直接在宿主机执行候选代码,因此只能运行可信代码。只在本地模式中得到的结果不应作为发布结果。

## 适用范围与现实差距

本基准刻画确定性的单位时间图移动、节点冲突、迎面边冲突和载荷变化,但不模拟连续动力学、加减速、定位误差、临时障碍、充电、通信延迟或硬件故障。真实部署仍需要运动控制、安全裕量和在线重规划。

在保证安全与载重合规的前提下降低路径距离,可以减少能耗和设备磨损,并支持履约效率。本 benchmark 引用 Ma 等人的 *Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks*(AAMAS 2017,arXiv:1705.10868)作为公开 MAPD 证据来源;上文已明确区分其在线吞吐量设定与本任务的离线距离设定。

<!-- BENCHGEN-PUBLIC-CONTRACT-START -->
## 评测契约

验证器会独立重算 `verified_total_move_distance_plus_one`,候选方案需要将其最小化。
每个有效实例按照相对基线的 `log2` 改进计分,最终取所有实例分数的平均;
无效方案得分为 `-1e18`。

本地执行只适用于已经审核的代码:

```bash
python verification/evaluator.py scripts/init.py --local
```

正式发布评测需要 Docker 和固定摘要的运行镜像:

```bash
docker pull python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7
python verification/evaluator.py scripts/init.py
```
<!-- BENCHGEN-PUBLIC-CONTRACT-END -->
Loading
Loading