diff --git a/benchmarks/Robotics/README.md b/benchmarks/Robotics/README.md index b3e834a5..b4cf8283 100644 --- a/benchmarks/Robotics/README.md +++ b/benchmarks/Robotics/README.md @@ -20,3 +20,9 @@ This domain contains robotics control and planning tasks for unified evaluation. - `QuadrupedGaitOptimization`: `.venvs/frontier-eval-driver/bin/python -m frontier_eval task=unified task.benchmark=Robotics/QuadrupedGaitOptimization task.runtime.env_name=frontier-v1-main algorithm.iterations=0` - `RobotArmCycleTimeOptimization`: `.venvs/frontier-eval-driver/bin/python -m frontier_eval task=unified task.benchmark=Robotics/RobotArmCycleTimeOptimization task.runtime.env_name=frontier-v1-main algorithm.iterations=0` - `UAVInspectionCoverageWithWind`: `python -m frontier_eval task=unified task.benchmark=Robotics/UAVInspectionCoverageWithWind algorithm.iterations=0` + +## Benchgen Tasks + + +- [WarehouseRobotRouting](WarehouseRobotRouting/README.md): 带碰撞与载重约束的多仓储机器人路径规划 - 给定离散仓库通行图、机器人初始位置、载重上限、规划时域,以及包含取货点、送货点和货物重量的订单,参赛系统输出每台机器人的逐时刻路径与取送货动作。实例覆盖不同仓库布局、机器人密度、订单规模和拥堵程度,并以全部订单完成后的机器人总移动距离衡量方案质量。 + diff --git a/benchmarks/Robotics/README_zh-CN.md b/benchmarks/Robotics/README_zh-CN.md new file mode 100644 index 00000000..05b9c3e8 --- /dev/null +++ b/benchmarks/Robotics/README_zh-CN.md @@ -0,0 +1,9 @@ +# Robotics + +该领域的工程 benchmark。 + +## Benchgen Tasks + + +- [WarehouseRobotRouting](WarehouseRobotRouting/README.md): 带碰撞与载重约束的多仓储机器人路径规划 - 给定离散仓库通行图、机器人初始位置、载重上限、规划时域,以及包含取货点、送货点和货物重量的订单,参赛系统输出每台机器人的逐时刻路径与取送货动作。实例覆盖不同仓库布局、机器人密度、订单规模和拥堵程度,并以全部订单完成后的机器人总移动距离衡量方案质量。 + diff --git a/benchmarks/Robotics/WarehouseRobotRouting/README.md b/benchmarks/Robotics/WarehouseRobotRouting/README.md new file mode 100644 index 00000000..8f177022 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/README.md @@ -0,0 +1,80 @@ +# Warehouse Robot Routing + +This benchmark asks a solver to assign pickup-and-delivery orders to warehouse robots and produce complete discrete-time paths without collisions or capacity violations. + +## Objective + +For a valid solution, the verifier independently counts every edge traversal by every robot: + +- `D`: verified total move distance; waiting does not contribute. +- Raw metric: `C = D + 1`, minimized. +- Baseline-normalized score: `log2(C_baseline / C)`. +- Dataset score: arithmetic mean across instances. +- Invalid solutions receive `-1e18`. + +A baseline-equivalent solution scores `0`; lower distance gives a positive score. + +## Instances + +`generate_instances(seed)` deterministically returns exactly two instances, matching the evaluator's `instances_per_seed` contract. Public seeds are `101`, `103`, `107`, `109`, and `113`. + +Each instance contains an undirected warehouse graph, robot start nodes and capacities, weighted pickup-and-delivery orders, and a finite horizon `H`. Generated layouts use aisle-like grids with narrow cross-aisles and private robot parking nodes. + +## Evaluation Design + +This benchmark deliberately uses offline batch planning: all orders in an instance are known before `solve(instance)` runs. It evaluates the coupled combinatorial core of warehouse planning - order assignment, service sequencing, capacity management, and collision-free fleet scheduling - without also requiring an online arrival process or a long-running dispatch policy. + +Total verified move distance is the primary objective because it is deterministic and independently recomputable from complete submitted paths. Within the fixed horizon and mandatory-delivery constraints, reducing movement is a useful proxy for fleet energy use, equipment wear, and aisle traffic. The benchmark does not claim that distance is a replacement for throughput in an online warehouse system. + +Ma et al. provide the MAPD task structure and warehouse coordination motivation. Their lifelong online formulation optimizes throughput, whereas this benchmark isolates an offline distance-minimization formulation so candidate quality can be compared with a compact, reproducible verifier. + +## Evaluation-Set Isolation + +`data/seeds.json` publishes the public seeds and the evaluation-set size, but not the evaluation seed values. Evaluation seeds remain in the frozen benchmark/verifier configuration for reproducibility. Frontier's `agent_files.txt` excludes `benchmark.yaml`, `data/`, `verification/`, `baseline/`, and `reference/`; an untrusted candidate container mounts only the candidate program and receives one current instance through standard input. This prevents routine agent or candidate access to the held-out seed list during an evaluation run. + +Because the benchmark implementation is open source, this is an execution-time isolation boundary rather than cryptographic secrecy against a person who inspects the repository before submitting hand-written code. Moving the seed material to platform-owned private evaluator assets would require corresponding Frontier support and can be done later without changing the candidate schema. + +## Solver contract + +The candidate entry point is `solve(instance)` in `scripts/init.py`. It returns one record for every robot. Each path must contain exactly `H + 1` node identifiers, representing times `0` through `H` inclusive. Actions refer to the robot position at their stated time. + +The supplied candidate is a deterministic feasible baseline. It greedily inserts orders into per-robot service sequences, routes one order at a time, reserves non-overlapping robot execution windows, and pads all paths to the horizon. + +`verification/problem.py` provides deterministic instance generation, random and baseline solvers, a stronger reference solver, strict validation, and evaluation. The reference uses subset dynamic programming to optimize assignment and macro-order sequencing, and safely avoids the baseline's unnecessary final return for the last active robot. Benchgen's smoke and calibration gates independently check baseline feasibility, determinism, and reference improvement. + +## Verification + +The verifier does not trust declared costs, loads, or feasibility. It checks the complete output structure, identifiers, starts, graph moves, vertex conflicts, opposite-direction edge swaps, action locations, action uniqueness and ordering, robot capacities, and completion of every order. It then recomputes distance from the submitted paths. + +## Running Modes + +Docker isolation is the publish and evaluation default. Run the normal Benchgen command without `--local` for publish-quality results. The immutable runtime image is declared in `benchmark.yaml`; networking is disabled. + +Trusted local development may append `--local` to the same Benchgen command. Local mode executes candidate code directly on the host and must only be used with code you trust. Do not publish results produced only in local mode. + +## Scope and Reality Gap + +The benchmark models deterministic unit-time graph movement, vertex conflicts, head-on edge conflicts, and payload changes. It does not model continuous dynamics, acceleration, localization error, temporary obstacles, charging, communication latency, or hardware failure. Deployment requires motion control, safety margins, and online replanning beyond this benchmark. + +Reducing valid route distance can lower energy use and equipment wear while supporting fulfillment efficiency. The benchmark cites Ma et al., *Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks* (AAMAS 2017, arXiv:1705.10868), as its public MAPD evidence source; the online-throughput and offline-distance formulations are intentionally distinguished above. + + +## Evaluation Contract + +The verifier recomputes `verified_total_move_distance_plus_one` and candidates must minimize it. +Each valid case is scored by `log2` improvement over the baseline and the final score is the +mean across cases. Invalid solutions receive `-1e18`. + +Local execution is only for reviewed code: + +```bash +python verification/evaluator.py scripts/init.py --local +``` + +Publish evaluation requires Docker and the pinned runtime image: + +```bash +docker pull python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7 +python verification/evaluator.py scripts/init.py +``` + diff --git a/benchmarks/Robotics/WarehouseRobotRouting/README_zh-CN.md b/benchmarks/Robotics/WarehouseRobotRouting/README_zh-CN.md new file mode 100644 index 00000000..1e9eb657 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/README_zh-CN.md @@ -0,0 +1,80 @@ +# 仓储机器人路径规划 + +本基准要求求解器为仓储机器人分配取送货订单,并输出完整的离散时间路径,同时满足碰撞与载重约束。 + +## 优化目标 + +对于合法方案,验证器独立统计所有机器人的通行边移动次数: + +- `D`:验证后的总移动距离;原地等待不计入距离。 +- 原始指标:`C = D + 1`,越小越好。 +- 相对基线分数:`log2(C_baseline / C)`。 +- 数据集分数:各实例分数的算术平均值。 +- 非法方案得分为 `-1e18`。 + +与基线距离相同的方案得分为 `0`;距离更短时得分为正。 + +## 实例 + +`generate_instances(seed)` 确定性地返回恰好两个实例,与评测器的 `instances_per_seed` 契约一致。公开种子为 `101`、`103`、`107`、`109` 和 `113`。 + +每个实例包含无向仓库图、机器人的初始节点和载重上限、带重量的取送货订单,以及有限规划时域 `H`。生成布局采用带狭窄横向通道的巷道网格,并为每台机器人设置独立停车节点。 + +## 评测设计 + +本 benchmark 有意采用离线批量规划:在调用 `solve(instance)` 前,实例中的全部订单均已给出。它集中评测仓储规划中的组合优化核心,即订单分配、服务顺序、载重管理和无碰撞的多机器人调度,而不同时引入在线订单到达过程或长期调度策略。 + +总移动距离是主要目标,因为验证器可以根据完整路径确定性地独立重算该指标。在固定规划时域和强制完成全部订单的约束下,减少移动是车队能耗、设备磨损和通道占用的实用代理指标。本 benchmark 并不主张离线距离可以替代在线仓储系统中的吞吐量目标。 + +Ma 等人的工作为本 benchmark 提供 MAPD 任务结构和仓储协同动机;其 lifelong online 设定优化吞吐量,而本 benchmark 有意隔离离线距离最小化问题,以便使用紧凑且可复现的验证器比较候选方案质量。 + +## 评测集隔离 + +`data/seeds.json` 只公开公共种子和评测集规模,不公开评测种子的具体数值。为保证复现性,评测种子仍保存在冻结的 benchmark/验证器配置中。Frontier 的 `agent_files.txt` 不包含 `benchmark.yaml`、`data/`、`verification/`、`baseline/` 和 `reference/`;不可信候选容器只挂载候选程序,并通过标准输入接收当前单个实例。这可以防止 agent 或候选程序在评测运行期间按常规方式读取保留种子列表。 + +由于 benchmark 实现是开源的,这一机制属于运行时隔离边界,而不是针对提交前人工检查仓库并编写硬编码代码的密码学保密。未来若 Frontier 支持平台私有的验证器资产,可以在不改变候选接口的前提下进一步迁移种子材料。 + +## 求解器接口 + +候选入口为 `scripts/init.py` 中的 `solve(instance)`。返回结果必须恰好包含每台机器人一条记录。每条路径必须包含 `H + 1` 个节点标识,依次对应时刻 `0` 到 `H`。动作发生位置由该机器人在动作时刻的路径节点确定。 + +随附候选实现是确定且可行的基线。它将订单贪心插入各机器人的服务序列,每次只运输一个订单,以互不重叠的执行窗口规避碰撞,并将所有路径补齐到规划时域。 + +`verification/problem.py` 提供确定性的实例生成器、随机求解器、基线求解器、更强的参考求解器、严格验证和评估逻辑。参考求解器使用子集动态规划优化订单分配与宏观服务顺序,并安全地省略最后一台活动机器人不必要的返航。Benchgen 的 smoke 与 calibration 门禁会独立检查基线可行性、确定性和参考解提升。 + +## 验证 + +验证器不信任提交方声明的成本、载荷或可行性。它会检查完整输出结构、标识符、起点、图上移动、节点冲突、反向边交换、动作位置、动作唯一性与先后关系、机器人载重,以及所有订单是否完成,然后根据路径重新计算距离。 + +## 运行模式 + +Docker 隔离是发布和正式评测的默认模式。发布结果时,应使用不带 `--local` 的常规 Benchgen 命令。不可变运行镜像在 `benchmark.yaml` 中声明,并禁用网络。 + +仅在可信的本地开发中,才可向同一 Benchgen 命令追加 `--local`。本地模式会直接在宿主机执行候选代码,因此只能运行可信代码。只在本地模式中得到的结果不应作为发布结果。 + +## 适用范围与现实差距 + +本基准刻画确定性的单位时间图移动、节点冲突、迎面边冲突和载荷变化,但不模拟连续动力学、加减速、定位误差、临时障碍、充电、通信延迟或硬件故障。真实部署仍需要运动控制、安全裕量和在线重规划。 + +在保证安全与载重合规的前提下降低路径距离,可以减少能耗和设备磨损,并支持履约效率。本 benchmark 引用 Ma 等人的 *Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks*(AAMAS 2017,arXiv:1705.10868)作为公开 MAPD 证据来源;上文已明确区分其在线吞吐量设定与本任务的离线距离设定。 + + +## 评测契约 + +验证器会独立重算 `verified_total_move_distance_plus_one`,候选方案需要将其最小化。 +每个有效实例按照相对基线的 `log2` 改进计分,最终取所有实例分数的平均; +无效方案得分为 `-1e18`。 + +本地执行只适用于已经审核的代码: + +```bash +python verification/evaluator.py scripts/init.py --local +``` + +正式发布评测需要 Docker 和固定摘要的运行镜像: + +```bash +docker pull python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7 +python verification/evaluator.py scripts/init.py +``` + diff --git a/benchmarks/Robotics/WarehouseRobotRouting/Task.md b/benchmarks/Robotics/WarehouseRobotRouting/Task.md new file mode 100644 index 00000000..2a6e77df --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/Task.md @@ -0,0 +1,271 @@ +# Task: Warehouse Robot Routing + +Implement `solve(instance)` in `scripts/init.py`. + +## Input + +The input object contains: + +- `instance_id`: a non-empty identifier. +- `graph.nodes`: all valid node identifiers. +- `graph.edges`: undirected traversable edges, each represented by two node identifiers. +- `robots`: objects with `id`, `start_node`, and positive integer `capacity`. +- `orders`: objects with `id`, `pickup_node`, `dropoff_node`, and positive integer `weight`. +- `horizon`: the final discrete time `H`. + +## Output + +Return an object with exactly one top-level field, `robots`. Its array must contain exactly one entry for every input robot and no unknown or duplicate robot. + +Each robot entry contains: + +- `robot_id`: the input robot identifier. +- `path`: exactly `H + 1` nodes. `path[t]` is the occupied node at time `t`. +- `actions`: zero or more objects with integer `time`, `type` equal to `pickup` or `dropoff`, and an input `order_id`. + +Do not include claimed loads, costs, scores, or feasibility flags; the verifier ignores such claims and the output schema rejects extra fields. + +## Hard Constraints + +1. Every path starts at its robot's specified start node. +2. Between adjacent times, a robot either waits or traverses one input edge. +3. No two robots occupy the same node at the same time. +4. Two robots may not swap endpoints of one edge during the same time step. +5. Every order is picked up exactly once and dropped off exactly once. +6. Pickup occurs at the order's pickup node. Dropoff occurs at its dropoff node. +7. Dropoff is strictly later than pickup and is performed by the same robot. +8. A robot may perform at most one action at any time. +9. A robot cannot drop an order it is not carrying. +10. Loads start at zero. Pickup adds the order weight and dropoff subtracts it. Load must remain between zero and robot capacity. +11. Every order must be delivered no later than time `H`. +12. Every node, robot identifier, and order identifier in the output must be known to the instance. + +Actions by different robots may occur simultaneously. Waiting is free in the objective but the finite horizon prevents unbounded schedules. + +## Objective and Score + +The verifier recomputes + +`D = sum over robots and t=1..H of [path[t] != path[t-1]]`. + +The minimized raw metric is `C = D + 1`. With baseline raw metric `C_baseline`, the normalized instance score is `log2(C_baseline / C)`. Scores are averaged across instances. Invalid solutions receive `-1e18`. + +Distance is the only primary objective. Completion time may be reported as a diagnostic but does not change the score. + +## Evaluation Design and Isolation + +The task is an offline batch MAPD variant: every order is known before planning. It isolates joint assignment, sequencing, capacity, and collision-free scheduling rather than modeling online arrivals. Total movement is used because complete paths make it deterministic and independently verifiable, and because movement is a useful proxy for fleet energy, wear, and aisle occupancy under mandatory delivery. The cited Ma et al. work motivates MAPD but studies a different lifelong online throughput objective. + +The public seed manifest contains public seeds and the evaluation-set size only. Frontier does not include `benchmark.yaml`, `data/`, `verification/`, `baseline/`, or `reference/` in the agent context, and the candidate container receives only its source file plus the current instance. This is runtime isolation, not cryptographic hiding of an open-source repository. + +## Implementation Rules + +`scripts/init.py` may contain only the `solve(instance)` function, with imports placed inside that function. Benchgen adds the fixed command-line wrapper. Candidate code must be deterministic for deterministic inputs, must not access the network, secrets, evaluator outputs, or absolute paths, and must finish within the configured resource limits. + +Docker is the default mode for publishing and formal evaluation. The `--local` option is only for trusted local development. + + +## Input Schema + +```json +{ + "type": "object", + "additionalProperties": false, + "required": [ + "instance_id", + "graph", + "robots", + "orders", + "horizon" + ], + "properties": { + "instance_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "graph": { + "type": "object", + "additionalProperties": false, + "required": [ + "nodes", + "edges" + ], + "properties": { + "nodes": { + "type": "array", + "minItems": 2, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + }, + "edges": { + "type": "array", + "items": { + "type": "array", + "minItems": 2, + "maxItems": 2, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + } + } + } + }, + "robots": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "start_node", + "capacity" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "start_node": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "capacity": { + "type": "integer", + "minimum": 1 + } + } + } + }, + "orders": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "pickup_node", + "dropoff_node", + "weight" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "pickup_node": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "dropoff_node": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "weight": { + "type": "integer", + "minimum": 1 + } + } + } + }, + "horizon": { + "type": "integer", + "minimum": 1 + } + } +} +``` + +## Output Schema + +```json +{ + "type": "object", + "additionalProperties": false, + "required": [ + "robots" + ], + "properties": { + "robots": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "robot_id", + "path", + "actions" + ], + "properties": { + "robot_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "path": { + "type": "array", + "minItems": 2, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + }, + "actions": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "time", + "type", + "order_id" + ], + "properties": { + "time": { + "type": "integer", + "minimum": 0 + }, + "type": { + "type": "string", + "enum": [ + "pickup", + "dropoff" + ] + }, + "order_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + } + } + } + } + } + } + } +} +``` + +## Constraints and Objective + +The output must satisfy every hard constraint described above. The frozen verifier independently +checks feasibility and recomputes `verified_total_move_distance_plus_one`. The objective is to minimize +that strictly positive raw metric. Valid cases use `log2` baseline improvement and are aggregated +with the mean; invalid solutions receive `-1e18`. + diff --git a/benchmarks/Robotics/WarehouseRobotRouting/Task_zh-CN.md b/benchmarks/Robotics/WarehouseRobotRouting/Task_zh-CN.md new file mode 100644 index 00000000..d32fe8b8 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/Task_zh-CN.md @@ -0,0 +1,270 @@ +# 任务:仓储机器人路径规划 + +请在 `scripts/init.py` 中实现 `solve(instance)`。 + +## 输入 + +输入对象包含: + +- `instance_id`:非空实例标识。 +- `graph.nodes`:全部合法节点标识。 +- `graph.edges`:无向可通行边,每条边由两个节点标识表示。 +- `robots`:机器人对象,字段为 `id`、`start_node` 和正整数 `capacity`。 +- `orders`:订单对象,字段为 `id`、`pickup_node`、`dropoff_node` 和正整数 `weight`。 +- `horizon`:离散规划的最终时刻 `H`。 + +## 输出 + +返回对象只能包含顶层字段 `robots`。该数组必须恰好包含每台输入机器人一条记录,不得缺失、重复或包含未知机器人。 + +每条机器人记录包含: + +- `robot_id`:输入中的机器人标识。 +- `path`:恰好 `H + 1` 个节点;`path[t]` 表示时刻 `t` 占据的节点。 +- `actions`:零个或多个动作对象,包含整数 `time`、取值为 `pickup` 或 `dropoff` 的 `type`,以及输入中的 `order_id`。 + +不要输出自行声明的载荷、成本、分数或可行性字段;验证器不信任这些声明,输出模式也不允许额外字段。 + +## 硬约束 + +1. 每条路径必须从该机器人的指定起点开始。 +2. 相邻时刻之间,机器人只能原地等待或沿输入图中的一条边移动。 +3. 任意时刻,两台机器人不得占据同一节点。 +4. 同一时间步内,两台机器人不得交换同一条边的两个端点。 +5. 每个订单必须且只能取货一次、送货一次。 +6. 取货必须发生在订单的取货节点,送货必须发生在送货节点。 +7. 送货必须严格晚于取货,并由同一台机器人完成。 +8. 同一机器人在同一时刻至多执行一个动作。 +9. 机器人不得送出尚未携带的订单。 +10. 初始载荷为零。取货增加订单重量,送货减少订单重量;载荷必须始终位于零和机器人载重上限之间。 +11. 所有订单必须在时刻 `H` 之前或恰在 `H` 完成。 +12. 输出中的节点、机器人标识和订单标识都必须来自输入实例。 + +不同机器人可以同时执行动作。原地等待不计入目标,但有限时域禁止无限等待。 + +## 目标与分数 + +验证器重新计算: + +`D = 所有机器人在 t=1..H 范围内满足 path[t] != path[t-1] 的次数之和`。 + +需要最小化的原始指标为 `C = D + 1`。设基线原始指标为 `C_baseline`,单实例归一化分数为 `log2(C_baseline / C)`。数据集取各实例分数的算术平均值。非法方案得分为 `-1e18`。 + +总移动距离是唯一主目标。完工时间可以作为诊断指标,但不影响分数。 + +## 评测设计与隔离 + +本任务是离线批量 MAPD 变体:规划开始前全部订单均已知。它隔离评测联合分配、服务顺序、载重和无碰撞调度,而不模拟在线订单到达。选择总移动距离,是因为验证器可以根据完整路径确定性地独立重算该指标,并且在强制完成订单的条件下,移动量可作为车队能耗、磨损和通道占用的实用代理。引用的 Ma 等人工作提供 MAPD 动机,但研究的是不同的 lifelong online 吞吐量目标。 + +公开种子清单只包含公共种子和评测集规模。Frontier 不会把 `benchmark.yaml`、`data/`、`verification/`、`baseline/` 或 `reference/` 放入 agent 上下文;候选容器只接收自身源码和当前实例。这是运行时隔离,而不是对开源仓库的密码学隐藏。 + +## 实现规则 + +`scripts/init.py` 只能包含 `solve(instance)` 函数,所需导入必须写在函数内部。固定命令行封装由 Benchgen 添加。候选代码对确定输入必须保持确定性,不得访问网络、秘密信息、评估器输出或绝对路径,并须在配置的资源限制内完成。 + +Docker 是发布和正式评测的默认模式。`--local` 仅用于可信的本地开发。 + + +## 输入 Schema + +```json +{ + "type": "object", + "additionalProperties": false, + "required": [ + "instance_id", + "graph", + "robots", + "orders", + "horizon" + ], + "properties": { + "instance_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "graph": { + "type": "object", + "additionalProperties": false, + "required": [ + "nodes", + "edges" + ], + "properties": { + "nodes": { + "type": "array", + "minItems": 2, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + }, + "edges": { + "type": "array", + "items": { + "type": "array", + "minItems": 2, + "maxItems": 2, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + } + } + } + }, + "robots": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "start_node", + "capacity" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "start_node": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "capacity": { + "type": "integer", + "minimum": 1 + } + } + } + }, + "orders": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "pickup_node", + "dropoff_node", + "weight" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "pickup_node": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "dropoff_node": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "weight": { + "type": "integer", + "minimum": 1 + } + } + } + }, + "horizon": { + "type": "integer", + "minimum": 1 + } + } +} +``` + +## 输出 Schema + +```json +{ + "type": "object", + "additionalProperties": false, + "required": [ + "robots" + ], + "properties": { + "robots": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "robot_id", + "path", + "actions" + ], + "properties": { + "robot_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "path": { + "type": "array", + "minItems": 2, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + }, + "actions": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "time", + "type", + "order_id" + ], + "properties": { + "time": { + "type": "integer", + "minimum": 0 + }, + "type": { + "type": "string", + "enum": [ + "pickup", + "dropoff" + ] + }, + "order_id": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + } + } + } + } + } + } + } +} +``` + +## 约束与目标 + +输出必须满足上文全部硬约束。冻结验证器会独立检查可行性并重算 +`verified_total_move_distance_plus_one`;优化目标是将这个严格为正的原始指标最小化。 +有效实例使用相对基线的 `log2` 改进值,并取平均;无效方案得分为 `-1e18`。 + diff --git a/benchmarks/Robotics/WarehouseRobotRouting/baseline/heuristic.py b/benchmarks/Robotics/WarehouseRobotRouting/baseline/heuristic.py new file mode 100644 index 00000000..d6641f55 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/baseline/heuristic.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from verification.problem import solve_baseline # noqa: E402 + + +def solve(instance: dict[str, Any]) -> dict[str, Any]: + return solve_baseline(instance) + + +if __name__ == "__main__": + json.dump(solve(json.load(sys.stdin)), sys.stdout, allow_nan=False) + sys.stdout.write("\n") diff --git a/benchmarks/Robotics/WarehouseRobotRouting/baseline/weak.py b/benchmarks/Robotics/WarehouseRobotRouting/baseline/weak.py new file mode 100644 index 00000000..87d24d4d --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/baseline/weak.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from verification.problem import solve_random # noqa: E402 + + +def solve(instance: dict[str, Any]) -> dict[str, Any]: + return solve_random(instance) + + +if __name__ == "__main__": + json.dump(solve(json.load(sys.stdin)), sys.stdout, allow_nan=False) + sys.stdout.write("\n") diff --git a/benchmarks/Robotics/WarehouseRobotRouting/benchmark.yaml b/benchmarks/Robotics/WarehouseRobotRouting/benchmark.yaml new file mode 100644 index 00000000..b330903a --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/benchmark.yaml @@ -0,0 +1,211 @@ +api_version: benchgen/v1 +benchmark_id: Robotics/WarehouseRobotRouting +title: 带碰撞与载重约束的多仓储机器人路径规划 +summary: 给定离散仓库通行图、机器人初始位置、载重上限、规划时域,以及包含取货点、送货点和货物重量的订单,参赛系统输出每台机器人的逐时刻路径与取送货动作。实例覆盖不同仓库布局、机器人密度、订单规模和拥堵程度,并以全部订单完成后的机器人总移动距离衡量方案质量。 +archetype: structured-solution +candidate: + path: scripts/init.py + function: solve + input_schema: + type: object + additionalProperties: false + required: + - instance_id + - graph + - robots + - orders + - horizon + properties: + instance_id: + type: string + minLength: 1 + maxLength: 128 + graph: + type: object + additionalProperties: false + required: + - nodes + - edges + properties: + nodes: + type: array + minItems: 2 + uniqueItems: true + items: + type: string + minLength: 1 + maxLength: 128 + edges: + type: array + items: + type: array + minItems: 2 + maxItems: 2 + items: + type: string + minLength: 1 + maxLength: 128 + robots: + type: array + minItems: 1 + items: + type: object + additionalProperties: false + required: + - id + - start_node + - capacity + properties: + id: + type: string + minLength: 1 + maxLength: 128 + start_node: + type: string + minLength: 1 + maxLength: 128 + capacity: + type: integer + minimum: 1 + orders: + type: array + minItems: 1 + items: + type: object + additionalProperties: false + required: + - id + - pickup_node + - dropoff_node + - weight + properties: + id: + type: string + minLength: 1 + maxLength: 128 + pickup_node: + type: string + minLength: 1 + maxLength: 128 + dropoff_node: + type: string + minLength: 1 + maxLength: 128 + weight: + type: integer + minimum: 1 + horizon: + type: integer + minimum: 1 + output_schema: + type: object + additionalProperties: false + required: + - robots + properties: + robots: + type: array + minItems: 1 + items: + type: object + additionalProperties: false + required: + - robot_id + - path + - actions + properties: + robot_id: + type: string + minLength: 1 + maxLength: 128 + path: + type: array + minItems: 2 + items: + type: string + minLength: 1 + maxLength: 128 + actions: + type: array + items: + type: object + additionalProperties: false + required: + - time + - type + - order_id + properties: + time: + type: integer + minimum: 0 + type: + type: string + enum: + - pickup + - dropoff + order_id: + type: string + minLength: 1 + maxLength: 128 + timeout_s: 15.0 + max_output_bytes: 8000000 +instances: + generator_module: verification.problem + public_seeds: + - 101 + - 103 + - 107 + - 109 + - 113 + evaluation_seeds: + - 1009 + - 1013 + - 1019 + - 1021 + - 1031 + instances_per_seed: 2 +objective: + raw_metric: verified_total_move_distance_plus_one + direction: minimize + aggregation: mean + normalization: log2_baseline_ratio + invalid_score: -1.0e+18 +baseline: + module: baseline.heuristic + callable: solve +reference: + module: reference.exact + callable: solve +runtime: + isolation: docker + docker_image: python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7 + timeout_s: 240.0 + cpus: 1.0 + memory_mb: 512 + pids_limit: 128 + network_disabled: true +provenance: + sources: + - id: src-e38f3ddb2a6f24cf + title: Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks + location: references/citations/src-e38f3ddb2a6f24cf.json + sha256: e38f3ddb2a6f24cf13669520173cd6a0ae8af9c3eac47ea93fa47c6ea93fb065 + license: citation-only + retrieved_at: '2026-07-14T09:48:03.938950+00:00' + origin: https://arxiv.org/abs/1705.10868 + final_url: https://arxiv.org/abs/1705.10868 + citation_only: true + publishable: true + data_license: CC0-1.0 + generated_from_seeds: true +frontier: + enabled: true + domain: Robotics + task: WarehouseRobotRouting + languages: + - en + - zh-CN +calibration: + minimum_reference_score: 0.05 + deterministic_tolerance: 1.0e-12 + unified_score_tolerance: 1.0e-09 diff --git a/benchmarks/Robotics/WarehouseRobotRouting/data/seeds.json b/benchmarks/Robotics/WarehouseRobotRouting/data/seeds.json new file mode 100644 index 00000000..8a634a29 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/data/seeds.json @@ -0,0 +1,14 @@ +{ + "evaluation": { + "count": 5, + "visibility": "frozen-verifier-only" + }, + "instances_per_seed": 2, + "public": [ + 101, + 103, + 107, + 109, + 113 + ] +} diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/agent_files.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/agent_files.txt new file mode 100644 index 00000000..5288d154 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/agent_files.txt @@ -0,0 +1,5 @@ +README.md +README_zh-CN.md +Task.md +Task_zh-CN.md +scripts/init.py diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/artifact_files.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/artifact_files.txt new file mode 100644 index 00000000..fb5eabb7 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/artifact_files.txt @@ -0,0 +1,3 @@ +artifacts.json +metrics.json +outputs/*.json diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/candidate_destination.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/candidate_destination.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/candidate_destination.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/constraints.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/constraints.txt new file mode 100644 index 00000000..c25f544a --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/constraints.txt @@ -0,0 +1,199 @@ +带碰撞与载重约束的多仓储机器人路径规划 constraints: +1) Candidate file is `scripts/init.py` and must expose `solve(instance)`. +2) Candidate input must match this JSON Schema: +{ + "additionalProperties": false, + "properties": { + "graph": { + "additionalProperties": false, + "properties": { + "edges": { + "items": { + "items": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "maxItems": 2, + "minItems": 2, + "type": "array" + }, + "type": "array" + }, + "nodes": { + "items": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "minItems": 2, + "type": "array", + "uniqueItems": true + } + }, + "required": [ + "nodes", + "edges" + ], + "type": "object" + }, + "horizon": { + "minimum": 1, + "type": "integer" + }, + "instance_id": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "orders": { + "items": { + "additionalProperties": false, + "properties": { + "dropoff_node": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "id": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "pickup_node": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "weight": { + "minimum": 1, + "type": "integer" + } + }, + "required": [ + "id", + "pickup_node", + "dropoff_node", + "weight" + ], + "type": "object" + }, + "minItems": 1, + "type": "array" + }, + "robots": { + "items": { + "additionalProperties": false, + "properties": { + "capacity": { + "minimum": 1, + "type": "integer" + }, + "id": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "start_node": { + "maxLength": 128, + "minLength": 1, + "type": "string" + } + }, + "required": [ + "id", + "start_node", + "capacity" + ], + "type": "object" + }, + "minItems": 1, + "type": "array" + } + }, + "required": [ + "instance_id", + "graph", + "robots", + "orders", + "horizon" + ], + "type": "object" +} +3) Candidate output must match this JSON Schema: +{ + "additionalProperties": false, + "properties": { + "robots": { + "items": { + "additionalProperties": false, + "properties": { + "actions": { + "items": { + "additionalProperties": false, + "properties": { + "order_id": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "time": { + "minimum": 0, + "type": "integer" + }, + "type": { + "enum": [ + "pickup", + "dropoff" + ], + "type": "string" + } + }, + "required": [ + "time", + "type", + "order_id" + ], + "type": "object" + }, + "type": "array" + }, + "path": { + "items": { + "maxLength": 128, + "minLength": 1, + "type": "string" + }, + "minItems": 2, + "type": "array" + }, + "robot_id": { + "maxLength": 128, + "minLength": 1, + "type": "string" + } + }, + "required": [ + "robot_id", + "path", + "actions" + ], + "type": "object" + }, + "minItems": 1, + "type": "array" + } + }, + "required": [ + "robots" + ], + "type": "object" +} +4) Every hard constraint is recomputed by the frozen verifier. +5) Candidate timeout is 15 seconds per instance. +6) Candidate output is limited to 8000000 bytes. +7) Read-only benchmark assets include verification/, baseline/, reference/, data/, and references/. +8) Publish evaluation requires Docker image `python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7`; candidate and verifier + run in separate network-disabled, non-root containers. +9) Frontier unified must use its process isolation mode because the benchmark evaluator owns the + inner candidate/verifier containers; do not wrap this evaluator in another Docker runtime. diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/copy_files.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/copy_files.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/copy_files.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/eval_command.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/eval_command.txt new file mode 100644 index 00000000..613443e7 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/eval_command.txt @@ -0,0 +1 @@ +{python} {benchmark}/verification/evaluator.py {candidate} diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/eval_cwd.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/eval_cwd.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/eval_cwd.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/initial_program.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/initial_program.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/initial_program.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/readonly_files.txt b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/readonly_files.txt new file mode 100644 index 00000000..ab2c6d75 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/frontier_eval/readonly_files.txt @@ -0,0 +1,8 @@ +baseline/ +data/ +reference/ +references/ +verification/docker/ +verification/evaluator.py +verification/problem.py +verification/process_runner.py diff --git a/benchmarks/Robotics/WarehouseRobotRouting/reference/exact.py b/benchmarks/Robotics/WarehouseRobotRouting/reference/exact.py new file mode 100644 index 00000000..191b5705 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/reference/exact.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from verification.problem import solve_reference # noqa: E402 + + +def solve(instance: dict[str, Any]) -> dict[str, Any]: + return solve_reference(instance) + + +if __name__ == "__main__": + json.dump(solve(json.load(sys.stdin)), sys.stdout, allow_nan=False) + sys.stdout.write("\n") diff --git a/benchmarks/Robotics/WarehouseRobotRouting/references/citations/src-e38f3ddb2a6f24cf.json b/benchmarks/Robotics/WarehouseRobotRouting/references/citations/src-e38f3ddb2a6f24cf.json new file mode 100644 index 00000000..8f93a87a --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/references/citations/src-e38f3ddb2a6f24cf.json @@ -0,0 +1,12 @@ +{ + "citation_only": true, + "final_url": "https://arxiv.org/abs/1705.10868", + "id": "src-e38f3ddb2a6f24cf", + "license": "citation-only", + "location": "sources/src-e38f3ddb2a6f24cf.raw", + "origin": "https://arxiv.org/abs/1705.10868", + "publishable": true, + "retrieved_at": "2026-07-14T09:48:03.938950+00:00", + "sha256": "e38f3ddb2a6f24cf13669520173cd6a0ae8af9c3eac47ea93fa47c6ea93fb065", + "title": "Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks" +} diff --git a/benchmarks/Robotics/WarehouseRobotRouting/references/provenance.json b/benchmarks/Robotics/WarehouseRobotRouting/references/provenance.json new file mode 100644 index 00000000..78127038 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/references/provenance.json @@ -0,0 +1,19 @@ +{ + "benchmark_id": "Robotics/WarehouseRobotRouting", + "data_license": "CC0-1.0", + "generated_from_seeds": true, + "sources": [ + { + "citation_only": true, + "final_url": "https://arxiv.org/abs/1705.10868", + "id": "src-e38f3ddb2a6f24cf", + "license": "citation-only", + "location": "references/citations/src-e38f3ddb2a6f24cf.json", + "origin": "https://arxiv.org/abs/1705.10868", + "publishable": true, + "retrieved_at": "2026-07-14T09:48:03.938950+00:00", + "sha256": "e38f3ddb2a6f24cf13669520173cd6a0ae8af9c3eac47ea93fa47c6ea93fb065", + "title": "Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks" + } + ] +} diff --git a/benchmarks/Robotics/WarehouseRobotRouting/scripts/init.py b/benchmarks/Robotics/WarehouseRobotRouting/scripts/init.py new file mode 100644 index 00000000..b22b9733 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/scripts/init.py @@ -0,0 +1,152 @@ +from __future__ import annotations + +import json +import sys +from typing import Any + + +# EVOLVE-BLOCK-START +def solve(instance): + from collections import deque + robots = {robot['id']: robot for robot in instance['robots']} + orders = {order['id']: order for order in instance['orders']} + robot_ids = sorted(robots) + adjacency = {node: [] for node in instance['graph']['nodes']} + for left, right in instance['graph']['edges']: + adjacency[left].append(right) + adjacency[right].append(left) + for node in adjacency: + adjacency[node] = sorted(set(adjacency[node])) + distances = {} + for source in sorted(adjacency): + seen = {source: 0} + queue = deque([source]) + while queue: + current = queue.popleft() + for neighbor in adjacency[current]: + if neighbor not in seen: + seen[neighbor] = seen[current] + 1 + queue.append(neighbor) + distances[source] = seen + + def sequence_cost(robot_id, sequence): + if not sequence: + return 0 + start = robots[robot_id]['start_node'] + current = start + total = 0 + for order_id in sequence: + order = orders[order_id] + pickup = order['pickup_node'] + dropoff = order['dropoff_node'] + if pickup not in distances[current] or dropoff not in distances[pickup]: + return float('inf') + total += distances[current][pickup] + total += distances[pickup][dropoff] + current = dropoff + if start not in distances[current]: + return float('inf') + return total + distances[current][start] + sequences = {robot_id: [] for robot_id in robot_ids} + for order_id in sorted(orders): + order = orders[order_id] + best_key = None + best_choice = None + for rank, robot_id in enumerate(robot_ids): + if order['weight'] > robots[robot_id]['capacity']: + continue + old_cost = sequence_cost(robot_id, sequences[robot_id]) + for position in range(len(sequences[robot_id]) + 1): + candidate = list(sequences[robot_id]) + candidate.insert(position, order_id) + new_cost = sequence_cost(robot_id, candidate) + key = (new_cost - old_cost, new_cost, rank, position, robot_id) + if new_cost != float('inf') and (best_key is None or key < best_key): + best_key = key + best_choice = (robot_id, position) + if best_choice is None: + raise ValueError('No capacity-feasible robot can serve every order') + chosen_robot, chosen_position = best_choice + sequences[chosen_robot].insert(chosen_position, order_id) + + def shortest_path(source, target, forbidden): + if source == target: + return [source] + parent = {source: None} + queue = deque([source]) + while queue: + current = queue.popleft() + for neighbor in adjacency[current]: + if neighbor in parent: + continue + if neighbor in forbidden and neighbor != target: + continue + parent[neighbor] = current + if neighbor == target: + path = [target] + while path[-1] != source: + path.append(parent[path[-1]]) + path.reverse() + return path + queue.append(neighbor) + raise ValueError('A required service node is unreachable') + starts = {robot['start_node'] for robot in robots.values()} + paths = {robot_id: [robots[robot_id]['start_node']] for robot_id in robot_ids} + actions = {robot_id: [] for robot_id in robot_ids} + cursor = 0 + for robot_id in robot_ids: + path = paths[robot_id] + while len(path) - 1 < cursor: + path.append(path[-1]) + forbidden = starts - {robots[robot_id]['start_node']} + last_action_time = None + for order_id in sequences[robot_id]: + order = orders[order_id] + segment = shortest_path(path[-1], order['pickup_node'], forbidden) + path.extend(segment[1:]) + if last_action_time is not None and len(path) - 1 == last_action_time: + path.append(path[-1]) + pickup_time = len(path) - 1 + actions[robot_id].append({'time': pickup_time, 'type': 'pickup', 'order_id': order_id}) + last_action_time = pickup_time + segment = shortest_path(path[-1], order['dropoff_node'], forbidden) + path.extend(segment[1:]) + if len(path) - 1 == last_action_time: + path.append(path[-1]) + dropoff_time = len(path) - 1 + actions[robot_id].append({'time': dropoff_time, 'type': 'dropoff', 'order_id': order_id}) + last_action_time = dropoff_time + if sequences[robot_id]: + segment = shortest_path(path[-1], robots[robot_id]['start_node'], forbidden) + path.extend(segment[1:]) + cursor = len(path) - 1 + horizon = instance['horizon'] + if cursor > horizon: + raise ValueError('The baseline route exceeds the instance horizon') + output = [] + for robot_id in robot_ids: + path = paths[robot_id] + path.extend([path[-1]] * (horizon + 1 - len(path))) + output.append({'robot_id': robot_id, 'path': path, 'actions': actions[robot_id]}) + return {'robots': output} +# EVOLVE-BLOCK-END + + +def main() -> int: + try: + instance = json.load(sys.stdin) + if not isinstance(instance, dict): + raise TypeError("input must be a JSON object") + solution = solve(instance) + if not isinstance(solution, dict): + raise TypeError("solve() must return a JSON object") + json.dump(solution, sys.stdout, allow_nan=False, separators=(",", ":")) + sys.stdout.write("\n") + return 0 + except Exception as exc: + print(f"candidate error: {type(exc).__name__}: {exc}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/Robotics/WarehouseRobotRouting/verification/docker/Dockerfile b/benchmarks/Robotics/WarehouseRobotRouting/verification/docker/Dockerfile new file mode 100644 index 00000000..9512d042 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/verification/docker/Dockerfile @@ -0,0 +1,13 @@ +FROM python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7 + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PYTHONHASHSEED=0 + +RUN groupadd --gid 65532 benchgen \ + && useradd --uid 65532 --gid 65532 --no-create-home --shell /usr/sbin/nologin benchgen + +WORKDIR /workspace +USER 65532:65532 + +CMD ["python", "--version"] diff --git a/benchmarks/Robotics/WarehouseRobotRouting/verification/evaluator.py b/benchmarks/Robotics/WarehouseRobotRouting/verification/evaluator.py new file mode 100644 index 00000000..4e909e1c --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/verification/evaluator.py @@ -0,0 +1,707 @@ +from __future__ import annotations + +import argparse +import copy +import importlib.util +import json +import math +import os +import statistics +import sys +import time +import uuid +from pathlib import Path +from types import ModuleType +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from verification.process_runner import BoundedResult, run_bounded # noqa: E402 + +SEEDS = [1009, 1013, 1019, 1021, 1031] +SMOKE_SEED = 2705467810 +INSTANCES_PER_SEED = 2 +INPUT_SCHEMA = json.loads( + "{\"additionalProperties\": false, \"propert" + "ies\": {\"graph\": {\"additionalProperties\":" + " false, \"properties\": {\"edges\": {\"items\"" + ": {\"items\": {\"maxLength\": 128, \"minLengt" + "h\": 1, \"type\": \"string\"}, \"maxItems\": 2," + " \"minItems\": 2, \"type\": \"array\"}, \"type\"" + ": \"array\"}, \"nodes\": {\"items\": {\"maxLeng" + "th\": 128, \"minLength\": 1, \"type\": \"strin" + "g\"}, \"minItems\": 2, \"type\": \"array\", \"un" + "iqueItems\": true}}, \"required\": [\"nodes\"" + ", \"edges\"], \"type\": \"object\"}, \"horizon\"" + ": {\"minimum\": 1, \"type\": \"integer\"}, \"in" + "stance_id\": {\"maxLength\": 128, \"minLengt" + "h\": 1, \"type\": \"string\"}, \"orders\": {\"it" + "ems\": {\"additionalProperties\": false, \"p" + "roperties\": {\"dropoff_node\": {\"maxLength" + "\": 128, \"minLength\": 1, \"type\": \"string\"" + "}, \"id\": {\"maxLength\": 128, \"minLength\":" + " 1, \"type\": \"string\"}, \"pickup_node\": {\"" + "maxLength\": 128, \"minLength\": 1, \"type\":" + " \"string\"}, \"weight\": {\"minimum\": 1, \"ty" + "pe\": \"integer\"}}, \"required\": [\"id\", \"pi" + "ckup_node\", \"dropoff_node\", \"weight\"], \"" + "type\": \"object\"}, \"minItems\": 1, \"type\":" + " \"array\"}, \"robots\": {\"items\": {\"additio" + "nalProperties\": false, \"properties\": {\"c" + "apacity\": {\"minimum\": 1, \"type\": \"intege" + "r\"}, \"id\": {\"maxLength\": 128, \"minLength" + "\": 1, \"type\": \"string\"}, \"start_node\": {" + "\"maxLength\": 128, \"minLength\": 1, \"type\"" + ": \"string\"}}, \"required\": [\"id\", \"start_" + "node\", \"capacity\"], \"type\": \"object\"}, \"" + "minItems\": 1, \"type\": \"array\"}}, \"requir" + "ed\": [\"instance_id\", \"graph\", \"robots\", " + "\"orders\", \"horizon\"], \"type\": \"object\"}" +) +OUTPUT_SCHEMA = json.loads( + "{\"additionalProperties\": false, \"propert" + "ies\": {\"robots\": {\"items\": {\"additionalP" + "roperties\": false, \"properties\": {\"actio" + "ns\": {\"items\": {\"additionalProperties\": " + "false, \"properties\": {\"order_id\": {\"maxL" + "ength\": 128, \"minLength\": 1, \"type\": \"st" + "ring\"}, \"time\": {\"minimum\": 0, \"type\": \"" + "integer\"}, \"type\": {\"enum\": [\"pickup\", \"" + "dropoff\"], \"type\": \"string\"}}, \"required" + "\": [\"time\", \"type\", \"order_id\"], \"type\":" + " \"object\"}, \"type\": \"array\"}, \"path\": {\"" + "items\": {\"maxLength\": 128, \"minLength\": " + "1, \"type\": \"string\"}, \"minItems\": 2, \"ty" + "pe\": \"array\"}, \"robot_id\": {\"maxLength\":" + " 128, \"minLength\": 1, \"type\": \"string\"}}" + ", \"required\": [\"robot_id\", \"path\", \"acti" + "ons\"], \"type\": \"object\"}, \"minItems\": 1," + " \"type\": \"array\"}}, \"required\": [\"robots" + "\"], \"type\": \"object\"}" +) +DIRECTION = "minimize" +AGGREGATION = "mean" +INVALID_SCORE = -1e+18 +TIMEOUT_S = 15.0 +MAX_OUTPUT_BYTES = 8000000 +RUNTIME_IMAGE = ( + "python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b87" + "31f1499a57e22e6c285135ae657bf7" +) +MEMORY_MB = 512 +CPUS = 1.0 +PIDS_LIMIT = 128 + + +def _load_module(relative_module: str) -> ModuleType: + path = ROOT / (relative_module.replace(".", "/") + ".py") + module_spec = importlib.util.spec_from_file_location("benchgen_task_module", path) + if module_spec is None or module_spec.loader is None: + raise RuntimeError(f"cannot load task module: {path}") + module = importlib.util.module_from_spec(module_spec) + module_spec.loader.exec_module(module) + return module + + +def _valid( + problem: ModuleType, instance: dict[str, Any], solution: dict[str, Any] +) -> tuple[bool, str]: + result = problem.validate_solution(instance, solution) + if result is None: + return True, "" + if isinstance(result, bool): + return result, "" if result else "solution rejected by verifier" + if isinstance(result, tuple) and len(result) == 2: + return bool(result[0]), str(result[1]) + if hasattr(result, "valid"): + return bool(result.valid), str(getattr(result, "reason", "")) + raise TypeError("invalid validate_solution return value") + + +def _finite(value: Any) -> bool: + if isinstance(value, float): + return math.isfinite(value) + if isinstance(value, dict): + return all(_finite(key) and _finite(item) for key, item in value.items()) + if isinstance(value, list): + return all(_finite(item) for item in value) + return True + + +def _validate_json_schema(value: Any, schema: dict[str, Any], path: str = "$") -> list[str]: + """Validate the dependency-free JSON Schema subset supported by benchgen v1.""" + + errors: list[str] = [] + expected_type = schema.get("type") + if expected_type and not _matches_type(value, expected_type): + return [f"{path}: expected {expected_type}, got {type(value).__name__}"] + if "enum" in schema and not any(_json_equal(value, item) for item in schema["enum"]): + errors.append(f"{path}: value is not in enum") + + if isinstance(value, dict): + properties = schema.get("properties", {}) + required = schema.get("required", []) + for key in required: + if key not in value: + errors.append(f"{path}: missing required property {key!r}") + for key, item in value.items(): + if key in properties: + errors.extend(_validate_json_schema(item, properties[key], f"{path}.{key}")) + elif schema.get("additionalProperties") is False: + errors.append(f"{path}: unexpected property {key!r}") + elif isinstance(value, list): + if "minItems" in schema and len(value) < schema["minItems"]: + errors.append(f"{path}: fewer than minItems") + if "maxItems" in schema and len(value) > schema["maxItems"]: + errors.append(f"{path}: more than maxItems") + if schema.get("uniqueItems"): + canonical = [_json_key(item) for item in value] + if len(canonical) != len(set(canonical)): + errors.append(f"{path}: items are not unique") + if isinstance(schema.get("items"), dict): + for index, item in enumerate(value): + errors.extend(_validate_json_schema(item, schema["items"], f"{path}[{index}]")) + elif isinstance(value, str): + if "minLength" in schema and len(value) < schema["minLength"]: + errors.append(f"{path}: shorter than minLength") + if "maxLength" in schema and len(value) > schema["maxLength"]: + errors.append(f"{path}: longer than maxLength") + elif isinstance(value, (int, float)) and not isinstance(value, bool): + if "minimum" in schema and value < schema["minimum"]: + errors.append(f"{path}: below minimum") + if "maximum" in schema and value > schema["maximum"]: + errors.append(f"{path}: above maximum") + return errors + + +def _json_equal(left: Any, right: Any) -> bool: + return _json_key(left) == _json_key(right) + + +def _json_key(value: Any) -> Any: + if value is None: + return ("null",) + if isinstance(value, bool): + return ("boolean", value) + if isinstance(value, (int, float)): + return ("number", value) + if isinstance(value, str): + return ("string", value) + if isinstance(value, list): + return ("array", tuple(_json_key(item) for item in value)) + if isinstance(value, dict): + return ( + "object", + tuple(sorted((str(key), _json_key(item)) for key, item in value.items())), + ) + return (type(value).__name__, repr(value)) + + +def _matches_type(value: Any, expected: str | list[str]) -> bool: + if isinstance(expected, list): + return any(_matches_type(value, item) for item in expected) + checks = { + "null": lambda item: item is None, + "boolean": lambda item: isinstance(item, bool), + "object": lambda item: isinstance(item, dict), + "array": lambda item: isinstance(item, list), + "string": lambda item: isinstance(item, str), + "integer": lambda item: isinstance(item, int) and not isinstance(item, bool), + "number": lambda item: isinstance(item, (int, float)) and not isinstance(item, bool), + } + return expected in checks and checks[expected](value) + + +def _require_schema(value: Any, schema: dict[str, Any], label: str) -> None: + errors = _validate_json_schema(value, schema) + if errors: + raise ValueError(f"{label} violates JSON Schema: {'; '.join(errors[:5])}") + + +def _metric_value(value: Any, label: str) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise TypeError(f"{label} must be an int or float, got {type(value).__name__}") + metric = float(value) + if not math.isfinite(metric): + raise ValueError(f"{label} must be finite") + if metric <= 0: + raise ValueError(f"{label} must be strictly positive") + return metric + + +def _raw_metric( + problem: ModuleType, + instance: dict[str, Any], + solution: dict[str, Any], + label: str, +) -> float: + value = problem.evaluate_solution(instance, solution) + return _metric_value(value, f"{label} evaluate_solution result") + + +def _canonical_json(value: Any, label: str) -> str: + try: + return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False) + except (TypeError, ValueError) as exc: + raise TypeError(f"{label} must be JSON serializable: {exc}") from exc + + +def _smoke_instances(problem: ModuleType) -> list[dict[str, Any]]: + generated = problem.generate_instances(SMOKE_SEED) + instances = [generated] if isinstance(generated, dict) else list(generated) + if len(instances) != INSTANCES_PER_SEED: + raise RuntimeError( + f"smoke generator returned {len(instances)} instances; " + f"expected exactly {INSTANCES_PER_SEED}" + ) + checked: list[dict[str, Any]] = [] + for instance in instances: + if not isinstance(instance, dict) or not _finite(instance): + raise TypeError("smoke instance must be a finite JSON object") + _require_schema(instance, INPUT_SCHEMA, "smoke instance") + _canonical_json(instance, "smoke instance") + checked.append(instance) + return checked + + +def _smoke_solver( + problem: ModuleType, + solver: Any, + instance: dict[str, Any], + label: str, +) -> tuple[dict[str, Any], float]: + solution = solver(copy.deepcopy(instance)) + if not isinstance(solution, dict) or not _finite(solution): + raise TypeError(f"{label} output must be a finite JSON object") + _require_schema(solution, OUTPUT_SCHEMA, f"{label} output") + _canonical_json(solution, f"{label} output") + valid, reason = _valid(problem, copy.deepcopy(instance), copy.deepcopy(solution)) + if not valid: + raise ValueError(f"{label} output is invalid: {reason or 'solution rejected'}") + metric = _raw_metric( + problem, + copy.deepcopy(instance), + copy.deepcopy(solution), + label, + ) + return solution, metric + + +def _smoke_payload() -> dict[str, Any]: + problem = _load_module("verification.problem") + first_instances = _smoke_instances(problem) + second_instances = _smoke_instances(problem) + if _canonical_json(first_instances, "generated instances") != _canonical_json( + second_instances, "repeated generated instances" + ): + raise ValueError("generate_instances is not deterministic") + + solvers = ( + ("random solver", problem.solve_random), + ("baseline solver", problem.solve_baseline), + ("reference solver", problem.solve_reference), + ) + solver_runs = 0 + for label, solver in solvers: + for instance in first_instances: + first_solution, first_metric = _smoke_solver(problem, solver, instance, label) + second_solution, second_metric = _smoke_solver(problem, solver, instance, label) + solver_runs += 2 + if _canonical_json(first_solution, f"{label} output") != _canonical_json( + second_solution, f"repeated {label} output" + ): + raise ValueError(f"{label} is not deterministic") + if first_metric != second_metric: + raise ValueError(f"{label} metric is not deterministic") + return { + "ok": True, + "instances_checked": len(first_instances), + "solver_runs": solver_runs, + "solvers_checked": [label for label, _solver in solvers], + } + + +def _score(baseline: float, candidate: float) -> float: + if not math.isfinite(baseline) or not math.isfinite(candidate): + raise ValueError("objective metrics must be finite") + if baseline <= 0 or candidate <= 0: + raise ValueError("log2 normalization requires positive metrics") + ratio = baseline / candidate if DIRECTION == "minimize" else candidate / baseline + return math.log2(ratio) + + +def _prepare_payload() -> dict[str, Any]: + problem = _load_module("verification.problem") + baseline_module = _load_module("baseline.heuristic") + baseline_solver = getattr(baseline_module, "solve") # noqa: B009 + cases: list[dict[str, Any]] = [] + for seed in SEEDS: + generated = problem.generate_instances(seed) + instances = [generated] if isinstance(generated, dict) else list(generated) + if len(instances) != INSTANCES_PER_SEED: + raise RuntimeError( + f"seed {seed} generated {len(instances)} instances; " + f"expected exactly {INSTANCES_PER_SEED}" + ) + for index, instance in enumerate(instances): + if not isinstance(instance, dict) or not _finite(instance): + raise TypeError("generated instance must be a finite JSON object") + _require_schema(instance, INPUT_SCHEMA, "generated instance") + baseline_solution = baseline_solver(instance) + if not isinstance(baseline_solution, dict) or not _finite(baseline_solution): + raise TypeError("baseline output must be a finite JSON object") + _require_schema(baseline_solution, OUTPUT_SCHEMA, "baseline output") + valid, reason = _valid(problem, instance, baseline_solution) + if not valid: + raise RuntimeError(f"invalid benchmark baseline: {reason}") + baseline_metric = _raw_metric( + problem, instance, baseline_solution, "baseline" + ) + cases.append( + { + "case_id": f"{seed}:{index}", + "seed": seed, + "index": index, + "instance": instance, + "baseline_metric": baseline_metric, + } + ) + return {"cases": cases} + + +def _score_payload(payload: dict[str, Any]) -> dict[str, Any]: + problem = _load_module("verification.problem") + prepared = payload.get("prepared", {}) + cases = prepared.get("cases", []) if isinstance(prepared, dict) else [] + submissions = payload.get("submissions", []) + if not isinstance(cases, list) or not isinstance(submissions, list): + raise TypeError("score payload must contain cases and submissions lists") + if len(cases) != len(submissions): + raise ValueError("submission count does not match prepared case count") + records: list[dict[str, Any]] = [] + for case, submission in zip(cases, submissions, strict=True): + started = time.monotonic() + record = {"case_index": len(records)} + try: + if submission.get("case_id") != case.get("case_id"): + raise ValueError("submission case id mismatch") + if submission.get("runner_error"): + raise RuntimeError(str(submission["runner_error"])) + solution = submission.get("solution") + if not isinstance(solution, dict) or not _finite(solution): + raise TypeError("candidate output must be a finite JSON object") + _require_schema(solution, OUTPUT_SCHEMA, "candidate output") + instance = case["instance"] + if not isinstance(instance, dict) or not _finite(instance): + raise TypeError("prepared instance must be a finite JSON object") + _require_schema(instance, INPUT_SCHEMA, "prepared instance") + valid, reason = _valid(problem, instance, solution) + if not valid: + raise ValueError(reason or "invalid candidate solution") + baseline_metric = _metric_value(case["baseline_metric"], "prepared baseline metric") + candidate_metric = _raw_metric(problem, instance, solution, "candidate") + record.update( + valid=True, + score=_score(baseline_metric, candidate_metric), + baseline_metric=baseline_metric, + candidate_metric=candidate_metric, + ) + except Exception as exc: + record.update(valid=False, score=INVALID_SCORE, reason=f"{type(exc).__name__}: {exc}") + record["runtime_s"] = time.monotonic() - started + records.append(record) + all_valid = bool(records) and all(record["valid"] for record in records) + scores = [float(record["score"]) for record in records] + combined = ( + (statistics.fmean(scores) if AGGREGATION == "mean" else statistics.median(scores)) + if all_valid + else INVALID_SCORE + ) + metrics = { + "valid": 1.0 if all_valid else 0.0, + "combined_score": combined, + "instances_total": len(records), + "instances_valid": sum(bool(record["valid"]) for record in records), + "runtime_s": sum(float(record["runtime_s"]) for record in records), + } + return {"metrics": metrics, "artifacts": {"instance_results": records}} + + +def _parse_json_result(result: BoundedResult, label: str) -> dict[str, Any]: + if result.timed_out: + raise TimeoutError(f"{label} timed out") + if result.output_truncated: + raise ValueError(f"{label} exceeded output limit") + if result.returncode != 0: + raise RuntimeError(f"{label} exited with {result.returncode}: {result.stderr[-2000:]}") + value = json.loads(result.stdout) + if not isinstance(value, dict) or not _finite(value): + raise TypeError(f"{label} output must be a finite JSON object") + return value + + +def _candidate_local( + candidate: Path, + instance: dict[str, Any], + *, + timeout_s: float = TIMEOUT_S, + max_output_bytes: int = MAX_OUTPUT_BYTES, +) -> dict[str, Any]: + result = run_bounded( + [sys.executable, str(candidate)], + json.dumps(instance, allow_nan=False), + timeout_s=timeout_s, + max_output_bytes=max_output_bytes, + ) + return _parse_json_result(result, "candidate") + + +def _docker_base(name: str) -> list[str]: + return [ + "docker", + "run", + "--rm", + "--interactive", + "--name", + name, + "--network", + "none", + "--read-only", + "--user", + "65532:65532", + "--cap-drop", + "ALL", + "--security-opt", + "no-new-privileges", + "--pids-limit", + str(PIDS_LIMIT), + "--memory", + f"{MEMORY_MB}m", + "--cpus", + str(CPUS), + "--tmpfs", + "/tmp:rw,noexec,nosuid,nodev,size=64m", + ] + + +def _docker_json( + command: list[str], + input_payload: dict[str, Any] | None, + *, + timeout_s: float, + label: str, + max_output_bytes: int = 2_000_000, +) -> dict[str, Any]: + name = f"benchgen-{label}-{uuid.uuid4().hex[:12]}" + full_command = _docker_base(name) + command + result = run_bounded( + full_command, + json.dumps(input_payload, allow_nan=False) if input_payload is not None else "", + timeout_s=timeout_s, + max_output_bytes=max_output_bytes, + ) + if result.timed_out: + run_bounded( + ["docker", "rm", "--force", name], + "", + timeout_s=10, + max_output_bytes=10_000, + ) + return _parse_json_result(result, label) + + +def _verifier_container(mode: str, payload: dict[str, Any] | None = None) -> dict[str, Any]: + if "," in str(ROOT) or "\n" in str(ROOT): + raise ValueError("benchmark path contains unsupported Docker mount characters") + command: list[str] = [] + for relative in ("verification", "baseline", "reference", "data", "references"): + source = ROOT / relative + if source.exists(): + command.extend( + ( + "--mount", + f"type=bind,src={source},dst=/workspace/benchmark/{relative},readonly", + ) + ) + command.extend( + ( + "--workdir", + "/workspace/benchmark", + RUNTIME_IMAGE, + "python", + "/workspace/benchmark/verification/evaluator.py", + mode, + ) + ) + return _docker_json(command, payload, timeout_s=240.0, label="verifier") + + +def _candidate_container( + candidate: Path, + case: dict[str, Any], + *, + timeout_s: float = TIMEOUT_S, + max_output_bytes: int = MAX_OUTPUT_BYTES, +) -> dict[str, Any]: + if "," in str(candidate) or "\n" in str(candidate): + raise ValueError("candidate path contains unsupported Docker mount characters") + trusted_solver = False + try: + relative = candidate.relative_to(ROOT) + trusted_solver = bool(relative.parts and relative.parts[0] in {"baseline", "reference"}) + except ValueError: + relative = Path(candidate.name) + if trusted_solver: + command = [ + "--mount", + f"type=bind,src={ROOT},dst=/workspace/benchmark,readonly", + "--workdir", + "/workspace/benchmark", + RUNTIME_IMAGE, + "python", + f"/workspace/benchmark/{relative.as_posix()}", + ] + else: + command = [ + "--mount", + f"type=bind,src={candidate},dst=/workspace/candidate.py,readonly", + "--workdir", + "/tmp", + RUNTIME_IMAGE, + "python", + "/workspace/candidate.py", + ] + return _docker_json( + command, + case["instance"], + timeout_s=timeout_s, + label="candidate", + max_output_bytes=max_output_bytes, + ) + + +def evaluate( + candidate: Path, + *, + local: bool = False, + candidate_timeout_s: float = TIMEOUT_S, + candidate_max_output_bytes: int = MAX_OUTPUT_BYTES, +) -> tuple[dict[str, Any], dict[str, Any]]: + candidate = candidate.resolve() + if not candidate.is_file(): + raise FileNotFoundError(f"candidate not found: {candidate}") + if local: + prepared = _prepare_payload() + submissions = [] + for case in prepared["cases"]: + try: + solution = _candidate_local( + candidate, + case["instance"], + timeout_s=candidate_timeout_s, + max_output_bytes=candidate_max_output_bytes, + ) + submissions.append({"case_id": case["case_id"], "solution": solution}) + except Exception as exc: + submissions.append( + {"case_id": case["case_id"], "runner_error": f"{type(exc).__name__}: {exc}"} + ) + result = _score_payload({"prepared": prepared, "submissions": submissions}) + else: + prepared = _verifier_container("--prepare") + submissions = [] + for case in prepared["cases"]: + try: + solution = _candidate_container( + candidate, + case, + timeout_s=candidate_timeout_s, + max_output_bytes=candidate_max_output_bytes, + ) + submissions.append({"case_id": case["case_id"], "solution": solution}) + except Exception as exc: + submissions.append( + {"case_id": case["case_id"], "runner_error": f"{type(exc).__name__}: {exc}"} + ) + result = _verifier_container( + "--score", {"prepared": prepared, "submissions": submissions} + ) + return result["metrics"], result["artifacts"] + + +def _atomic_json(path: Path, payload: dict[str, Any]) -> None: + from tempfile import NamedTemporaryFile + + path.parent.mkdir(parents=True, exist_ok=True) + with NamedTemporaryFile("w", encoding="utf-8", dir=path.parent, delete=False) as handle: + json.dump(payload, handle, indent=2, sort_keys=True, allow_nan=False) + handle.write("\n") + handle.flush() + os.fsync(handle.fileno()) + temporary = Path(handle.name) + temporary.replace(path) + + +def main() -> int: + if len(sys.argv) == 2 and sys.argv[1] == "--smoke": + try: + payload = _verifier_container("--smoke-local") + except Exception as exc: + message = f"{type(exc).__name__}: {exc}" + print(json.dumps({"ok": False, "error": message}, sort_keys=True)) + print(message, file=sys.stderr) + return 1 + print(json.dumps(payload, sort_keys=True, allow_nan=False)) + return 0 + if len(sys.argv) == 2 and sys.argv[1] == "--smoke-local": + try: + payload = _smoke_payload() + except Exception as exc: + message = f"{type(exc).__name__}: {exc}" + print(json.dumps({"ok": False, "error": message}, sort_keys=True)) + print(message, file=sys.stderr) + return 1 + print(json.dumps(payload, sort_keys=True, allow_nan=False)) + return 0 + if len(sys.argv) == 2 and sys.argv[1] == "--prepare": + print(json.dumps(_prepare_payload(), sort_keys=True, allow_nan=False)) + return 0 + if len(sys.argv) == 2 and sys.argv[1] == "--score": + payload = json.load(sys.stdin) + print(json.dumps(_score_payload(payload), sort_keys=True, allow_nan=False)) + return 0 + parser = argparse.ArgumentParser(description="Evaluate one structured-solution candidate.") + parser.add_argument("candidate", type=Path) + parser.add_argument("--local", action="store_true") + parser.add_argument( + "--_benchgen-probe-timeout-s", + type=float, + default=TIMEOUT_S, + help=argparse.SUPPRESS, + ) + parser.add_argument( + "--_benchgen-probe-max-output-bytes", + type=int, + default=MAX_OUTPUT_BYTES, + help=argparse.SUPPRESS, + ) + arguments = parser.parse_args() + if not 0 < arguments._benchgen_probe_timeout_s <= TIMEOUT_S: + parser.error("probe timeout must be positive and cannot relax the benchmark limit") + if not 0 < arguments._benchgen_probe_max_output_bytes <= MAX_OUTPUT_BYTES: + parser.error("probe output limit must be positive and cannot relax the benchmark limit") + metrics, artifacts = evaluate( + arguments.candidate, + local=arguments.local, + candidate_timeout_s=arguments._benchgen_probe_timeout_s, + candidate_max_output_bytes=arguments._benchgen_probe_max_output_bytes, + ) + _atomic_json(Path.cwd() / "metrics.json", metrics) + _atomic_json(Path.cwd() / "artifacts.json", artifacts) + print(json.dumps(metrics, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/Robotics/WarehouseRobotRouting/verification/problem.py b/benchmarks/Robotics/WarehouseRobotRouting/verification/problem.py new file mode 100644 index 00000000..ce63b186 --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/verification/problem.py @@ -0,0 +1,620 @@ +from collections import deque +import math +import random + + +RAW_METRIC = 'verified_total_move_distance_plus_one' +INVALID_SCORE = -1e18 + + +def _adjacency(instance): + adjacency = {node: [] for node in instance['graph']['nodes']} + for left, right in instance['graph']['edges']: + adjacency[left].append(right) + adjacency[right].append(left) + for node in adjacency: + adjacency[node] = tuple(sorted(set(adjacency[node]))) + return adjacency + + +def _all_distances(adjacency): + distances = {} + for source in sorted(adjacency): + seen = {source: 0} + queue = deque([source]) + while queue: + current = queue.popleft() + for neighbor in adjacency[current]: + if neighbor not in seen: + seen[neighbor] = seen[current] + 1 + queue.append(neighbor) + distances[source] = seen + return distances + + +def _shortest_path(adjacency, source, target, forbidden): + if source == target: + return [source] + parent = {source: None} + queue = deque([source]) + while queue: + current = queue.popleft() + for neighbor in adjacency[current]: + if neighbor in parent: + continue + if neighbor in forbidden and neighbor != target: + continue + parent[neighbor] = current + if neighbor == target: + path = [target] + while path[-1] != source: + path.append(parent[path[-1]]) + path.reverse() + return path + queue.append(neighbor) + raise ValueError('A required service node is unreachable') + + +def _sequence_cost(sequence, robot_id, robots, orders, distances, close_route=True): + if not sequence: + return 0 + start = robots[robot_id]['start_node'] + current = start + total = 0 + for order_id in sequence: + order = orders[order_id] + pickup = order['pickup_node'] + dropoff = order['dropoff_node'] + if pickup not in distances[current] or dropoff not in distances[pickup]: + return float('inf') + total += distances[current][pickup] + total += distances[pickup][dropoff] + current = dropoff + if close_route: + if start not in distances[current]: + return float('inf') + total += distances[current][start] + return total + + +def _greedy_sequences(instance, order_ids, robot_priority): + robots = {robot['id']: robot for robot in instance['robots']} + orders = {order['id']: order for order in instance['orders']} + adjacency = _adjacency(instance) + distances = _all_distances(adjacency) + sequences = {robot_id: [] for robot_id in sorted(robots)} + + for order_id in order_ids: + order = orders[order_id] + best_key = None + best_choice = None + for rank, robot_id in enumerate(robot_priority): + if order['weight'] > robots[robot_id]['capacity']: + continue + old_cost = _sequence_cost( + sequences[robot_id], robot_id, robots, orders, distances, True + ) + for position in range(len(sequences[robot_id]) + 1): + candidate = list(sequences[robot_id]) + candidate.insert(position, order_id) + new_cost = _sequence_cost( + candidate, robot_id, robots, orders, distances, True + ) + key = (new_cost - old_cost, new_cost, rank, position, robot_id) + if new_cost != float('inf') and (best_key is None or key < best_key): + best_key = key + best_choice = (robot_id, position) + if best_choice is None: + raise ValueError('No capacity-feasible robot can serve order ' + order_id) + chosen_robot, chosen_position = best_choice + sequences[chosen_robot].insert(chosen_position, order_id) + return sequences + + +def _build_serial_solution(instance, sequences, schedule_order, open_robot=None): + robots = {robot['id']: robot for robot in instance['robots']} + orders = {order['id']: order for order in instance['orders']} + robot_ids = sorted(robots) + if sorted(schedule_order) != robot_ids: + raise ValueError('The schedule must contain every robot exactly once') + + adjacency = _adjacency(instance) + starts = {robot['start_node'] for robot in robots.values()} + paths = {robot_id: [robots[robot_id]['start_node']] for robot_id in robot_ids} + actions = {robot_id: [] for robot_id in robot_ids} + cursor = 0 + + for robot_id in schedule_order: + path = paths[robot_id] + while len(path) - 1 < cursor: + path.append(path[-1]) + forbidden = starts - {robots[robot_id]['start_node']} + last_action_time = None + + for order_id in sequences[robot_id]: + order = orders[order_id] + segment = _shortest_path( + adjacency, path[-1], order['pickup_node'], forbidden + ) + path.extend(segment[1:]) + if last_action_time is not None and len(path) - 1 == last_action_time: + path.append(path[-1]) + pickup_time = len(path) - 1 + actions[robot_id].append( + {'time': pickup_time, 'type': 'pickup', 'order_id': order_id} + ) + last_action_time = pickup_time + + segment = _shortest_path( + adjacency, path[-1], order['dropoff_node'], forbidden + ) + path.extend(segment[1:]) + if len(path) - 1 == last_action_time: + path.append(path[-1]) + dropoff_time = len(path) - 1 + actions[robot_id].append( + {'time': dropoff_time, 'type': 'dropoff', 'order_id': order_id} + ) + last_action_time = dropoff_time + + if sequences[robot_id] and robot_id != open_robot: + segment = _shortest_path( + adjacency, path[-1], robots[robot_id]['start_node'], forbidden + ) + path.extend(segment[1:]) + cursor = len(path) - 1 + + horizon = instance['horizon'] + if cursor > horizon: + raise ValueError('Constructed route exceeds the instance horizon') + + result = [] + for robot_id in robot_ids: + path = paths[robot_id] + path.extend([path[-1]] * (horizon + 1 - len(path))) + result.append( + {'robot_id': robot_id, 'path': path, 'actions': actions[robot_id]} + ) + return {'robots': result} + + +def _stable_seed(seed, text): + value = int(seed) & ((1 << 64) - 1) + for byte in text.encode('utf-8'): + value ^= byte + value = (value * 1099511628211) & ((1 << 64) - 1) + return value + + +def _route_table(instance, robot_id, close_route): + robots = {robot['id']: robot for robot in instance['robots']} + order_list = sorted(instance['orders'], key=lambda order: order['id']) + adjacency = _adjacency(instance) + distances = _all_distances(adjacency) + robot = robots[robot_id] + count = len(order_list) + full = 1 << count + dynamic = [dict() for _ in range(full)] + + for index, order in enumerate(order_list): + if order['weight'] > robot['capacity']: + continue + start = robot['start_node'] + pickup = order['pickup_node'] + dropoff = order['dropoff_node'] + if pickup not in distances[start] or dropoff not in distances[pickup]: + continue + cost = distances[start][pickup] + distances[pickup][dropoff] + dynamic[1 << index][index] = (cost, (order['id'],)) + + for mask in range(1, full): + for last, state in list(dynamic[mask].items()): + cost, sequence = state + last_dropoff = order_list[last]['dropoff_node'] + for next_index, order in enumerate(order_list): + bit = 1 << next_index + if mask & bit or order['weight'] > robot['capacity']: + continue + pickup = order['pickup_node'] + dropoff = order['dropoff_node'] + if pickup not in distances[last_dropoff] or dropoff not in distances[pickup]: + continue + new_mask = mask | bit + new_cost = ( + cost + + distances[last_dropoff][pickup] + + distances[pickup][dropoff] + ) + candidate = (new_cost, sequence + (order['id'],)) + previous = dynamic[new_mask].get(next_index) + if previous is None or candidate < previous: + dynamic[new_mask][next_index] = candidate + + options = {0: (0, ())} + for mask in range(1, full): + best = None + for last, state in dynamic[mask].items(): + cost, sequence = state + if close_route: + dropoff = order_list[last]['dropoff_node'] + start = robot['start_node'] + if start not in distances[dropoff]: + continue + cost += distances[dropoff][start] + candidate = (cost, sequence) + if best is None or candidate < best: + best = candidate + if best is not None: + options[mask] = best + return options + + +def _exact_reference_sequences(instance): + robot_ids = sorted(robot['id'] for robot in instance['robots']) + order_ids = sorted(order['id'] for order in instance['orders']) + count = len(order_ids) + full_mask = (1 << count) - 1 + closed_tables = { + robot_id: _route_table(instance, robot_id, True) for robot_id in robot_ids + } + open_tables = { + robot_id: _route_table(instance, robot_id, False) for robot_id in robot_ids + } + + best_key = None + best_payload = None + for last_robot in robot_ids: + states = {0: (0, ())} + for robot_id in robot_ids: + table = open_tables[robot_id] if robot_id == last_robot else closed_tables[robot_id] + next_states = {} + for covered, state in states.items(): + accumulated_cost, subsets = state + remaining = full_mask ^ covered + subset = remaining + while True: + if not (robot_id == last_robot and subset == 0) and subset in table: + route_cost = table[subset][0] + new_covered = covered | subset + candidate = ( + accumulated_cost + route_cost, + subsets + (subset,), + ) + previous = next_states.get(new_covered) + if previous is None or candidate < previous: + next_states[new_covered] = candidate + if subset == 0: + break + subset = (subset - 1) & remaining + states = next_states + + if full_mask in states: + cost, subsets = states[full_mask] + key = (cost, last_robot, subsets) + if best_key is None or key < best_key: + best_key = key + best_payload = (last_robot, subsets) + + if best_payload is None: + raise ValueError('No capacity-feasible reference partition exists') + + last_robot, subsets = best_payload + sequences = {} + for index, robot_id in enumerate(robot_ids): + table = open_tables[robot_id] if robot_id == last_robot else closed_tables[robot_id] + sequences[robot_id] = list(table[subsets[index]][1]) + return sequences, last_robot + + +def solve_random(instance): + seed = 0 + robot_ids = sorted(robot['id'] for robot in instance['robots']) + order_ids = sorted(order['id'] for order in instance['orders']) + rng = random.Random(_stable_seed(seed, instance['instance_id'])) + rng.shuffle(robot_ids) + rng.shuffle(order_ids) + sequences = _greedy_sequences(instance, order_ids, robot_ids) + return _build_serial_solution(instance, sequences, robot_ids, None) + + +def solve_baseline(instance): + robot_ids = sorted(robot['id'] for robot in instance['robots']) + order_ids = sorted(order['id'] for order in instance['orders']) + sequences = _greedy_sequences(instance, order_ids, robot_ids) + return _build_serial_solution(instance, sequences, robot_ids, None) + + +def solve_reference(instance): + robot_ids = sorted(robot['id'] for robot in instance['robots']) + order_ids = sorted(order['id'] for order in instance['orders']) + baseline_sequences = _greedy_sequences(instance, order_ids, robot_ids) + candidates = [] + + baseline = _build_serial_solution(instance, baseline_sequences, robot_ids, None) + if validate_solution(instance, baseline)[0]: + candidates.append(baseline) + + used_robots = [robot_id for robot_id in robot_ids if baseline_sequences[robot_id]] + for last_robot in used_robots: + schedule = [robot_id for robot_id in robot_ids if robot_id != last_robot] + schedule.append(last_robot) + candidate = _build_serial_solution( + instance, baseline_sequences, schedule, last_robot + ) + if validate_solution(instance, candidate)[0]: + candidates.append(candidate) + + if len(order_ids) <= 10: + try: + exact_sequences, last_robot = _exact_reference_sequences(instance) + schedule = [robot_id for robot_id in robot_ids if robot_id != last_robot] + schedule.append(last_robot) + candidate = _build_serial_solution( + instance, exact_sequences, schedule, last_robot + ) + if validate_solution(instance, candidate)[0]: + candidates.append(candidate) + except ValueError: + pass + + if not candidates: + return baseline + return min(candidates, key=lambda solution: (_move_distance(solution), repr(solution))) + + +def validate_solution(instance, solution): + if type(solution) is not dict or set(solution) != {'robots'}: + return False, 'Solution must be an object containing only robots' + submitted_robots = solution['robots'] + if type(submitted_robots) is not list: + return False, 'robots must be an array' + + robot_map = {robot['id']: robot for robot in instance['robots']} + order_map = {order['id']: order for order in instance['orders']} + node_set = set(instance['graph']['nodes']) + horizon = instance['horizon'] + expected_robot_ids = set(robot_map) + entries = {} + + for index, entry in enumerate(submitted_robots): + if type(entry) is not dict or set(entry) != {'robot_id', 'path', 'actions'}: + return False, 'Each robot entry must contain only robot_id, path, and actions' + robot_id = entry['robot_id'] + if type(robot_id) is not str or robot_id not in robot_map: + return False, 'Unknown robot at output index ' + str(index) + if robot_id in entries: + return False, 'Duplicate robot ' + robot_id + + path = entry['path'] + if type(path) is not list or len(path) != horizon + 1: + return False, 'Robot ' + robot_id + ' path must have horizon + 1 nodes' + for time, node in enumerate(path): + if type(node) is not str or node not in node_set: + return False, 'Robot ' + robot_id + ' uses an unknown node at time ' + str(time) + + actions = entry['actions'] + if type(actions) is not list: + return False, 'Robot ' + robot_id + ' actions must be an array' + actions_by_time = {} + for action_index, action in enumerate(actions): + if type(action) is not dict or set(action) != {'time', 'type', 'order_id'}: + return False, 'Malformed action for robot ' + robot_id + time = action['time'] + action_type = action['type'] + order_id = action['order_id'] + if type(time) is not int or time < 0 or time > horizon: + return False, 'Action time is outside the horizon for robot ' + robot_id + if action_type not in ('pickup', 'dropoff'): + return False, 'Unknown action type for robot ' + robot_id + if type(order_id) is not str or order_id not in order_map: + return False, 'Unknown order in action ' + str(action_index) + if time in actions_by_time: + return False, 'Robot ' + robot_id + ' performs multiple actions at one time' + actions_by_time[time] = action + entries[robot_id] = (path, actions_by_time) + + if set(entries) != expected_robot_ids: + missing = sorted(expected_robot_ids - set(entries)) + return False, 'Missing robots: ' + ','.join(missing) + + directed_edges = set() + for left, right in instance['graph']['edges']: + directed_edges.add((left, right)) + directed_edges.add((right, left)) + + robot_ids = sorted(robot_map) + for robot_id in robot_ids: + path = entries[robot_id][0] + if path[0] != robot_map[robot_id]['start_node']: + return False, 'Robot ' + robot_id + ' starts at the wrong node' + for time in range(1, horizon + 1): + if path[time] != path[time - 1] and (path[time - 1], path[time]) not in directed_edges: + return False, 'Robot ' + robot_id + ' makes an illegal move at time ' + str(time) + + for time in range(horizon + 1): + occupied = {} + for robot_id in robot_ids: + node = entries[robot_id][0][time] + if node in occupied: + return False, 'Vertex collision at time ' + str(time) + occupied[node] = robot_id + + for time in range(1, horizon + 1): + for left_index in range(len(robot_ids)): + left_id = robot_ids[left_index] + left_path = entries[left_id][0] + for right_index in range(left_index + 1, len(robot_ids)): + right_id = robot_ids[right_index] + right_path = entries[right_id][0] + if ( + left_path[time - 1] == right_path[time] + and right_path[time - 1] == left_path[time] + and left_path[time - 1] != left_path[time] + ): + return False, 'Opposite-direction edge collision at time ' + str(time) + + picked = {} + dropped = {} + for robot_id in robot_ids: + path, actions_by_time = entries[robot_id] + capacity = robot_map[robot_id]['capacity'] + carried = set() + load = 0 + for time in range(horizon + 1): + action = actions_by_time.get(time) + if action is None: + continue + order_id = action['order_id'] + order = order_map[order_id] + if action['type'] == 'pickup': + if order_id in picked: + return False, 'Order ' + order_id + ' is picked up more than once' + if path[time] != order['pickup_node']: + return False, 'Order ' + order_id + ' is picked up at the wrong node' + load += order['weight'] + if load > capacity: + return False, 'Robot ' + robot_id + ' exceeds its capacity' + carried.add(order_id) + picked[order_id] = (robot_id, time) + else: + if order_id not in carried: + return False, 'Robot ' + robot_id + ' drops an order it is not carrying' + if order_id in dropped: + return False, 'Order ' + order_id + ' is dropped more than once' + if path[time] != order['dropoff_node']: + return False, 'Order ' + order_id + ' is dropped at the wrong node' + pickup_robot, pickup_time = picked[order_id] + if pickup_robot != robot_id or time <= pickup_time: + return False, 'Order ' + order_id + ' has an invalid pickup/dropoff order' + carried.remove(order_id) + load -= order['weight'] + if load < 0: + return False, 'Robot ' + robot_id + ' has a negative load' + dropped[order_id] = (robot_id, time) + if load != 0 or carried: + return False, 'Robot ' + robot_id + ' finishes with undelivered cargo' + + expected_orders = set(order_map) + if set(picked) != expected_orders: + return False, 'Not every order is picked up exactly once' + if set(dropped) != expected_orders: + return False, 'Not every order is dropped off exactly once' + return True, 'ok' + + +def _move_distance(solution): + total = 0 + for entry in solution['robots']: + path = entry['path'] + total += sum(path[index] != path[index - 1] for index in range(1, len(path))) + return total + + +def _completion_time(solution): + times = [ + action['time'] + for entry in solution['robots'] + for action in entry['actions'] + if action['type'] == 'dropoff' + ] + return max(times) if times else 0 + + +def evaluate_solution(instance, solution): + valid, message = validate_solution(instance, solution) + if not valid: + raise ValueError(message) + return _move_distance(solution) + 1 + + +def _make_instance(seed, index): + mixed_seed = _stable_seed(seed, 'warehouse-routing-' + str(index)) + rng = random.Random(mixed_seed) + width = 5 + rng.randrange(3) + height = 5 + rng.randrange(3) + robot_count = 2 + ((index + rng.randrange(3)) % 3) + order_count = 5 + index + + def grid_node(x, y): + return 'n' + str(x) + '_' + str(y) + + grid_nodes = [grid_node(x, y) for x in range(width) for y in range(height)] + edge_set = set() + + def add_edge(left, right): + if left != right: + edge_set.add(tuple(sorted((left, right)))) + + for x in range(width): + for y in range(height - 1): + add_edge(grid_node(x, y), grid_node(x, y + 1)) + + cross_rows = {0, height // 2, height - 1} + for y in sorted(cross_rows): + for x in range(width - 1): + add_edge(grid_node(x, y), grid_node(x + 1, y)) + + anchor_candidates = [ + grid_node(0, 0), + grid_node(width - 1, 0), + grid_node(0, height - 1), + grid_node(width - 1, height - 1), + grid_node(width // 2, 0), + ] + anchors = anchor_candidates[:robot_count] + parking_nodes = ['park_' + str(index) + '_' + str(i) for i in range(robot_count)] + for parking, anchor in zip(parking_nodes, anchors): + add_edge(parking, anchor) + + capacities = [4 + rng.randrange(4) for _ in range(robot_count)] + robots = [ + { + 'id': 'r' + str(i).zfill(2), + 'start_node': parking_nodes[i], + 'capacity': capacities[i], + } + for i in range(robot_count) + ] + + service_nodes = [node for node in grid_nodes if node not in set(anchors)] + orders = [] + maximum_weight = min(capacities) + for order_index in range(order_count): + pickup = service_nodes[rng.randrange(len(service_nodes))] + dropoff = service_nodes[rng.randrange(len(service_nodes))] + while dropoff == pickup: + dropoff = service_nodes[rng.randrange(len(service_nodes))] + orders.append( + { + 'id': 'o' + str(order_index).zfill(2), + 'pickup_node': pickup, + 'dropoff_node': dropoff, + 'weight': 1 + rng.randrange(maximum_weight), + } + ) + + nodes = sorted(grid_nodes + parking_nodes) + edges = [list(edge) for edge in sorted(edge_set)] + node_count = len(nodes) + horizon = ( + (2 * order_count + robot_count + 1) * (node_count - 1) + + 2 * order_count + + 10 + ) + return { + 'instance_id': 'warehouse-routing-' + str(seed) + '-' + str(index), + 'graph': {'nodes': nodes, 'edges': edges}, + 'robots': robots, + 'orders': orders, + 'horizon': horizon, + } + + +def generate_instances(seed): + if type(seed) is not int: + raise TypeError('seed must be an integer') + instances = [] + for index in range(2): + instance = _make_instance(seed, index) + instances.append(instance) + return instances diff --git a/benchmarks/Robotics/WarehouseRobotRouting/verification/process_runner.py b/benchmarks/Robotics/WarehouseRobotRouting/verification/process_runner.py new file mode 100644 index 00000000..3498497b --- /dev/null +++ b/benchmarks/Robotics/WarehouseRobotRouting/verification/process_runner.py @@ -0,0 +1,90 @@ +from __future__ import annotations + +import os +import signal +import subprocess +import threading +from dataclasses import dataclass + + +@dataclass(frozen=True) +class BoundedResult: + returncode: int + stdout: str + stderr: str + timed_out: bool + output_truncated: bool + + +def run_bounded( + command: list[str], input_text: str, *, timeout_s: float, max_output_bytes: int +) -> BoundedResult: + process = subprocess.Popen( + command, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + start_new_session=True, + ) + stdout = bytearray() + stderr = bytearray() + truncated = [False, False] + readers = [ + threading.Thread( + target=_drain, + args=(process.stdout, stdout, max_output_bytes, truncated, 0), + daemon=True, + ), + threading.Thread( + target=_drain, + args=(process.stderr, stderr, max_output_bytes, truncated, 1), + daemon=True, + ), + ] + for reader in readers: + reader.start() + writer = threading.Thread(target=_write, args=(process, input_text.encode("utf-8")), daemon=True) + writer.start() + timed_out = False + try: + process.wait(timeout=timeout_s) + except subprocess.TimeoutExpired: + timed_out = True + try: + os.killpg(process.pid, signal.SIGKILL) + except (ProcessLookupError, PermissionError, OSError): + process.kill() + process.wait() + for reader in readers: + reader.join() + return BoundedResult( + returncode=process.returncode, + stdout=stdout.decode("utf-8", errors="replace"), + stderr=stderr.decode("utf-8", errors="replace"), + timed_out=timed_out, + output_truncated=any(truncated), + ) + + +def _drain(stream, target: bytearray, limit: int, truncated: list[bool], index: int) -> None: + if stream is None: + return + while True: + chunk = stream.read(64 * 1024) + if not chunk: + return + remaining = limit - len(target) + if remaining > 0: + target.extend(chunk[:remaining]) + if len(chunk) > remaining: + truncated[index] = True + + +def _write(process: subprocess.Popen[bytes], content: bytes) -> None: + if process.stdin is None: + return + try: + process.stdin.write(content) + process.stdin.close() + except (BrokenPipeError, OSError): + pass