feat(WaterDistribution): add PumpScheduling benchmark

[Yunn1633]

Background

This task optimizes a causal closed-loop pump controller for a water-distribution network. The evaluator runs multiple demand and tariff scenarios and checks pressure, tank-level, terminal-storage, and control constraints.

Source

The hydraulic model is based on EPA Net3 and is evaluated with WNTR 1.4.0 and EPANET. Scenario definitions, third-party notices, and provenance are included under references/ and THIRD_PARTY_NOTICES.md.

Validation

From the benchmark directory:

../../../.venvs/frontier-wntr/bin/python verification/evaluator.py scripts/init.py --json-out metrics.json --artifacts-out artifacts.json

From the repository root:

python -m frontier_eval task=unified task.benchmark=WaterDistribution/PumpScheduling task.runtime.python_path=uv-env:frontier-wntr algorithm=openevolve algorithm.iterations=0

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new benchmark task, WaterDistribution/PumpScheduling, designed to evaluate closed-loop control algorithms for water utility pump management. It utilizes the EPANET engine via the WNTR library to simulate hydraulic behavior under varying demand and electricity tariffs.
• Modified File Structure & Modifications:
  • benchmarks/WaterDistribution/PumpScheduling/README.md & README_zh-CN.md: Documentation for environment setup and evaluation commands.
  • benchmarks/WaterDistribution/PumpScheduling/Task.md & Task_zh-CN.md: Detailed task description, physical model constraints, and scoring metrics.
  • benchmarks/WaterDistribution/PumpScheduling/frontier_eval/: Metadata files for integration with the frontier_eval framework (e.g., agent_files.txt, eval_command.txt).
  • benchmarks/WaterDistribution/PumpScheduling/references/: Static data including public scenarios, JSON schemas, and service node definitions.
  • benchmarks/WaterDistribution/PumpScheduling/scripts/init.py: The baseline/initial program for the agent to modify.
  • benchmarks/WaterDistribution/PumpScheduling/verification/: Core evaluation logic, including rollout.py (simulation wrapper), scoring.py (metric calculation), and controller_worker.py (process isolation).

2. AI Content Analysis

• Estimated AI Component: 15%
• Reasoning & Evidence: The code exhibits high domain specificity and sophisticated engineering patterns. The use of selectors.DefaultSelector for non-blocking I/O in controller_worker.py and the specific integration with the ENepanet toolkit (EPANET C-API) suggest human architectural design. AI influence is likely limited to standard boilerplate in argparse setups or basic docstring structures. The logic for hydraulic rollout and the specific scoring ratios (0.55 energy, 0.20 peak, etc.) are highly customized.

3. Engineering & Economic Assessment

• Engineering Reality Check: This is a production-grade engineering problem. Unlike "toy" control tasks, it accounts for real-world complexities: electricity tariffs, hydraulic pressure constraints at specific service nodes, tank level limits, and terminal storage recovery. The inclusion of "small leaks" in the scenario schema demonstrates attention to realistic edge cases.
• Economic Value: High. Pump operation is one of the largest electricity costs for water utilities. An algorithm that optimizes for both energy cost and peak power while maintaining service levels has direct industrial application and cost-reduction potential.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: WaterDistribution/PumpScheduling
  • Execution & Dependencies: The README.md clearly documents the creation of a virtual environment and the installation of dependencies via benchmarks/WaterDistribution/PumpScheduling/verification/requirements.txt. It provides specific commands for both direct evaluation and unified baseline evaluation.
• Documentation Quality: High. The documentation is bilingual (EN/ZH), provides a clear input/output table for the control function, and explicitly defines hard constraints and scoring formulas. No significant grammatical errors or formatting inconsistencies were detected.
• Organizational Structure: The structure is logical and modular. It separates simulation logic (rollout.py), scoring logic (scoring.py), and the isolation layer (controller_worker.py), which facilitates maintenance and scalability.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were found.
• Absolute Paths: None detected. The code uses pathlib and relative path resolution (e.g., Path(__file__).parents[1]) to ensure portability.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个新的基准测试任务 WaterDistribution/PumpScheduling，旨在评估配水系统泵站管理的闭环控制算法。它通过 WNTR 库调用 EPANET 引擎，模拟不同需求和电价下的水力行为。
• 修改的文件结构与变更摘要:
  • benchmarks/WaterDistribution/PumpScheduling/README.md & README_zh-CN.md: 环境配置和评测命令文档。
  • benchmarks/WaterDistribution/PumpScheduling/Task.md & Task_zh-CN.md: 任务详细说明、物理模型约束及评分指标。
  • benchmarks/WaterDistribution/PumpScheduling/frontier_eval/: 用于集成到 frontier_eval 框架的元数据文件（如 agent_files.txt, eval_command.txt）。
  • benchmarks/WaterDistribution/PumpScheduling/references/: 静态数据，包括公开场景、JSON Schema 和服务节点定义。
  • benchmarks/WaterDistribution/PumpScheduling/scripts/init.py: 供智能体修改的基线/初始程序。
  • benchmarks/WaterDistribution/PumpScheduling/verification/: 核心评测逻辑，包含 rollout.py（仿真封装）、scoring.py（指标计算）和 controller_worker.py（进程隔离）。

2. AI 成分分析

• 预估 AI 含量: 15%
• 判断依据与证据: 代码表现出极高的领域专业性和复杂的工程模式。在 controller_worker.py 中使用 selectors.DefaultSelector 进行非阻塞 I/O，以及与 ENepanet 工具包（EPANET C-API）的深度集成，表明了人工架构设计。AI 的影响可能仅限于 argparse 设置或基础文档字符串的模板化生成。水力仿真逻辑和特定的评分权重（0.55 能源、0.20 峰值等）具有高度定制化特征。

3. 工程与经济评估

• 工程现实检验: 这是一个生产级别的工程问题。与“玩具型”控制任务不同，它考虑了现实世界的复杂性：电价政策、特定服务节点的水力压力约束、水箱水位限制以及终态储水恢复。场景架构中包含“小型泄漏”的设计体现了对现实边缘情况的关注。
• 经济价值: 高。泵站运行是水务公司最大的电力支出之一。在保证服务水平的同时优化能源成本和峰值功率的算法具有直接的工业应用价值和成本优化潜力。

4. Quality Assurance

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: WaterDistribution/PumpScheduling
  • 运行与依赖: README.md 清晰地记录了虚拟环境的创建以及通过 verification/requirements.txt 安装依赖的步骤。提供了直接评测和统一基线评测的具体命令。
• 文档质量: 高。文档采用双语（中英）编写，为 control 函数提供了清晰的输入/输出表，并明确定义了硬约束和评分公式。未发现明显的语法错误或格式不一致。
• 组织结构: 文件组织逻辑清晰且模块化。将仿真逻辑 (rollout.py)、评分逻辑 (scoring.py) 和隔离层 (controller_worker.py) 分离，有利于维护和扩展。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未提交 .env、API 密钥或 IDE 配置文件。
• 绝对路径: 未检测到。代码使用 pathlib 和相对路径解析（如 Path(__file__).parents[1]）以确保可移植性。

[wrh-human]

Review — PumpScheduling (PR #91)

Thank you for contributing this benchmark. Water distribution network pump scheduling is a real engineering problem, and using WNTR/EPANET as the hydraulic simulation model is a reasonable choice. The overall direction is solid, but several issues need to be addressed before merging.

---

1. Domain, Economic Value, and Frontier-Eng Fit ✅

Water distribution network pump scheduling is a genuine municipal engineering problem — water utilities must minimize pumping energy costs while maintaining service pressure and storage reserves. The task uses EPANET / WNTR 1.4.0 developed by the US EPA for hydraulic simulation, which is an industry-standard tool. Optimizing pump scheduling directly translates to reduced electricity costs and decreased mechanical wear, giving the task clear economic value.

2. Not purely numerical ✅

The agent writes a closed-loop control policy (control(observation)) that outputs pump speeds based on causal observations such as tank levels, electricity tariffs, and demand multipliers. The policy code must reflect an understanding of hydraulic system behavior (e.g., off-peak replenishment, tank level management, leak response), rather than outputting static numerical values.

3. Search space ✅

2 pumps × 24 hours = 48 decision points per scenario, each pump speed continuous in [0, 1]. Since the control policy is causal (decisions are based on tank levels, tariffs, and other state), and is subject to hydraulic constraints (pressure ≥ 20m, tank bounds, terminal storage requirements), brute-force search is infeasible.

4. Evaluator and engineering verification ✅

The evaluator uses WNTR/EPANET for continuous 24-hour hydraulic simulation (5-minute steps), with verification including:

• Pressure ≥ 20m at all service nodes
• Tank levels within bounds
• Terminal levels no more than 0.25m below initial
• EPANET convergence
• Candidate isolation via subprocess + JSON communication
• controller_worker.py restricts candidate builtins via SAFE_BUILTINS
• 3 public scenarios + 3 hidden scenarios (parameter shifts + leak injection)

5. Constraints and verifier enforcement ✅

Pressure constraints, tank bounds, and terminal storage constraints are checked at every hydraulic time step in rollout(), with violations raised as exceptions. Output shape, pump speed range, and finite value checks are thorough. Hidden scenarios generated via parameter shifts and leak injection help detect overfitting.

6. Baseline experiment ❌ Missing

There is no baseline/ directory and no result_log.json. The baseline reference code lives in verification/baseline.py, but no actual run results are recorded — no energy cost, peak power, switching count, or other metrics are provided.

---

Issues to address

Issue 1 (most critical): Missing EVOLVE-BLOCK boundary validation

Neither the evaluator nor controller_worker.py validates the EVOLVE-BLOCK boundary. constraints.txt explicitly requires "Modify only code between EVOLVE-BLOCK-START and EVOLVE-BLOCK-END", but the evaluator never enforces this. An agent could modify the control() function signature, add code outside the EVOLVE-BLOCK, or modify/remove the EVOLVE markers themselves without triggering any checks.

Furthermore, scripts/init.py and verification/baseline.py have already diverged — init.py contains EVOLVE markers and comments, while baseline.py is a simplified version. The evaluator loads both files separately but never validates that init.py's EVOLVE-BLOCK shell matches baseline.py.

Suggestion: add EVOLVE-BLOCK boundary validation before importing the candidate source, and verify that code outside the EVOLVE-BLOCK matches the baseline.

Issue 2: Missing baseline run results

There is no baseline/result_log.json or baseline/ directory. verification/baseline.py is only a code file with no attached run results (energy cost, peak power, switching count, terminal deficit, etc.). Other benchmarks all include a baseline/ directory with run result records.

Suggestion: create a baseline/ directory recording the output of python verification/evaluator.py scripts/init.py.

Issue 3: Candidate subprocess lacks resource isolation

IsolatedController uses subprocess.Popen() to launch the candidate without any resource limits:

• No RLIMIT_NPROC
• No RLIMIT_AS
• No RLIMIT_CPU
• No start_new_session=True
• No preexec_fn

While controller_worker.py applies some Python-level code isolation via SAFE_BUILTINS, there are no physical resource protections.

Suggestion: add preexec_fn to subprocess.Popen with RLIMIT_NPROC, RLIMIT_AS, and RLIMIT_CPU, and use start_new_session=True.

Issue 4: Scoring discrimination lacks empirical support

The scoring formula's weight distribution (energy 55%, peak power 20%, switching 10%, terminal deficit 15%) is reasonable from an engineering perspective, but has the following structural issues:

• Uneven ratio sensitivity: The baseline / candidate ratio means that if a baseline component is very small (e.g., switching count of only 1.2), even a tiny absolute improvement by the candidate pushes the ratio to the 1.5 cap, beyond which further optimization yields no score gain. Conversely, if a baseline component is very large, candidate improvements are barely reflected in the ratio. Sensitivity varies significantly across baseline components.
• 1.5 cap produces a narrow score range: All ratios are hard-clipped to [0, 1.5]. The theoretical maximum scenario score is 150, and combined_score ranges from [1.0, 1.5] — a narrower band compared to other benchmarks.
• 0.01 floor for switching and terminal_deficit_m creates nonlinearity: Both metrics are floored via max(value, 0.01) in rollout(). If the baseline's switching count is already close to 0.01, the candidate has almost no room for improvement; if the baseline switching count is larger (e.g., 20), the candidate has a meaningful range for differentiation.

The scoring formula's theoretical framework is reasonable, but without a baseline/result_log.json recording each scenario's actual baseline metrics (energy cost, peak power, switching count, terminal deficit), it is impossible to verify whether the scoring function can effectively differentiate candidates of varying quality under real baseline values. Fixing Issue 2 (adding baseline run results) will simultaneously address this concern.

Issue 5: scripts/init.py and verification/baseline.py are separate but lack synchronization

The control logic in both files is identical, but init.py has EVOLVE-BLOCK markers and comments while baseline.py does not. If the baseline policy is updated, both files must be modified in sync, otherwise the combined_score reference baseline will diverge from the actual baseline. Suggestion: use a single source file reference, or add an automated check to ensure logical consistency between the two files.

---

Non-blocking suggestions

1. The call_timeout_s=1.0 in rollout() is hardcoded. Consider reading it from farm_config.json for consistency with other configuration.
2. Hidden scenario leak parameters could benefit from more diversity (different nodes, time windows, and leak areas).
3. WNTR is multi-threaded (numpy/scipy). Consider setting OMP_NUM_THREADS=1 and MKL_NUM_THREADS=1 in the evaluator to avoid simulation time variance from parallelism.

---

Summary

The core direction is sound, and using WNTR/EPANET as the hydraulic simulation model is a strength. The 6-scenario design (including hidden scenarios) helps generalization. Issues 1 (missing EVOLVE-BLOCK validation), 2 (missing baseline run results), and 4 (scoring discrimination lacking empirical data support) are the highest priority. Ready to merge once the above issues are addressed.