Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions benchmarks/PowerSystems/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,8 @@ Current tasks emphasize realistic operational constraints, economic objectives,
- Unified benchmark: `task=unified task.benchmark=PowerSystems/EV2GymSmartCharging`
- Quick run: `python -m frontier_eval task=unified task.benchmark=PowerSystems/EV2GymSmartCharging task.runtime.env_name=frontier-eval-driver algorithm.iterations=0`
- Description: upstream-aligned EV smart charging with transformer constraints in the real `EV2Gym` simulator

- `SecurityConstrainedDispatch`
- Unified benchmark: `task=unified task.benchmark=PowerSystems/SecurityConstrainedDispatch`
- Runtime: install the pinned packages in `verification/requirements.txt` and set `task.runtime.python_path` to that environment
- Description: cost-and-security dispatch across normal and N-1 PGLib operating scenarios with frozen AC power-flow verification
5 changes: 5 additions & 0 deletions benchmarks/PowerSystems/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,8 @@
- `frontier_eval` 任务:`task=unified task.benchmark=PowerSystems/EV2GymSmartCharging`
- 快速运行:`python -m frontier_eval task=unified task.benchmark=PowerSystems/EV2GymSmartCharging task.runtime.env_name=frontier-eval-driver algorithm.iterations=0`
- 简介:在真实上游 `EV2Gym` 模拟器中进行、与上游数据对齐的 EV 智能充电与变压器约束优化

- `SecurityConstrainedDispatch`
- `frontier_eval` 任务:`task=unified task.benchmark=PowerSystems/SecurityConstrainedDispatch`
- 运行环境:安装 `verification/requirements.txt` 中的固定依赖,并通过 `task.runtime.python_path` 指定解释器
- 简介:在正常与 N-1 PGLib 场景中优化成本和安全裕度,并由冻结交流潮流验证器评分
92 changes: 92 additions & 0 deletions benchmarks/PowerSystems/SecurityConstrainedDispatch/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Security-Constrained Dispatch

This benchmark asks an agent to improve generator active-power and voltage
setpoints across normal and N-1 operating scenarios derived from PGLib-OPF.
Every candidate is checked by an independent AC power-flow verifier.

## Benchmark ID

```text
PowerSystems/SecurityConstrainedDispatch
```

## Data provenance

The bundled MATPOWER cases come from PGLib-OPF release `v23.07`, commit
`dc6be4b2f85ca0e776952ec22cbd4c22396ea5a3`:

- `pglib_opf_case24_ieee_rts.m`
- `pglib_opf_case57_ieee.m`
- `pglib_opf_case73_ieee_rts.m`

PGLib-OPF is maintained by the IEEE PES Task Force on Benchmarks for Validation
of Emerging Power System Algorithms. See `references/pglib/LICENSE` and
`references/PROVENANCE.md`.

## Environment

Use Python 3.12 on Linux. Install the pinned dependencies:

```bash
python3.12 -m venv .venvs/frontier-pglib-scd
.venvs/frontier-pglib-scd/bin/python -m pip install \
-r benchmarks/PowerSystems/SecurityConstrainedDispatch/verification/requirements.txt
```

The evaluator is CPU-only and requires Linux seccomp support. Candidate code is
run in a restricted worker process; Docker and Linux namespaces are not used.
The worker blocks file access, process creation, and networking from `solve()`.
Because namespace isolation is unavailable on the target runtime, this is not
claimed to be container-equivalent isolation. Evaluation fails closed if the
seccomp filter cannot be installed.

A typical baseline evaluation of all 12 scenarios uses less than 2 GB RAM and
should complete within seconds on a modern CPU. Candidate import has a separate
10-second infrastructure timeout. Each `solve()` call retains its full
contract-defined two-second budget.

## Direct evaluation

From the benchmark directory:

```bash
../../../.venvs/frontier-pglib-scd/bin/python \
verification/evaluator.py scripts/init.py \
--metrics-out metrics.json \
--artifacts-out artifacts.json
```

## Frontier Eval

From the repository root, use the repository-owned task runtime:

```bash
python -m frontier_eval \
task=unified \
task.benchmark=PowerSystems/SecurityConstrainedDispatch \
task.runtime.python_path=uv-env:frontier-pglib-scd \
algorithm=openevolve \
algorithm.iterations=0
```

Baseline-only validation does not require a model API key.

## Isolation tests

From the benchmark directory:

```bash
../../../.venvs/frontier-pglib-scd/bin/python -m pytest -q \
verification/test_candidate_isolation.py
```

## Layout

- `scripts/init.py`: editable dispatch policy.
- `verification/`: frozen AC power-flow evaluator.
- `references/pglib/`: version-pinned PGLib source cases and license.
- `references/scenarios.json`: deterministic scenario baselines and reference costs.
- `references/build_scenarios.py`: reproducible scenario-generation script.
- `frontier_eval/`: unified-task metadata.

See `Task.md` for the complete contract and scoring definition.
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# 安全约束电力调度

本任务要求 Agent 改进发电机有功功率和电压设定值,在由 PGLib-OPF
构建的正常及 N-1 故障场景中降低发电成本。冻结验证器使用独立交流潮流
检查每个候选方案。

## Benchmark ID

```text
PowerSystems/SecurityConstrainedDispatch
```

## 数据来源

任务内置的 MATPOWER 案例来自 PGLib-OPF `v23.07`,提交:
`dc6be4b2f85ca0e776952ec22cbd4c22396ea5a3`。

- `pglib_opf_case24_ieee_rts.m`
- `pglib_opf_case57_ieee.m`
- `pglib_opf_case73_ieee_rts.m`

许可证和文件哈希见 `references/pglib/LICENSE` 与
`references/PROVENANCE.md`。

## 环境

推荐 Linux 与 Python 3.12。本任务仅使用 CPU:

```bash
python -m venv .venv-pglib-scd
source .venv-pglib-scd/bin/activate
python -m pip install -r benchmarks/PowerSystems/SecurityConstrainedDispatch/verification/requirements.txt
```

评测器仅使用 CPU,并要求 Linux seccomp 支持。候选程序在受限 worker
进程中运行,不使用 Docker 或 Linux namespace。worker 会禁止 `solve()`
读取或写入文件、创建进程和访问网络;如果 seccomp 过滤器无法安装,评测将
直接失败,不会降级为无保护执行。由于运行环境不提供 namespace 隔离,本方案
不宣称具备与容器完全等价的安全边界。

候选模块加载使用独立的 10 秒基础设施超时;每次 `solve()` 仍拥有任务合同
规定的完整 2 秒预算,模块加载时间不会占用该预算。

完整12个场景的 baseline 评测通常只需数秒,内存低于2 GB。

## 直接评测

在任务目录执行:

```bash
python verification/evaluator.py scripts/init.py \
--metrics-out metrics.json \
--artifacts-out artifacts.json
```

## Frontier Eval

在仓库根目录执行,并传入任务环境的 Python:

```bash
python -m frontier_eval \
task=unified \
task.benchmark=PowerSystems/SecurityConstrainedDispatch \
task.runtime.python_path=/absolute/path/to/.venv-pglib-scd/bin/python \
algorithm=openevolve \
algorithm.iterations=0
```

若环境位于 `.venvs/frontier-pglib-scd`,可以改用:

```text
task.runtime.python_path=uv-env:frontier-pglib-scd
```

完整任务契约和评分方法见 `Task_zh-CN.md`。
152 changes: 152 additions & 0 deletions benchmarks/PowerSystems/SecurityConstrainedDispatch/Task.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
# Task: Security-Constrained Dispatch

## Background

Power-system operators must dispatch generators economically while respecting
network physics and remaining feasible after credible component outages. A
dispatch that is cheap in the intact network may overload a transmission line,
violate voltage limits, or become infeasible after a line outage.

This benchmark uses three PGLib-OPF systems and twelve deterministic operating
scenarios. Scenarios combine normal operation, load variation, and selected N-1
line outages that were verified to admit AC-OPF solutions.

## Editable program

Edit only the logic between `EVOLVE-BLOCK-START` and `EVOLVE-BLOCK-END` in
`scripts/init.py`. Preserve:

```python
def solve(case: dict) -> dict:
...
```

The function is called once per scenario. It must return:

```python
{
"pg_mw": [generator active-power setpoints],
"vg_pu": [generator voltage-magnitude setpoints],
}
```

Both lists must contain exactly one finite value per generator, in the order
given by `case["generators"]`. All online generators at the same bus must use
the same voltage setpoint.

## Input

The `case` dictionary has the following exact top-level fields:

- `case_id: str`: PGLib system identifier.
- `scenario_id: str`: deterministic operating-scenario identifier.
- `base_mva: float`: system power base in MVA.
- `load_scale: float`: demand multiplier applied to the source case.
- `outage_branch: int | None`: zero-based source-case branch index removed in
this scenario, or `None` for an intact network.
- `total_active_load_mw: float`: sum of active bus demand after scaling.
- `buses: list[dict]`: buses in MATPOWER row order.
- `generators: list[dict]`: generators in MATPOWER row order. Output arrays
must use this same order.
- `branches: list[dict]`: branches in MATPOWER row order after applying the
scenario outage.

Each entry in `case["buses"]` contains:

- `bus_id: int`, `type: int`;
- `pd_mw: float`, `qd_mvar: float`;
- `base_kv: float`;
- `vmin_pu: float`, `vmax_pu: float`.

Each entry in `case["generators"]` contains:

- `index: int`, `bus_id: int`, `online: bool`;
- `pmin_mw: float`, `pmax_mw: float`;
- `qmin_mvar: float`, `qmax_mvar: float`;
- `baseline_pg_mw: float`, `baseline_vg_pu: float`;
- `cost_model: list[float]`: the unmodified MATPOWER `gencost` row.

`cost_model` uses one of the MATPOWER encodings below. Polynomial rows are
`[2, startup, shutdown, n, c_(n-1), ..., c_0]`, representing coefficients in
descending power order. Piecewise-linear rows are
`[1, startup, shutdown, n, x_1, y_1, ..., x_n, y_n]`.

Each entry in `case["branches"]` contains:

- `index: int`, `from_bus: int`, `to_bus: int`;
- `resistance_pu: float`, `reactance_pu: float`, `charging_pu: float`;
- `rate_a_mva: float` (`0` means no active `RATE_A` limit);
- `tap_ratio: float`, `phase_shift_deg: float`;
- `in_service: bool`;
- `angle_min_deg: float`, `angle_max_deg: float`.

The baseline setpoints are feasible but deliberately non-optimal. They are a
starting point, not reference answers.

## Hard constraints

A candidate is valid only if every scenario satisfies all of the following:

1. AC Newton power flow converges.
2. Generator active and reactive powers remain within limits.
3. Bus voltage magnitudes remain within PGLib limits.
4. Apparent power at both ends of every rated in-service branch stays within
`RATE_A`.
5. Branch angle differences remain within `ANGMIN` and `ANGMAX`.
6. Output is finite, has the exact required shape, and returns within two seconds
per scenario.

Any hard violation makes the overall candidate invalid and sets
`combined_score` to zero.

## Scenarios

The benchmark evaluates three systems:

- IEEE RTS 24-bus;
- IEEE 57-bus;
- IEEE RTS 73-bus.

Each system has four scenarios:

- 98% load, intact network;
- 100% load, intact network;
- 100% load, selected line outage A;
- 102% load, selected line outage B.

The selected outages and fixed reference values are stored in the readonly
scenario manifest.

## Score

For each feasible scenario, three normalized components are computed:

```text
cost_efficiency = min(1.05, reference_AC_OPF_cost / candidate_cost)
thermal_margin = clip((100 - max_branch_loading_percent) / 20, 0, 1)
voltage_margin = 10th percentile normalized distance from voltage limits

scenario_score = 70 * cost_efficiency
+ 20 * thermal_margin
+ 10 * voltage_margin
```

The aggregate score emphasizes both average and worst-case behavior:

```text
combined_score = 0.75 * mean(scenario_score)
+ 0.25 * min(scenario_score)
```

Higher is better. The cost term rewards proximity to the frozen AC-OPF
references; the two margin terms reward dispatches that do not merely sit on
thermal or voltage limits. The cost component is capped to limit sensitivity to
numerical solver differences.

## Prohibited behavior

- Do not modify the evaluator, PGLib cases, scenario manifest, or metadata.
- Do not read or write files, launch subprocesses, access the network, or use
wall-clock-dependent behavior from `solve`.
- Do not hard-code evaluator outputs or bypass AC power flow.
- Keep the policy deterministic.
Loading
Loading