diff --git a/benchmarks/WindEnergy/README.md b/benchmarks/WindEnergy/README.md new file mode 100644 index 00000000..e6dcd037 --- /dev/null +++ b/benchmarks/WindEnergy/README.md @@ -0,0 +1,9 @@ +# Wind Energy + +Wind-energy benchmarks evaluate control and design decisions with executable, +physics-based wind-plant models. + +## Tasks + +- [`WakeSteeringControl`](WakeSteeringControl/README.md): optimize turbine yaw + control to increase weighted farm energy while limiting control effort. diff --git a/benchmarks/WindEnergy/README_zh-CN.md b/benchmarks/WindEnergy/README_zh-CN.md new file mode 100644 index 00000000..bb968aa3 --- /dev/null +++ b/benchmarks/WindEnergy/README_zh-CN.md @@ -0,0 +1,8 @@ +# 风能工程 + +风能工程任务使用可执行的风电场物理模型，评测控制与设计策略的优化能力。 + +## 任务 + +- [`WakeSteeringControl`](WakeSteeringControl/README_zh-CN.md)：优化风机偏航控制， + 在限制控制代价的同时提升风电场加权发电量。 diff --git a/benchmarks/WindEnergy/WakeSteeringControl/README.md b/benchmarks/WindEnergy/WakeSteeringControl/README.md new file mode 100644 index 00000000..844f489a --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/README.md @@ -0,0 +1,69 @@ +# Wake Steering Control + +This benchmark asks an agent to improve a deterministic yaw-control policy for +a nine-turbine wind farm. A frozen FLORIS 4.6.6 verifier evaluates the policy +over multiple wind directions and speeds. + +## Structure + +```text +scripts/init.py editable yaw policy +references/farm_config.json public farm and score contract +verification/evaluator.py frozen FLORIS evaluator +verification/run_candidate.py isolated candidate runner +frontier_eval/ unified-task metadata +baseline/result_log.json measured zero-yaw baseline +``` + +## Environment + +From the Frontier-Engineering repository root, create the task runtime. The task +is tested with Python 3.12 on Linux. + +```bash +python3.12 -m venv .venvs/frontier-wake-steering +.venvs/frontier-wake-steering/bin/python -m pip install \ + -r benchmarks/WindEnergy/WakeSteeringControl/verification/requirements.txt +``` + +The task is CPU-only, and FLORIS itself does not require a GPU. A baseline +evaluation typically completes within seconds. No external dataset, Docker, or +runtime network access is required. + +## Direct Evaluation + +From this benchmark directory: + +```bash +../../../.venvs/frontier-wake-steering/bin/python \ + verification/evaluator.py scripts/init.py +``` + +The evaluator writes `metrics.json` and `artifacts.json`. The shipped zero-yaw +policy is feasible and has a combined score of approximately zero by design. + +## Unified Evaluation + +The unified benchmark ID is `WindEnergy/WakeSteeringControl`: + +```bash +python -m frontier_eval \ + task=unified \ + task.benchmark=WindEnergy/WakeSteeringControl \ + task.runtime.python_path=uv-env:frontier-wake-steering \ + algorithm=openevolve \ + algorithm.iterations=0 +``` + +Optimization runs require the normal model API configuration used by +Frontier Eval. Baseline-only validation does not require an API key. + +## Reproducibility and Security + +- FLORIS is pinned to version 4.6.6. +- The verifier uses the built-in NREL 5 MW turbine model and a fixed farm. +- Candidate code runs in a separate subprocess with a wall-clock timeout. +- The unified task checks the verifier and public contract as read-only and + invalidates candidates that modify them. +- Process isolation is not a hardened security sandbox. Run untrusted candidate + code in a container or disposable host. diff --git a/benchmarks/WindEnergy/WakeSteeringControl/README_zh-CN.md b/benchmarks/WindEnergy/WakeSteeringControl/README_zh-CN.md new file mode 100644 index 00000000..62fbb05e --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/README_zh-CN.md @@ -0,0 +1,62 @@ +# 风电场尾流偏航控制 + +本任务要求 Agent 优化一个确定性的偏航控制策略。冻结的 FLORIS 4.6.6 +验证器在多种风向和风速下评估九台风机组成的风电场。 + +## 目录 + +```text +scripts/init.py 可编辑偏航策略 +references/farm_config.json 公开风场与评分契约 +verification/evaluator.py 冻结的 FLORIS 验证器 +verification/run_candidate.py 候选代码隔离运行器 +frontier_eval/ unified-task 元数据 +baseline/result_log.json 零偏航基线结果 +``` + +## 环境 + +在 Frontier-Engineering 仓库根目录执行： + +```bash +python -m venv .venvs/frontier-wake-steering +.venvs/frontier-wake-steering/bin/python -m pip install \ + -r benchmarks/WindEnergy/WakeSteeringControl/verification/requirements.txt +``` + +本任务仅需 CPU，FLORIS 不要求 GPU。 + +## 直接评测 + +在任务目录中执行： + +```bash +../../../.venvs/frontier-wake-steering/bin/python \ + verification/evaluator.py scripts/init.py +``` + +验证器生成 `metrics.json` 和 `artifacts.json`。仓库自带的零偏航策略保证可行， +其综合分数按定义约为零。 + +## Unified 评测 + +任务 ID 为 `WindEnergy/WakeSteeringControl`： + +```bash +python -m frontier_eval \ + task=unified \ + task.benchmark=WindEnergy/WakeSteeringControl \ + task.runtime.python_path=uv-env:frontier-wake-steering \ + algorithm=openevolve \ + algorithm.iterations=0 +``` + +零迭代验证不需要模型 API Key；正式优化使用 Frontier Eval 的常规模型配置。 + +## 复现与安全 + +- FLORIS 固定为 4.6.6。 +- 验证器使用内置 NREL 5 MW 风机和固定风场。 +- 候选代码在带超时的独立子进程中运行。 +- unified task 检查验证器与公开契约是否保持只读，修改这些文件的候选会被判为无效。 +- 进程隔离不是强安全沙箱；不可信候选应在容器或一次性主机中运行。 diff --git a/benchmarks/WindEnergy/WakeSteeringControl/Task.md b/benchmarks/WindEnergy/WakeSteeringControl/Task.md new file mode 100644 index 00000000..84dd31f3 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/Task.md @@ -0,0 +1,63 @@ +# Task: Wind-Farm Wake-Steering Control + +## Background + +An upstream turbine extracts energy and leaves a slower wake. When turbines are +aligned with the wind, that wake can reduce the production of downstream +turbines. Deliberately yawing an upstream rotor can redirect its wake. The +upstream turbine loses some power, but the farm may gain more power overall. + +## Objective + +Implement `yaw_policy()` in `scripts/init.py`. For each wind condition, return +one yaw angle per turbine. Maximize frequency-weighted farm energy relative to +the zero-yaw baseline while avoiding excessive yaw and abrupt changes between +nearby wind directions. + +## Inputs + +The policy receives Python lists: + +- `wind_directions_deg`: meteorological wind directions in degrees; +- `wind_speeds_mps`: wind speeds in metres per second; +- `turbulence_intensities`: dimensionless turbulence intensity values; +- `layout_x_m`, `layout_y_m`: fixed turbine coordinates in metres. + +There are `n_conditions` wind conditions and `n_turbines` layout points. + +## Output + +Return a rectangular nested list or compatible numeric array with shape: + +```text +(n_conditions, n_turbines) +``` + +Every value is a yaw angle in degrees and must lie in `[-25, 25]`. Positive and +negative signs represent opposite steering directions under FLORIS conventions. + +## Score + +The verifier computes weighted expected farm power for the candidate and the +zero-yaw baseline. The higher-is-better score is: + +```text +relative energy gain in percentage points +- 0.01 * frequency-weighted mean absolute yaw in degrees +- 0.002 * mean adjacent-direction yaw change in degrees +``` + +The zero-yaw baseline therefore scores approximately `0.0`. Raw energy and +penalty components are reported separately. + +## Hard Constraints + +- Return the exact required shape and only finite numeric values. +- Keep every yaw angle within `[-25, 25]` degrees. +- Return identical results for identical inputs. +- Complete within the candidate timeout. +- Do not modify benchmark, reference, or verifier files. +- Keep the function signature and EVOLVE markers unchanged. + +The verifier determines feasibility and score. During unified evaluation, +Frontier Eval also invalidates candidates that modify files declared read-only. diff --git a/benchmarks/WindEnergy/WakeSteeringControl/Task_zh-CN.md b/benchmarks/WindEnergy/WakeSteeringControl/Task_zh-CN.md new file mode 100644 index 00000000..698064f1 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/Task_zh-CN.md @@ -0,0 +1,59 @@ +# 任务：风电场尾流偏航控制 + +## 背景 + +上游风机提取能量后会形成低速尾流。当风机沿风向排列时，尾流会降低下游风机 +发电量。上游风机主动偏航可以改变尾流方向；虽然上游风机会损失部分功率，整个 +风电场的总功率可能增加。 + +## 目标 + +实现 `scripts/init.py` 中的 `yaw_policy()`。针对每个风况，为每台风机返回一个 +偏航角。在限制偏航幅度和相邻风向策略突变的同时，相对零偏航基线最大化加权 +风电场发电量。 + +## 输入 + +策略接收以下 Python 列表： + +- `wind_directions_deg`：气象风向，单位为度； +- `wind_speeds_mps`：风速，单位为米每秒； +- `turbulence_intensities`：无量纲湍流强度； +- `layout_x_m`、`layout_y_m`：固定风机坐标，单位为米。 + +风况数量为 `n_conditions`，风机数量为 `n_turbines`。 + +## 输出 + +返回矩形嵌套列表或兼容数值数组，形状必须为： + +```text +(n_conditions, n_turbines) +``` + +每个值表示偏航角，范围为 `[-25, 25]` 度。正负号表示 FLORIS 约定下两个相反 +的尾流转向方向。 + +## 评分 + +验证器分别计算候选策略和零偏航基线的加权期望功率。综合分数越高越好： + +```text +相对发电量增益（百分点） +- 0.01 * 频率加权平均绝对偏航角 +- 0.002 * 相邻风向平均偏航变化 +``` + +因此零偏航基线分数约为 `0.0`。验证器会分别报告原始发电量和惩罚项。 + +## 硬约束 + +- 返回精确形状和有限数值； +- 所有偏航角均在 `[-25, 25]` 度内； +- 相同输入必须返回相同结果； +- 必须在候选超时前完成； +- 不得修改任务、参考资料及验证器文件； +- 不得改变函数签名和 EVOLVE 标记。 + +验证器负责判定可行性和最终分数。在 unified 评测中，Frontier Eval 还会将修改 +只读声明文件的候选判为无效。 diff --git a/benchmarks/WindEnergy/WakeSteeringControl/baseline/result_log.json b/benchmarks/WindEnergy/WakeSteeringControl/baseline/result_log.json new file mode 100644 index 00000000..a8a55e0e --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/baseline/result_log.json @@ -0,0 +1,15 @@ +{ + "baseline_expected_power_w": 14562813.187886339, + "candidate_expected_power_w": 14562813.187886339, + "candidate_runtime_s": 0.05653033405542374, + "combined_score": 0.0, + "evaluator_runtime_s": 0.2937989681959152, + "n_conditions": 24.0, + "n_turbines": 9.0, + "relative_energy_gain_pct": 0.0, + "valid": 1.0, + "weighted_mean_abs_yaw_deg": 0.0, + "yaw_effort_penalty": 0.0, + "yaw_smoothness_deg": 0.0, + "yaw_smoothness_penalty": 0.0 +} diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/agent_files.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/agent_files.txt new file mode 100644 index 00000000..1cbcba12 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/agent_files.txt @@ -0,0 +1,7 @@ +README.md +README_zh-CN.md +Task.md +Task_zh-CN.md +references/README.md +references/farm_config.json +frontier_eval/constraints.txt diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/artifact_files.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/artifact_files.txt new file mode 100644 index 00000000..76dc893a --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/artifact_files.txt @@ -0,0 +1,2 @@ +metrics.json +artifacts.json diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/candidate_destination.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/candidate_destination.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/candidate_destination.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/constraints.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/constraints.txt new file mode 100644 index 00000000..cd7e27e1 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/constraints.txt @@ -0,0 +1,7 @@ +WakeSteeringControl constraints: +1) Only modify the EVOLVE block in `scripts/init.py`. +2) Keep the `yaw_policy()` signature unchanged and return shape `(n_conditions, n_turbines)`. +3) Return only finite yaw angles in `[-25, 25]` degrees. +4) Keep the policy deterministic and complete within the candidate timeout. +5) Do not modify task, reference, or verification files; unified evaluation checks them as read-only. +6) Maximize weighted farm-energy gain while limiting yaw effort and directional discontinuity. diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/copy_files.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/copy_files.txt new file mode 100644 index 00000000..1f1d4caf --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/copy_files.txt @@ -0,0 +1,8 @@ +README.md +README_zh-CN.md +Task.md +Task_zh-CN.md +references +verification +scripts +frontier_eval/run_eval.sh diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/eval_command.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/eval_command.txt new file mode 100644 index 00000000..edd26896 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/eval_command.txt @@ -0,0 +1 @@ +bash frontier_eval/run_eval.sh {python} {candidate} diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/eval_cwd.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/eval_cwd.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/eval_cwd.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/initial_program.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/initial_program.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/initial_program.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/readonly_files.txt b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/readonly_files.txt new file mode 100644 index 00000000..1e747d81 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/readonly_files.txt @@ -0,0 +1,7 @@ +README.md +README_zh-CN.md +Task.md +Task_zh-CN.md +references +verification +frontier_eval/run_eval.sh diff --git a/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/run_eval.sh b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/run_eval.sh new file mode 100755 index 00000000..0f3361d2 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/frontier_eval/run_eval.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail + +PYTHON_CMD="${1:?missing Python interpreter}" +CANDIDATE_PATH="${2:?missing candidate path}" + +exec "${PYTHON_CMD}" verification/evaluator.py "${CANDIDATE_PATH}" \ + --json-out metrics.json \ + --artifacts-out artifacts.json diff --git a/benchmarks/WindEnergy/WakeSteeringControl/references/README.md b/benchmarks/WindEnergy/WakeSteeringControl/references/README.md new file mode 100644 index 00000000..2cc3cd00 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/references/README.md @@ -0,0 +1,8 @@ +# Public Configuration + +`farm_config.json` defines the public farm geometry, yaw limits, timeout, and +score coefficients. Coordinates are expressed in rotor-diameter multiples and +are converted to metres using the NREL 5 MW rotor diameter loaded by FLORIS. + +The evaluator owns the deterministic wind-condition ensemble. A candidate +receives every condition it must control through the documented policy inputs. diff --git a/benchmarks/WindEnergy/WakeSteeringControl/references/farm_config.json b/benchmarks/WindEnergy/WakeSteeringControl/references/farm_config.json new file mode 100644 index 00000000..f8b1c3e6 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/references/farm_config.json @@ -0,0 +1,20 @@ +{ + "floris_version": "4.6.6", + "turbine_model": "nrel_5MW", + "layout_rotor_diameters": [ + [0.0, 0.0], + [5.0, 0.0], + [10.0, 0.0], + [0.0, 5.0], + [5.0, 5.0], + [10.0, 5.0], + [0.0, 10.0], + [5.0, 10.0], + [10.0, 10.0] + ], + "yaw_min_deg": -25.0, + "yaw_max_deg": 25.0, + "candidate_timeout_s": 5.0, + "yaw_effort_coefficient": 0.01, + "smoothness_coefficient": 0.002 +} diff --git a/benchmarks/WindEnergy/WakeSteeringControl/scripts/init.py b/benchmarks/WindEnergy/WakeSteeringControl/scripts/init.py new file mode 100644 index 00000000..7fbab649 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/scripts/init.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +from typing import Sequence + + +def yaw_policy( + wind_directions_deg: Sequence[float], + wind_speeds_mps: Sequence[float], + turbulence_intensities: Sequence[float], + layout_x_m: Sequence[float], + layout_y_m: Sequence[float], +) -> list[list[float]]: + """Return one yaw angle per wind condition and turbine. + + The shipped baseline applies no wake steering. Keep this function + deterministic and preserve its signature. + """ + + n_conditions = len(wind_directions_deg) + n_turbines = len(layout_x_m) + + # EVOLVE-BLOCK-START + yaw_angles = [ + [0.0 for _ in range(n_turbines)] + for _ in range(n_conditions) + ] + # EVOLVE-BLOCK-END + + return yaw_angles diff --git a/benchmarks/WindEnergy/WakeSteeringControl/verification/evaluator.py b/benchmarks/WindEnergy/WakeSteeringControl/verification/evaluator.py new file mode 100644 index 00000000..a7c2517b --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/verification/evaluator.py @@ -0,0 +1,255 @@ +from __future__ import annotations + +import argparse +import json +import math +import os +from pathlib import Path +import subprocess +import sys +import time +from typing import Any + +import floris +from floris import FlorisModel +import numpy as np + + +INVALID_SCORE = -1_000_000.0 +TASK_ROOT = Path(__file__).resolve().parents[1] +CONFIG_PATH = TASK_ROOT / "references" / "farm_config.json" +RUNNER_PATH = TASK_ROOT / "verification" / "run_candidate.py" + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Evaluate a FLORIS wake-steering policy.") + parser.add_argument("candidate", type=Path) + parser.add_argument("--json-out", type=Path, default=Path("metrics.json")) + parser.add_argument("--artifacts-out", type=Path, default=Path("artifacts.json")) + return parser.parse_args() + + +def _write_json(path: Path, payload: dict[str, Any]) -> None: + if not path.is_absolute(): + path = (Path.cwd() / path).resolve() + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps(payload, indent=2, ensure_ascii=False, sort_keys=True, default=str) + "\n", + encoding="utf-8", + ) + + +def _load_config() -> dict[str, Any]: + return json.loads(CONFIG_PATH.read_text(encoding="utf-8")) + + +def _build_conditions() -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: + directions = np.asarray( + [0.0, 8.0, 82.0, 90.0, 98.0, 172.0, 180.0, 188.0, 262.0, 270.0, 278.0, 352.0], + dtype=float, + ) + direction_weights = np.asarray( + [0.11, 0.04, 0.07, 0.13, 0.05, 0.06, 0.12, 0.04, 0.07, 0.14, 0.05, 0.12], + dtype=float, + ) + speed_values = np.asarray([8.0, 10.0], dtype=float) + speed_weights = np.asarray([0.65, 0.35], dtype=float) + + wind_directions = np.tile(directions, speed_values.size) + wind_speeds = np.repeat(speed_values, directions.size) + turbulence = np.repeat(np.asarray([0.06, 0.08], dtype=float), directions.size) + frequency = np.concatenate( + [direction_weights * speed_weight for speed_weight in speed_weights] + ) + frequency /= frequency.sum() + return wind_directions, wind_speeds, turbulence, frequency + + +def _run_candidate( + candidate_path: Path, + payload: dict[str, Any], + timeout_s: float, +) -> tuple[np.ndarray, float, dict[str, Any]]: + env = os.environ.copy() + env["NO_PROXY"] = "*" + env["no_proxy"] = "*" + start = time.perf_counter() + proc = subprocess.run( + [sys.executable, str(RUNNER_PATH), str(candidate_path)], + input=json.dumps(payload), + text=True, + capture_output=True, + timeout=timeout_s, + cwd=str(TASK_ROOT), + env=env, + ) + runtime_s = time.perf_counter() - start + if proc.returncode != 0: + raise RuntimeError(f"candidate runner exited with code {proc.returncode}") + lines = [line for line in proc.stdout.splitlines() if line.strip()] + if not lines: + raise RuntimeError("candidate runner produced no JSON result") + result = json.loads(lines[-1]) + if not result.get("ok"): + raise RuntimeError(str(result.get("error", "candidate failed"))) + + first = np.asarray(result["first"], dtype=float) + second = np.asarray(result["second"], dtype=float) + if not np.all(np.isfinite(first)) or not np.all(np.isfinite(second)): + raise ValueError("yaw output contains NaN or infinity") + if first.shape != second.shape or not np.array_equal(first, second): + raise ValueError("candidate is nondeterministic for identical inputs") + diagnostics = { + "candidate_stdout": str(result.get("captured_stdout", ""))[-2000:], + "candidate_stderr": str(result.get("captured_stderr", ""))[-2000:], + } + return first, runtime_s, diagnostics + + +def _smoothness(yaw: np.ndarray, wind_directions: np.ndarray, wind_speeds: np.ndarray) -> float: + changes: list[np.ndarray] = [] + for speed in np.unique(wind_speeds): + indices = np.where(wind_speeds == speed)[0] + ordered = indices[np.argsort(wind_directions[indices])] + values = yaw[ordered] + changes.append(np.abs(np.diff(values, axis=0))) + changes.append(np.abs(values[:1] - values[-1:])) + if not changes: + return 0.0 + return float(np.mean(np.concatenate([item.reshape(-1) for item in changes]))) + + +def _invalid(error: str, runtime_s: float) -> tuple[dict[str, float], dict[str, Any]]: + metrics = { + "valid": 0.0, + "combined_score": INVALID_SCORE, + "evaluator_runtime_s": float(runtime_s), + } + artifacts = {"error_message": error} + return metrics, artifacts + + +def evaluate(candidate_path: Path) -> tuple[dict[str, float], dict[str, Any]]: + start = time.perf_counter() + config = _load_config() + if str(floris.__version__) != str(config["floris_version"]): + return _invalid( + f"FLORIS version mismatch: expected {config['floris_version']}, got {floris.__version__}", + time.perf_counter() - start, + ) + + model = FlorisModel("defaults") + rotor_diameter = float(model.core.farm.turbine_definitions[0]["rotor_diameter"]) + layout = np.asarray(config["layout_rotor_diameters"], dtype=float) * rotor_diameter + layout_x = layout[:, 0] + layout_y = layout[:, 1] + wind_directions, wind_speeds, turbulence, frequency = _build_conditions() + + payload = { + "wind_directions_deg": wind_directions.tolist(), + "wind_speeds_mps": wind_speeds.tolist(), + "turbulence_intensities": turbulence.tolist(), + "layout_x_m": layout_x.tolist(), + "layout_y_m": layout_y.tolist(), + } + + diagnostics: dict[str, Any] = {} + try: + yaw, candidate_runtime_s, diagnostics = _run_candidate( + candidate_path, + payload, + float(config["candidate_timeout_s"]), + ) + expected_shape = (wind_directions.size, layout_x.size) + if yaw.shape != expected_shape: + raise ValueError(f"wrong yaw shape: expected {expected_shape}, got {yaw.shape}") + if not np.all(np.isfinite(yaw)): + raise ValueError("yaw output contains NaN or infinity") + yaw_min = float(config["yaw_min_deg"]) + yaw_max = float(config["yaw_max_deg"]) + if float(np.min(yaw)) < yaw_min or float(np.max(yaw)) > yaw_max: + raise ValueError(f"yaw output must stay within [{yaw_min}, {yaw_max}] degrees") + + model.set( + layout_x=layout_x, + layout_y=layout_y, + wind_directions=wind_directions, + wind_speeds=wind_speeds, + turbulence_intensities=turbulence, + ) + baseline_yaw = np.zeros_like(yaw) + model.set(yaw_angles=baseline_yaw) + model.run() + baseline_power_by_condition = np.sum(model.get_turbine_powers(), axis=1) + + model.set(yaw_angles=yaw) + model.run() + candidate_power_by_condition = np.sum(model.get_turbine_powers(), axis=1) + + baseline_power = float(np.dot(frequency, baseline_power_by_condition)) + candidate_power = float(np.dot(frequency, candidate_power_by_condition)) + if not math.isfinite(candidate_power) or candidate_power <= 0.0: + raise ValueError("FLORIS returned invalid candidate farm power") + + relative_gain_pct = 100.0 * (candidate_power / baseline_power - 1.0) + weighted_abs_yaw = float(np.sum(frequency[:, None] * np.abs(yaw)) / yaw.shape[1]) + smoothness_deg = _smoothness(yaw, wind_directions, wind_speeds) + effort_penalty = float(config["yaw_effort_coefficient"]) * weighted_abs_yaw + smoothness_penalty = float(config["smoothness_coefficient"]) * smoothness_deg + combined_score = relative_gain_pct - effort_penalty - smoothness_penalty + + metrics = { + "valid": 1.0, + "combined_score": float(combined_score), + "relative_energy_gain_pct": float(relative_gain_pct), + "baseline_expected_power_w": baseline_power, + "candidate_expected_power_w": candidate_power, + "weighted_mean_abs_yaw_deg": weighted_abs_yaw, + "yaw_effort_penalty": effort_penalty, + "yaw_smoothness_deg": smoothness_deg, + "yaw_smoothness_penalty": smoothness_penalty, + "candidate_runtime_s": float(candidate_runtime_s), + "evaluator_runtime_s": float(time.perf_counter() - start), + "n_conditions": float(wind_directions.size), + "n_turbines": float(layout_x.size), + } + artifacts = { + "summary": "candidate evaluated successfully with frozen FLORIS", + "floris_version": str(floris.__version__), + "turbine_model": str(config["turbine_model"]), + "candidate_stdout": diagnostics.get("candidate_stdout", ""), + "candidate_stderr": diagnostics.get("candidate_stderr", ""), + } + return metrics, artifacts + except subprocess.TimeoutExpired: + return _invalid( + f"candidate exceeded {config['candidate_timeout_s']} second timeout", + time.perf_counter() - start, + ) + except Exception as exc: + metrics, artifacts = _invalid( + f"{type(exc).__name__}: {exc}", + time.perf_counter() - start, + ) + artifacts.update(diagnostics) + return metrics, artifacts + + +def main() -> int: + args = _parse_args() + candidate_path = args.candidate.expanduser().resolve() + if not candidate_path.is_file(): + metrics, artifacts = _invalid( + f"candidate file not found: {candidate_path}", + 0.0, + ) + else: + metrics, artifacts = evaluate(candidate_path) + _write_json(args.json_out, metrics) + _write_json(args.artifacts_out, artifacts) + print(json.dumps(metrics, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/WindEnergy/WakeSteeringControl/verification/requirements.txt b/benchmarks/WindEnergy/WakeSteeringControl/verification/requirements.txt new file mode 100644 index 00000000..a5dc1185 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/verification/requirements.txt @@ -0,0 +1 @@ +floris==4.6.6 diff --git a/benchmarks/WindEnergy/WakeSteeringControl/verification/run_candidate.py b/benchmarks/WindEnergy/WakeSteeringControl/verification/run_candidate.py new file mode 100644 index 00000000..8c1224b3 --- /dev/null +++ b/benchmarks/WindEnergy/WakeSteeringControl/verification/run_candidate.py @@ -0,0 +1,80 @@ +from __future__ import annotations + +import contextlib +import importlib.util +import io +import json +import os +from pathlib import Path +import socket +import sys +from typing import Any + + +def _blocked_network(*args: Any, **kwargs: Any) -> Any: + raise RuntimeError("network access is disabled during candidate evaluation") + + +def _load_candidate(path: Path): + spec = importlib.util.spec_from_file_location("wake_steering_candidate", path) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot import candidate: {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _jsonable(value: Any) -> Any: + if hasattr(value, "tolist"): + return value.tolist() + return value + + +def main() -> int: + if len(sys.argv) != 2: + print(json.dumps({"ok": False, "error": "expected candidate path"})) + return 0 + + os.environ["NO_PROXY"] = "*" + os.environ["no_proxy"] = "*" + socket.socket = _blocked_network # type: ignore[assignment] + socket.create_connection = _blocked_network # type: ignore[assignment] + + stdout_capture = io.StringIO() + stderr_capture = io.StringIO() + try: + payload = json.loads(sys.stdin.read()) + candidate_path = Path(sys.argv[1]).expanduser().resolve() + with contextlib.redirect_stdout(stdout_capture), contextlib.redirect_stderr(stderr_capture): + module = _load_candidate(candidate_path) + policy = getattr(module, "yaw_policy") + args = ( + payload["wind_directions_deg"], + payload["wind_speeds_mps"], + payload["turbulence_intensities"], + payload["layout_x_m"], + payload["layout_y_m"], + ) + first = _jsonable(policy(*args)) + second = _jsonable(policy(*args)) + result = { + "ok": True, + "first": first, + "second": second, + "captured_stdout": stdout_capture.getvalue()[-2000:], + "captured_stderr": stderr_capture.getvalue()[-2000:], + } + except Exception as exc: + result = { + "ok": False, + "error": f"{type(exc).__name__}: {exc}", + "captured_stdout": stdout_capture.getvalue()[-2000:], + "captured_stderr": stderr_capture.getvalue()[-2000:], + } + + print(json.dumps(result, ensure_ascii=False, default=str)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())