Feat/computer systems/index optimization

.gitignore

@@ -16,6 +16,7 @@ benchmarks/Astrodynamics/MannedLunarLanding/results.txt
 benchmarks/Astrodynamics/MannedLunarLanding/outputlog.txt
 .vscode
 *.csv
+!leaderboard/*.csv
 *.h5ad
 *.h5ad.tmp
 *.h5ad.tmp.*
@@ -46,6 +47,7 @@ outputlog.txt
 frontier_eval/conf/batch/*
 !frontier_eval/conf/batch/example_matrix.yaml
 !frontier_eval/conf/batch/v1.yaml
+!frontier_eval/conf/batch/v1_lite.yaml
 metrics.json
 artifacts.json
 debug-*.log
@@ -54,3 +56,6 @@ debug-*.log
 **/outputs/
 **/artifacts/
 **/last_eval.json
+
+# IndexOptimization generated data
+benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/tpch_sf1.dump

README.md

@@ -14,6 +14,10 @@ Frontier-Eng is a benchmark for **generative optimization**: agents iteratively
 
 The benchmark currently covers **47 tasks** across computing, quantum information, operations research, robotics and control, optics and communications, and physical sciences. The project homepage and paper frame it as a missing evaluation axis between pass/fail coding benchmarks and real engineering work: most engineering problems start from a feasible baseline and reward iterative improvement, not one-shot correctness.
 
+## News
+
+- **2026-06-30** — **New scoring metric: the Medal Score (gold/silver/bronze).** Alongside average rank, we now release a peer-relative *Medal Score* (normalized to `[0,1]`). On each task the top-3 best-feasible scores in the v1 snapshot are frozen as gold/silver/bronze baselines; a model earns 1.00 / 0.67 / 0.33 for reaching each, averaged over the task set, and is reported on both v1 (47 tasks) and v1-lite (10 tasks). It rewards only reaching each task's frontier and ignores negligible long-tail margins, making cross-task aggregation fairer. Per-task podium values and the leaderboard live in [`leaderboard/`](leaderboard/README.md).
+- **2026-06-30** — **`v1-lite` released.** A 10-task representative subset of `v1` covering all five categories with distinct benchmark families, selected for tasks whose scores climb gradually under budget (not one-shot-saturated or all-or-nothing). Run it with `frontier_eval/conf/batch/v1_lite.yaml`.
 
 ## 0. Host Requirements
 
@@ -97,6 +101,14 @@ bash scripts/batch/validate_v1_task_envs.sh
 
 That command runs the batch config for the `v1` problem set with `algorithm.iterations=0`, which evaluates each task's shipped baseline instead of asking an LLM to improve it.
 
+### `v1-lite` quick subset
+
+For fast iteration and ablations, use the 10-task `v1-lite` matrix
+([`frontier_eval/conf/batch/v1_lite.yaml`](frontier_eval/conf/batch/v1_lite.yaml))
+instead of the full `v1` config. It spans all five categories with distinct
+benchmark families and favors tasks whose scores improve gradually under budget,
+so a short run still exercises the full optimization loop.
+
 If you want the full `v1` problem set with normal optimization runs later, see [`run.md`](run.md).
 
 ## Where To Go Next
@@ -108,18 +120,20 @@ If you want the full `v1` problem set with normal optimization runs later, see [
 
 ## Leaderboard
 
-Detailed leaderboard: [lab.einsia.ai/frontier-eng/leaderboard.html](https://lab.einsia.ai/frontier-eng/leaderboard.html)
-
-| Rank | Model | Average Rank |
-| :--: | :--- | --: |
-| 1 | GPT-5.4 | 3.54 |
-| 2 | Claude Opus 4.6 | 3.63 |
-| 3 | GLM-5 | 4.34 |
-| 4 | DeepSeek V3.2 | 4.76 |
-| 5 | Gemini 3.1 Pro Preview | 5.53 |
-| 6 | Grok 4.20 | 5.82 |
-| 7 | SEED 2.0 Pro | 5.86 |
-| 8 | Qwen3 Coder Next | 6.71 |
+Detailed leaderboard (incl. average rank): [lab.einsia.ai/frontier-eng/leaderboard](https://lab.einsia.ai/frontier-eng/leaderboard). Released score tables and the per-task medal podium: [`leaderboard/`](leaderboard/README.md).
+
+**Medal Score** (gold/silver/bronze podium, higher is better, normalized to `[0,1]` = mean per-task podium credit). On each task the top-3 best scores in the **v1 snapshot (2026-04-14)** are frozen as gold/silver/bronze baselines; a model earns 1.00 / 0.67 / 0.33 for reaching each. Reported on both the full **v1** set (47 tasks) and the **v1-lite** subset (10 tasks); gold/silver/bronze counts are for v1 (see [`leaderboard/`](leaderboard/README.md)):
+
+| Rank | Model | Medal (v1) | Medal (v1-lite) | 🥇 | 🥈 | 🥉 |
+| :--: | :--- | --: | --: | --: | --: | --: |
+| 1 | GPT-5.4 | 0.596 | 0.667 | 24 | 5 | 2 |
+| 2 | Claude Opus 4.6 | 0.490 | 0.501 | 9 | 18 | 6 |
+| 3 | GLM-5 | 0.312 | 0.233 | 4 | 10 | 12 |
+| 4 | DeepSeek V3.2 | 0.248 | 0.166 | 3 | 9 | 8 |
+| 5 | Gemini 3.1 Pro Preview | 0.213 | 0.200 | 3 | 6 | 9 |
+| 6 | Seed 2.0 Pro | 0.185 | 0.100 | 3 | 7 | 3 |
+| 7 | Grok 4.20 | 0.184 | 0.133 | 3 | 6 | 5 |
+| 8 | Qwen3 Coder Next | 0.121 | 0.000 | 3 | 3 | 2 |
 
 ## Contributing
 

README_zh-CN.md

@@ -8,6 +8,11 @@ Frontier-Eng 是一个面向 **generative optimization** 的 benchmark：Agent 
 
 当前版本包含 **47 个任务**，覆盖计算系统、量子信息、运筹优化、机器人控制、光学通信、物理与工程设计。主页和论文的核心观点是：真实工程问题通常从一个可行 baseline 出发，价值来自持续优化，而不是 pass/fail。
 
+## News
+
+- **2026-06-30** — **新增评测指标：金银铜 Medal Score。** 在 average rank 之外，我们发布同侪相对的 *Medal Score*（归一化到 `[0,1]`）：每道题取 v1 snapshot 中最好的前三名分数冻结为金/银/铜 baseline，模型达到金/银/铜分别得 1.00 / 0.67 / 0.33，对题集求均值；同时汇报 v1（47 题）与 v1-lite（10 题）。它只奖励"达到该题最前沿（领奖台）"，忽略长尾里可忽略的微小差距，使跨题汇总更公平。每题 podium 分数与榜单见 [`leaderboard/`](leaderboard/README.md)。
+- **2026-06-30** — **发布 `v1-lite`。** `v1` 的 10 题代表性子集，覆盖全部五大类、family 各不相同，专选"分数随预算逐步提升（而非一步做满或非高即低）"的题，配置见 `frontier_eval/conf/batch/v1_lite.yaml`。
+
 ## 这个 benchmark 在测什么
 
 和传统 agent benchmark 相比，Frontier-Eng 更关注三件事：
@@ -110,18 +115,20 @@ bash scripts/batch/validate_v1_task_envs.sh
 
 ## Leaderboard
 
-详细榜单见 [lab.einsia.ai/frontier-eng/leaderboard.html](https://lab.einsia.ai/frontier-eng/leaderboard.html)。
-
-| 排名 | Model | Average Rank |
-| :--: | :--- | --: |
-| 1 | Claude Opus 4.6 | 3.18 |
-| 2 | GLM-5 | 4.02 |
-| 3 | DeepSeek V3.2 | 4.41 |
-| 4 | Gemini 3.1 Pro Preview | 5.34 |
-| 5 | Grok 4.20 | 5.60 |
-| 6 | SEED 2.0 Pro | 5.63 |
-| 7 | GPT-5.4 | 5.68 |
-| 8 | Qwen3 Coder Next | 6.68 |
+详细榜单（含 average rank）见 [lab.einsia.ai/frontier-eng/leaderboard](https://lab.einsia.ai/frontier-eng/leaderboard)。发布的分数表与每题金银铜 podium 见 [`leaderboard/`](leaderboard/README.md)。
+
+**Medal Score**（金银铜 podium，越高越好，归一化到 `[0,1]`，即每题领奖台得分的均值）。每题取 **v1 snapshot (2026-04-14)** 的前三名分数冻结为金/银/铜 baseline，模型达到金/银/铜分别得 1.00 / 0.67 / 0.33。同时汇报 **v1**（47 题）与 **v1-lite**（10 题）两个集合；金银铜次数为 v1（`gpt-5.4` 采用其 47 题全量重测结果）：
+
+| 排名 | Model | Medal (v1) | Medal (v1-lite) | 🥇 | 🥈 | 🥉 |
+| :--: | :--- | --: | --: | --: | --: | --: |
+| 1 | GPT-5.4 | 0.596 | 0.667 | 24 | 5 | 2 |
+| 2 | Claude Opus 4.6 | 0.490 | 0.501 | 9 | 18 | 6 |
+| 3 | GLM-5 | 0.312 | 0.233 | 4 | 10 | 12 |
+| 4 | DeepSeek V3.2 | 0.248 | 0.166 | 3 | 9 | 8 |
+| 5 | Gemini 3.1 Pro Preview | 0.213 | 0.200 | 3 | 6 | 9 |
+| 6 | Seed 2.0 Pro | 0.185 | 0.100 | 3 | 7 | 3 |
+| 7 | Grok 4.20 | 0.184 | 0.133 | 3 | 6 | 5 |
+| 8 | Qwen3 Coder Next | 0.121 | 0.000 | 3 | 3 | 2 |
 
 ## 贡献
 

TASK_DETAILS.md

@@ -204,14 +204,18 @@ We welcome new engineering problem ideas — even without complete verification
       <td>Polarization-multiplexed holography</td>
     </tr>
     <tr>
-      <td rowspan="2"><b>ComputerSystems</b></td>
+      <td rowspan="3"><b>ComputerSystems</b></td>
       <td><code>MallocLab</code></td>
       <td>High-performance C memory allocator (utilization &amp; throughput)</td>
     </tr>
     <tr>
       <td><code>DuckDBWorkloadOptimization</code></td>
       <td>Index / materialized-view selection and query rewriting on official DuckDB workloads</td>
     </tr>
+    <tr>
+      <td><code>CacheReplacement</code></td>
+      <td>CPU cache replacement policy optimization via ChampSim on SPEC CPU traces</td>
+    </tr>
     <tr>
       <td><b>EngDesign</b></td>
       <td><code>CY_03, WJ_01, XY_05, AM_02, AM_03, YJ_02, YJ_03</code></td>

benchmarks/ComputerSystems/CacheReplacement/.gitignore

@@ -0,0 +1,9 @@
+data/traces/
+data/ChampSim/
+data/baseline_ipcs/
+references/generated_baseline.json
+__pycache__/
+*.pyc
+*.log
+temp/
+.DS_Store