Feat/computer systems/index optimization

[wrh-human]

Summary

Add IndexOptimization — a new database index selection benchmark under ComputerSystems/. This task evaluates an agent's ability to optimize PostgreSQL query performance through intelligent B-tree index configuration, a core database administration problem with direct operational and economic impact.

Benchmark ID

ComputerSystems/IndexOptimization

Engineering Problem

Given a PostgreSQL 16 database with TPC-H SF1 schema and a mixed analytical SQL workload, find a high-quality B-tree index configuration that improves query execution time compared to a heuristic baseline, subject to:

• Index count limit: max 10 indexes
• Storage constraint: max 500 MB index storage

Economic Relevance

Database index optimization is a core engineering task in production environments:

• Poor indexing can degrade query performance by 10–100× compared to a well-tuned configuration.
• Excess indexes waste storage and slow write operations; missing indexes force full-table scans on large tables.
• Automated index selection reduces reliance on manual DBA tuning, which does not scale to modern database fleets.
• TPC-H is the industry-standard decision-support benchmark; results are transferable to real-world analytical workloads (reporting, BI, data warehousing).

Workload

6 TPC-H SF1 queries (~1.3 GB data, 6 million lineitems):

Query	Pattern	Optimization Target
Q1	Single-table aggregate (full scan)	Recognizing when indexes are not beneficial
Q3	3-table JOIN + filter + aggregate + ORDER BY	Multi-join, range filter indexing
Q5	6-table star-schema JOIN + aggregate	Star join key indexing
Q6	Single-table range filter + aggregate	Selective single-column index
Q10	4-table JOIN + aggregate + ORDER BY	Join + filter optimization
Q12	2-table JOIN + dual-condition filter + aggregate	Filter selectivity and covering indexes

Scoring

combined_score = log2(speedup) × (1 - storage_penalty - count_penalty)

• speedup = baseline_time / candidate_time (heuristic vs agent)
• storage_penalty = 0.3 × (storage_mb / 500)
• count_penalty = 0.1 × (n_indexes / 10)
• Hard constraints: result correctness, ≤10 indexes, ≤500 MB storage

Task Design

• Agent evolution target: scripts/init.py (EVOLVE-BLOCK wrapped, matching DuckDBWorkloadOptimization pattern)
• Evaluator: Python + psycopg2, controls PostgreSQL Docker lifecycle
• Database isolation: Baseline and candidate each run on independent database instances
• Measurement: Median of 3 runs per query, 2 warmup rounds, single-threaded mode
• Correctness: ORDER BY queries compared in-order; unordered queries sorted before comparison; 1e-6 numeric tolerance

Repository Structure

benchmarks/ComputerSystems/IndexOptimization/
├── README.md / README_zh-CN.md
├── Task.md / Task_zh-CN.md
├── scripts/init.py                        # EVOLVE-BLOCK (lines 1-92)
├── data/
│   ├── raw_task.json                      # Configuration with schema + constraints
│   └── tpch_sf1/
│       ├── schema.sql                     # 8 TPC-H tables with primary keys
│       ├── gen_data.sh                    # One-time data generation script
│       └── queries/                       # 6 JSON query files (SQL + metadata)
├── references/constants.json              # Scoring parameters with documentation
├── frontier_eval/                         # 9 metadata files (DuckDB-compatible)
├── verification/
│   ├── evaluator.py                       # PostgreSQL evaluator (672 lines)
│   ├── requirements.txt
│   └── docker/Dockerfile                  # postgres:16 + python3-psycopg2
└── baseline/
    ├── heuristic.py                       # Filter/JOIN column heuristic
    └── result_log.txt

Relation to LLMIA

This task is inspired by LLMIA (Zhao et al., arXiv:2503.07884, 2025), which demonstrated the effectiveness of LLM-based index recommendation. This benchmark is algorithm-agnostic and does not include LLMIA's source code, prompt templates, or demonstration pool.

Testing

• Local evaluator pipeline: valid=1.0, correct=1.0
• Verified on real TPC-H SF1 data (~1.3 GB, 6,001,215 lineitems)
• Baseline heuristic generates 10 indexes; candidate (0 indexes) achieves 1.67× speedup
• Docker lifecycle (start → restore → measure → stop) verified
• All 6 queries execute correctly, results match across configurations
• Scoring formula produces continuous, meaningful values

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new benchmark task, IndexOptimization, focused on PostgreSQL database tuning. It also implements a new evaluation metric called the "Medal Score" and a v1-lite task subset to facilitate faster iteration and fairer cross-task aggregation.
• Modified File Structure & Modifications:
  • .gitignore: Added rules to exclude generated database dumps (*.dump) and include specific leaderboard CSVs and the new v1_lite.yaml config.
  • README.md / README_zh-CN.md: Updated with "News" regarding the Medal Score and v1-lite. Updated the leaderboard table to reflect these new metrics.
  • benchmarks/ComputerSystems/IndexOptimization/: Created a complete new benchmark suite including:
    • README.md & Task.md: Detailed English/Chinese documentation of the task, constraints, and economic relevance.
    • baseline/heuristic.py: A Python implementation of a frequency-based index recommendation strategy.
    • data/tpch_sf1/: Scripts for TPC-H data generation (gen_data.sh) and structured SQL query metadata.
  • frontier_eval/conf/batch/v1_lite.yaml: (Implied/Added) A new configuration file for the 10-task subset.

2. AI Content Analysis

• Estimated AI Component: 15%
• Reasoning & Evidence: The core logic, specifically the heuristic.py baseline and the TPC-H integration scripts, demonstrates high domain-specific nuance (e.g., handling PostgreSQL pg_dump, TPC-H pipe-delimited formatting, and specific SQL metadata extraction). However, the documentation structure (Task.md) and the boilerplate in heuristic.py (standard argparse and json loading patterns) exhibit a highly organized, standard style often seen in AI-assisted documentation or scaffolding. The "Medal Score" concept appears to be a custom research-driven design rather than a generic AI suggestion.

3. Engineering & Economic Assessment

• Engineering Reality Check: High. This is a production-grade problem. Index selection is a non-trivial DBA task. The PR effectively handles realistic constraints: maximum index counts, storage limits (500MB), and the requirement for result parity. Using TPC-H SF1 (~1.3GB data) ensures the benchmark is not a "toy example" but requires actual performance consideration.
• Economic Value: High. Database optimization directly impacts infrastructure costs (storage/compute) and application latency. Automating this via LLM agents could significantly reduce technical debt and manual DBA overhead in enterprise environments.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: ComputerSystems/IndexOptimization
  • Execution & Dependencies: The README.md and gen_data.sh provide explicit, step-by-step instructions for environment setup (Docker), data generation, and execution commands.
• Documentation Quality: Excellent. The documentation includes both English and Chinese versions. It clearly defines the scoring formula, hard constraints, and optimization directions. No significant redundant information or grammatical errors were detected in the provided diff.
• Organizational Structure: Logical and Scalable. The task is neatly encapsulated within its own directory under benchmarks/ComputerSystems/, following the project's modular pattern.

5. Security & Privacy Check

• Sensitive Files: Clean. The .gitignore was proactively updated to exclude tpch_sf1.dump, preventing large binary data or potential sensitive data snapshots from being committed.
• Absolute Paths: None detected. The scripts (e.g., gen_data.sh, heuristic.py) use relative path resolution via dirname "$0" or Path(__file__).parent, which is best practice for portability.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个全新的评测任务 IndexOptimization（PostgreSQL 数据库索引优化）。同时，引入了新的评分维度 "Medal Score"（金银铜牌分）以及 v1-lite 任务子集，旨在实现更快的迭代速度和更公平的跨任务指标聚合。
• 修改的文件结构与变更摘要:
  • .gitignore: 更新了规则以排除生成的数据库 Dump 文件 (*.dump)，并允许提交特定的排行榜 CSV 和新的 v1_lite.yaml 配置。
  • README.md / README_zh-CN.md: 新增了关于 Medal Score 和 v1-lite 的新闻动态，并更新了排行榜表格以反映新指标。
  • benchmarks/ComputerSystems/IndexOptimization/: 创建了完整的评测套件，包括：
    • README.md & Task.md: 任务、约束和经济价值的中英文详细说明。
    • baseline/heuristic.py: 基于频率的索引推荐策略的 Python 实现。
    • data/tpch_sf1/: TPC-H 数据生成脚本 (gen_data.sh) 和结构化的 SQL 查询元数据。
  • frontier_eval/conf/batch/v1_lite.yaml: (新增) 包含 10 个代表性任务的精简版配置文件。

2. AI 成分分析

• 预估 AI 含量: 15%
• 判断依据与证据: 核心逻辑（特别是 heuristic.py 基准线和 TPC-H 集成脚本）展现了极高的领域特定知识（如处理 PostgreSQL pg_dump、TPC-H 的管道分隔符格式以及特定的 SQL 元数据提取）。然而，文档结构 (Task.md) 和 heuristic.py 中的模板代码（如标准的 argparse 和 json 加载模式）表现出 AI 辅助生成文档或脚手架时常见的标准风格。Medal Score 的概念更像是基于研究的自定义设计，而非通用的 AI 建议。

3. 工程与经济评估

• 工程现实检验: 高。这是一个生产级别的工程问题。索引选择是 DBA 的一项非琐碎任务。该 PR 有效处理了现实约束：最大索引数量、存储限制 (500MB) 以及查询结果一致性要求。使用 TPC-H SF1（约 1.3GB 数据）确保了该基准测试不是“玩具示例”，而是需要真正的性能权衡。
• 经济价值: 高。数据库优化直接影响基础设施成本（存储/计算）和应用延迟。通过 LLM Agent 自动化这一过程可以显著减少企业环境中的技术债务和人工 DBA 开销。

4. Quality Assurance

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: ComputerSystems/IndexOptimization
  • 运行与依赖: README.md 和 gen_data.sh 提供了明确的步骤说明，涵盖环境配置 (Docker)、数据生成和运行命令。
• 文档质量: 优秀。文档包含中英文版本，清晰定义了评分公式、硬性约束和优化方向。在提供的 Diff 中未发现明显的冗余信息或语法错误。
• 组织结构: 逻辑清晰且具备可扩展性。任务被整齐地封装在 benchmarks/ComputerSystems/ 下的独立目录中，遵循了项目的模块化模式。

5. 安全与隐私检查

• 敏感文件: 未发现异常。.gitignore 已主动更新以排除 tpch_sf1.dump，防止大型二进制数据或潜在的敏感数据快照被提交。
• 绝对路径: 未检测到。脚本（如 gen_data.sh, heuristic.py）使用 dirname "$0" 或 Path(__file__).parent 进行相对路径解析，符合可移植性最佳实践。

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a significant update to the evaluation framework, including a new scoring metric ("Medal Score"), a lightweight benchmark subset (v1-lite), and a new high-fidelity engineering task (CacheReplacement) focused on CPU microarchitecture.
• Modified File Structure & Modifications:
  • .gitignore: Added exclusions for benchmark-specific generated data and allowed specific leaderboard CSVs and config files.
  • README.md & README_zh-CN.md: Updated with "News" section, detailed explanation of the Medal Score metric, and an updated leaderboard featuring GPT-5.4 and other models.
  • TASK_DETAILS.md: Registered the new CacheReplacement task under the ComputerSystems category.
  • benchmarks/ComputerSystems/CacheReplacement/: Created a new benchmark directory containing README.md, Task.md, and evaluation scripts (pipeline logic for ChampSim simulator).

2. AI Content Analysis

• Estimated AI Component: 35%
• Reasoning & Evidence: The documentation structure (especially in Task.md and the README.md for the new task) follows a highly standardized, clean format typical of AI-assisted technical writing. The ASCII architecture diagram and the "EVOLVE-BLOCK" contract explanation exhibit high clarity and boilerplate-like perfection often seen in LLM outputs. However, the domain-specific integration with ChampSim, SPEC CPU 2017 traces, and the specific 4-layer validation logic suggest deep human architectural design and expert domain knowledge.

3. Engineering & Economic Assessment

• Engineering Reality Check: High. This is a production-grade engineering problem. Unlike "toy" coding tasks, this requires optimizing C++ code within a cycle-accurate simulator (ChampSim) using industry-standard traces (SPEC CPU 2017). The implementation of a 4-layer validation pipeline (Static -> Compile -> Quick Verify -> Full Eval) effectively handles edge cases like non-deterministic code, forbidden APIs, and resource exhaustion.
• Economic Value: High. CPU cache optimization is a multi-billion dollar concern for cloud providers and chip designers. A 1% improvement in LLC (Last Level Cache) hit rates can result in significant power savings and performance gains in data centers. Providing a benchmark for this enables the development of agents capable of high-value microarchitectural optimization.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: ComputerSystems/CacheReplacement
  • Execution & Dependencies: The benchmarks/ComputerSystems/CacheReplacement/README.md provides explicit, step-by-step instructions for environment setup, including vcpkg bootstrapping, trace downloading, and Docker instructions.
• Documentation Quality: Excellent. The PR provides comprehensive documentation in both English and Chinese. It clearly defines the "EVOLVE-BLOCK" contract, which is essential for LLM agents to understand their operational boundaries.
• Organizational Structure: Logical and Scalable. The new task is correctly placed within the ComputerSystems category, following the established modular pattern of the repository.

5. Security & Privacy Check

• Sensitive Files: Clean. The .gitignore has been proactively updated to prevent the accidental commitment of large trace files (data/traces/), simulator binaries, and local logs.
• Absolute Paths: None detected. The setup scripts and documentation use relative paths or environment-agnostic commands.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 为评测框架引入了重大更新，包括全新的评分维度（“奖牌分” Medal Score）、轻量化评测子集（v1-lite），以及一个专注于 CPU 微架构的高保真工程任务 CacheReplacement。
• 修改的文件结构与变更摘要:
  • .gitignore: 增加了对 benchmark 生成数据的排除规则，并允许特定的排行榜 CSV 和配置文件提交。
  • README.md & README_zh-CN.md: 增加了“新闻”栏目，详细解释了 Medal Score 指标，并更新了包含 GPT-5.4 等模型的排行榜。
  • TASK_DETAILS.md: 在 ComputerSystems 类别下注册了新的 CacheReplacement 任务。
  • benchmarks/ComputerSystems/CacheReplacement/: 创建了新的 benchmark 目录，包含中英文文档、任务说明及基于 ChampSim 模拟器的评测管线逻辑。

2. AI 成分分析

• 预估 AI 含量: 35%
• 判断依据与证据: 文档结构（特别是新任务的 Task.md 和 README.md）遵循了非常标准、整洁的格式，这是 AI 辅助技术写作的典型特征。其中的 ASCII 架构图和 “EVOLVE-BLOCK” 契约说明表现出 LLM 输出中常见的极高清晰度和模板化风格。然而，与 ChampSim、SPEC CPU 2017 trace 的领域特定集成以及四层验证逻辑表明了深厚的人工架构设计和专家领域知识。

3. 工程与经济评估

• 工程现实检验: 高。这是一个生产级别的工程问题。与“玩具型”编程任务不同，该任务要求在周期精确模拟器（ChampSim）中优化 C++ 代码，并使用行业标准的 SPEC CPU 2017 轨迹。实现的四层验证管线（静态扫描 -> 编译 -> 快速验证 -> 全量评测）有效地处理了非确定性代码、禁用 API 和资源耗尽等边缘情况。
• 经济价值: 高。CPU 缓存优化是云服务商和芯片设计商关注的价值数十亿美元的问题。LLC（末级缓存）命中率提升 1% 即可在数据中心实现显著的节能和性能提升。为此提供 benchmark 能够推动开发具备高价值微架构优化能力的 Agent。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: ComputerSystems/CacheReplacement
  • 运行与依赖: benchmarks/ComputerSystems/CacheReplacement/README.md 提供了详尽的环境搭建步骤，包括 vcpkg 引导、trace 下载及 Docker 运行指令。
• 文档质量: 优秀。PR 提供了完整的中英文文档。明确定义了 “EVOLVE-BLOCK” 契约，这对于 LLM Agent 理解其操作边界至关重要。
• 组织结构: 逻辑清晰且具备可扩展性。新任务被正确放置在 ComputerSystems 类别下，遵循了仓库既有的模块化模式。

5. 安全与隐私检查

• 敏感文件: 未发现异常。.gitignore 已主动更新，防止了大型 trace 文件 (data/traces/)、模拟器二进制文件和本地日志的意外提交。
• 绝对路径: 未检测到。安装脚本和文档均使用相对路径或与环境无关的命令。