Add kernel block encoding benchmark

[hyizhak]

Summary

• add a CPU-only KernelBlockEncoding quantum-computing benchmark for optimizing partial-Hadamard basis selection, block-encoding normalization, and sparse quantized FABLE rotations
• independently reconstruct every encoded matrix in Python, enforce Frobenius block-error bounds, and score a frozen logical gate/depth resource model
• add six deterministic QML kernel and Gram workloads, frozen hashes, a C++17 starter and baseline, hard process limits, focused tests, unified-task metadata, research references, and English/Chinese repository index updates

Motivation and impact

Dense kernel block encodings often hide data-loading and normalization costs behind an oracle assumption. This task makes those choices directly optimizable and independently measurable without relying on a state-vector simulator or candidate-reported metrics.

The scored path needs only Python's standard library and g++. It runs on one CPU core in under 1 GiB and evaluates the frozen starter in about 2.7 seconds on the current AMD EPYC host. An exploratory policy that was not included in the starter reached a 2.7177 combined score, confirming meaningful optimization headroom.

Validation

• clang-format --dry-run --Werror on all C++ sources
• g++ -std=c++17 -O2 -Wall -Wextra -Wpedantic starter compilation
• 4 evaluator/construction tests under Python 3.10.12 and Python 3.12.12
• direct evaluator: valid=1.0, combined_score=1.0, 6 scenarios, 8,448 checked matrix entries
• unified task adapter: valid=1.0, combined_score=1.0, including read-only path checks
• bash -n and git diff --check

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new engineering benchmark titled "Kernel Block Encoding" under the Quantum Computing category. It challenges developers to optimize the construction of FABLE-style quantum circuits for encoding classical kernel matrices, balancing circuit depth/gate count against approximation error and normalization scales.
• Modified File Structure & Modifications:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: Added the KernelBlockEncoding task to the Quantum Computing section in both English and Chinese documentation.
  • benchmarks/QuantumComputing/KernelBlockEncoding/README.md: Provided a high-level overview, environment requirements (C++17, Python 3.10+), and execution commands for both direct and unified evaluation.
  • benchmarks/QuantumComputing/KernelBlockEncoding/Task.md: Defined the rigorous mathematical framework, including Walsh-Hadamard transforms, Gray-code permutations, and the Frobenius norm error budget.
  • benchmarks/QuantumComputing/KernelBlockEncoding/scripts/init.cpp: Created a starter C++ policy that implements a baseline binary-search thresholding algorithm.
  • benchmarks/QuantumComputing/KernelBlockEncoding/baseline/: Included a reference solution and a result log for normalization and scoring.
  • benchmarks/QuantumComputing/KernelBlockEncoding/verification/: Added the core evaluation logic, including a Python-based analytic reconstructor and a C++ runtime header.
  • benchmarks/QuantumComputing/KernelBlockEncoding/references/: Included a problem configuration JSON and a bibliography of relevant quantum research (FABLE, QSVT).

2. AI Content Analysis

• Estimated AI Component: 15%
• Reasoning & Evidence: The overall structure and the complex mathematical logic (e.g., the specific implementation of the FABLE oracle and the normalization-aware objective function) suggest high-level human domain expertise in Quantum Information Science. AI patterns are primarily visible in the boilerplate C++ code in init.cpp (e.g., standard argument parsing in main) and the highly structured Markdown tables. The core algorithmic logic and the specific integration with frontier_eval appear bespoke and domain-specific.

3. Engineering & Economic Assessment

• Engineering Reality Check: This is a high-fidelity, production-grade engineering problem. It addresses the "data-loading bottleneck" in Quantum Machine Learning (QML), which is a significant real-world hurdle. The task correctly identifies that circuit complexity is often dominated by data loading rather than the algorithm itself. It handles edge cases like basis-dependent minimum scales and enforces strict Frobenius norm constraints.
• Economic Value: High. Optimizing block encoding directly translates to reduced resource requirements (qubits, gate depth) for future fault-tolerant quantum applications. This reduces the "cost of entry" for running QML algorithms on near-term and future quantum hardware.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes
  • task_name: KernelBlockEncoding
  • Execution & Dependencies: The README.md clearly documents the requirements (C++17, Python 3.10) and provides exact bash commands for both standalone and unified testing. No external heavy dependencies (like Qiskit or simulators) are required, making it highly portable.
• Documentation Quality: Excellent. The Task.md provides a comprehensive mathematical contract. The README.md is concise and actionable. No significant spelling or grammatical errors were detected. The use of EVOLVE-BLOCK markers clearly defines the candidate's scope.
• Organizational Structure: The structure is logical and follows the repository's established modular pattern. Separating verification, baseline, and scripts ensures a clean boundary between the evaluator and the candidate code.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were found.
• Absolute Paths: None detected. All file references in the C++ and Python scripts use relative paths or are passed via command-line arguments.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 在量子计算类别下引入了一个名为 “Kernel Block Encoding” (核矩阵块编码) 的新工程基准任务。该任务要求开发者优化 FABLE 风格量子线路的构造，用于编码经典机器学习中的核矩阵，在保持近似误差和归一化比例达标的前提下，最小化线路深度和门数量。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: 在中英文任务列表中同步新增了 KernelBlockEncoding 任务。
  • benchmarks/QuantumComputing/KernelBlockEncoding/README.md: 提供了任务概览、环境依赖（C++17, Python 3.10+）以及直接运行和统一评测的执行命令。
  • benchmarks/QuantumComputing/KernelBlockEncoding/Task.md: 定义了严谨的数学框架，包括 Walsh-Hadamard 变换、格雷码排列以及 Frobenius 范数误差预算。
  • benchmarks/QuantumComputing/KernelBlockEncoding/scripts/init.cpp: 提供了一个 C++ 初始策略，实现了基于二分查找阈值的基准算法。
  • benchmarks/QuantumComputing/KernelBlockEncoding/baseline/: 包含参考解决方案和结果日志，用于分数归一化。
  • benchmarks/QuantumComputing/KernelBlockEncoding/verification/: 增加了核心评测逻辑，包括 Python 编写的解析重构器和 C++ 运行时头文件。
  • benchmarks/QuantumComputing/KernelBlockEncoding/references/: 包含问题配置 JSON 和相关量子计算研究（FABLE, QSVT）的参考文献。

2. AI 成分分析

• 预估 AI 含量: 15%
• 判断依据与证据: 整体结构和复杂的数学逻辑（如 FABLE oracle 的具体实现和归一化感知的目标函数）表明了极高的量子信息科学领域专业性。AI 特征主要体现在 init.cpp 中的模板化 C++ 代码（如 main 函数中标准的参数解析）以及高度结构化的 Markdown 表格。核心算法逻辑和 frontier_eval 的具体集成表现出明显的定制化特征。

3. 工程与经济评估

• 工程现实检验: 这是一个高保真、生产级别的工程问题。它解决了量子机器学习 (QML) 中的“数据加载瓶颈”，这是现实中的主要障碍。该任务准确识别出线路复杂度往往由数据加载而非算法本身主导。它妥善处理了基组相关的最小比例等边缘情况，并强制执行严格的 Frobenius 范数约束。
• 经济价值: 高。优化块编码直接意味着减少未来容错量子应用所需的资源（量子比特、门深度）。这降低了在近期和未来量子硬件上运行 QML 算法的“准入门槛”。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: KernelBlockEncoding
  • 运行与依赖: README.md 清晰记录了依赖要求（C++17, Python 3.10），并提供了独立测试和统一测试的确切命令。无需 Qiskit 或模拟器等外部重型依赖，移植性极佳。
• 文档质量: 优秀。Task.md 提供了详尽的数学契约。README.md 简洁且具备可操作性。未发现明显的拼写或语法错误。EVOLVE-BLOCK 标记清晰定义了候选代码的修改范围。
• 组织结构: 文件组织符合逻辑，遵循了仓库既有的模块化模式。将 verification、baseline 和 scripts 分离，确保了评测器与候选代码之间的清晰边界。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未发现 .env、API 密钥或 IDE 配置文件。
• 绝对路径: 未检测到。C++ 和 Python 脚本中的所有文件引用均使用相对路径或通过命令行参数传递。

[wrh-human]

Thank you for contributing KernelBlockEncoding! Quantum circuit construction is a valuable engineering direction. The overall implementation quality is high — the baseline data is included, the evaluator's independent reconstruction mechanism is solid, and the documentation is thorough.

The following issues should be addressed before merging:

1. Per-workload error isolation is insufficient

The current evaluate() workload loop lacks try/except isolation. If workload 3 fails during candidate execution or certificate validation, the exception propagates directly to the outer except Exception, terminating the entire evaluation. As a result:

• The agent cannot determine which workload failed or why.
• Even if 5 out of 6 workloads succeed, the 6th failure causes valid=0 with no partial results retained.
• The agent receives no partial feedback to guide the next iteration.

Suggestion: wrap each workload in a try/except block, record a workload-level error field, and continue evaluating the remaining workloads. Successfully completed workloads should retain their scores and contribute to the combined score.

2. limited_exec.py lacks an RLIMIT_NPROC constraint

The current implementation sets RLIMIT_AS, RLIMIT_CPU, RLIMIT_FSIZE, and other resource limits, but does not restrict process creation. A candidate program can fork child processes without limit within the evaluation window, posing a risk of exhausting system resources. Suggestion:

try:
    resource.setrlimit(resource.RLIMIT_NPROC, (64, 64))
except (ValueError, OSError):
    pass

---

Non-blocking suggestions:

3. Add an end-to-end evaluator regression test

The current 4 unit tests cover the correctness of the mathematical transforms, but none of them call the full evaluate(scripts/init.cpp) pipeline to verify that the baseline returns valid=1.0 and combined_score=1.0. Suggestion:

def test_baseline_evaluates_to_valid_one(self) -> None:
    metrics, artifacts = evaluator.evaluate(
        evaluator.TASK_DIR / "scripts" / "init.cpp"
    )
    self.assertEqual(metrics["valid"], 1.0)
    self.assertAlmostEqual(metrics["combined_score"], 1.0)

(Note: this test requires g++ in the environment, but the README already lists g++ as a prerequisite, so this is reasonable.)

4. frontier_eval/artifact_files.txt format may not work with the framework

The current content is a single comment line:

# metrics.json and artifacts.json are collected by UnifiedTask directly.

The UnifiedTask documentation and implementation do not indicate support for # comment lines. If the framework parses each line as a file path, the comment line would be interpreted as a filename — specifically, "# metrics.json and artifacts.json are collected by UnifiedTask directly." — which does not exist. This could cause both metrics.json and artifacts.json to be skipped by the framework's artifact collection, meaning the evaluation runs successfully but the results are never persisted, effectively breaking the experiment pipeline.

Suggestion: replace the comment with explicit filenames matching the convention used by all other benchmarks:

metrics.json
artifacts.json

---

The overall direction is sound. Once the four items above are addressed, the PR is ready to merge.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🤖 LLM 调用失败: HTTPSConnectionPool(host='litellm-dev.vida.app', port=443): Read timed out. (read timeout=120)

[hyizhak]

Thank you for contributing KernelBlockEncoding! Quantum circuit construction is a valuable engineering direction. The overall implementation quality is high — the baseline data is included, the evaluator's independent reconstruction mechanism is solid, and the documentation is thorough.

The following issues should be addressed before merging:

1. Per-workload error isolation is insufficient

The current evaluate() workload loop lacks try/except isolation. If workload 3 fails during candidate execution or certificate validation, the exception propagates directly to the outer except Exception, terminating the entire evaluation. As a result:

• The agent cannot determine which workload failed or why.
• Even if 5 out of 6 workloads succeed, the 6th failure causes valid=0 with no partial results retained.
• The agent receives no partial feedback to guide the next iteration.

Suggestion: wrap each workload in a try/except block, record a workload-level error field, and continue evaluating the remaining workloads. Successfully completed workloads should retain their scores and contribute to the combined score.

2. limited_exec.py lacks an RLIMIT_NPROC constraint

The current implementation sets RLIMIT_AS, RLIMIT_CPU, RLIMIT_FSIZE, and other resource limits, but does not restrict process creation. A candidate program can fork child processes without limit within the evaluation window, posing a risk of exhausting system resources. Suggestion:

try:
    resource.setrlimit(resource.RLIMIT_NPROC, (64, 64))
except (ValueError, OSError):
    pass

Non-blocking suggestions:

3. Add an end-to-end evaluator regression test

The current 4 unit tests cover the correctness of the mathematical transforms, but none of them call the full evaluate(scripts/init.cpp) pipeline to verify that the baseline returns valid=1.0 and combined_score=1.0. Suggestion:

def test_baseline_evaluates_to_valid_one(self) -> None:
    metrics, artifacts = evaluator.evaluate(
        evaluator.TASK_DIR / "scripts" / "init.cpp"
    )
    self.assertEqual(metrics["valid"], 1.0)
    self.assertAlmostEqual(metrics["combined_score"], 1.0)

(Note: this test requires g++ in the environment, but the README already lists g++ as a prerequisite, so this is reasonable.)

4. frontier_eval/artifact_files.txt format may not work with the framework

The current content is a single comment line:

# metrics.json and artifacts.json are collected by UnifiedTask directly.

The UnifiedTask documentation and implementation do not indicate support for # comment lines. If the framework parses each line as a file path, the comment line would be interpreted as a filename — specifically, "# metrics.json and artifacts.json are collected by UnifiedTask directly." — which does not exist. This could cause both metrics.json and artifacts.json to be skipped by the framework's artifact collection, meaning the evaluation runs successfully but the results are never persisted, effectively breaking the experiment pipeline.

Suggestion: replace the comment with explicit filenames matching the convention used by all other benchmarks:

metrics.json
artifacts.json

The overall direction is sound. Once the four items above are addressed, the PR is ready to merge.

Thank you for the detailed review. I addressed the feedback in commit 466c6ef.

1. Per-workload error isolation

   Each baseline and candidate workload is now evaluated independently. Failures record baseline_error,
   candidate_error, and a combined workload-level error, while evaluation continues through the remaining
   workloads.

   Successful workload scores are retained in partial_combined_score, with failed_workloads and
   failure_summary included in the artifacts. The official combined_score remains 0.0 and valid=0.0 if any
   workload fails, preventing candidates from improving their score by deliberately skipping difficult
   scenarios.

2. Process-count limit

   Added a frozen max_processes=64 setting and applied it through RLIMIT_NPROC in limited_exec.py, with
   portability guards for unsupported platforms.

3. End-to-end regression coverage

   Added:

   • a full baseline evaluation test asserting valid=1.0 and combined_score=1.0;
   • an isolation regression test that intentionally fails workload 3 and verifies that workloads 4–6 still
     run and retain their results.

   The suite now contains six tests and passes under Python 3.10 and 3.12.

4. artifact_files.txt

   I verified that the current comment-only file is intentional. UnifiedTask’s _read_list_file() explicitly
   ignores lines beginning with #. Furthermore, metrics.json and artifacts.json are loaded separately through
   the dedicated metrics_json_rel and artifacts_json_rel paths.

   I confirmed this end to end: artifact_files resolves to an empty optional-artifact tuple, while both
   metrics and user artifacts are successfully loaded and persisted. Listing them again in artifact_files.txt
   would only duplicate their collection.

Additional validation included strict C++17 compilation, JSON and shell checks, git diff --check, direct
evaluation, and unified-adapter evaluation. The baseline remains valid=1.0 with combined_score=1.0.

[wrh-human]

Final Review — KernelBlockEncoding (PR #96)

Thank you for the timely updates!

All issues raised in the previous review have been addressed:

1. Per-workload error isolation ✅ — Each workload now has independent try/except blocks for both baseline and candidate, with per-workload error tracking and partial_combined_score.
2. RLIMIT_NPROC ✅ — Added to limited_exec.py, configured via max_processes: 64 in problem_config.json.
3. End-to-end regression tests ✅ — Two new tests verify baseline integrity and workload failure isolation.
4. Documentation sync ✅ — Task.md, result_log.txt, and problem_config.json updated accordingly.
5. artifact_files.txt ✅ — Confirmed intentional; the framework ignores # lines and loads metrics and artifacts through dedicated paths.

All items verified. I believe this is ready to merge.