feat(ProteinDesign): add NCAAInsertion task (Case B) by wrh-human · Pull Request #86 · EinsiaLab/Frontier-Engineering

wrh-human · 2026-07-09T14:05:19Z

Summary

Add NCAAInsertion task (Case B) — Non-canonical amino acid (TRF) insertion design for fixed-backbone protein scaffolds. This task corresponds to Case B of the Agent Rosetta paper (arXiv:2603.15952, ICML 2026).

Task Details

Domain: ProteinDesign
Task: NCAAInsertion (Non-Canonical Amino Acid Insertion)
Benchmark ID: ProteinDesign/NCAAInsertion
Scoring: PyRosetta ref2015 with loaded TRF residue parameters
Runtime: Docker (rosettacommons/rosetta:serial)

Test Proteins (4 total)

Protein	PDB ID	Length	TRF Position	Fold Type
Ubiquitin	1ubq	76	4	α/β mixed
Protein G B1 domain	1bdd	56	3	α/β mixed
Engrailed homeodomain	1enh	54	5	All-α
Transmembrane β-barrel	8uzl	286×4	58	β-barrel (de novo)

Verification Metrics

valid — All designs structurally valid
trf_present — TRF correctly placed at target position (verified per-protein)
total_energy — Rosetta ref2015 score (lower is better)
improvement — Energy improvement over native baseline
combined_score — Normalized improvement across all proteins

Key Improvements Over FixedBackboneDesign (Case A)

Multi-protein architecture — evaluator automatically iterates over 4 scaffolds
TRF presence validation — trf_present check ensures correct NCAA placement
NCAA parameter documentation — TRF.sdf + GENERATE.md for reproducibility
Detailed biological background in Task.md explaining NCAA significance
8 constraints (vs 6 in Case A)

Testing

Local pipeline test (4/4 proteins, all valid, all TRF inserted)
Unified task integration (algorithm.iterations=0)
Docker isolation mode verified

- Medal Score: peer-relative gold/silver/bronze podium (normalized to [0,1]), reported on v1 (47 tasks) and the v1-lite subset (10 tasks). READMEs now lead with Medal Score; average rank stays on the website leaderboard. - leaderboard/: ship the frozen podium baselines (medal_podium.csv), published leaderboard (medal_leaderboard.csv), raw score table (exp1_models_raw.csv), a submission scorer (score_submission.py), and an example submission. Un-ignore leaderboard/*.csv. - v1-lite: add frontier_eval/conf/batch/v1_lite.yaml (10-task subset across all five categories, distinct families, gradual-improvement tasks). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-09T14:05:59Z

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

Core Purpose: This PR introduces a new benchmark task named NCAAInsertion (Non-Canonical Amino Acid Insertion) under the ProteinDesign category. The task evaluates an agent's ability to use Rosetta for inserting a specific non-canonical amino acid (TRF) into three different protein scaffolds and redesigning the surrounding environment to stabilize the structure.
Modified File Structure & Modifications:
- benchmarks/ProteinDesign/NCAAInsertion/README.md: Created a quick-start guide, environment setup (Docker), and benchmark ID.
- benchmarks/ProteinDesign/NCAAInsertion/README_zh-CN.md: Chinese translation of the README.
- benchmarks/ProteinDesign/NCAAInsertion/Task.md: Detailed technical documentation including biological background, scoring metrics (Rosetta ref2015), and constraints.
- benchmarks/ProteinDesign/NCAAInsertion/Task_zh-CN.md: Chinese translation of the Task documentation.
- benchmarks/ProteinDesign/NCAAInsertion/data/pdbs/1bdd.pdb: Added the structural data file for the Protein G B1 domain.

2. AI Content Analysis

Estimated AI Component: 40%
Reasoning & Evidence: The documentation (especially Task.md) exhibits a highly structured, "textbook-style" layout typical of AI assistance. Specifically, the "Improvement Directions for Agents" section uses generic but technically accurate bullet points (e.g., "Monte Carlo simulated annealing," "Iterative design cycles") that follow standard LLM instructional patterns. However, the domain-specific details regarding TRF (1-formyl-tryptophan) and the specific Rosetta ref2015 scoring logic suggest significant human expert oversight and configuration.

3. Engineering & Economic Assessment

Engineering Reality Check: This is a high-quality, production-grade engineering task. Unlike "toy" protein design tasks that only use standard amino acids, this addresses a specific limitation of current ML models (like ProteinMPNN) which cannot handle NCAAs. The use of Docker (rosettacommons/rosetta:serial) and the requirement to handle three distinct scaffolds (Ubiquitin, Protein G, Engrailed homeodomain) ensures the task is robust and scientifically valid.
Economic Value: High. The ability to computationally design proteins with non-canonical amino acids is of significant value in the pharmaceutical and biotech industries (e.g., creating more stable peptide drugs or novel enzymes). Providing a standardized benchmark for AI agents in this niche reduces technical debt for researchers building Rosetta-integrated agents.

4. Quality Assurance

Verification & Testing:
- frontier_eval Integration: Yes.
- task_name: ProteinDesign/NCAAInsertion
- Execution & Dependencies: The README.md clearly documents the execution flow using evaluator.py and init.py. It specifies the exact Docker image required for the environment, which is excellent for reproducibility.
Documentation Quality: High. The documentation provides both English and Chinese versions. It includes a clear biological rationale, a breakdown of scoring metrics, and explicit constraints. No spelling or grammatical errors were detected in the provided diff.
Organizational Structure: Logical and scalable. The separation of data/, scripts/, and verification/ (implied by the README paths) follows standard benchmark repository conventions.

5. Security & Privacy Check

Sensitive Files: Clean. No .env, API keys, or temporary files (__pycache__, .log) were detected in the diff.
Absolute Paths: None detected. The README and Task files use relative paths (e.g., data/raw_task.json, outputs/prepared.pdb).

🇨🇳 中文分析

1. 摘要

核心目的: 此 PR 在 ProteinDesign 类别下引入了一个名为 NCAAInsertion（非规范氨基酸插入）的新基准测试任务。该任务评估智能体利用 Rosetta 在三种不同的蛋白质支架中插入特定的非规范氨基酸（TRF），并重新设计周围环境以稳定结构的能力。
修改的文件结构与变更摘要:
- benchmarks/ProteinDesign/NCAAInsertion/README.md: 创建了快速入门指南、环境配置（Docker）和基准测试 ID。
- benchmarks/ProteinDesign/NCAAInsertion/README_zh-CN.md: README 的中文翻译。
- benchmarks/ProteinDesign/NCAAInsertion/Task.md: 详细的技术文档，包括生物学背景、评分指标（Rosetta ref2015）和约束条件。
- benchmarks/ProteinDesign/NCAAInsertion/Task_zh-CN.md: Task 文档的中文翻译。
- benchmarks/ProteinDesign/NCAAInsertion/data/pdbs/1bdd.pdb: 添加了 Protein G B1 结构域的结构数据文件。

2. AI 成分分析

预估 AI 含量: 40%
判断依据与证据: 文档（尤其是 Task.md）表现出高度结构化、类似教科书的布局，这是典型的 AI 辅助特征。具体而言，“Agent 可优化的方向”部分使用了通用但技术准确的要点（如“蒙特卡洛模拟退火”、“多轮迭代设计”），符合标准 LLM 的指令模式。然而，关于 TRF（N1-甲酰基色氨酸）的领域特定细节和具体的 Rosetta ref2015 评分逻辑表明存在显著的人类专家监督和配置。

3. 工程与经济评估

工程现实检验: 这是一个高质量、生产级别的工程任务。与仅使用标准氨基酸的“玩具”蛋白质设计任务不同，它解决了当前机器学习模型（如 ProteinMPNN）无法处理 NCAA 的特定局限性。使用 Docker (rosettacommons/rosetta:serial) 并要求处理三个不同的支架（泛素、Protein G、Engrailed homeodomain）确保了任务的鲁棒性和科学有效性。
经济价值: 高。通过计算设计含有非规范氨基酸的蛋白质在制药和生物技术行业具有重要价值（例如，创建更稳定的多肽药物或新型酶）。为该领域的 AI 智能体提供标准化的基准测试，可以为开发集成 Rosetta 的智能体的研究人员减少技术债务。

4. 质量保证

验证与测试:
- frontier_eval 集成: 是
- task_name: ProteinDesign/NCAAInsertion
- 运行与依赖: README.md 清晰地记录了使用 evaluator.py 和 init.py 的执行流程。它指定了环境所需的确切 Docker 镜像，这对于可重复性非常有益。
文档质量: 高。文档提供了中英文版本。包括清晰的生物学原理、评分指标分解和明确的约束条件。在提供的 diff 中未检测到拼写或语法错误。
组织结构: 逻辑清晰且具备可扩展性。data/、scripts/ 和 verification/（由 README 路径推断）的分离符合标准基准测试仓库的惯例。

5. 安全与隐私检查

敏感文件: 未发现异常。在 diff 中未检测到 .env、API 密钥或临时文件（__pycache__、.log）。
绝对路径: 未检测到。README 和 Task 文件均使用相对路径（例如 data/raw_task.json、outputs/prepared.pdb）。

…onstants weights

github-actions · 2026-07-14T02:55:01Z

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

Core Purpose: This PR introduces a new benchmark task (NCAAInsertion), a new evaluation metric ("Medal Score"), and a lightweight evaluation subset (v1-lite). It also updates the project leaderboard and documentation to reflect these changes.
Modified File Structure & Modifications:
- .gitignore: Added exceptions for leaderboard CSVs and the v1_lite.yaml config; added a database dump file to the ignore list.
- README.md & README_zh-CN.md: Added a "News" section, detailed the "Medal Score" logic, introduced the v1-lite subset, and updated the leaderboard table with new rankings and medal counts.
- benchmarks/ProteinDesign/NCAAInsertion/README.md (and _zh-CN.md): Provided quick start guides and Docker environment setup instructions for the new task.
- benchmarks/ProteinDesign/NCAAInsertion/Task.md (and _zh-CN.md): Detailed the biological background of Non-Canonical Amino Acids (NCAAs), scoring metrics (Rosetta energy), and constraints.
- benchmarks/ProteinDesign/NCAAInsertion/data/pdbs/1bdd.pdb: Added a standard protein structure file required for the task.

2. AI Content Analysis

Estimated AI Component: 15%
Reasoning & Evidence: The majority of the PR consists of domain-specific scientific documentation and standard configuration updates. The "Improvement Directions for Agents" section in Task.md (e.g., "Monte Carlo simulated annealing", "Iterative design cycles") follows a highly structured, bulleted format typical of AI-assisted brainstorming. However, the technical accuracy regarding Rosetta and TRF (1-formyl-tryptophan) suggests significant human oversight or expert prompting. The PDB files and leaderboard data are clearly human-curated or programmatically generated from experimental results.

3. Engineering & Economic Assessment

Engineering Reality Check: High. This is not a "toy example." The task utilizes Rosetta (a production-grade protein modeling suite) and addresses a complex, real-world problem: designing proteins with non-canonical amino acids, which is a frontier in drug discovery and synthetic biology. The inclusion of specific PDB scaffolds (1ubq, 1bdd, 1enh) demonstrates a rigorous scientific approach.
Economic Value: High.
- Technical Debt/Efficiency: The v1-lite subset (10 tasks) significantly reduces the computational cost and time required for model iteration and ablation studies.
- Evaluation Accuracy: The "Medal Score" provides a more robust aggregation method than simple averaging, as it rewards reaching the "frontier" of a task while ignoring insignificant noise in the long tail of scores.

4. Quality Assurance

Verification & Testing:
- frontier_eval Integration: Yes.
- task_name: ProteinDesign/NCAAInsertion
- Execution & Dependencies: The .md files clearly document the docker pull command for the Rosetta environment and provide the exact Python commands for preparation, initialization, and evaluation.
Documentation Quality: Excellent. The documentation is bilingual (English/Chinese), provides deep biological context, and includes clear constraints and scoring definitions. No significant grammatical errors or formatting inconsistencies were detected.
Organizational Structure: Logical and Modular. The new task follows the established directory structure of the repository, making it easy to navigate and scale.

5. Security & Privacy Check

Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were found. The .gitignore was correctly updated to prevent the accidental commitment of large data dumps (tpch_sf1.dump).
Absolute Paths: None detected. All paths in the documentation and scripts appear to be relative to the benchmark root.

🇨🇳 中文分析

1. 摘要

核心目的: 此 PR 引入了一个新的评测任务（NCAAInsertion）、一种新的评分指标（“Medal Score” 金银铜牌分）以及一个轻量级评测子集（v1-lite）。同时更新了项目主页的榜单和文档。
修改的文件结构与变更摘要:
- .gitignore: 为榜单 CSV 和 v1_lite.yaml 配置添加了排除规则；将数据库 dump 文件加入忽略列表。
- README.md & README_zh-CN.md: 新增“新闻”栏目，详细说明了“Medal Score”的逻辑，介绍了 v1-lite 子集，并更新了包含奖牌数的模型排名表。
- benchmarks/ProteinDesign/NCAAInsertion/README.md (及中文版): 提供了新任务的快速启动指南和 Docker 环境配置说明。
- benchmarks/ProteinDesign/NCAAInsertion/Task.md (及中文版): 详细介绍了非规范氨基酸 (NCAA) 的生物学背景、评分指标（Rosetta 能量函数）及任务约束。
- benchmarks/ProteinDesign/NCAAInsertion/data/pdbs/1bdd.pdb: 添加了任务所需的标准蛋白质结构文件。

2. AI 成分分析

预估 AI 含量: 15%
判断依据与证据: PR 的大部分内容是特定领域的科学文档和标准配置更新。Task.md 中的“Agent 可优化的方向”部分（如“蒙特卡洛模拟退火”、“多轮迭代设计”）采用了典型的 AI 辅助生成的结构化列表格式。然而，关于 Rosetta 和 TRF（N1-甲酰基色氨酸）的技术准确性表明存在显著的人工审核或专家级提示词引导。PDB 文件和榜单数据显然是人工整理或通过程序从实验结果中提取的。

3. 工程与经济评估

工程现实检验: 高。这不是一个“玩具示例”。该任务使用了 Rosetta（生产级蛋白质建模套件），并解决了现实世界中的复杂问题：设计含有非规范氨基酸的蛋白质，这是药物研发和合成生物学的前沿领域。包含特定的 PDB 支架（1ubq, 1bdd, 1enh）展示了严谨的科学方法。
经济价值: 高。
- 技术债务/效率: v1-lite 子集（10 个任务）显著降低了模型迭代和消融实验所需的计算成本和时间。
- 评估准确性: “Medal Score” 提供了一种比简单平均更稳健的聚合方法，因为它奖励模型达到任务的“前沿”水平，同时忽略了长尾分数中无关紧要的噪声。

4. 质量保证

验证与测试:
- frontier_eval 集成: 是
- task_name: ProteinDesign/NCAAInsertion
- 运行与依赖: .md 文件清晰地记录了 Rosetta 环境的 docker pull 命令，并提供了准备、初始化和评估的具体 Python 命令。
文档质量: 优秀。 文档提供中英双语，具有深厚的生物学背景介绍，并包含清晰的约束和评分定义。未发现明显的语法错误或格式不一致。
组织结构: 逻辑清晰且模块化。 新任务遵循仓库既有的目录结构，易于导航和扩展。

5. 安全与隐私检查

敏感文件: 未发现异常。 未发现 .env、API 密钥或 IDE 配置文件。.gitignore 已正确更新以防止意外提交大型数据转储文件 (tpch_sf1.dump)。
绝对路径: 未检测到。 文档和脚本中的所有路径均相对于 benchmark 根目录。

github-actions · 2026-07-14T08:32:57Z

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

Core Purpose: This PR introduces a significant update to the benchmark suite, including a new high-stakes engineering task (CacheReplacement), a new evaluation metric (Medal Score), and a lightweight execution subset (v1-lite). It also updates the leaderboard to reflect these changes.
Modified File Structure & Modifications:
- .gitignore: Updated to allow specific leaderboard CSVs and the new v1_lite.yaml config; added a database dump to the ignore list.
- README.md & README_zh-CN.md: Added "News" section, detailed the "Medal Score" logic, introduced the v1-lite subset, and updated the leaderboard table with new rankings and medal counts.
- benchmarks/ComputerSystems/CacheReplacement/: Created a new benchmark directory containing:
  - README.md / Task.md (Bilingual): Comprehensive documentation of the CPU cache optimization task.
  - baseline/: C++ implementations for LRU and Random policies.
  - data/: Scripts for downloading SPEC CPU 2017 traces and checksum verification.
- baseline_archive/experiment2/openevolve/.DS_Store: Accidental inclusion of a macOS system metadata file.

2. AI Content Analysis

Estimated AI Component: 15%
Reasoning & Evidence:
- The documentation (READMEs and Task descriptions) follows a highly structured, professional template that is consistent with AI-assisted technical writing.
- The C++ code in baseline/lru.cc and random.cc uses standard boilerplate for ChampSim modules.
- However, the domain-specific integration (ChampSim commit hashes, SPEC 2017 trace URLs, and the 4-layer validation logic) suggests significant human expert intervention and domain knowledge that goes beyond generic AI generation.

3. Engineering & Economic Assessment

Engineering Reality Check: High Reality. This is not a "toy example." It utilizes ChampSim, a standard simulator used in computer architecture research, and SPEC CPU 2017 traces, which are industry-standard workloads. The 4-layer validation pipeline (Static Scan -> Compile -> Quick Verify -> Full Eval) is a production-grade approach to evaluating untrusted agent-generated code.
Economic Value: High. CPU cache efficiency is a critical bottleneck in modern computing. Even a 1% improvement in LLC (Last Level Cache) hit rates can translate to significant performance gains and cost savings in data center environments. Providing a standardized benchmark for this encourages the development of more efficient systems.

4. Quality Assurance

Verification & Testing:
- frontier_eval Integration: Yes.
- task_name: ComputerSystems/CacheReplacement (and the v1_lite batch config).
- Execution & Dependencies: The README.md within the CacheReplacement folder provides explicit, step-by-step commands for cloning ChampSim, downloading traces, and running the evaluator.
Documentation Quality: Excellent. The documentation is bilingual, includes architectural diagrams (ASCII), and clearly defines the "EVOLVE-BLOCK" contract and forbidden APIs.
- Minor Note: The "News" section is dated 2026-06-30, which appears to be a future-dated placeholder or a typo.
Organizational Structure: Logical and modular. The separation of baseline, data, and verification within the benchmark folder follows best practices for reproducibility.

5. Security & Privacy Check

Sensitive Files:
- Flagged: baseline_archive/experiment2/openevolve/.DS_Store was accidentally committed. This should be removed.
- Note: benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/tpch_sf1.dump was added to .gitignore, which is correct for preventing large/sensitive data leaks.
Absolute Paths: None detected. The scripts use relative paths (e.g., SCRIPT_DIR="$(cd "$(dirname ...)") which is good for portability.

🇨🇳 中文分析

1. 摘要

核心目的: 此 PR 为评测集引入了重大更新，包括一个新的高难度工程任务（CacheReplacement）、一个新的评分维度（Medal Score/奖牌分）以及一个轻量级运行子集（v1-lite）。同时更新了主页榜单以反映这些变化。
修改的文件结构与变更摘要:
- .gitignore: 更新以允许特定的排行榜 CSV 和新的 v1_lite.yaml 配置；将数据库 dump 文件加入忽略列表。
- README.md & README_zh-CN.md: 新增“News”板块，详细阐述了“奖牌分”逻辑，介绍了 v1-lite 子集，并更新了包含奖牌计数的榜单表格。
- benchmarks/ComputerSystems/CacheReplacement/: 创建了新的 benchmark 目录，包含：
  - README.md / Task.md (中英双语): CPU 缓存优化任务的详尽文档。
  - baseline/: LRU 和 Random 策略的 C++ 实现。
  - data/: 用于下载 SPEC CPU 2017 trace 和校验和验证的脚本。
- baseline_archive/experiment2/openevolve/.DS_Store: 意外提交了 macOS 系统元数据文件。

2. AI 成分分析

预估 AI 含量: 15%
判断依据与证据:
- 文档（README 和 Task 说明）遵循高度结构化、专业的模板，符合 AI 辅助技术写作的特征。
- baseline/lru.cc 和 random.cc 中的 C++ 代码使用了 ChampSim 模块的标准模板代码。
- 然而，特定领域的集成（如 ChampSim 的 commit hash、SPEC 2017 trace 的 URL 以及四层验证逻辑）表明了显著的人类专家干预，这些领域知识超出了通用 AI 的生成能力。

3. 工程与经济评估

工程现实检验: 高度真实。这绝非“玩具示例”。它使用了计算机体系结构研究中的标准模拟器 ChampSim 以及工业级负载 SPEC CPU 2017 traces。其设计的四层验证管线（静态扫描 -> 编译 -> 快速验证 -> 正式评测）是评估不可信 Agent 生成代码的生产级方案。
经济价值: 高。CPU 缓存效率是现代计算的关键瓶颈。LLC（末级缓存）命中率即使提升 1%，也能在数据中心环境中转化为显著的性能提升和成本节约。为此提供标准化基准有助于推动更高效系统的开发。

4. 质量保证

验证与测试:
- frontier_eval 集成: 是。
- task_name: ComputerSystems/CacheReplacement (以及 v1_lite 批处理配置)。
- 运行与依赖: CacheReplacement 目录下的 README.md 提供了明确的步骤，包括克隆 ChampSim、下载 trace 和运行评估器的命令。
文档质量: 优秀。文档采用双语编写，包含架构图（ASCII），并清晰定义了 “EVOLVE-BLOCK” 契约和禁止使用的 API。
- 微小瑕疵: “News” 部分的日期为 2026-06-30，似乎是未来日期的占位符或笔误。
组织结构: 逻辑清晰且模块化。benchmark 目录内 baseline、data 和 verification 的分离符合可复现性的最佳实践。

5. Security & Privacy Check

敏感文件:
- 警告: baseline_archive/experiment2/openevolve/.DS_Store 被意外提交，应予删除。
- 注意: benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/tpch_sf1.dump 已加入 .gitignore，这对于防止大型或敏感数据泄露是正确的。
绝对路径: 未检测到。脚本使用了相对路径（如 SCRIPT_DIR="$(cd "$(dirname ...)"），有利于移植。

…onstants weights

github-actions · 2026-07-14T08:43:41Z

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

Core Purpose: This PR introduces a new benchmark task for protein design, updates the evaluation metric system, and provides a lightweight version of the benchmark suite.
Modified File Structure & Modifications:
- .gitignore: Updated to allow leaderboard CSVs and the new v1_lite configuration while explicitly ignoring a large database dump file.
- README.md & README_zh-CN.md: Added a "News" section, introduced the "Medal Score" (Gold/Silver/Bronze) metric, defined the v1-lite subset (10 tasks), and updated the leaderboard with new model performance data.
- benchmarks/ProteinDesign/NCAAInsertion/: A completely new task directory containing:
  - README.md / README_zh-CN.md: Quick start guides and environment setup (Docker).
  - Task.md / Task_zh-CN.md: Detailed technical specifications for the "Non-Canonical Amino Acid Insertion" task, including biological background and scoring logic.
  - baseline/result_log.txt: Execution logs for the baseline implementation.
  - data/pdbs/: Structural data files (PDB format) for the proteins involved.

2. AI Content Analysis

Estimated AI Component: 35%
Reasoning & Evidence:
- The documentation structure in Task.md (e.g., the "Biological Background" and "Improvement Directions" sections) follows a highly standardized, pedagogical format typical of LLM-assisted writing.
- The tables in the READMEs are perfectly formatted, which is common in AI-generated boilerplate.
- However, the domain-specific technical details (Rosetta ref2015 score function, specific PDB IDs like 1ubq, and the chemistry of TRF) and the specific integration with the frontier_eval framework indicate significant human engineering and domain expertise. The AI was likely used for documentation scaffolding and translation.

3. Engineering & Economic Assessment

Engineering Reality Check: High. This is not a "toy example." It addresses a sophisticated problem in computational biology: designing proteins with non-canonical amino acids (NCAAs). Most ML models are limited to the 20 standard AAs; using Rosetta to handle NCAAs represents a production-grade engineering challenge. The inclusion of multiple protein scaffolds (Ubiquitin, Protein G, etc.) ensures the solution must generalize.
Economic Value: High. Protein engineering is a multi-billion dollar industry. Improving the ability of AI agents to perform iterative optimization in physics-based environments (like Rosetta) directly impacts drug discovery, enzyme design, and synthetic biology, potentially reducing R&D costs and timelines.

4. Quality Assurance

Verification & Testing:
- frontier_eval Integration: Yes.
- task_name: ProteinDesign/NCAAInsertion
- Execution & Dependencies: The README.md within the task folder clearly documents the docker pull command for the Rosetta environment and the specific Python commands for preparation, initialization, and evaluation.
Documentation Quality: Excellent. The PR provides comprehensive bilingual (English/Chinese) documentation. The "Task.md" is particularly thorough, explaining the "why" behind the task.
- Minor Inconsistency: The baseline/result_log.txt mentions 4 proteins (including 8uzl), while Task.md only lists 3 in the "Test Proteins" table. This should be reconciled.
Organizational Structure: Logical and Scalable. The task follows the established hierarchical structure of the repository (benchmarks/{Category}/{TaskName}), making it easy for new developers to navigate.

5. Security & Privacy Check

Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were detected. The .gitignore was correctly updated to prevent the accidental upload of the tpch_sf1.dump file.
Absolute Paths: None detected. The verification commands and scripts use relative paths (e.g., data/raw_task.json), which is appropriate for containerized or shared environments.

🇨🇳 中文分析

1. 摘要

核心目的: 此 PR 引入了一个新的蛋白质设计基准任务，更新了评估指标体系，并发布了基准测试集的轻量化版本（v1-lite）。
修改的文件结构与变更摘要:
- .gitignore: 更新以允许提交排行榜 CSV 文件和新的 v1_lite 配置，同时明确忽略大型数据库 dump 文件。
- README.md & README_zh-CN.md: 新增“新闻”栏目，引入“勋章分”（金/银/铜）指标，定义了包含 10 个任务的 v1-lite 子集，并根据新指标更新了模型排行榜。
- benchmarks/ProteinDesign/NCAAInsertion/: 全新的任务目录，包含：
  - README.md / README_zh-CN.md: 快速入门指南和环境配置（Docker）。
  - Task.md / Task_zh-CN.md: “非规范氨基酸插入”任务的详细技术规范，包括生物学背景和评分逻辑。
  - baseline/result_log.txt: 基线实现的运行日志。
  - data/pdbs/: 相关蛋白质的结构数据文件（PDB 格式）。

2. AI 成分分析

预估 AI 含量: 35%
判断依据与证据:
- Task.md 中的文档结构（如“生物学背景”和“Agent 可优化方向”章节）遵循高度标准化的教学格式，这是典型的 LLM 辅助写作特征。
- README 中的表格格式非常规整，符合 AI 生成模板的习惯。
- 然而，领域特定的技术细节（Rosetta ref2015 评分函数、特定的 PDB ID 如 1ubq 以及 TRF 的化学特性）以及与 frontier_eval 框架的深度集成，表明了显著的人工工程投入和领域专业知识。AI 很可能被用于文档脚手架的搭建和翻译工作。

3. 工程与经济评估

工程现实检验: 高。这并非“玩具示例”。它解决了计算生物学中的一个复杂问题：设计含有非规范氨基酸（NCAA）的蛋白质。大多数机器学习模型仅限于 20 种标准氨基酸；利用 Rosetta 处理 NCAA 代表了生产级别的工程挑战。包含多个蛋白质骨架（泛素、蛋白 G 等）确保了解决方案必须具备泛化能力。
经济价值: 高。蛋白质工程是一个价值数十亿美元的行业。提高 AI Agent 在基于物理的环境（如 Rosetta）中进行迭代优化的能力，直接影响药物研发、酶设计和合成生物学，有望降低研发成本并缩短周期。

4. 质量保证

验证与测试:
- frontier_eval 集成: 是
- task_name: ProteinDesign/NCAAInsertion
- 运行与依赖: 任务文件夹内的 README.md 清晰地记录了 Rosetta 环境的 docker pull 命令，以及用于准备、初始化和评估的具体 Python 命令。
文档质量: 优秀。该 PR 提供了详尽的中英双语文档。Task.md 尤为出色，解释了任务背后的科学意义。
- 微小不一致: baseline/result_log.txt 提到了 4 个蛋白质（包括 8uzl），而 Task.md 的“测试蛋白质”表格中仅列出了 3 个。这需要统一。
组织结构: 逻辑清晰且具备可扩展性。任务遵循仓库既有的层级结构（benchmarks/{Category}/{TaskName}），便于新开发者理解和维护。

5. 安全与隐私检查

敏感文件: 未发现异常。未检测到 .env、API 密钥或 IDE 特定配置。.gitignore 已正确更新，以防止意外上传 tpch_sf1.dump 文件。
绝对路径: 未检测到。验证命令和脚本均使用相对路径（如 data/raw_task.json），适用于容器化或共享环境。

c7w and others added 2 commits June 30, 2026 20:37

feat(ProteinDesign): add NCAAInsertion task (Case B)

d1e6ed3

c7w force-pushed the main branch from 7d5c041 to 9d4513d Compare July 12, 2026 08:06

Einsia-nana force-pushed the main branch 2 times, most recently from f9aafac to 7c61ef6 Compare July 12, 2026 08:09

fix: add constraint verification, hash protection, random seed, fix c…

4c089d6

…onstants weights

fix: add constraint verification, hash protection, random seed, fix c…

6016c15

…onstants weights

wrh-human force-pushed the feat/ProteinDesign/NCAAInsertion branch from 7a9a9a0 to 6016c15 Compare July 14, 2026 08:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ProteinDesign): add NCAAInsertion task (Case B)#86

feat(ProteinDesign): add NCAAInsertion task (Case B)#86
wrh-human wants to merge 4 commits into
EinsiaLab:mainfrom
wrh-human:feat/ProteinDesign/NCAAInsertion

wrh-human commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wrh-human commented Jul 9, 2026

Summary

Task Details

Test Proteins (4 total)

Verification Metrics

Key Improvements Over FixedBackboneDesign (Case A)

Testing

Uh oh!

github-actions Bot commented Jul 9, 2026

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

2. AI Content Analysis

3. Engineering & Economic Assessment

4. Quality Assurance

5. Security & Privacy Check

🇨🇳 中文分析

1. 摘要

2. AI 成分分析

3. 工程与经济评估

4. 质量保证

5. 安全与隐私检查

Uh oh!

github-actions Bot commented Jul 14, 2026

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

2. AI Content Analysis

3. Engineering & Economic Assessment

4. Quality Assurance

5. Security & Privacy Check

🇨🇳 中文分析

1. 摘要

2. AI 成分分析

3. 工程与经济评估

4. 质量保证

5. 安全与隐私检查

Uh oh!

github-actions Bot commented Jul 14, 2026

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

2. AI Content Analysis

3. Engineering & Economic Assessment

4. Quality Assurance

5. Security & Privacy Check

🇨🇳 中文分析

1. 摘要

2. AI 成分分析

3. 工程与经济评估

4. 质量保证

5. Security & Privacy Check

Uh oh!

github-actions Bot commented Jul 14, 2026

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

2. AI Content Analysis

3. Engineering & Economic Assessment

4. Quality Assurance

5. Security & Privacy Check

🇨🇳 中文分析

1. 摘要

2. AI 成分分析

3. 工程与经济评估

4. 质量保证

5. 安全与隐私检查

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants