feat(ProteinDesign): add FixedBackboneDesign task#83
Open
wrh-human wants to merge 5 commits into
Open
Conversation
- Medal Score: peer-relative gold/silver/bronze podium (normalized to [0,1]), reported on v1 (47 tasks) and the v1-lite subset (10 tasks). READMEs now lead with Medal Score; average rank stays on the website leaderboard. - leaderboard/: ship the frozen podium baselines (medal_podium.csv), published leaderboard (medal_leaderboard.csv), raw score table (exp1_models_raw.csv), a submission scorer (score_submission.py), and an example submission. Un-ignore leaderboard/*.csv. - v1-lite: add frontier_eval/conf/batch/v1_lite.yaml (10-task subset across all five categories, distinct families, gradual-improvement tasks). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. 质量保证
5. 安全与隐私检查
|
Einsia-nana
force-pushed
the
main
branch
2 times, most recently
from
July 12, 2026 08:09
f9aafac to
7c61ef6
Compare
There was a problem hiding this comment.
Pull request overview
Adds a new ProteinDesign benchmark domain with a FixedBackboneDesign task (PyRosetta/Rosetta Docker-based evaluation), and introduces v1-lite plus a frozen Medal Score leaderboard artifact + local scoring script.
Changes:
- Add
ProteinDesign/FixedBackboneDesignbenchmark (task docs, baseline candidate, evaluator, reference inputs, frontier_eval wiring) and a unified Hydra task config. - Introduce
v1-lite10-task batch matrix for faster runs. - Add Medal Score leaderboard artifacts (
leaderboard/*.csv), documentation, and a submission scoring script; update root READMEs accordingly.
Reviewed changes
Copilot reviewed 33 out of 34 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Add News section, v1-lite quick subset docs, and Medal Score leaderboard presentation |
| README_zh-CN.md | Chinese README updates mirroring News/Medal Score leaderboard updates |
| leaderboard/submission_example.csv | Example submission CSV for Medal Score scoring |
| leaderboard/score_submission.py | Script to score a submission CSV against frozen Medal podium |
| leaderboard/README.md | Documentation for Medal Score artifacts and how to score your model |
| leaderboard/medal_podium.csv | Frozen per-task gold/silver/bronze thresholds |
| leaderboard/medal_leaderboard.csv | Frozen per-model Medal Score table (v1 and v1-lite) |
| leaderboard/exp1_models_raw.csv | Released per-model raw best-feasible scores used to derive podium |
| frontier_eval/conf/task/protein_design_fixed_backbone.yaml | Hydra config to run ProteinDesign/FixedBackboneDesign under unified task with Docker |
| frontier_eval/conf/batch/v1_lite.yaml | New v1-lite batch matrix configuration |
| benchmarks/ProteinDesign/README.md | New ProteinDesign domain overview + setup/run instructions |
| benchmarks/ProteinDesign/README_zh-CN.md | Chinese version of ProteinDesign domain overview |
| benchmarks/ProteinDesign/FixedBackboneDesign/verification/requirements.txt | PyRosetta dependency note and version constraint |
| benchmarks/ProteinDesign/FixedBackboneDesign/verification/evaluator.py | PyRosetta evaluator implementing prepare/evaluate pipeline and metrics emission |
| benchmarks/ProteinDesign/FixedBackboneDesign/verification/docker/Dockerfile | Dockerfile based on Rosetta image with PyRosetta import/init check |
| benchmarks/ProteinDesign/FixedBackboneDesign/Task.md | Task spec (inputs/outputs/scoring/constraints) |
| benchmarks/ProteinDesign/FixedBackboneDesign/Task_zh-CN.md | Chinese task spec |
| benchmarks/ProteinDesign/FixedBackboneDesign/scripts/init.py | Baseline candidate (EVOLVE-BLOCK) for fixed-backbone sequence design |
| benchmarks/ProteinDesign/FixedBackboneDesign/references/petrobind/2ci2.pdb | Reference PDB input |
| benchmarks/ProteinDesign/FixedBackboneDesign/references/petrobind/1ubq.pdb | Reference PDB input (used by raw_task.json) |
| benchmarks/ProteinDesign/FixedBackboneDesign/references/constants.json | Reference constants for ref2015 score terms |
| benchmarks/ProteinDesign/FixedBackboneDesign/README.md | Quick-start instructions for task directory |
| benchmarks/ProteinDesign/FixedBackboneDesign/README_zh-CN.md | Chinese quick-start + repository structure guide |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/readonly_files.txt | Mark readonly directories for the task |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/initial_program.txt | Initial editable program path |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/eval_cwd.txt | Evaluation working directory |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/eval_command.txt | Evaluation command template |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/copy_files.txt | Files to copy into eval sandbox |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/constraints.txt | Runtime/agent constraints (editable file + rules) |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/candidate_destination.txt | Candidate destination path |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/artifact_files.txt | Artifacts to collect from runs |
| benchmarks/ProteinDesign/FixedBackboneDesign/frontier_eval/agent_files.txt | Files exposed to the agent |
| benchmarks/ProteinDesign/FixedBackboneDesign/data/raw_task.json | Task input definition (PDB + design positions + metadata) |
| .gitignore | Allow committing leaderboard CSV artifacts and v1_lite batch config |
Comment on lines
+105
to
+109
| # Run Packer | ||
| packer = PackRotamersMover() | ||
| packer.score_function(scorefxn) | ||
| packer.task_factory(tf) | ||
| packer.apply(pose) |
Comment on lines
+151
to
+163
| metrics: dict[str, Any] = {"valid": True} | ||
|
|
||
| # Total energies | ||
| metrics["native_energy"] = round(native_energy, 6) | ||
| metrics["total_energy"] = round(candidate_energy, 6) | ||
| metrics["improvement"] = round(native_energy - candidate_energy, 6) | ||
|
|
||
| # Combined score: normalized improvement | ||
| if abs(native_energy) > 1e-6: | ||
| metrics["combined_score"] = round((native_energy - candidate_energy) / abs(native_energy), 6) | ||
| else: | ||
| metrics["combined_score"] = 0.0 | ||
|
|
| @@ -0,0 +1 @@ | |||
| {python} {benchmark}/verification/evaluator.py {candidate} | |||
Comment on lines
+57
to
+62
| with open(path, encoding="utf-8-sig") as f: | ||
| reader = csv.reader(f) | ||
| first = next(reader) | ||
| if not (first[1].strip().lower() in ("score", "best", "value")): | ||
| f.seek(0) # no recognizable header -> treat all rows as data | ||
| reader = csv.reader(f) |
Comment on lines
+17
to
+20
| ## News | ||
|
|
||
| - **2026-06-30** — **New scoring metric: the Medal Score (gold/silver/bronze).** Alongside average rank, we now release a peer-relative *Medal Score* (normalized to `[0,1]`). On each task the top-3 best-feasible scores in the v1 snapshot are frozen as gold/silver/bronze baselines; a model earns 1.00 / 0.67 / 0.33 for reaching each, averaged over the task set, and is reported on both v1 (47 tasks) and v1-lite (10 tasks). It rewards only reaching each task's frontier and ignores negligible long-tail margins, making cross-task aggregation fairer. Per-task podium values and the leaderboard live in [`leaderboard/`](leaderboard/README.md). | ||
| - **2026-06-30** — **`v1-lite` released.** A 10-task representative subset of `v1` covering all five categories with distinct benchmark families, selected for tasks whose scores climb gradually under budget (not one-shot-saturated or all-or-nothing). Run it with `frontier_eval/conf/batch/v1_lite.yaml`. |
jdp22
dismissed
their stale review
July 13, 2026 09:58
Review published without prior approval; withdrawn by reviewer.
jdp22
requested changes
Jul 13, 2026
jdp22
left a comment
jdp22
left a comment
Collaborator
There was a problem hiding this comment.
感谢贡献 FixedBackboneDesign。这个任务方向很有价值，双语文档、unified task 元数据和 EVOLVE-BLOCK 的整体结构也比较完整。
在合并前，建议先修复以下三个与评测正确性直接相关的问题：
1. Verifier 需要检查任务约束
当前 evaluator 主要比较 native 和 candidate 的 Rosetta 能量，但没有验证 candidate 是否仍然是同一个固定骨架。
建议在评分前至少检查：
- candidate 与 reference 的残基数量一致；
- 只有
design_positions的氨基酸类型允许变化； - 非设计位置的氨基酸类型保持不变；
- N、CA、C、O 等骨架原子的坐标保持不变；
- candidate 只包含允许的标准氨基酸。
检查失败时应返回 valid=false，避免其他 PDB 也能被当作有效结果评分。
2. Candidate 不应能够修改 reference
当前流程把 prepared.pdb 的路径传给 candidate，candidate 运行结束后 evaluator 又从同一路径读取 native。理论上 candidate 可以覆盖这个文件，从而影响基准能量。
建议在运行 candidate 后从只读原始数据重新构建 reference，或者校验 reference 在运行前后的 hash，确保它没有被修改。
3. Baseline 的 PackerTask 配置没有传给 mover
init.py 中修改了局部 task，对非设计位置调用了 prevent_repacking()；但 PackRotamersMover 最终使用的是原始 TaskFactory，因此这些限制可能没有实际生效。
建议把配置后的 task 直接传给 mover，或者将对应的 TaskOperations 加入 TaskFactory，并确认输出只改变允许的设计位置。
建议补充的简单验证
修复后，希望提供以下结果：
python verification/evaluator.py scripts/init.py
python -m frontier_eval \
task=unified \
task.benchmark=ProteinDesign/FixedBackboneDesign \
task.runtime.isolation_mode=docker \
task.runtime.docker_image=rosettacommons/rosetta:serial \
algorithm=openevolve \
algorithm.iterations=0另外建议增加两个轻量测试：
- 修改一个非设计位置时，evaluator 应拒绝；
- 修改一个 backbone 坐标时，evaluator 应拒绝。
非阻塞建议
以下内容不会阻止本次合并，但后续可以顺手完善：
- 设置固定的 Rosetta 随机种子，提高评分稳定性；
- 检查
constants.json中的 ref2015 权重； - 如果
2ci2.pdb暂时没有用途，可以删除或补充说明； - 文档可说明这是受 Agent Rosetta 启发的轻量化任务。
整体方向没有问题。核心是确保 evaluator 真正执行“固定骨架、仅修改指定位置”的任务约束；完成上述三项修复后即可继续审核。
…protection, fix PackerTask config, set random seed, fix constants weights
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. Quality Assurance
5. 安全与隐私检查
|
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. 质量保证
5. 安全与隐私检查
|
y-ji24
reviewed
Jul 15, 2026
Collaborator
There was a problem hiding this comment.
Thank you for the update. This revision is much more complete. The constraint validation, reference file protection, and task metadata are now in place, and the three issues raised in the previous review have all been addressed.
There is one remaining suggestion that I recommend resolving before merging.
load_design_positions() should not silently fall back
Currently, load_design_positions() in scripts/init.py falls back to treating all residues as design positions when meta.json is missing. While this allows the workflow to continue, it can mask configuration errors and may cause the evaluator to score candidates under an incorrect task configuration.
In addition, the read-only portion of the benchmark should not be responsible for inferring design positions through PyRosetta. The set of design positions should always come directly from the task metadata.
I would recommend removing this fallback behavior and raising a FileNotFoundError (or a similar exception) when meta.json is missing. This allows configuration problems to be detected early instead of silently degrading to a different behavior.
| import pyrosetta | ||
| pyrosetta.init(silent=True) | ||
| pose = pyrosetta.pose_from_file(str(prepared_pdb)) | ||
| return list(range(1, pose.total_residue() + 1)) |
Collaborator
There was a problem hiding this comment.
Silent fallback may mask configuration errors
The current load_design_positions() function falls back to designing all positions if the meta file is missing. While a warning is printed to stderr, this silent fallback could go unnoticed in automated runs where stderr isn't actively monitored—potentially causing incorrect results.
Since the evaluator always creates this meta file during prepare(), it should be considered mandatory. Suggest raising FileNotFoundError instead:
def load_design_positions(prepared_pdb: str | Path) -> list[int]:
"""
Read design positions from the meta JSON accompanying the prepared PDB.
The meta file is at the same path but with .meta.json extension.
Raises:
FileNotFoundError: If the meta file cannot be found at either expected location.
"""
p = Path(prepared_pdb)
meta_path = p.with_suffix(".pdb.meta.json")
if not meta_path.exists():
meta_path = p.with_suffix(".meta.json")
if not meta_path.exists():
raise FileNotFoundError(
f"Meta file not found at {p.with_suffix('.pdb.meta.json')} or {p.with_suffix('.meta.json')}. "
f"Ensure evaluator.prepare() was called first to generate the prepared PDB and metadata."
)
meta = load_json(meta_path)
return list(meta.get("design_positions", []))This makes configuration bugs explicit and easier to debug, especially in CI/automated pipelines.
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. 质量保证
5. 安全与隐私检查
|
y-ji24
reviewed
Jul 15, 2026
Collaborator
There was a problem hiding this comment.
Thank you for the update. This revision has addressed the issues raised in the previous review.
After another round of verification, the constraint validation, reference file protection, task metadata loading, and the implementation of load_design_positions() all look correct. The previous silent fallback behavior has also been removed. The implementation is now consistent with the task definition, and I did not find any new issues affecting evaluation correctness or benchmark integrity.
The benchmark is in good shape now, and I don't have any further blocking comments for this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Add FixedBackboneDesign task based on Agent Rosetta paper (arXiv:2603.15952, ICML 2026).
Files