Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions benchmarks/ProteinDesign/FixedBackboneDesign/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# FixedBackboneDesign

## Quick Start

```bash
cd benchmarks/ProteinDesign/FixedBackboneDesign
mkdir -p outputs
python verification/evaluator.py prepare --raw-task data/raw_task.json --prepared-output outputs/prepared.pdb
python scripts/init.py --prepared-input outputs/prepared.pdb --solution-output outputs/solution.pdb
python verification/evaluator.py evaluate --native outputs/prepared.pdb --candidate outputs/solution.pdb --result-output outputs/result.json
```

## Background

This task is inspired by Agent Rosetta (Teneggi et al., arXiv:2603.15952, ICML 2026), which demonstrates LLM-driven protein design using Rosetta. This benchmark adapts the fixed-backbone sequence design problem as a standalone engineering optimization task.

## Unified Benchmark ID

`ProteinDesign/FixedBackboneDesign`

## Environment

```bash
docker pull rosettacommons/rosetta:serial
```
81 changes: 81 additions & 0 deletions benchmarks/ProteinDesign/FixedBackboneDesign/README_zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# FixedBackboneDesign - 固定骨架蛋白质序列设计

## 一句话解释

给定一个蛋白质的骨架结构(backbone),设计出最适合这个骨架的氨基酸序列,使 Rosetta 能量得分最低。

## 任务描述

本任务基于 Agent Rosetta 论文 (arXiv:2603.15952) 的 Case A:固定骨架规范氨基酸序列设计。Agent 需要从初始 baseline 出发,通过迭代优化找到总能量更低的氨基酸序列。

## 文件结构

```
FixedBackboneDesign/
├── Task.md # 任务详情(英文)
├── Task_zh-CN.md # 任务详情(中文)
├── README.md # 本文件(导航)
├── scripts/
│ └── init.py # [可编辑] baseline 解法(含 EVOLVE-BLOCK)
├── data/
│ └── raw_task.json # 任务配置
├── references/
│ ├── constants.json # Rosetta 能量函数参数
│ └── petrobind/ # PDB 结构文件
├── frontier_eval/ # unified task 元数据
├── verification/
│ ├── evaluator.py # [核心] PyRosetta 评分入口
│ ├── requirements.txt
│ └── docker/
│ └── Dockerfile
└── baseline/ # [可选] 参考实现存档
```

## 运行方式

### 本地三步运行

```bash
cd benchmarks/ProteinDesign/FixedBackboneDesign
mkdir -p outputs

# 1. prepare(由 evaluator 完成数据准备)
python verification/evaluator.py prepare \
--raw-task data/raw_task.json \
--prepared-output outputs/prepared.pdb

# 2. 运行 baseline 生成设计结果
python scripts/init.py \
--prepared-input outputs/prepared.pdb \
--solution-output outputs/solution.pdb

# 3. 评估设计结果
python verification/evaluator.py evaluate \
--native outputs/prepared.pdb \
--candidate outputs/solution.pdb \
--result-output outputs/result.json
```

### Unified task 方式

```bash
python -m frontier_eval \
task=unified \
task.benchmark=ProteinDesign/FixedBackboneDesign \
task.runtime.isolation_mode=docker \
task.runtime.docker_image=rosettacommons/rosetta:serial \
algorithm=openevolve \
algorithm.iterations=0
```

## 统一 benchmark ID

- `ProteinDesign/FixedBackboneDesign`

## 环境准备

本任务需要 Rosetta 环境,推荐使用官方 Docker 镜像:

```bash
docker pull rosettacommons/rosetta:serial
```
37 changes: 37 additions & 0 deletions benchmarks/ProteinDesign/FixedBackboneDesign/Task.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# FixedBackboneDesign

## Overview

Design the optimal amino acid sequence for a given fixed protein backbone. This task corresponds to Case A (Fixed-Backbone Canonical Amino Acid Sequence Design) of the Agent Rosetta paper (arXiv:2603.15952, ICML 2026).

## Input

The candidate script (`scripts/init.py`) receives:

- `--prepared-input`: Path to a PDB file with the target backbone and design position annotations
- `--solution-output`: Path where the designed structure should be written

## Output

The candidate must output a PDB file at `--solution-output` containing:
1. The original backbone coordinates (unchanged)
2. Designed amino acid substitutions at specified positions
3. Only standard 20 amino acids (no non-canonical residues)

## Scoring

The evaluator uses PyRosetta's `ref2015` score function to compute:

- **total_energy**: Sum of all energy terms
- **baseline_energy**: Energy of the native (starting) sequence
- **improvement**: baseline_energy - total_energy (positive = better)
- **combined_score**: Normalized improvement = improvement / |baseline_energy|

Energy terms reported: `fa_atr`, `fa_rep`, `fa_sol`, `fa_elec`, `hbond_bb_sc`, `hbond_sc`, `p_aa_pp`, `ref`

## Constraints

- Only modify `scripts/init.py`
- Do NOT modify backbone coordinates
- Only use the 20 standard amino acids
- Output must be a valid PDB file
37 changes: 37 additions & 0 deletions benchmarks/ProteinDesign/FixedBackboneDesign/Task_zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# FixedBackboneDesign - 固定骨架蛋白质序列设计

## 概述

给定一个固定的蛋白质骨架(backbone),设计出最稳定的氨基酸序列。本任务对应 Agent Rosetta 论文(arXiv:2603.15952, ICML 2026)的 Case A:固定骨架规范氨基酸序列设计。

## 输入

candidate 脚本(`scripts/init.py`)接收以下参数:

- `--prepared-input`:包含目标骨架和设计位置标注的 PDB 文件路径
- `--solution-output`:设计结果 PDB 的输出路径

## 输出

candidate 必须在 `--solution-output` 输出一个 PDB 文件,包含:
1. 原始的骨架坐标(保持不变)
2. 在指定位置替换为设计后的氨基酸
3. 只能使用 20 种标准氨基酸(不允许非规范氨基酸)

## 评分方式

评估器使用 PyRosetta 的 `ref2015` 能量函数计算:

- **total_energy**:所有能量项的加和
- **baseline_energy**:初始序列的能量
- **improvement**:baseline_energy - total_energy(正数表示改进)
- **combined_score**:归一化改进值 = improvement / |baseline_energy|

报告的能量项:`fa_atr`、`fa_rep`、`fa_sol`、`fa_elec`、`hbond_bb_sc`、`hbond_sc`、`p_aa_pp`、`ref`

## 约束

- 只能修改 `scripts/init.py`
- 不得修改骨架坐标
- 只能使用 20 种标准氨基酸
- 输出必须是有效的 PDB 文件
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
FixedBackboneDesign baseline (no additional indexes beyond initial)
==============================================================
Date: 2026-07-14
Evaluator: verification/evaluator.py (evaluate() function)
Candidate: scripts/init.py (baseline: single-round PackRotamers with default settings)
Environment: Docker (rosettacommons/rosetta:serial), PyRosetta ref2015

Results:
--------
valid: 1.0
combined_score: 0.025
native_energy: -169.18
total_energy: -173.48
improvement: +4.30

Energy terms:
fa_atr: -377.73
fa_rep: 110.97
fa_sol: 245.04
fa_elec: -133.95
hbond_bb_sc: -4.62
hbond_sc: -6.87
p_aa_pp: -23.02
ref: 12.74

Note:
- The baseline uses a single-round PackRotamersMover with packer.task().
- Non-design positions are prevented from repacking via prevent_repacking().
- Constraint verification (_verify_constraints) passed: no backbone movement,
no unexpected mutations at non-design positions.
- Random seed fixed to 42 for reproducibility.
14 changes: 14 additions & 0 deletions benchmarks/ProteinDesign/FixedBackboneDesign/data/raw_task.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"task_name": "fixed_backbone_design_demo",
"description": "Fixed-backbone protein sequence design using PyRosetta ref2015 scoring",
"pdb_id": "1ubq",
"pdb_path": "references/petrobind/1ubq.pdb",
"design_positions": [2, 4, 6, 8, 10, 12, 14, 16],
"native_sequence": "MQIFVKTLTGKTITLEVEPSDTIENVKAKIQDKEGIPPDQQRLIFAGKQLEDGRTLSDYNIQKESTLHLVLRLRGG",
"energy_function": "ref2015",
"task_config": {
"n_design_positions": 8,
"allow_design_of_native": true,
"description": "Design 8 core positions of ubiquitin (PDB 1UBQ)"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Task.md
Task_zh-CN.md
scripts/init.py
verification/evaluator.py
verification/requirements.txt
data/raw_task.json
references/constants.json
references/petrobind/1ubq.pdb
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
metrics.json
outputs/*.pdb
outputs/*.meta.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
scripts/init.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
1) Only modify `scripts/init.py` — it is the ONLY editable file.
2) Keep CLI contract unchanged: the candidate must accept `--prepared-input` and `--solution-output`.
3) Output must be a PDB file with designed amino acid substitutions.
4) Only use the 20 standard amino acids (no non-canonical residues).
5) Do NOT modify the backbone coordinates (fixed-backbone design).
6) Optimize the Rosetta total_score (lower is better). Use the ref2015 score function.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
scripts
data
verification
references
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{python} {benchmark}/verification/evaluator.py {candidate}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
scripts/init.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
data
verification
references
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"energy_function": "ref2015",
"score_terms": {
"fa_atr": {"weight": 1.0, "description": "Lennard-Jones attractive (van der Waals)"},
"fa_rep": {"weight": 0.55, "description": "Lennard-Jones repulsive (steric clash)"},
"fa_sol": {"weight": 1.0, "description": "Lazaridis-Karplus solvation energy"},
"fa_elec": {"weight": 1.0, "description": "Coulomb electrostatic potential"},
"hbond_bb_sc": {"weight": 1.0, "description": "Backbone-sidechain hydrogen bonds"},
"hbond_sc": {"weight": 1.0, "description": "Sidechain-sidechain hydrogen bonds"},
"p_aa_pp": {"weight": 0.6, "description": "Ramachandran torsion preference"},
"ref": {"weight": 1.0, "description": "Per-residue reference energy"}
}
}
Loading
Loading