Skip to content

Feat/issue 91 eval optimize loop#132

Open
zzp1221 wants to merge 3 commits into
trpc-group:mainfrom
zzp1221:feat/issue-91-eval-optimize-loop
Open

Feat/issue 91 eval optimize loop#132
zzp1221 wants to merge 3 commits into
trpc-group:mainfrom
zzp1221:feat/issue-91-eval-optimize-loop

Conversation

@zzp1221

@zzp1221 zzp1221 commented Jul 7, 2026

Copy link
Copy Markdown

概要

本 PR 在 examples/optimization/eval_optimize_loop/ 下新增一个可复现的“评测 -> 失败归因 -> prompt 优化 -> 验证 -> 接
受/拒绝 gate -> 审计报告”闭环示例。

示例覆盖:

  • 使用 AgentEvaluator 执行 baseline 评测
  • 按最终回复不匹配、工具调用错误、工具参数错误、rubric 不达标、知识召回不足、格式不符合要求进行失败归因
  • 默认使用离线 deterministic fake backend 进行 prompt 优化,避免依赖 API key
  • 可选支持 --backend agent_optimizer 调用真实 AgentOptimizer
  • 使用独立 validation split 验证候选 prompt
  • 通过可配置 gate 判断 accept/reject
  • 输出 optimization_report.jsonoptimization_report.md 和 trace evalset 审计产物

交付物

包含以下文件:

  • run_pipeline.py
  • train.evalset.json
  • val.evalset.json
  • prompts/system.md
  • prompts/skill.md
  • optimizer.json
  • README.md
  • design.md
  • optimization_report.json
  • optimization_report.md
  • trace_evalsets/ 示例输出

样例 evalset 共 6 条 case:3 条训练、3 条验证,覆盖可优化成功、优化无效、优化后退化三类情况。

验证

本地已运行:

py -3.14 examples\optimization\eval_optimize_loop\run_pipeline.py

输出:

Optimization gate decision: reject
Report written to: optimization_report.json

这里的 reject 是预期结果:候选 prompt 修复了训练集和一个验证集 case,但引入了验证集退化,包括关键 case 退化,因此 gate
正确拒绝。

@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@099b571). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #132   +/-   ##
==========================================
  Coverage        ?   87.86295%           
==========================================
  Files           ?         467           
  Lines           ?       44014           
  Branches        ?           0           
==========================================
  Hits            ?       38672           
  Misses          ?        5342           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@zzp1221 zzp1221 force-pushed the feat/issue-91-eval-optimize-loop branch from 04c1f74 to 1808173 Compare July 7, 2026 08:02
@zzp1221

zzp1221 commented Jul 7, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant