Feat/issue 91 eval optimize loop

examples/optimization/eval_optimize_loop/README.md

@@ -0,0 +1,81 @@
+# Evaluation + Optimization closed loop
+
+This example implements a reproducible "evaluate -> attribute failures -> optimize prompt -> validate candidate -> gate -> audit" loop for issue #91.
+
+It is intentionally runnable without a real API key. The default backend is deterministic fake/trace mode: it generates baseline and candidate traces, evaluates both with `AgentEvaluator`, applies the acceptance gate, and writes `optimization_report.json` plus `optimization_report.md`. The fake trace model recognizes the public sample IDs and also has query/trace-based fallbacks for JSON-format, tool-argument, private-knowledge, and validation-regression cases, so the gate logic is not tied only to the checked-in IDs.
+
+The same script also supports `--backend agent_optimizer`. That path calls `AgentOptimizer.optimize(...)` against the configured `TargetPrompt` files and uses the optimized prompt text to regenerate candidate traces before the final validation/gate/report stages.
+
+## Files
+
+```text
+examples/optimization/eval_optimize_loop/
+|-- run_pipeline.py
+|-- optimizer.json
+|-- train.evalset.json
+|-- val.evalset.json
+|-- optimization_report.json
+|-- optimization_report.md
+|-- prompts/
+|   |-- system.md
+|   `-- skill.md
+`-- trace_evalsets/
+    |-- baseline_train.evalset.json
+    |-- baseline_val.evalset.json
+    |-- candidate_train.evalset.json
+    `-- candidate_val.evalset.json
+```
+
+## Run offline
+
+From the repository root:
+
+```bash
+python examples/optimization/eval_optimize_loop/run_pipeline.py
+```
+
+If `python` is not on PATH, use your active interpreter, for example:
+
+```bash
+py -3.14 examples/optimization/eval_optimize_loop/run_pipeline.py
+```
+
+The script writes reports into this directory by default. The expected gate decision for the public sample is `reject`: the candidate improves the training split and one validation case, but introduces validation regressions, including a critical case.
+
+## Data design
+
+The sample has six public cases:
+
+- `train_format_json`: optimizable response-format failure.
+- `train_tool_args`: optimizable tool-argument failure.
+- `train_knowledge_gap`: optimization is ineffective because missing private knowledge cannot be fixed by prompt wording alone.
+- `val_format_json`: validation case that improves.
+- `val_critical_discount`: validation hard regression and critical-case regression.
+- `val_stable_refund`: validation regression caused by overfitting.
+
+## Report contract
+
+`optimization_report.json` includes:
+
+- `baseline`: train/validation scores, pass/fail, metric scores, failure reasons, and traces.
+- `optimization`: every round's input failure attribution, full candidate prompts, token usage, model-call count, seed, and cost.
+- `candidate`: train/validation scores after the candidate.
+- `delta`: per-case deltas, new passes, new failures, score improvements, and score regressions.
+- `gate_decision`: accept/reject decision with every configured gate check and reason.
+- `failure_attribution`: failure-type counts for every phase.
+- `audit`: reproducibility config and generated trace evalset artifacts.
+
+## Optional real optimizer backend
+
+The checked-in example does not store secrets. To run the real optimizer backend, pass credentials through environment variables and keep `optimizer.json` using environment references.
+
+For the DeepSeek OpenAI-compatible endpoint:
+
+```powershell
+$env:TRPC_AGENT_OPT_BASE_URL="https://api.deepseek.com/v1"
+$env:TRPC_AGENT_OPT_MODEL="deepseek-v4-pro"
+$env:TRPC_AGENT_OPT_API_KEY="<your key>"
+py -3.14 examples/optimization/eval_optimize_loop/run_pipeline.py --backend agent_optimizer
+```
+
+The optimizer stage uses response-only metrics because `AgentOptimizer` runs through a black-box `call_agent` callback. The final report still reruns trace-mode baseline/candidate validation with the full metric set, including tool trajectory checks. The default `backend` remains `fake` so CI and reviewer machines can run without network access.

examples/optimization/eval_optimize_loop/design.md

@@ -0,0 +1,7 @@
+# Design note
+
+本示例把“评测、归因、优化、验证、准入、审计”串成可复现闭环。入口脚本先把 train 和 validation 引用转成 trace evalset，用 `AgentEvaluator` 评估 baseline，再按 case 汇总失败原因。归因不只看总分，而是同时检查最终回答、工具轨迹、工具参数和 fake judge rubric：文本不一致归为 response mismatch，工具名或调用顺序错误归为 tool call error，参数差异归为 tool argument error，JSON 契约失败归为 format noncompliance，缺少私有知识且提示词无法补足时归为 knowledge recall gap。
+
+接受策略由 `optimizer.json` 的 gate 控制。候选必须在验证集达到最小分数提升，不能引入新的 hard failure，关键 case 不能退化，并且成本必须在预算内；任一条件失败都会拒绝。该样例故意让候选修复训练集和一个验证集格式问题，同时使 `val_critical_discount` 与 `val_stable_refund` 退化，因此最终 gate 应拒绝，用来演示“训练收益不等于可发布”。
+
+防过拟合策略包括 train/validation 分离、关键验证 case 白名单、逐 case delta 对比、记录新增失败和退化分数，并把不可由 prompt 解决的 knowledge gap 与可优化格式/参数问题分开处理。产物审计方面，报告保存 baseline、candidate、每轮优化输入、候选 prompt、seed、成本、模型调用数、门禁检查、失败归因统计和生成的 trace evalset 路径；默认 fake backend 无需密钥，便于 CI 和评审者复跑比对。