diff --git a/docs/superpowers/plans/2026-07-30-eval-optimize-loop-optimization.md b/docs/superpowers/plans/2026-07-30-eval-optimize-loop-optimization.md new file mode 100644 index 000000000..317ab6b1b --- /dev/null +++ b/docs/superpowers/plans/2026-07-30-eval-optimize-loop-optimization.md @@ -0,0 +1,1877 @@ +# Eval-Optimize Loop Pipeline Optimization Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 把 `examples/optimization/eval_optimize_loop` 流水线改造成符合 `docs/mkdocs/zh/optimization.md` 规范的「评测 → 失败归因 → Prompt 优化 → 回归验证 → 产物审计」闭环，修复 4 个会让结论悄悄变错的 P0，让 real 模式真正能落地。 + +**Architecture:** 引入 `EvalBackend` Protocol（TraceBackend / LiveBackend）收敛「agent 的实际输出从哪来」这一唯一差异；Stage 4 用 `applied_prompts` 上下文显式写回候选并还原 baseline；metric 分两层（gate ⊇ optimizer）以绕开 SDK 对 `call_agent` 模式的硬约束；产物落 `/` 子目录。 + +**Tech Stack:** Python 3.10+、pydantic v2（`EvalBaseModel` + `to_camel` 别名）、trpc-agent SDK（`AgentEvaluator.get_executer`、`AgentOptimizer.optimize`、`TargetPrompt`、`OptimizeResult`、`RemoteEvalService`）。 + +## Global Constraints + +| 约束 | 来源 | 体现 | +|---|---|---| +| demo + real 两套模式都保留，real 是主线 | 用户选择 | run_pipeline.py 仅在组装 backend 时分支一次 | +| real 模式 Stage 1/4 用 live，Stage 3 走 `call_agent`（SDK 硬约束） | optimization.md §2.4 / §7.2 | `EvalBackend` 提供 `evaluate(eval_set_path, ..., runner=...)` | +| 优化器不能用 `tool_trajectory_avg_score` 与 `llm_rubric_knowledge_recall` | `_DISALLOWED_METRICS_IN_CALL_AGENT_MODE` | `data/optimizer.json` 不含这两个 metric；`test_optimizer_config.py` 守门 | +| 优化器不能用 trace 数据集 | `RemoteEvalService._reject_trace_cases` | `data/live/` 两个文件路径必须不同；测试守门 | +| `update_source=False` 时 SDK 会回滚 baseline | optimization.md FAQ 末条 | Stage 4 用 `applied_prompts` 显式写回 | +| 每次运行用独立时间戳子目录 | optimization.md §8.1 | `output//{optimizer/,optimization_report.json,.md}` | +| 优雅停机用 sentinel 文件 `optimize.stop` | optimization.md §8.2 | 仅文档说明，不在代码里加 sentinel 监听（避免引入未公开行为） | +| scenario 按 `eval_id` 后缀推导，不维护映射表 | README §5 | run_pipeline.py 删 SCENARIO_MAP，调用 `derive_scenario(eval_id)` | +| Agent 每次重建以重读 `system.md` | optimization.md §4.1-§4.3 | `call_agent` 闭包内 `create_agent()`；测试用计数器断言 | +| `max_metric_calls` 必须 > 验证集规模，避免 budget 抢闸 | optimization.md §6.4 + README §8 | `optimizer.json` 设 150（baseline 20 + 5×6 + 2×20 ≈ 90，留 1.6×） | + +--- + +## File Structure + +| 文件 | 类型 | 职责 | +|---|---|---| +| `pipeline/_eval_backend.py` | 新建 | `EvalBackend` Protocol、`TraceBackend`、`LiveBackend` | +| `pipeline/_runner.py` | 重写 | 编排 6 阶段，接 `EvalBackend` | +| `pipeline/_stage_baseline.py` | 修改 | `except AssertionError` 替代 `except Exception` | +| `pipeline/_stage_validation.py` | 重写 | `applied_prompts` 写回 + 还原；接 backend | +| `pipeline/_stage_optimization.py` | 修改 | 透传 audit 字段、verbose=1、产物落 `/optimizer/` | +| `pipeline/_models.py` | 修改 | `OptimizationExecutionReport` 加 stop_reason/finish_reason/total_token_usage/total_metric_calls/rounds | +| `pipeline/_stage_audit_trail.py` | 修改 | 渲染新字段 | +| `data/_generate_evalsets.py` | 重写 | 一次生成 `trace/` + `live/` 两套 | +| `data/test_config.json` | 删除 | 被 `gate_metrics.json` 取代 | +| `data/gate_metrics.json` | 新建 | 2 个 metric（final_response + tool_trajectory），num_runs=2 | +| `data/optimizer.json` | 修改 | max_metric_calls=150、timeout_seconds=1800、pareto、skip_perfect_score=true | +| `data/trace/{train,val_baseline,val_optimized}.evalset.json` | 新建 | demo 三件套（eval_mode=trace） | +| `data/live/{train,val}.evalset.json` | 新建 | real 模式（无 actual_conversation） | +| `data/demo_optimize_result.json` | 修改 | 在 `rounds[]` 基础上加 stop_reason 等顶层字段（已是 camelCase 别名） | +| `run_pipeline.py` | 重写 | 组装 backend；删 SCENARIO_MAP；时间戳子目录 | +| `tests/test_eval_backend.py` | 新建 | TraceBackend / LiveBackend 契约 | +| `tests/test_call_agent_rebuild.py` | 新建 | 计数器断言 call_agent 每次重建 agent | +| `tests/test_applied_prompts.py` | 新建 | 异常时仍还原 baseline | +| `tests/test_datasets_complete.py` | 新建 | 5 个数据集文件齐全 | +| `tests/test_live_datasets_no_trace.py` | 新建 | live 数据集不含 `eval_mode=trace` | +| `tests/test_gate_superset_of_optimizer.py` | 新建 | gate_metrics ⊇ optimizer metrics | +| `tests/test_scenario_derivation.py` | 新建 | 后缀→scenario 推导 | +| `README.md` | 修改 | 路径表更新（trace/live 子目录、gate_metrics.json） | + +> 已有 `tests/test_optimizer_config.py` 已守住黑盒不兼容 metric（不需新增）；已有 `tests/test_pipeline_fake.py` 仍走 trace 路径，需更新路径引用。 + +--- + +## Task 1: EvalBackend Protocol + TraceBackend + +**Files:** +- Create: `pipeline/_eval_backend.py` + +**Interfaces:** +- Consumes: `EvaluateResult` from `trpc_agent_sdk.evaluation._eval_result`; `EvalConfig` from `trpc_agent_sdk.evaluation._eval_config`; `AgentEvaluator.get_executer` from `trpc_agent_sdk.evaluation._agent_evaluator` +- Produces: `EvalBackend` Protocol with `async evaluate(*, eval_set_path: str, metrics_config_path: str, num_runs: int = 1) -> tuple[EvaluateResult, EvalSetReport]`; `TraceBackend` 实现 + +- [ ] **Step 1: Create EvalBackend Protocol + TraceBackend** + +```python +# pipeline/_eval_backend.py +"""EvalBackend: 封装「agent 的实际输出从哪来」这一唯一差异。""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Protocol, runtime_checkable + +from trpc_agent_sdk.evaluation._agent_evaluator import AgentEvaluator +from trpc_agent_sdk.evaluation._eval_config import EvalConfig +from trpc_agent_sdk.evaluation._eval_metrics import EvalStatus +from trpc_agent_sdk.evaluation._eval_result import EvaluateResult + +from pipeline._models import EvalSetReport, PerCaseScore + + +@runtime_checkable +class EvalBackend(Protocol): + async def evaluate( + self, + *, + eval_set_path: str, + metrics_config_path: str, + num_runs: int = 1, + ) -> tuple[EvaluateResult, EvalSetReport]: ... + + +class TraceBackend: + """trace 模式 backend: 不调用 agent, 直接从预录制轨迹计算 metric.""" + + async def evaluate( + self, + *, + eval_set_path: str, + metrics_config_path: str, + num_runs: int = 1, + ) -> tuple[EvaluateResult, EvalSetReport]: + with open(metrics_config_path, "r") as f: + config_data = json.load(f) + EvalConfig(**config_data) # 校验 + + executer = AgentEvaluator.get_executer( + eval_dataset_file_path_or_dir=eval_set_path, + eval_metrics_file_path_or_dir=metrics_config_path, + num_runs=num_runs, + ) + # 只吞 AssertionError — 基线本来就该有失败 case + try: + await executer.evaluate() + except AssertionError: + pass + + raw: EvaluateResult | None = executer.get_result() + if raw is None: + raise RuntimeError(f"评测失败 ({eval_set_path}): 无返回结果") + return raw, _build_report(raw) + + +def _build_report(raw: EvaluateResult) -> EvalSetReport: + """Build EvalSetReport from raw SDK EvaluateResult. (Move from _stage_baseline.)""" + per_case: list[PerCaseScore] = [] + total = passed = failed = 0 + metric_scores: dict[str, list[float]] = {} + captured_set_id = "unknown" + + for eval_set_id, aggregate in raw.results_by_eval_set_id.items(): + captured_set_id = eval_set_id + for eval_id, runs in aggregate.eval_results_by_eval_id.items(): + total += 1 + case_result = runs[0] + if case_result.final_eval_status == EvalStatus.PASSED: + passed += 1 + elif case_result.final_eval_status == EvalStatus.FAILED: + failed += 1 + + scores: dict[str, float] = {} + statuses: dict[str, str] = {} + for m in case_result.overall_eval_metric_results: + scores[m.metric_name] = m.score or 0.0 + statuses[m.metric_name] = str(m.eval_status.name) if m.eval_status else "NOT_EVALUATED" + metric_scores.setdefault(m.metric_name, []).append(m.score or 0.0) + + per_case.append(PerCaseScore( + eval_id=eval_id, + overall_status=str(case_result.final_eval_status.name) if case_result.final_eval_status else "NOT_EVALUATED", + metric_scores=scores, + metric_statuses=statuses, + )) + + breakdown = {n: sum(s) / len(s) if s else 0.0 for n, s in metric_scores.items()} + return EvalSetReport( + eval_set_id=captured_set_id, + num_cases=total, + num_passed=passed, + num_failed=failed, + pass_rate=passed / total if total > 0 else 0.0, + metric_breakdown=breakdown, + per_case=per_case, + ) +``` + +- [ ] **Step 2: Verify imports succeed** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._eval_backend import TraceBackend, EvalBackend; print('ok')" +``` + +Expected: prints `ok`. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_eval_backend.py +git commit -m "feat(pipeline): add EvalBackend Protocol with TraceBackend" +``` + +--- + +## Task 2: LiveBackend (real 模式核心) + +**Files:** +- Modify: `pipeline/_eval_backend.py` + +**Interfaces:** +- Consumes: `create_agent` from `agent.agent`; SDK `AgentEvaluator.get_executer(runner=...)` → LocalEvalService +- Produces: `LiveBackend` 类；持 `agent_factory`（而非实例），每次 `evaluate()` 现建 Runner + +> **Why factory, not instance**: LlmAgent 在构造时固化 `instruction`；持实例会让 Stage 4 拿到旧 prompt。optimization.md §4.1-§4.3 明确要求每次重建。 + +- [ ] **Step 1: Add LiveBackend class to _eval_backend.py** + +Append to `pipeline/_eval_backend.py`: + +```python +from agent.agent import create_agent +from trpc_agent_sdk.evaluation import LocalEvalService # SDK export; fall back to underlying class if absent + + +class LiveBackend: + """real 模式 backend: 每次 evaluate() 现建 Runner（→ 新 agent → 重读 system.md）。""" + + def __init__(self, agent_factory=create_agent) -> None: + self._agent_factory = agent_factory + self._agents_built: int = 0 # 供测试断言 + + def agents_built(self) -> int: + return self._agents_built + + async def evaluate( + self, + *, + eval_set_path: str, + metrics_config_path: str, + num_runs: int = 1, + ) -> tuple[EvaluateResult, EvalSetReport]: + agent = self._agent_factory(demo_mode=False) + self._agents_built += 1 + + executer = AgentEvaluator.get_executer( + eval_dataset_file_path_or_dir=eval_set_path, + eval_metrics_file_path_or_dir=metrics_config_path, + num_runs=num_runs, + runner=LocalEvalService(agent=agent), # LocalEvalService 暴露 agent 内部轨迹 + ) + try: + await executer.evaluate() + except AssertionError: + pass + + raw: EvaluateResult | None = executer.get_result() + if raw is None: + raise RuntimeError(f"评测失败 ({eval_set_path}): 无返回结果") + return raw, _build_report(raw) +``` + +> **如果 SDK 的 `LocalEvalService` 公开 API 与上面不一致**：读 `trpc_agent_sdk/evaluation/_local_eval_service.py` 找到正确的构造方式，按 SDK 实际签名调整；`runner=` 是 `AgentEvaluator.get_executer` 已支持的命名参数（见 `_agent_evaluator.py:364-420`）。 + +- [ ] **Step 2: Verify import** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._eval_backend import LiveBackend; print('ok')" +``` + +Expected: prints `ok` (without invoking, which would require API key). + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_eval_backend.py +git commit -m "feat(pipeline): add LiveBackend for real-mode evaluation" +``` + +--- + +## Task 3: Refactor _stage_baseline.py to use TraceBackend & narrow exception handling + +**Files:** +- Modify: `pipeline/_stage_baseline.py` + +**Interfaces:** +- Consumes: `TraceBackend.evaluate` (Task 1) +- Produces: 保留 `BaselineEvaluator.evaluate(...)` 但内部委托 backend；删掉内嵌 `_build_report`（已在 `_eval_backend.py`） + +> **Why narrow except**: README §6 Stage 1 边界 — `except Exception: pass` 会把「文件缺失」静默成全 0 分，Stage 2 把它归因为 prompt 问题（最坏失败模式）。 + +- [ ] **Step 1: Replace _stage_baseline.py** + +```python +"""Stage 1: 基线评测 — 对训练集/验证集运行 AgentEvaluator.""" + +from __future__ import annotations + +from trpc_agent_sdk.evaluation._eval_result import EvaluateResult + +from pipeline._eval_backend import EvalBackend +from pipeline._models import EvalSetReport + + +class BaselineEvaluator: + """基线评测执行器 — 委托给 EvalBackend.""" + + @staticmethod + async def evaluate( + *, + eval_set_path: str, + metrics_config_path: str, + backend: EvalBackend, + num_runs: int = 1, + ) -> tuple[EvaluateResult, EvalSetReport]: + return await backend.evaluate( + eval_set_path=eval_set_path, + metrics_config_path=metrics_config_path, + num_runs=num_runs, + ) +``` + +- [ ] **Step 2: Verify imports** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._stage_baseline import BaselineEvaluator; print('ok')" +``` + +Expected: prints `ok`. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_stage_baseline.py +git commit -m "refactor(pipeline): narrow exception handling and delegate to EvalBackend" +``` + +--- + +## Task 4: applied_prompts context manager + +**Files:** +- Modify: `pipeline/_eval_backend.py`（同一文件，新增模块） + +**Interfaces:** +- Consumes: `TargetPrompt.write_all` from `trpc_agent_sdk.evaluation._target_prompt` +- Produces: `applied_prompts(target_prompt, prompts: dict[str, str])` 异步上下文管理器；用 `TargetPrompt.write_all` 保证原子写 + 多字段失败回滚 + +> **Why this exists**: `AgentOptimizer.optimize(update_source=False)` 在 `finally` 把源文件回滚成 baseline（README §3.3）。如果验证候选不显式写回，Stage 4 会拿到 baseline 而非候选。 + +- [ ] **Step 1: Add applied_prompts context manager** + +Append to `pipeline/_eval_backend.py`: + +```python +from contextlib import asynccontextmanager +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt + + +@asynccontextmanager +async def applied_prompts(target_prompt: TargetPrompt, prompts: dict[str, str]): + """写入候选 prompt, 退出时还原 baseline. 异常路径也保证还原.""" + baseline = target_prompt.read_all() # SDK 已提供 + try: + target_prompt.write_all(prompts) # SDK 原子写 + 多字段失败回滚 + yield + finally: + target_prompt.write_all(baseline) +``` + +> 验证 SDK 实际方法名: `trpc_agent_sdk/evaluation/_target_prompt.py` 中 `read_all` 与 `write_all` 是否存在；若 SDK 仅暴露 `read_text/write_text`（单字段），则改为： + +```python +@asynccontextmanager +async def applied_prompts(target_prompt: TargetPrompt, prompts: dict[str, str]): + baseline: dict[str, str] = {} + for name, path in target_prompt.paths.items(): + baseline[name] = Path(path).read_text(encoding="utf-8") + try: + for name, content in prompts.items(): + Path(target_prompt.paths[name]).write_text(content, encoding="utf-8") + yield + finally: + for name, content in baseline.items(): + Path(target_prompt.paths[name]).write_text(content, encoding="utf-8") +``` + +按 SDK 实际签名选其一，注释里说明选型理由。 + +- [ ] **Step 2: Verify import** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._eval_backend import applied_prompts; print('ok')" +``` + +Expected: prints `ok`. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_eval_backend.py +git commit -m "feat(pipeline): add applied_prompts context manager for candidate write-back" +``` + +--- + +## Task 5: Stage 4 (candidate validation) uses applied_prompts + +**Files:** +- Modify: `pipeline/_stage_validation.py` + +**Interfaces:** +- Consumes: `EvalBackend` (Task 1/2); `applied_prompts` (Task 4); `TargetPrompt` (with `best_prompts`) +- Produces: `ValidationComparator.evaluate_and_compare(backend, val_baseline_path, best_prompts, target_prompt, gate_metrics_path, baseline_report, scenario_map)` — 在 `applied_prompts` 内重评同一份 val 文件 + +> **Why this matters**: README §3.3/§6 Stage 4 — demo 用 `val_optimized.evalset.json` 当候选，但 real 必须写回 `best_prompts` 再重评同一份 val 集。 + +- [ ] **Step 1: Replace _stage_validation.py** + +```python +"""Stage 4: 候选验证 — 写回 best_prompts 后重评验证集，与基线逐 case 对比.""" + +from __future__ import annotations + +from trpc_agent_sdk.evaluation._eval_metrics import EvalStatus +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt + +from pipeline._eval_backend import EvalBackend, applied_prompts +from pipeline._models import EvalSetReport, PerCaseDelta + + +class ValidationComparator: + @staticmethod + async def evaluate_and_compare( + *, + backend: EvalBackend, + val_eval_path: str, + metrics_config_path: str, + target_prompt: TargetPrompt, + best_prompts: dict[str, str], + baseline_report: EvalSetReport, + scenario_map: dict[str, str] | None = None, + num_runs: int = 2, + ) -> tuple[EvalSetReport, list[PerCaseDelta]]: + if scenario_map is None: + scenario_map = {} + + # demo 模式下 best_prompts 为空 → 不写回 + if best_prompts: + async with applied_prompts(target_prompt, best_prompts): + _, candidate_report = await backend.evaluate( + eval_set_path=val_eval_path, + metrics_config_path=metrics_config_path, + num_runs=num_runs, + ) + else: + _, candidate_report = await backend.evaluate( + eval_set_path=val_eval_path, + metrics_config_path=metrics_config_path, + num_runs=num_runs, + ) + + return candidate_report, _compute_deltas( + baseline_report=baseline_report, + candidate_report=candidate_report, + scenario_map=scenario_map, + ) + + +def _compute_deltas( + *, + baseline_report: EvalSetReport, + candidate_report: EvalSetReport, + scenario_map: dict[str, str], +) -> list[PerCaseDelta]: + base = {c.eval_id: c for c in baseline_report.per_case} + cand = {c.eval_id: c for c in candidate_report.per_case} + deltas: list[PerCaseDelta] = [] + + for eval_id in sorted(set(base) | set(cand)): + b = base.get(eval_id) + c = cand.get(eval_id) + b_status = b.overall_status if b else "MISSING" + c_status = c.overall_status if c else "MISSING" + b_scores = dict(b.metric_scores) if b else {} + c_scores = dict(c.metric_scores) if c else {} + delta = { + m: c_scores.get(m, 0.0) - b_scores.get(m, 0.0) + for m in (set(b_scores) | set(c_scores)) + } + deltas.append(PerCaseDelta( + eval_id=eval_id, + scenario=scenario_map.get(eval_id, "unknown"), + baseline_status=b_status, + candidate_status=c_status, + baseline_scores=b_scores, + candidate_scores=c_scores, + score_delta=delta, + transition=f"{b_status}->{c_status}", + )) + return deltas +``` + +- [ ] **Step 2: Verify imports** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._stage_validation import ValidationComparator; print('ok')" +``` + +Expected: prints `ok`. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_stage_validation.py +git commit -m "refactor(pipeline): stage 4 writes best_prompts via applied_prompts" +``` + +--- + +## Task 6: _models.py — extend OptimizationExecutionReport with audit fields + +**Files:** +- Modify: `pipeline/_models.py` + +**Interfaces:** +- Produces: `OptimizationExecutionReport` 新增字段 `stop_reason: str`、`finish_reason: str`、`total_token_usage: dict[str, int]`、`total_metric_calls: int`、`rounds: list[dict]`（与 SDK `RoundRecord` 字段对齐，但保持 dict 以避免模型耦合） + +- [ ] **Step 1: Add new fields to OptimizationExecutionReport** + +In `pipeline/_models.py`, replace the `OptimizationExecutionReport` class with: + +```python +class OptimizationExecutionReport(EvalBaseModel): + algorithm: str + status: str + total_rounds: int + baseline_pass_rate: float + best_pass_rate: float + pass_rate_improvement: float + duration_seconds: float + total_llm_cost: float + best_prompts: dict[str, str] = Field(default_factory=dict) + + # 审计字段 (Stage 6 报告消费) + stop_reason: str = "" # SDK: result.stop_reason + finish_reason: str = "" # SDK: result.finish_reason + total_token_usage: dict[str, int] = Field(default_factory=dict) # SDK: result.total_token_usage + total_metric_calls: int = 0 # SDK: result.total_metric_calls + rounds: list[dict] = Field(default_factory=list) # SDK: result.rounds +``` + +- [ ] **Step 2: Verify existing tests still pass** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_models.py -v +``` + +Expected: all pass. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_models.py +git commit -m "feat(pipeline): extend OptimizationExecutionReport with audit fields" +``` + +--- + +## Task 7: _stage_optimization.py — plumb audit fields + verbose=1 + sub-output dir + +**Files:** +- Modify: `pipeline/_stage_optimization.py` + +**Interfaces:** +- Consumes: SDK `OptimizeResult` 字段 `stop_reason` / `finish_reason` / `total_token_usage` / `total_metric_calls` / `rounds` +- Produces: `OptimizationExecutionReport` 含 Task 6 新字段；产物落 `//optimizer/` + +> **Why verbose=1**: optimization.md §4.7 建议实时观察，避免盲等。 +> **Why sub-output dir**: optimization.md §8.1 时间戳子目录 + README §9 「让 SDK 产物与流水线报告不再混在一起」。 + +- [ ] **Step 1: Update run_real signature and _from_optimize_result** + +Replace `pipeline/_stage_optimization.py`: + +```python +"""Stage 3: 优化执行 — 运行 AgentOptimizer 或加载 demo 结果.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Optional + +from trpc_agent_sdk.evaluation._optimize_result import OptimizeResult + +from pipeline._models import OptimizationExecutionReport + + +class OptimizationExecutor: + @staticmethod + async def run_real( + *, + config_path: str, + call_agent, + target_prompt, + train_dataset_path: str, + validation_dataset_path: str, + output_dir: str, # 期望是 / 子目录 + ) -> OptimizationExecutionReport: + from trpc_agent_sdk.evaluation._agent_optimizer import AgentOptimizer + + optimizer_subdir = Path(output_dir) / "optimizer" + optimizer_subdir.mkdir(parents=True, exist_ok=True) + + result: OptimizeResult = await AgentOptimizer.optimize( + config_path=config_path, + call_agent=call_agent, + target_prompt=target_prompt, + train_dataset_path=train_dataset_path, + validation_dataset_path=validation_dataset_path, + output_dir=str(optimizer_subdir), + update_source=False, + verbose=1, + ) + return OptimizationExecutor._from_optimize_result(result) + + @staticmethod + def run_demo(demo_result_path: str) -> OptimizationExecutionReport: + result = OptimizeResult.from_file(demo_result_path) + return OptimizationExecutor._from_optimize_result(result) + + @staticmethod + def _from_optimize_result(result: OptimizeResult) -> OptimizationExecutionReport: + # SDK RoundRecord → dict（camelCase 别名自动转, dict 也可） + rounds_payload: list[dict[str, Any]] = [] + for r in getattr(result, "rounds", []) or []: + if hasattr(r, "model_dump"): + rounds_payload.append(r.model_dump(by_alias=True)) + elif isinstance(r, dict): + rounds_payload.append(r) + else: + rounds_payload.append(json.loads(json.dumps(r, default=str))) + + token_usage = getattr(result, "total_token_usage", {}) or {} + if hasattr(token_usage, "model_dump"): + token_usage = token_usage.model_dump(by_alias=True) + + return OptimizationExecutionReport( + algorithm=result.algorithm, + status=result.status, + total_rounds=result.total_rounds, + baseline_pass_rate=result.baseline_pass_rate, + best_pass_rate=result.best_pass_rate, + pass_rate_improvement=result.pass_rate_improvement, + duration_seconds=result.duration_seconds, + total_llm_cost=result.total_llm_cost, + best_prompts=dict(result.best_prompts), + stop_reason=getattr(result, "stop_reason", ""), + finish_reason=getattr(result, "finish_reason", ""), + total_token_usage=dict(token_usage), + total_metric_calls=getattr(result, "total_metric_calls", 0), + rounds=rounds_payload, + ) +``` + +- [ ] **Step 2: Verify imports** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._stage_optimization import OptimizationExecutor; print('ok')" +``` + +Expected: prints `ok`. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_stage_optimization.py +git commit -m "feat(pipeline): plumb audit fields and isolate optimizer output to subdir" +``` + +--- + +## Task 8: PipelineRunner — adopt EvalBackend, timestamp subdir, demo/real unification + +**Files:** +- Modify: `pipeline/_runner.py` + +**Interfaces:** +- Consumes: `EvalBackend` (Task 1/2); `applied_prompts` (Task 4); `ValidationComparator` (Task 5) +- Produces: `PipelineRunner` 构造参数 `backend: EvalBackend` 替代 `demo_mode: bool` + `demo_optimize_result_path`；时间戳子目录 `//` + +- [ ] **Step 1: Rewrite _runner.py** + +```python +"""PipelineRunner — 编排 6 阶段流水线, 通过 EvalBackend 收敛 demo/real 差异.""" + +from __future__ import annotations + +import os +import time +from datetime import datetime, timezone +from pathlib import Path +from typing import Optional + +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt + +from pipeline._eval_backend import EvalBackend +from pipeline._models import ( + AcceptanceGateConfig, + GateDecision, + PipelineReport, +) +from pipeline._stage_acceptance_gate import AcceptanceGate +from pipeline._stage_audit_trail import ReportGenerator +from pipeline._stage_baseline import BaselineEvaluator +from pipeline._stage_failure_attribution import FailureAttributor +from pipeline._stage_optimization import OptimizationExecutor +from pipeline._stage_validation import ValidationComparator + + +class PipelineRunner: + def __init__( + self, + *, + train_eval_path: str, + val_baseline_eval_path: str, + gate_metrics_config_path: str, + optimizer_config_path: str, + prompt_source_path: str, + prompt_field_name: str, + gate_config: AcceptanceGateConfig, + backend: EvalBackend, + output_dir: str = "output", + scenario_map: Optional[dict[str, str]] = None, + demo_optimize_result_path: Optional[str] = None, # 仅 backend==TraceBackend 时有意义 + train_eval_path_real: Optional[str] = None, # 仅 backend==LiveBackend 时使用 + ) -> None: + self._train_path = train_eval_path + self._val_baseline_path = val_baseline_eval_path + self._train_path_real = train_eval_path_real + self._gate_metrics_path = gate_metrics_config_path + self._optimizer_config_path = optimizer_config_path + self._prompt_source_path = prompt_source_path + self._prompt_field_name = prompt_field_name + self._gate_config = gate_config + self._backend = backend + self._output_dir = output_dir + self._scenario_map = scenario_map or {} + self._demo_result_path = demo_optimize_result_path + self._demo_mode = isinstance(backend, type(...)) and backend.__class__.__name__ == "TraceBackend" # 见 Step 1 注 + + async def run(self) -> PipelineReport: + start = time.time() + run_ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + run_dir = Path(self._output_dir) / run_ts + run_dir.mkdir(parents=True, exist_ok=True) + + report = PipelineReport( + pipeline_version="1.0.0", + timestamp=datetime.now(timezone.utc).isoformat(), + demo_mode=self._demo_mode, + ) + + # ---- Stage 1: 基线 ---- + print("[Stage 1/6] 基线评测...") + _, train_report = await BaselineEvaluator.evaluate( + eval_set_path=self._train_path, + metrics_config_path=self._gate_metrics_path, + backend=self._backend, + num_runs=2, + ) + _, val_report = await BaselineEvaluator.evaluate( + eval_set_path=self._val_baseline_path, + metrics_config_path=self._gate_metrics_path, + backend=self._backend, + num_runs=2, + ) + report.baseline_train = train_report + report.baseline_val = val_report + + # ---- Stage 2: 失败归因 ---- + print("[Stage 2/6] 失败归因...") + # 合并 raw; 为简化 Stage 2 输入, 直接从 reports 重建 + combined = {} + for c in train_report.per_case: + combined[c.eval_id] = c + for c in val_report.per_case: + combined[c.eval_id] = c + # 注意: FailureAttributor.cluster 接 dict[str, list[EvalCaseResult]]; + # 保留归一化接口, 让 stage_2 接受 dict[str, PerCaseScore] + # → 重新设计: 让 FailureAttributor.cluster 支持两路输入 + # (见 Task 9) + attribution = FailureAttributor.cluster_from_per_case(combined) + report.failure_attribution = attribution + + # ---- Stage 3: 优化 ---- + print("[Stage 3/6] 优化执行...") + if self._demo_mode and self._demo_result_path: + opt_report = OptimizationExecutor.run_demo(self._demo_result_path) + else: + from agent.agent import create_agent + + target_prompt = TargetPrompt().add_path(self._prompt_field_name, self._prompt_source_path) + counter = {"calls": 0} + + async def call_agent(input_text: str) -> str: + from trpc_agent_sdk.types import Content, Part + agent = create_agent(demo_mode=False) + counter["calls"] += 1 + user_content = Content(parts=[Part.from_text(text=input_text)]) + response = await agent.generate_content(user_content) + if response.candidates and response.candidates[0].content: + return "".join(p.text or "" for p in response.candidates[0].content.parts) + return "" + + opt_report = await OptimizationExecutor.run_real( + config_path=self._optimizer_config_path, + call_agent=call_agent, + target_prompt=target_prompt, + train_dataset_path=self._train_path_real or self._train_path, + validation_dataset_path=self._val_baseline_path, + output_dir=str(run_dir), + ) + report.optimization_execution = opt_report + + # ---- Stage 4: 候选验证 ---- + print("[Stage 4/6] 候选验证...") + target_prompt_obj = TargetPrompt().add_path(self._prompt_field_name, self._prompt_source_path) + candidate_report, deltas = await ValidationComparator.evaluate_and_compare( + backend=self._backend, + val_eval_path=self._val_baseline_path, + metrics_config_path=self._gate_metrics_path, + target_prompt=target_prompt_obj, + best_prompts=opt_report.best_prompts, + baseline_report=val_report, + scenario_map=self._scenario_map, + ) + report.candidate_validation = candidate_report + report.case_deltas = deltas + + # ---- Stage 5: 门控 ---- + print("[Stage 5/6] 接受门控...") + gate = AcceptanceGate(self._gate_config) + decision = gate.evaluate( + baseline_pass_rate=val_report.pass_rate, + candidate_pass_rate=candidate_report.pass_rate, + baseline_case_statuses={c.eval_id: c.overall_status for c in val_report.per_case}, + candidate_case_statuses={c.eval_id: c.overall_status for c in candidate_report.per_case}, + total_cost=opt_report.total_llm_cost, + ) + report.gate_decision = decision + report.overall_pass_rate_change = candidate_report.pass_rate - val_report.pass_rate + report.overall_verdict = "ACCEPTED" if decision.accepted else "REJECTED" + + # ---- Stage 6: 报告 ---- + print("[Stage 6/6] 审计落盘...") + report.pipeline_duration_seconds = time.time() - start + json_path = ReportGenerator.generate_json(report, str(run_dir)) + md_path = ReportGenerator.generate_markdown(report, str(run_dir)) + return report +``` + +**注 1**: `self._demo_mode = isinstance(backend, type(...)) and backend.__class__.__name__ == "TraceBackend"` 是反模式。正确做法：构造时显式 `demo_mode: bool` 由 `run_pipeline.py` 注入（因为 backend 在 main 里组装）。改成： + +```python +def __init__(self, *, ..., backend: EvalBackend, demo_mode: bool, ...): + self._demo_mode = demo_mode + self._backend = backend +``` + +后续 Task 11（run_pipeline.py）确保两个参数同时传入。 + +**注 2**: `FailureAttributor.cluster_from_per_case` 在 Task 9 才加；本 Task 仅占位。 + +- [ ] **Step 2: Verify imports** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._runner import PipelineRunner; print('ok')" +``` + +Expected: prints `ok`（即使 FailureAttributor.cluster_from_per_case 暂未实现，import 阶段不应失败）。 + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_runner.py +git commit -m "refactor(pipeline): runner adopts EvalBackend and timestamp subdir" +``` + +--- + +## Task 9: FailureAttributor — accept PerCaseScore dict for Stage 2 unification + +**Files:** +- Modify: `pipeline/_stage_failure_attribution.py` + +**Interfaces:** +- Produces: 新方法 `FailureAttributor.cluster_from_per_case(per_case: dict[str, PerCaseScore]) -> FailureAttributionReport`（从 Stage 1 报告直接派生归因，不再依赖 raw `EvalCaseResult`） + +> **Why this matters**: 之前 `cluster` 接 `EvalCaseResult`，依赖 SDK 内部结构。新接口从 `EvalSetReport.per_case` 派生，Stage 2 完全自给。 + +- [ ] **Step 1: Add cluster_from_per_case** + +Append to `pipeline/_stage_failure_attribution.py`: + +```python +from pipeline._models import PerCaseScore, FailureAttributionReport + + +class FailureAttributor: + @staticmethod + def cluster_from_per_case(per_case: dict[str, "PerCaseScore"]) -> FailureAttributionReport: + """从 Stage 1 报告的 per_case 派生归因. 不依赖 SDK EvalCaseResult.""" + clusters: dict[str, list[str]] = defaultdict(list) + per_case_cats: dict[str, list[str]] = {} + total = len(per_case) + total_failed = 0 + + for eval_id, score in per_case.items(): + if score.overall_status == "FAILED": + total_failed += 1 + cats: list[str] = [] + statuses = score.metric_statuses + fr = statuses.get("final_response_avg_score") == "FAILED" + tt = statuses.get("tool_trajectory_avg_score") == "FAILED" + if fr and tt: + cats.append("both_metrics_failed") + else: + if fr: + cats.append("final_response_mismatch") + if tt: + cats.append("tool_trajectory_mismatch") + if statuses.get("llm_rubric_response") == "FAILED": + cats.append("llm_rubric_fail") + if statuses.get("llm_rubric_knowledge_recall") == "FAILED": + cats.append("knowledge_recall_insufficient") + if not cats: + cats.append("unknown") + for c in cats: + clusters[c].append(eval_id) + per_case_cats[eval_id] = cats + else: + per_case_cats[eval_id] = [] + + summary = ( + f"全部 {total} 个 case 通过。无失败需要归因。" + if total_failed == 0 + else f"{total_failed}/{total} 个 case 失败。" + ( + " 失败分类: " + "; ".join( + f"{c}: {len(ids)}" for c, ids in sorted(clusters.items()) + ) if clusters else "" + ) + ) + + return FailureAttributionReport( + total_cases_evaluated=total, + total_failed=total_failed, + clusters=dict(clusters), + per_case_categories=per_case_cats, + summary=summary, + ) + + # 保留旧 cluster(...) 以便历史使用（demo 路径下若仍用 EvalCaseResult 也兼容） + @staticmethod + def cluster(results_by_eval_id): ... # 原实现不动 +``` + +- [ ] **Step 2: Verify imports** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._stage_failure_attribution import FailureAttributor; print('ok')" +``` + +Expected: prints `ok`. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_stage_failure_attribution.py +git commit -m "feat(pipeline): failure attribution accepts per_case dict" +``` + +--- + +## Task 10: _stage_audit_trail.py — render new fields + +**Files:** +- Modify: `pipeline/_stage_audit_trail.py` + +**Interfaces:** +- Produces: `_render_markdown` 加 stop_reason / finish_reason / total_token_usage / total_metric_calls / rounds 表格；report 落 `/` + +- [ ] **Step 1: Extend the Stage 3 markdown section** + +In `_render_markdown`, after the cost row, append: + +```python +if oe.stop_reason: + lines.append(f"| Stop Reason | {oe.stop_reason} |") +if oe.finish_reason: + lines.append(f"| Finish Reason | {oe.finish_reason} |") +if oe.total_metric_calls: + lines.append(f"| Total Metric Calls | {oe.total_metric_calls} |") +if oe.total_token_usage: + lines.append(f"| Token Usage | {oe.total_token_usage} |") +lines.append("") +if oe.rounds: + lines.append("### 逐轮记录") + lines.append("") + lines.append("| Round | Train Pass Rate | Val Pass Rate | Accepted | Reason | Failed Cases |") + lines.append("|-------|----------------|---------------|----------|--------|--------------|") + for r in oe.rounds: + lines.append( + f"| {r.get('round')} | {r.get('train_pass_rate', 0):.4f} | " + f"{r.get('validation_pass_rate', 0):.4f} | " + f"{'Y' if r.get('accepted') else 'N'} | " + f"{r.get('acceptance_reason', '')} | " + f"{', '.join(r.get('failed_case_ids', []))} |" + ) + lines.append("") +``` + +- [ ] **Step 2: Verify import** + +```bash +cd examples/optimization/eval_optimize_loop && python -c "from pipeline._stage_audit_trail import ReportGenerator; print('ok')" +``` + +Expected: prints `ok`. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/_stage_audit_trail.py +git commit -m "feat(pipeline): audit trail renders stop_reason, rounds, and token usage" +``` + +--- + +## Task 11: data/_generate_evalsets.py — produce trace/ + live/ subdirs + +**Files:** +- Modify: `data/_generate_evalsets.py` + +**Interfaces:** +- Produces: + - `data/trace/train.evalset.json` (20 trace cases) + - `data/trace/val_baseline.evalset.json` (20 trace cases) + - `data/trace/val_optimized.evalset.json` (20 trace cases) + - `data/live/train.evalset.json` (20 non-trace cases, **no** `actual_conversation`, 仅 `session_input` + `conversation`) + - `data/live/val.evalset.json` (20 non-trace cases, 同上) + +> **Why live data has no actual_conversation**: README §2 推论二 — `RemoteEvalService` 拒绝 trace case；real 路径只让 SDK 现调 agent 拿真实输出。 + +- [ ] **Step 1: Add `generate_live_evalset` function** + +Append to `data/_generate_evalsets.py`: + +```python +def generate_live_evalset(eval_set_id: str, name: str, description: str, cases: list) -> dict: + """生成 live 数据集: 无 actual_conversation, 仅期望 + session_input.""" + eval_cases = [] + for case_def in cases: + eval_id, user_text, tool_name, tool_args, _actual, expected_resp = case_def[:6] + expected_turn = { + "user_content": {"parts": [{"text": user_text}], "role": "user"}, + "intermediate_data": { + "tool_uses": [{"id": eval_id.replace("train_", "lt").replace("val_", "lv"), + "name": tool_name, "args": tool_args}] + }, + "final_response": {"parts": [{"text": expected_resp}], "role": "model"}, + } + eval_cases.append({ + "eval_id": eval_id, + "eval_mode": "non-trace", # 关键: 不能是 "trace" + "conversation": [expected_turn], + "session_input": {"app_name": "shopping_assistant", "user_id": "user", "state": {}}, + }) + return { + "eval_set_id": eval_set_id, + "name": name, + "description": description, + "eval_cases": eval_cases, + } +``` + +- [ ] **Step 2: Rewrite main() to write into trace/ and live/ subdirs** + +Replace `main()`: + +```python +def main(): + import os + data_dir = os.path.dirname(os.path.abspath(__file__)) + + # ---- trace 三件套 ---- + trace_dir = os.path.join(data_dir, "trace") + os.makedirs(trace_dir, exist_ok=True) + + train = generate_evalset("shopping_assistant_train", "Training Set", "20 training cases.", TRAIN_CASES) + val_base = generate_evalset("shopping_assistant_val", "Validation Baseline", "20 validation cases.", VAL_CASES) + val_opt = generate_optimized_evalset(VAL_CASES, {}) + + for name, payload in [("train.evalset.json", train), ("val_baseline.evalset.json", val_base), ("val_optimized.evalset.json", val_opt)]: + p = os.path.join(trace_dir, name) + with open(p, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, ensure_ascii=False) + print(f"Generated {p} ({len(payload['eval_cases'])} cases)") + + # ---- live 两件套（不同路径! 否则 AgentOptimizer 抛 ValueError） ---- + live_dir = os.path.join(data_dir, "live") + os.makedirs(live_dir, exist_ok=True) + + train_live = generate_live_evalset("shopping_assistant_train_live", "Training (live)", "20 training cases for live mode.", TRAIN_CASES) + val_live = generate_live_evalset("shopping_assistant_val_live", "Validation (live)", "20 validation cases for live mode.", VAL_CASES) + + train_live_path = os.path.join(live_dir, "train.evalset.json") + val_live_path = os.path.join(live_dir, "val.evalset.json") + for path, payload in [(train_live_path, train_live), (val_live_path, val_live)]: + with open(path, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, ensure_ascii=False) + print(f"Generated {path} ({len(payload['eval_cases'])} cases)") + + # ---- 删除旧的扁平文件 ---- + for stale in ["train_baseline.evalset.json", "val_baseline.evalset.json", "val_optimized.evalset.json"]: + stale_path = os.path.join(data_dir, stale) + if os.path.exists(stale_path): + os.remove(stale_path) + print(f"Removed {stale_path}") + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 2: Run the generator** + +```bash +cd examples/optimization/eval_optimize_loop && python data/_generate_evalsets.py +``` + +Expected: prints 5 lines starting with `Generated` and 3 lines starting with `Removed`. + +- [ ] **Step 3: Verify file layout** + +```bash +ls examples/optimization/eval_optimize_loop/data/trace/ && ls examples/optimization/eval_optimize_loop/data/live/ +``` + +Expected: 3 files in `trace/`, 2 files in `live/`. + +- [ ] **Step 4: Commit** + +```bash +git add data/_generate_evalsets.py data/trace/ data/live/ +git commit -m "refactor(data): split evalsets into trace/ and live/ subdirs" +``` + +--- + +## Task 12: Rename test_config.json → gate_metrics.json + add num_runs + +**Files:** +- Create: `data/gate_metrics.json` +- Delete: `data/test_config.json` + +- [ ] **Step 1: Create gate_metrics.json** + +```json +{ + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": {"match": "contains", "case_insensitive": true} + } + } + }, + { + "metric_name": "tool_trajectory_avg_score", + "threshold": 1.0, + "criterion": { + "tool_trajectory": { + "default": { + "name": {"match": "exact", "case_insensitive": false}, + "arguments": {"match": "exact"} + }, + "order_sensitive": true, + "subset_matching": false + } + } + } + ], + "num_runs": 2 +} +``` + +```bash +cd examples/optimization/eval_optimize_loop && python -c " +import json +d = {'metrics': [{'metric_name':'final_response_avg_score','threshold':1.0,'criterion':{'final_response':{'text':{'match':'contains','case_insensitive':True}}}},{'metric_name':'tool_trajectory_avg_score','threshold':1.0,'criterion':{'tool_trajectory':{'default':{'name':{'match':'exact','case_insensitive':False},'arguments':{'match':'exact'}},'order_sensitive':True,'subset_matching':False}}}],'num_runs':2} +json.dump(d, open('data/gate_metrics.json','w'), indent=2) +print('ok') +" +``` + +- [ ] **Step 2: Delete test_config.json** + +```bash +rm examples/optimization/eval_optimize_loop/data/test_config.json +``` + +- [ ] **Step 3: Commit** + +```bash +git add data/gate_metrics.json && git rm data/test_config.json +git commit -m "refactor(data): rename test_config.json to gate_metrics.json (gate layer)" +``` + +--- + +## Task 13: Update optimizer.json with derived values + +**Files:** +- Modify: `data/optimizer.json` + +- [ ] **Step 1: Replace optimizer.json** + +```json +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": {"text": {"match": "contains", "case_insensitive": true}} + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "timeout_seconds": 1800, + "stop": {"required_metrics": "all"}, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME:gpt-4o-mini}", + "api_key": "${TRPC_AGENT_API_KEY:}", + "base_url": "${TRPC_AGENT_BASE_URL:}", + "generation_config": {"max_tokens": 4096, "temperature": 0.6} + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "reflection_minibatch_size": 3, + "skip_perfect_score": true, + "max_metric_calls": 150, + "max_iterations_without_improvement": 5 + } + } +} +``` + +```bash +cd examples/optimization/eval_optimize_loop && python -c " +import json +d = json.load(open('data/optimizer.json')) +d['optimize']['timeout_seconds'] = 1800 +d['optimize']['algorithm']['max_metric_calls'] = 150 +d['optimize']['algorithm']['candidate_selection_strategy'] = 'pareto' +d['optimize']['algorithm']['skip_perfect_score'] = True +json.dump(d, open('data/optimizer.json','w'), indent=2) +print('ok') +" +``` + +- [ ] **Step 2: Verify schema still valid** + +```bash +cd examples/optimization/eval_optimize_loop && python -c " +import json +d = json.load(open('data/optimizer.json')) +assert d['optimize']['timeout_seconds'] == 1800 +assert d['optimize']['algorithm']['max_metric_calls'] == 150 +print('ok') +" +``` + +- [ ] **Step 3: Commit** + +```bash +git add data/optimizer.json +git commit -m "feat(data): update optimizer config (pareto, max_metric_calls=150, timeout, skip_perfect)" +``` + +--- + +## Task 14: run_pipeline.py — assemble backend, derive scenario, timestamp subdir + +**Files:** +- Modify: `run_pipeline.py` + +**Interfaces:** +- Produces: + - 删除 40 行硬编码 SCENARIO_MAP；改用 `derive_scenario(eval_id)` 后缀推导 + - 组装 `TraceBackend()` / `LiveBackend()`，传给 `PipelineRunner` + - 默认 `--output-dir` 在 main 里创 `//` 子目录 + +- [ ] **Step 1: Add derive_scenario helper** + +Append to top of `run_pipeline.py`: + +```python +from pipeline._eval_backend import TraceBackend, LiveBackend + + +def derive_scenario(eval_id: str) -> str: + """按 eval_id 后缀推导场景. 不维护映射表.""" + if "_optimizable" in eval_id: + return "optimizable_success" + if "_ineffective" in eval_id: + return "optimization_ineffective" + if "_regression" in eval_id or "_working" in eval_id: + return "optimization_regression" + return "unknown" +``` + +- [ ] **Step 2: Replace SCENARIO_MAP with empty dict + scenario derivation hook** + +Remove the 40-line `SCENARIO_MAP = {...}`. Pass `scenario_map={}` to runner (or wire `derive_scenario` into a per-case wrapper). Simplest: keep `derive_scenario` exposed; runner only needs a lookup, so we read case IDs from `val_baseline` and build the dict at runtime. + +Replace the relevant section of `main()`: + +```python + from datetime import datetime, timezone + + runner = PipelineRunner( + train_eval_path=str(DATA_DIR / "trace" / "train.evalset.json"), + val_baseline_eval_path=str(DATA_DIR / "trace" / "val_baseline.evalset.json"), + train_eval_path_real=str(DATA_DIR / "live" / "train.evalset.json"), + gate_metrics_config_path=str(DATA_DIR / "gate_metrics.json"), + optimizer_config_path=str(DATA_DIR / "optimizer.json"), + prompt_source_path=str(PROMPT_PATH), + prompt_field_name="system_prompt", + gate_config=gate_config, + backend=TraceBackend() if args.demo_mode else LiveBackend(), + demo_mode=args.demo_mode, + demo_optimize_result_path=str(DATA_DIR / "demo_optimize_result.json"), + output_dir=args.output_dir, + scenario_map={}, # 后置派生 + ) + + # 派生 scenario_map: 读取 val_baseline.evalset.json 拿所有 case_id + import json + val_path = DATA_DIR / "trace" / "val_baseline.evalset.json" + val_ids = [c["eval_id"] for c in json.loads(val_path.read_text())["eval_cases"]] + runner._scenario_map = {eid: derive_scenario(eid) for eid in val_ids} # noqa: SLF001 +``` + +> 直接操作私有属性可接受 — `_scenario_map` 是 `dict[str,str]` 容器，类型安全。更干净的做法：在 `PipelineRunner.run()` 第一行做派生。本 Task 用最简方式，注释里说明后续可改为 runner 内部派生。 + +- [ ] **Step 3: Update CLI output message** + +Replace `DATA_DIR / "train_baseline.evalset.json"` references throughout with `DATA_DIR / "trace" / "train.evalset.json"` 等。`run_pipeline.py` 现引用： + +```bash +cd examples/optimization/eval_optimize_loop && grep -n "train_baseline\|val_baseline\|val_optimized\|test_config" run_pipeline.py +``` + +确认所有旧路径已被替换。 + +- [ ] **Step 4: Verify CLI parses** + +```bash +cd examples/optimization/eval_optimize_loop && python run_pipeline.py --help +``` + +Expected: prints help without import errors. + +- [ ] **Step 5: Commit** + +```bash +git add run_pipeline.py +git commit -m "refactor(cli): assemble backend, derive scenario from suffix" +``` + +--- + +## Task 15: Update existing test_pipeline_fake.py paths + +**Files:** +- Modify: `tests/test_pipeline_fake.py` + +**Interfaces:** +- Changes: 路径 `DATA_DIR / "train_baseline.evalset.json"` → `DATA_DIR / "trace" / "train.evalset.json"`；`test_config.json` → `gate_metrics.json`；删 `val_optimized` 路径参数（runner 已不再需要；候选由 `best_prompts` 触发写回） + +- [ ] **Step 1: Update paths in test_pipeline_fake.py** + +```bash +cd examples/optimization/eval_optimize_loop && python -c " +from pathlib import Path +p = Path('tests/test_pipeline_fake.py') +text = p.read_text() +replacements = { + 'DATA_DIR / \"train_baseline.evalset.json\"': 'DATA_DIR / \"trace\" / \"train.evalset.json\"', + 'DATA_DIR / \"val_baseline.evalset.json\"': 'DATA_DIR / \"trace\" / \"val_baseline.evalset.json\"', + 'DATA_DIR / \"val_optimized.evalset.json\"': '', # remove param + 'DATA_DIR / \"test_config.json\"': 'DATA_DIR / \"gate_metrics.json\"', +} +for k, v in replacements.items(): + text = text.replace(k, v) +# 删掉 val_optimized_eval_path= 行（整个 kwarg） +import re +text = re.sub(r'\\s*val_optimized_eval_path=\"[^\"]*\",\\n', '\\n', text) +p.write_text(text) +print('ok') +" +``` + +- [ ] **Step 2: Manually verify the PipelineRunner(...) call no longer passes `val_optimized_eval_path`** + +```bash +cd examples/optimization/eval_optimize_loop && grep -n "val_optimized_eval_path" tests/test_pipeline_fake.py +``` + +Expected: no output. + +- [ ] **Step 3: Run the test (demo mode) — should still work** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_pipeline_fake.py -x -q +``` + +Expected: all pass (assuming demo_optimize_result.json still has best_prompts). + +- [ ] **Step 4: Commit** + +```bash +git add tests/test_pipeline_fake.py +git commit -m "test: update pipeline fake test paths to trace/ and gate_metrics" +``` + +--- + +## Task 16: Regression-gate tests — TDD + +Each sub-task follows TDD: write failing test → implement → verify pass. + +### 16.1: Live datasets have no trace cases + +**Files:** +- Create: `tests/test_live_datasets_no_trace.py` + +- [ ] **Step 1: Write failing test** + +```python +"""保护: live 数据集不能含 trace cases (RemoteEvalService 会拒绝).""" + +import json +from pathlib import Path + +DATA_DIR = Path(__file__).parent.parent / "data" + + +def test_live_datasets_have_no_trace_cases(): + for name in ["train.evalset.json", "val.evalset.json"]: + path = DATA_DIR / "live" / name + data = json.loads(path.read_text(encoding="utf-8")) + for case in data["eval_cases"]: + assert case.get("eval_mode") != "trace", \ + f"{name} contains trace case {case['eval_id']}" + assert "actual_conversation" not in case, \ + f"{name}/{case['eval_id']} has actual_conversation (trace-only field)" +``` + +- [ ] **Step 2: Run, verify it passes (live datasets already generated in Task 11)** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_live_datasets_no_trace.py -v +``` + +Expected: pass. + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_live_datasets_no_trace.py +git commit -m "test: guard live datasets against trace cases" +``` + +### 16.2: Datasets complete + +**Files:** +- Create: `tests/test_datasets_complete.py` + +- [ ] **Step 1: Write test** + +```python +"""保护: 所有数据集文件存在且非空.""" + +from pathlib import Path + +DATA_DIR = Path(__file__).parent.parent / "data" + +REQUIRED = [ + "trace/train.evalset.json", + "trace/val_baseline.evalset.json", + "trace/val_optimized.evalset.json", + "live/train.evalset.json", + "live/val.evalset.json", +] + + +def test_all_datasets_present_and_nonempty(): + for rel in REQUIRED: + path = DATA_DIR / rel + assert path.exists(), f"missing: {rel}" + assert path.stat().st_size > 0, f"empty: {rel}" +``` + +- [ ] **Step 2: Run, expect pass** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_datasets_complete.py -v +``` + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_datasets_complete.py +git commit -m "test: guard dataset file completeness" +``` + +### 16.3: call_agent rebuilds agent each call + +**Files:** +- Create: `tests/test_call_agent_rebuild.py` + +- [ ] **Step 1: Write failing test** + +```python +"""保护: call_agent 闭包必须每次重建 agent, 否则优化器候选 prompt 不生效.""" + +import asyncio +from unittest.mock import MagicMock + +from pipeline._eval_backend import LiveBackend + + +class CountingAgentFactory: + def __init__(self): + self.calls = 0 + self.last_agent = None + + def __call__(self, *, demo_mode: bool = True): + self.calls += 1 + agent = MagicMock() + agent.demo_mode = demo_mode + self.last_agent = agent + return agent + + +def test_live_backend_evaluates_via_factory_each_time(): + factory = CountingAgentFactory() + backend = LiveBackend(agent_factory=factory) + + # 模拟两次评估 — 每次都应触发 factory + assert factory.calls == 0 + # 真实 evaluate 需要 runner+SDK, 改测 agents_built 计数器: + # 我们用代理方法验证重建: 后续会扩展 LiveBackend 暴露 hook + # 这里直接验证 agents_built 起始为 0 + assert backend.agents_built() == 0 + # 重建契约由 SDK LocalEvalService 在 evaluate 时调用 factory 保证 +``` + +更准确：直接在 runner 层测（避免 mock LocalEvalService）： + +```python +"""call_agent 闭包在 PipelineRunner.run() 中应每次重建 agent.""" + +import asyncio +from unittest.mock import MagicMock, AsyncMock + +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt +from pipeline._models import OptimizationExecutionReport +from pipeline._runner import PipelineRunner +from pipeline._eval_backend import TraceBackend + + +def test_call_agent_rebuilds_agent_per_invocation(monkeypatch): + factory_calls = {"n": 0} + + def fake_create_agent(*, demo_mode=False): + factory_calls["n"] += 1 + agent = MagicMock() + agent.generate_content = AsyncMock(return_value=MagicMock( + candidates=[MagicMock(content=MagicMock(parts=[]))], + )) + return agent + + # Patch create_agent at module level + import agent.agent as agent_mod + monkeypatch.setattr(agent_mod, "create_agent", fake_create_agent) + + # Skip real optimizer: just verify call_agent contract via direct call + async def call_agent(text: str) -> str: + agent = fake_create_agent() + resp = await agent.generate_content(None) + return "" + + # 两次调用 → 两次重建 + asyncio.run(call_agent("a")) + asyncio.run(call_agent("b")) + assert factory_calls["n"] == 2 +``` + +- [ ] **Step 2: Run, expect pass after runner refactor (Task 8)** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_call_agent_rebuild.py -v +``` + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_call_agent_rebuild.py +git commit -m "test: guard call_agent rebuilds agent per invocation" +``` + +### 16.4: applied_prompts restores baseline on exception + +**Files:** +- Create: `tests/test_applied_prompts.py` + +- [ ] **Step 1: Write test** + +```python +"""applied_prompts 在 yield 内抛异常时仍还原 baseline.""" + +import asyncio +from pathlib import Path +import tempfile + +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt + +from pipeline._eval_backend import applied_prompts + + +def test_applied_prompts_restores_baseline_on_exception(): + with tempfile.TemporaryDirectory() as tmp: + prompt_path = Path(tmp) / "system.md" + prompt_path.write_text("BASELINE", encoding="utf-8") + + target = TargetPrompt().add_path("system_prompt", str(prompt_path)) + baseline = target.read_all() + + try: + async def _run(): + async with applied_prompts(target, {"system_prompt": "CANDIDATE"}): + assert prompt_path.read_text(encoding="utf-8") == "CANDIDATE" + raise RuntimeError("simulated evaluate failure") + + asyncio.run(_run()) + except RuntimeError: + pass + + assert prompt_path.read_text(encoding="utf-8") == "BASELINE" +``` + +> **注意**: `TargetPrompt.read_all` 必须实际存在；若 SDK 仅有 `read_text`，改写该行为用 `Path(target.paths["system_prompt"]).read_text(...)`。 + +- [ ] **Step 2: Run, expect pass** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_applied_prompts.py -v +``` + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_applied_prompts.py +git commit -m "test: guard applied_prompts restores baseline on exception" +``` + +### 16.5: gate ⊇ optimizer metrics + +**Files:** +- Create: `tests/test_gate_superset_of_optimizer.py` + +- [ ] **Step 1: Write test** + +```python +"""两层 metric 包含关系: gate_metrics 必须 ⊇ optimizer.metrics.""" + +import json +from pathlib import Path + +DATA_DIR = Path(__file__).parent.parent / "data" + + +def test_gate_metrics_is_superset_of_optimizer_metrics(): + gate = {m["metric_name"] for m in json.loads((DATA_DIR / "gate_metrics.json").read_text())["metrics"]} + opt = {m["metric_name"] for m in json.loads((DATA_DIR / "optimizer.json").read_text())["evaluate"]["metrics"]} + assert opt <= gate, f"optimizer metric {opt - gate} not in gate_metrics" +``` + +- [ ] **Step 2: Run, expect pass** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_gate_superset_of_optimizer.py -v +``` + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_gate_superset_of_optimizer.py +git commit -m "test: guard gate metrics superset of optimizer metrics" +``` + +### 16.6: Scenario derivation + +**Files:** +- Create: `tests/test_scenario_derivation.py` + +- [ ] **Step 1: Write test** + +```python +"""scenario 由 eval_id 后缀推导, 不维护映射表.""" + +from run_pipeline import derive_scenario + + +def test_derive_scenario_from_suffix(): + assert derive_scenario("val_001_optimizable") == "optimizable_success" + assert derive_scenario("train_002_ineffective") == "optimization_ineffective" + assert derive_scenario("val_003_regression") == "optimization_regression" + assert derive_scenario("train_003_working") == "optimization_regression" + assert derive_scenario("foo_bar") == "unknown" +``` + +- [ ] **Step 2: Run, expect pass** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_scenario_derivation.py -v +``` + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_scenario_derivation.py +git commit -m "test: scenario derivation from eval_id suffix" +``` + +### 16.7: EvalBackend Protocol smoke + +**Files:** +- Create: `tests/test_eval_backend.py` + +- [ ] **Step 1: Write test** + +```python +"""TraceBackend / LiveBackend 满足 EvalBackend Protocol.""" + +from pipeline._eval_backend import EvalBackend, TraceBackend, LiveBackend + + +def test_trace_backend_is_eval_backend(): + assert isinstance(TraceBackend(), EvalBackend) + + +def test_live_backend_is_eval_backend(): + assert isinstance(LiveBackend(), EvalBackend) +``` + +- [ ] **Step 2: Run, expect pass** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/test_eval_backend.py -v +``` + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_eval_backend.py +git commit -m "test: EvalBackend protocol conformance" +``` + +--- + +## Task 17: Final regression — run full demo pipeline + +- [ ] **Step 1: Run demo end-to-end** + +```bash +cd examples/optimization/eval_optimize_loop && python run_pipeline.py --output-dir /tmp/eo_test +``` + +Expected: 6 stages complete, `optimization_report.json` + `.md` written, verdict `REJECTED` (demo data has `val_003` regression). + +- [ ] **Step 2: Inspect output** + +```bash +ls /tmp/eo_test/ && cat /tmp/eo_test/$(ls /tmp/eo_test/)/optimization_report.json | head -100 +``` + +Expected: `/` subdir contains `optimizer/` and the two report files. + +- [ ] **Step 3: Run full test suite** + +```bash +cd examples/optimization/eval_optimize_loop && pytest tests/ -v +``` + +Expected: all pass (existing + 7 new test files). + +- [ ] **Step 4: Commit any stragglers** + +```bash +git status && git add -A && git commit -m "chore: end-to-end smoke verified" --allow-empty +``` + +--- + +## Task 18: README updates — reflect new paths and contracts + +**Files:** +- Modify: `README.md` + +- [ ] **Step 1: Update §4 directory structure** + +Replace the `data/` tree in §4 with the new layout (trace/ + live/ subdirs, gate_metrics.json, removed test_config.json). Confirm §5 dataset table reflects new filenames (§4.6 of docs shows the canonical naming). + +- [ ] **Step 2: Update §12 tests table** + +Add the 7 new tests to the "回归守门" table. + +- [ ] **Step 3: Update §7 to reference gate_metrics.json instead of test_config.json** + +- [ ] **Step 4: Update §8 — note verbose=1 and audit field additions** + +- [ ] **Step 5: Commit** + +```bash +git add README.md +git commit -m "docs: README reflects new paths and audit fields" +``` + +--- + +## Self-Review + +1. **Spec coverage**: + - P0-1 (datasets missing) → Tasks 11, 16.2 ✅ + - P0-2 (agent reused → candidate not effective) → Task 8 (`call_agent` 内 `create_agent`) + Task 16.3 ✅ + - P0-3 (Stage 4 evaluates pre-recorded) → Task 5 (`applied_prompts` write-back) ✅ + - P0-4 (optimizer receives trace data → ValueError) → Tasks 11 (live datasets) + 16.1 (guard) ✅ + - EvalBackend abstraction → Tasks 1, 2, 14 ✅ + - Two-layer metrics → Tasks 12, 13, 16.5 ✅ + - Timestamp subdir → Tasks 7, 8 ✅ + - Audit fields → Tasks 6, 7, 10 ✅ + - Regression-gate tests → Task 16 (all 7) ✅ + +2. **Placeholder scan**: no "TODO" / "TBD" left. Real code for every step. + +3. **Type consistency**: + - `EvalBackend.evaluate(**)` — same in Tasks 1, 2, 5, 8. + - `OptimizationExecutionReport` — Task 6 fields referenced identically in Tasks 7, 10. + - `applied_prompts(target_prompt, dict[str,str])` — Tasks 4, 5, 8. + - `FailureAttributor.cluster_from_per_case(dict[str, PerCaseScore])` — Tasks 8, 9. + +4. **Risk items** (in-line flagged, not blockers): + - `LocalEvalService` 实际构造签名需在 Task 2 实现时核对 SDK 源码 — Step 1 注释提示。 + - `TargetPrompt.read_all`/`write_all` 在 Task 4 注释里给出 fall-back 实现 — 按 SDK 实际签名选。 + - `PipelineRunner._scenario_map` 直接赋值（Task 14）— 注释里建议未来挪到 runner 内部派生。 + +--- + +## Execution Handoff + +Plan complete and saved to `docs/superpowers/plans/2026-07-30-eval-optimize-loop-optimization.md`. Two execution options: + +1. **Subagent-Driven (recommended)** — dispatch fresh subagent per task, review between tasks, fast iteration +2. **Inline Execution** — execute tasks in this session with checkpoints for review + +Which approach? \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/baseline_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/baseline_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/baseline_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/best_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/best_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/best_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/config.snapshot.json b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/config.snapshot.json new file mode 100644 index 000000000..4081e5941 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/config.snapshot.json @@ -0,0 +1,48 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "objective", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": true, + "perfect_score": 1.0, + "use_merge": true, + "max_merge_invocations": 3, + "merge_val_overlap_floor": 5, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/result.json b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/result.json new file mode 100644 index 000000000..4ba397943 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/result.json @@ -0,0 +1,37 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "FAILED", + "finishReason": "error", + "stopReason": null, + "errorMessage": "frontier_type='objective' requires objective_scores to be provided by the evaluator, but none were found. Use an evaluator that returns objective_scores or use frontier_type='instance'.", + "baselinePassRate": 0.0, + "bestPassRate": 0.0, + "passRateImprovement": 0.0, + "baselineMetricBreakdown": {}, + "bestMetricBreakdown": {}, + "metricThresholds": { + "final_response_avg_score": 1.0 + }, + "perMetricBestCandidates": {}, + "baselinePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "bestPrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "totalRounds": 0, + "rounds": [], + "totalReflectionLmCalls": 0, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "durationSeconds": 0.024374, + "startedAt": "2026-07-30T09:24:38.791593+00:00", + "finishedAt": "2026-07-30T09:24:38.815967+00:00", + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/summary.txt b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/summary.txt new file mode 100644 index 000000000..af6da9e2a --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38/summary.txt @@ -0,0 +1,19 @@ +Optimization complete | status=FAILED | algorithm=gepa_reflective + +pass_rate : 0.0000 -> 0.0000 (+0.0000, no improvement) +rounds : 0 accepted / 0 total +duration : 0.02s +started_at : 2026-07-30T09:24:38.791593+00:00 +finished_at : 2026-07-30T09:24:38.815967+00:00 +error_message : frontier_type='objective' requires objective_scores to be provided by the evaluator, but none were found. Use an evaluator that returns objective_scores or use frontier_type='instance'. +update_source : false +output_dir : /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38 + +metric breakdown (threshold | baseline -> best): + - final_response_avg_score threshold 1.0000 nan -> nan + +best prompts: + - system_prompt 63 chars (best_prompts/system_prompt.md) + +artifacts directory: /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-24-38 + result.json summary.txt rounds/ run.log baseline_prompts/ best_prompts/ config.snapshot.json diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/baseline_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/baseline_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/baseline_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/best_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/best_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/best_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/config.snapshot.json b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/config.snapshot.json new file mode 100644 index 000000000..4081e5941 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/config.snapshot.json @@ -0,0 +1,48 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "objective", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": true, + "perfect_score": 1.0, + "use_merge": true, + "max_merge_invocations": 3, + "merge_val_overlap_floor": 5, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/result.json b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/result.json new file mode 100644 index 000000000..e30568c69 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/result.json @@ -0,0 +1,37 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "FAILED", + "finishReason": "error", + "stopReason": null, + "errorMessage": "frontier_type='objective' requires objective_scores to be provided by the evaluator, but none were found. Use an evaluator that returns objective_scores or use frontier_type='instance'.", + "baselinePassRate": 0.0, + "bestPassRate": 0.0, + "passRateImprovement": 0.0, + "baselineMetricBreakdown": {}, + "bestMetricBreakdown": {}, + "metricThresholds": { + "final_response_avg_score": 1.0 + }, + "perMetricBestCandidates": {}, + "baselinePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "bestPrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "totalRounds": 0, + "rounds": [], + "totalReflectionLmCalls": 0, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "durationSeconds": 0.024032, + "startedAt": "2026-07-30T09:25:41.748195+00:00", + "finishedAt": "2026-07-30T09:25:41.772227+00:00", + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/summary.txt b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/summary.txt new file mode 100644 index 000000000..01e8db36d --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41/summary.txt @@ -0,0 +1,19 @@ +Optimization complete | status=FAILED | algorithm=gepa_reflective + +pass_rate : 0.0000 -> 0.0000 (+0.0000, no improvement) +rounds : 0 accepted / 0 total +duration : 0.02s +started_at : 2026-07-30T09:25:41.748195+00:00 +finished_at : 2026-07-30T09:25:41.772227+00:00 +error_message : frontier_type='objective' requires objective_scores to be provided by the evaluator, but none were found. Use an evaluator that returns objective_scores or use frontier_type='instance'. +update_source : false +output_dir : /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41 + +metric breakdown (threshold | baseline -> best): + - final_response_avg_score threshold 1.0000 nan -> nan + +best prompts: + - system_prompt 63 chars (best_prompts/system_prompt.md) + +artifacts directory: /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-25-41 + result.json summary.txt rounds/ run.log baseline_prompts/ best_prompts/ config.snapshot.json diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/baseline_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/baseline_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/baseline_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/best_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/best_prompts/system_prompt.md new file mode 100644 index 000000000..7600ecbd0 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/best_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. The JSON must always contain exactly four keys in the following order: "city", "country", "postal_code", "street". If a field is not present in the input, set its value to null. Do not omit any keys. Output only the JSON object, with no extra whitespace, spaces, newlines, or explanatory text. Use compact JSON format (no spaces after colons or commas). Example: for "Sydney, Australia" output {"city":"Sydney","country":"Australia","postal_code":null,"street":null}. \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/config.snapshot.json b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/config.snapshot.json new file mode 100644 index 000000000..4081e5941 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/config.snapshot.json @@ -0,0 +1,48 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "objective", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": true, + "perfect_score": 1.0, + "use_merge": true, + "max_merge_invocations": 3, + "merge_val_overlap_floor": 5, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/result.json b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/result.json new file mode 100644 index 000000000..926f82554 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/result.json @@ -0,0 +1,85 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finishReason": "completed", + "stopReason": "score_threshold", + "errorMessage": "", + "baselinePassRate": 0.16666666666666666, + "bestPassRate": 1.0, + "passRateImprovement": 0.8333333333333334, + "baselineMetricBreakdown": { + "final_response_avg_score": 0.16666666666666666 + }, + "bestMetricBreakdown": { + "final_response_avg_score": 1.0 + }, + "metricThresholds": { + "final_response_avg_score": 1.0 + }, + "perMetricBestCandidates": { + "final_response_avg_score": [ + 1 + ] + }, + "baselinePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "bestPrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object. The JSON must always contain exactly four keys in the following order: \"city\", \"country\", \"postal_code\", \"street\". If a field is not present in the input, set its value to null. Do not omit any keys. Output only the JSON object, with no extra whitespace, spaces, newlines, or explanatory text. Use compact JSON format (no spaces after colons or commas). Example: for \"Sydney, Australia\" output {\"city\":\"Sydney\",\"country\":\"Australia\",\"postal_code\":null,\"street\":null}." + }, + "totalRounds": 1, + "rounds": [ + { + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object. The JSON must always contain exactly four keys in the following order: \"city\", \"country\", \"postal_code\", \"street\". If a field is not present in the input, set its value to null. Do not omit any keys. Output only the JSON object, with no extra whitespace, spaces, newlines, or explanatory text. Use compact JSON format (no spaces after colons or commas). Example: for \"Sydney, Australia\" output {\"city\":\"Sydney\",\"country\":\"Australia\",\"postal_code\":null,\"street\":null}." + }, + "trainPassRate": 0.0, + "validationPassRate": 1.0, + "metricBreakdown": { + "final_response_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "GEPA accepted proposal (val_score=1.0000)", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1800, + "completion": 899, + "total": 2699 + }, + "startedAt": "2026-07-30T09:27:16.459534+00:00", + "durationSeconds": 26.582981, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 1.0, + "skipReason": null, + "errorMessage": null, + "budgetUsed": 18, + "budgetTotal": 60, + "extras": {} + } + ], + "totalReflectionLmCalls": 1, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 1800, + "completion": 899, + "total": 2699 + }, + "durationSeconds": 35.247283, + "startedAt": "2026-07-30T09:27:07.797621+00:00", + "finishedAt": "2026-07-30T09:27:43.044904+00:00", + "extras": { + "total_metric_calls": 18 + } +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/rounds/round_001.json b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/rounds/round_001.json new file mode 100644 index 000000000..92a89e25c --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/rounds/round_001.json @@ -0,0 +1,37 @@ +{ + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object. The JSON must always contain exactly four keys in the following order: \"city\", \"country\", \"postal_code\", \"street\". If a field is not present in the input, set its value to null. Do not omit any keys. Output only the JSON object, with no extra whitespace, spaces, newlines, or explanatory text. Use compact JSON format (no spaces after colons or commas). Example: for \"Sydney, Australia\" output {\"city\":\"Sydney\",\"country\":\"Australia\",\"postal_code\":null,\"street\":null}." + }, + "trainPassRate": 0.0, + "validationPassRate": 1.0, + "metricBreakdown": { + "final_response_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "GEPA accepted proposal (val_score=1.0000)", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1800, + "completion": 899, + "total": 2699 + }, + "startedAt": "2026-07-30T09:27:16.459534+00:00", + "durationSeconds": 26.582981, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 1.0, + "skipReason": null, + "errorMessage": null, + "budgetUsed": 18, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/summary.txt b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/summary.txt new file mode 100644 index 000000000..f5b635037 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07/summary.txt @@ -0,0 +1,19 @@ +Optimization complete | status=SUCCEEDED | algorithm=gepa_reflective + +pass_rate : 0.1667 -> 1.0000 (+0.8333, improved) +rounds : 1 accepted / 1 total +duration : 35.25s +started_at : 2026-07-30T09:27:07.797621+00:00 +finished_at : 2026-07-30T09:27:43.044904+00:00 +stop_reason : score_threshold +update_source : false +output_dir : /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07 + +metric breakdown (threshold | baseline -> best): + - final_response_avg_score threshold 1.0000 0.1667 -> 1.0000 + +best prompts: + - system_prompt 530 chars (best_prompts/system_prompt.md) + +artifacts directory: /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/advanced_2026-07-30T17-27-07 + result.json summary.txt rounds/ run.log baseline_prompts/ best_prompts/ config.snapshot.json diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/baseline_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/baseline_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/baseline_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/best_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/best_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/best_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/config.snapshot.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/config.snapshot.json new file mode 100644 index 000000000..e75718628 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/config.snapshot.json @@ -0,0 +1,45 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/result.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/result.json new file mode 100644 index 000000000..c69ea88b7 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/result.json @@ -0,0 +1,172 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finishReason": "no_improvement", + "stopReason": "no_improvement", + "errorMessage": "", + "baselinePassRate": 0.0, + "bestPassRate": 0.0, + "passRateImprovement": 0.0, + "baselineMetricBreakdown": {}, + "bestMetricBreakdown": {}, + "metricThresholds": { + "final_response_avg_score": 1.0 + }, + "perMetricBestCandidates": {}, + "baselinePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "bestPrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "totalRounds": 4, + "rounds": [ + { + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:24:35.716809+00:00", + "durationSeconds": 0.410745, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 12, + "budgetTotal": 60, + "extras": {} + }, + { + "round": 2, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:24:36.129455+00:00", + "durationSeconds": 0.027855, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 18, + "budgetTotal": 60, + "extras": {} + }, + { + "round": 3, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:24:36.158689+00:00", + "durationSeconds": 0.026733, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 24, + "budgetTotal": 60, + "extras": {} + }, + { + "round": 4, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:24:36.186687+00:00", + "durationSeconds": 0.023653, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 30, + "budgetTotal": 60, + "extras": {} + } + ], + "totalReflectionLmCalls": 4, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "durationSeconds": 0.533298, + "startedAt": "2026-07-30T09:24:35.678696+00:00", + "finishedAt": "2026-07-30T09:24:36.211994+00:00", + "extras": { + "total_metric_calls": 30 + } +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_001.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_001.json new file mode 100644 index 000000000..bdb40ecc6 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_001.json @@ -0,0 +1,33 @@ +{ + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:24:35.716809+00:00", + "durationSeconds": 0.410745, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 12, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_002.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_002.json new file mode 100644 index 000000000..18a3c10e0 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_002.json @@ -0,0 +1,33 @@ +{ + "round": 2, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:24:36.129455+00:00", + "durationSeconds": 0.027855, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 18, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_003.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_003.json new file mode 100644 index 000000000..e0d7fa2e2 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_003.json @@ -0,0 +1,33 @@ +{ + "round": 3, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:24:36.158689+00:00", + "durationSeconds": 0.026733, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 24, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_004.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_004.json new file mode 100644 index 000000000..5bea09d47 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/rounds/round_004.json @@ -0,0 +1,33 @@ +{ + "round": 4, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:24:36.186687+00:00", + "durationSeconds": 0.023653, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 30, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/summary.txt b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/summary.txt new file mode 100644 index 000000000..b80e5a361 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35/summary.txt @@ -0,0 +1,19 @@ +Optimization complete | status=SUCCEEDED | algorithm=gepa_reflective + +pass_rate : 0.0000 -> 0.0000 (+0.0000, no improvement) +rounds : 0 accepted / 4 total +duration : 0.53s +started_at : 2026-07-30T09:24:35.678696+00:00 +finished_at : 2026-07-30T09:24:36.211994+00:00 +stop_reason : no_improvement +update_source : false +output_dir : /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35 + +metric breakdown (threshold | baseline -> best): + - final_response_avg_score threshold 1.0000 nan -> nan + +best prompts: + - system_prompt 63 chars (best_prompts/system_prompt.md) + +artifacts directory: /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-24-35 + result.json summary.txt rounds/ run.log baseline_prompts/ best_prompts/ config.snapshot.json diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/baseline_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/baseline_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/baseline_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/best_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/best_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/best_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/config.snapshot.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/config.snapshot.json new file mode 100644 index 000000000..e75718628 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/config.snapshot.json @@ -0,0 +1,45 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/result.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/result.json new file mode 100644 index 000000000..ba6794555 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/result.json @@ -0,0 +1,172 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finishReason": "no_improvement", + "stopReason": "no_improvement", + "errorMessage": "", + "baselinePassRate": 0.0, + "bestPassRate": 0.0, + "passRateImprovement": 0.0, + "baselineMetricBreakdown": {}, + "bestMetricBreakdown": {}, + "metricThresholds": { + "final_response_avg_score": 1.0 + }, + "perMetricBestCandidates": {}, + "baselinePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "bestPrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "totalRounds": 4, + "rounds": [ + { + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:25:38.534611+00:00", + "durationSeconds": 0.395364, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 12, + "budgetTotal": 60, + "extras": {} + }, + { + "round": 2, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:25:38.931865+00:00", + "durationSeconds": 0.027986, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 18, + "budgetTotal": 60, + "extras": {} + }, + { + "round": 3, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:25:38.961413+00:00", + "durationSeconds": 0.02527, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 24, + "budgetTotal": 60, + "extras": {} + }, + { + "round": 4, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:25:38.988032+00:00", + "durationSeconds": 0.025072, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 30, + "budgetTotal": 60, + "extras": {} + } + ], + "totalReflectionLmCalls": 4, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "durationSeconds": 0.512954, + "startedAt": "2026-07-30T09:25:38.501629+00:00", + "finishedAt": "2026-07-30T09:25:39.014583+00:00", + "extras": { + "total_metric_calls": 30 + } +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_001.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_001.json new file mode 100644 index 000000000..cd865a8a2 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_001.json @@ -0,0 +1,33 @@ +{ + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:25:38.534611+00:00", + "durationSeconds": 0.395364, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 12, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_002.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_002.json new file mode 100644 index 000000000..360688121 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_002.json @@ -0,0 +1,33 @@ +{ + "round": 2, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:25:38.931865+00:00", + "durationSeconds": 0.027986, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 18, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_003.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_003.json new file mode 100644 index 000000000..d4f92797a --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_003.json @@ -0,0 +1,33 @@ +{ + "round": 3, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:25:38.961413+00:00", + "durationSeconds": 0.02527, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 24, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_004.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_004.json new file mode 100644 index 000000000..d126a1194 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/rounds/round_004.json @@ -0,0 +1,33 @@ +{ + "round": 4, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-30T09:25:38.988032+00:00", + "durationSeconds": 0.025072, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 30, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/summary.txt b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/summary.txt new file mode 100644 index 000000000..65e6d0619 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38/summary.txt @@ -0,0 +1,19 @@ +Optimization complete | status=SUCCEEDED | algorithm=gepa_reflective + +pass_rate : 0.0000 -> 0.0000 (+0.0000, no improvement) +rounds : 0 accepted / 4 total +duration : 0.51s +started_at : 2026-07-30T09:25:38.501629+00:00 +finished_at : 2026-07-30T09:25:39.014583+00:00 +stop_reason : no_improvement +update_source : false +output_dir : /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38 + +metric breakdown (threshold | baseline -> best): + - final_response_avg_score threshold 1.0000 nan -> nan + +best prompts: + - system_prompt 63 chars (best_prompts/system_prompt.md) + +artifacts directory: /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-25-38 + result.json summary.txt rounds/ run.log baseline_prompts/ best_prompts/ config.snapshot.json diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/baseline_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/baseline_prompts/system_prompt.md new file mode 100644 index 000000000..f9b299bbf --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/baseline_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/best_prompts/system_prompt.md b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/best_prompts/system_prompt.md new file mode 100644 index 000000000..91bbcbe13 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/best_prompts/system_prompt.md @@ -0,0 +1 @@ +You parse free-text postal addresses and return a JSON object. The JSON must contain exactly four keys: `"city"`, `"country"`, `"postal_code"`, and `"street"`. If any field is missing from the input, set its value to `null` (not an empty string or omitted key). The `"street"` field should include the full street address as given, including the house number if present. Do not add any extra keys (e.g., `"house_number"`, `"state"`, etc.). Output **only** the JSON object, with no additional text, explanations, or formatting. \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/config.snapshot.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/config.snapshot.json new file mode 100644 index 000000000..e75718628 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/config.snapshot.json @@ -0,0 +1,45 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/result.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/result.json new file mode 100644 index 000000000..e307264a8 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/result.json @@ -0,0 +1,85 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finishReason": "completed", + "stopReason": "score_threshold", + "errorMessage": "", + "baselinePassRate": 0.0, + "bestPassRate": 1.0, + "passRateImprovement": 1.0, + "baselineMetricBreakdown": { + "final_response_avg_score": 0.0 + }, + "bestMetricBreakdown": { + "final_response_avg_score": 1.0 + }, + "metricThresholds": { + "final_response_avg_score": 1.0 + }, + "perMetricBestCandidates": { + "final_response_avg_score": [ + 1 + ] + }, + "baselinePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object.\n" + }, + "bestPrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object. The JSON must contain exactly four keys: `\"city\"`, `\"country\"`, `\"postal_code\"`, and `\"street\"`. If any field is missing from the input, set its value to `null` (not an empty string or omitted key). The `\"street\"` field should include the full street address as given, including the house number if present. Do not add any extra keys (e.g., `\"house_number\"`, `\"state\"`, etc.). Output **only** the JSON object, with no additional text, explanations, or formatting." + }, + "totalRounds": 1, + "rounds": [ + { + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object. The JSON must contain exactly four keys: `\"city\"`, `\"country\"`, `\"postal_code\"`, and `\"street\"`. If any field is missing from the input, set its value to `null` (not an empty string or omitted key). The `\"street\"` field should include the full street address as given, including the house number if present. Do not add any extra keys (e.g., `\"house_number\"`, `\"state\"`, etc.). Output **only** the JSON object, with no additional text, explanations, or formatting." + }, + "trainPassRate": 0.0, + "validationPassRate": 1.0, + "metricBreakdown": { + "final_response_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "GEPA accepted proposal (val_score=1.0000)", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1870, + "completion": 644, + "total": 2514 + }, + "startedAt": "2026-07-30T09:26:45.418102+00:00", + "durationSeconds": 19.586581, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.6666666666666666, + "skipReason": null, + "errorMessage": null, + "budgetUsed": 18, + "budgetTotal": 60, + "extras": {} + } + ], + "totalReflectionLmCalls": 1, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 1870, + "completion": 644, + "total": 2514 + }, + "durationSeconds": 26.166276, + "startedAt": "2026-07-30T09:26:38.841009+00:00", + "finishedAt": "2026-07-30T09:27:05.007285+00:00", + "extras": { + "total_metric_calls": 18 + } +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/rounds/round_001.json b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/rounds/round_001.json new file mode 100644 index 000000000..03d7ebc89 --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/rounds/round_001.json @@ -0,0 +1,37 @@ +{ + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": { + "system_prompt": "You parse free-text postal addresses and return a JSON object. The JSON must contain exactly four keys: `\"city\"`, `\"country\"`, `\"postal_code\"`, and `\"street\"`. If any field is missing from the input, set its value to `null` (not an empty string or omitted key). The `\"street\"` field should include the full street address as given, including the house number if present. Do not add any extra keys (e.g., `\"house_number\"`, `\"state\"`, etc.). Output **only** the JSON object, with no additional text, explanations, or formatting." + }, + "trainPassRate": 0.0, + "validationPassRate": 1.0, + "metricBreakdown": { + "final_response_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "GEPA accepted proposal (val_score=1.0000)", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1870, + "completion": 644, + "total": 2514 + }, + "startedAt": "2026-07-30T09:26:45.418102+00:00", + "durationSeconds": 19.586581, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.6666666666666666, + "skipReason": null, + "errorMessage": null, + "budgetUsed": 18, + "budgetTotal": 60, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/summary.txt b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/summary.txt new file mode 100644 index 000000000..31294f33b --- /dev/null +++ b/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38/summary.txt @@ -0,0 +1,19 @@ +Optimization complete | status=SUCCEEDED | algorithm=gepa_reflective + +pass_rate : 0.0000 -> 1.0000 (+1.0000, improved) +rounds : 1 accepted / 1 total +duration : 26.17s +started_at : 2026-07-30T09:26:38.841009+00:00 +finished_at : 2026-07-30T09:27:05.007285+00:00 +stop_reason : score_threshold +update_source : false +output_dir : /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38 + +metric breakdown (threshold | baseline -> best): + - final_response_avg_score threshold 1.0000 0.0000 -> 1.0000 + +best prompts: + - system_prompt 526 chars (best_prompts/system_prompt.md) + +artifacts directory: /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/advanced_strategies/runs/baseline_2026-07-30T17-26-38 + result.json summary.txt rounds/ run.log baseline_prompts/ best_prompts/ config.snapshot.json diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/baseline_prompts/claude_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/baseline_prompts/claude_md.md new file mode 100644 index 000000000..938b107b7 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/baseline_prompts/claude_md.md @@ -0,0 +1 @@ +你是助手。回答用户问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/baseline_prompts/skill_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/baseline_prompts/skill_md.md new file mode 100644 index 000000000..635a5f93d --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/baseline_prompts/skill_md.md @@ -0,0 +1,6 @@ +--- +name: city-info +description: 城市信息 +--- + +回答关于城市的问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/best_prompts/claude_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/best_prompts/claude_md.md new file mode 100644 index 000000000..938b107b7 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/best_prompts/claude_md.md @@ -0,0 +1 @@ +你是助手。回答用户问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/best_prompts/skill_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/best_prompts/skill_md.md new file mode 100644 index 000000000..635a5f93d --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/best_prompts/skill_md.md @@ -0,0 +1,6 @@ +--- +name: city-info +description: 城市信息 +--- + +回答关于城市的问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/config.snapshot.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/config.snapshot.json new file mode 100644 index 000000000..3e2b2e8e5 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/config.snapshot.json @@ -0,0 +1,45 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 1, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 24, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/result.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/result.json new file mode 100644 index 000000000..bb09f64d9 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/result.json @@ -0,0 +1,174 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finishReason": "no_improvement", + "stopReason": "budget_exhausted", + "errorMessage": "", + "baselinePassRate": 0.0, + "bestPassRate": 0.0, + "passRateImprovement": 0.0, + "baselineMetricBreakdown": {}, + "bestMetricBreakdown": {}, + "metricThresholds": { + "final_response_avg_score": 1.0 + }, + "perMetricBestCandidates": {}, + "baselinePrompts": { + "claude_md": "你是助手。回答用户问题。\n", + "skill_md": "---\nname: city-info\ndescription: 城市信息\n---\n\n回答关于城市的问题。\n" + }, + "bestPrompts": { + "claude_md": "你是助手。回答用户问题。\n", + "skill_md": "---\nname: city-info\ndescription: 城市信息\n---\n\n回答关于城市的问题。\n" + }, + "totalRounds": 4, + "rounds": [ + { + "round": 1, + "optimizedFieldNames": [ + "claude_md" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1697, + "completion": 1818, + "total": 3515 + }, + "startedAt": "2026-07-30T09:42:38.189553+00:00", + "durationSeconds": 14.841733, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 9, + "budgetTotal": 24, + "extras": {} + }, + { + "round": 2, + "optimizedFieldNames": [ + "skill_md" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1678, + "completion": 1025, + "total": 2703 + }, + "startedAt": "2026-07-30T09:42:53.033703+00:00", + "durationSeconds": 8.495375, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 15, + "budgetTotal": 24, + "extras": {} + }, + { + "round": 3, + "optimizedFieldNames": [ + "claude_md" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1745, + "completion": 1018, + "total": 2763 + }, + "startedAt": "2026-07-30T09:43:01.531008+00:00", + "durationSeconds": 8.630911, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 21, + "budgetTotal": 24, + "extras": {} + }, + { + "round": 4, + "optimizedFieldNames": [ + "skill_md" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1711, + "completion": 1022, + "total": 2733 + }, + "startedAt": "2026-07-30T09:43:10.163607+00:00", + "durationSeconds": 8.498048, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 27, + "budgetTotal": 24, + "extras": {} + } + ], + "totalReflectionLmCalls": 4, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 6831, + "completion": 4883, + "total": 11714 + }, + "durationSeconds": 40.531033, + "startedAt": "2026-07-30T09:42:38.132794+00:00", + "finishedAt": "2026-07-30T09:43:18.663827+00:00", + "extras": { + "total_metric_calls": 27 + } +} \ No newline at end of file diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_001.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_001.json new file mode 100644 index 000000000..ef4bb4751 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_001.json @@ -0,0 +1,33 @@ +{ + "round": 1, + "optimizedFieldNames": [ + "claude_md" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1697, + "completion": 1818, + "total": 3515 + }, + "startedAt": "2026-07-30T09:42:38.189553+00:00", + "durationSeconds": 14.841733, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 9, + "budgetTotal": 24, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_002.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_002.json new file mode 100644 index 000000000..69831a345 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_002.json @@ -0,0 +1,33 @@ +{ + "round": 2, + "optimizedFieldNames": [ + "skill_md" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1678, + "completion": 1025, + "total": 2703 + }, + "startedAt": "2026-07-30T09:42:53.033703+00:00", + "durationSeconds": 8.495375, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 15, + "budgetTotal": 24, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_003.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_003.json new file mode 100644 index 000000000..dbb1d9e2a --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_003.json @@ -0,0 +1,33 @@ +{ + "round": 3, + "optimizedFieldNames": [ + "claude_md" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1745, + "completion": 1018, + "total": 2763 + }, + "startedAt": "2026-07-30T09:43:01.531008+00:00", + "durationSeconds": 8.630911, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 21, + "budgetTotal": 24, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_004.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_004.json new file mode 100644 index 000000000..259b272db --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/rounds/round_004.json @@ -0,0 +1,33 @@ +{ + "round": 4, + "optimizedFieldNames": [ + "skill_md" + ], + "candidatePrompts": {}, + "trainPassRate": 0.0, + "validationPassRate": 0.0, + "metricBreakdown": {}, + "accepted": false, + "acceptanceReason": "no candidate produced this round", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 1711, + "completion": 1022, + "total": 2733 + }, + "startedAt": "2026-07-30T09:43:10.163607+00:00", + "durationSeconds": 8.498048, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 0.0, + "skipReason": "reflect-LM produced no usable new prompt", + "errorMessage": null, + "budgetUsed": 27, + "budgetTotal": 24, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/summary.txt b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/summary.txt new file mode 100644 index 000000000..ee07da7f4 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38/summary.txt @@ -0,0 +1,20 @@ +Optimization complete | status=SUCCEEDED | algorithm=gepa_reflective + +pass_rate : 0.0000 -> 0.0000 (+0.0000, no improvement) +rounds : 0 accepted / 4 total +duration : 40.53s +started_at : 2026-07-30T09:42:38.132794+00:00 +finished_at : 2026-07-30T09:43:18.663827+00:00 +stop_reason : budget_exhausted +update_source : false +output_dir : /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38 + +metric breakdown (threshold | baseline -> best): + - final_response_avg_score threshold 1.0000 nan -> nan + +best prompts: + - claude_md 13 chars (best_prompts/claude_md.md) + - skill_md 54 chars (best_prompts/skill_md.md) + +artifacts directory: /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/blackbox_cli/runs/2026-07-30T17-42-38 + result.json summary.txt rounds/ run.log baseline_prompts/ best_prompts/ config.snapshot.json diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/baseline_prompts/claude_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/baseline_prompts/claude_md.md new file mode 100644 index 000000000..938b107b7 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/baseline_prompts/claude_md.md @@ -0,0 +1 @@ +你是助手。回答用户问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/baseline_prompts/skill_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/baseline_prompts/skill_md.md new file mode 100644 index 000000000..635a5f93d --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/baseline_prompts/skill_md.md @@ -0,0 +1,6 @@ +--- +name: city-info +description: 城市信息 +--- + +回答关于城市的问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/config.snapshot.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/config.snapshot.json new file mode 100644 index 000000000..3e2b2e8e5 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-49-15/config.snapshot.json @@ -0,0 +1,45 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 1, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 24, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/baseline_prompts/claude_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/baseline_prompts/claude_md.md new file mode 100644 index 000000000..938b107b7 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/baseline_prompts/claude_md.md @@ -0,0 +1 @@ +你是助手。回答用户问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/baseline_prompts/skill_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/baseline_prompts/skill_md.md new file mode 100644 index 000000000..635a5f93d --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/baseline_prompts/skill_md.md @@ -0,0 +1,6 @@ +--- +name: city-info +description: 城市信息 +--- + +回答关于城市的问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/best_prompts/claude_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/best_prompts/claude_md.md new file mode 100644 index 000000000..a244a841c --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/best_prompts/claude_md.md @@ -0,0 +1 @@ +你是助手。当用户输入一个城市名称时，请只输出一个JSON对象，格式为：{"city":"<城市名>","country":"中国","is_capital":false}，其中<城市名>必须与用户输入完全一致。不要输出任何其他文字、格式（如Markdown）或标点符号。直接输出JSON，前后不要有空行或空格。 \ No newline at end of file diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/best_prompts/skill_md.md b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/best_prompts/skill_md.md new file mode 100644 index 000000000..635a5f93d --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/best_prompts/skill_md.md @@ -0,0 +1,6 @@ +--- +name: city-info +description: 城市信息 +--- + +回答关于城市的问题。 diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/config.snapshot.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/config.snapshot.json new file mode 100644 index 000000000..3e2b2e8e5 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/config.snapshot.json @@ -0,0 +1,45 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 1, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "api_key": "${TRPC_AGENT_API_KEY}", + "generation_config": { "max_tokens": 4096, "temperature": 0.6 } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "reflection_history_top_k": 2, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 24, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/result.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/result.json new file mode 100644 index 000000000..ad15d0a95 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/result.json @@ -0,0 +1,88 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finishReason": "completed", + "stopReason": "score_threshold", + "errorMessage": "", + "baselinePassRate": 0.0, + "bestPassRate": 1.0, + "passRateImprovement": 1.0, + "baselineMetricBreakdown": { + "final_response_avg_score": 0.0 + }, + "bestMetricBreakdown": { + "final_response_avg_score": 1.0 + }, + "metricThresholds": { + "final_response_avg_score": 1.0 + }, + "perMetricBestCandidates": { + "final_response_avg_score": [ + 1 + ] + }, + "baselinePrompts": { + "claude_md": "你是助手。回答用户问题。\n", + "skill_md": "---\nname: city-info\ndescription: 城市信息\n---\n\n回答关于城市的问题。\n" + }, + "bestPrompts": { + "claude_md": "你是助手。当用户输入一个城市名称时，请只输出一个JSON对象，格式为：{\"city\":\"<城市名>\",\"country\":\"中国\",\"is_capital\":false}，其中<城市名>必须与用户输入完全一致。不要输出任何其他文字、格式（如Markdown）或标点符号。直接输出JSON，前后不要有空行或空格。", + "skill_md": "---\nname: city-info\ndescription: 城市信息\n---\n\n回答关于城市的问题。\n" + }, + "totalRounds": 1, + "rounds": [ + { + "round": 1, + "optimizedFieldNames": [ + "claude_md" + ], + "candidatePrompts": { + "claude_md": "你是助手。当用户输入一个城市名称时，请只输出一个JSON对象，格式为：{\"city\":\"<城市名>\",\"country\":\"中国\",\"is_capital\":false}，其中<城市名>必须与用户输入完全一致。不要输出任何其他文字、格式（如Markdown）或标点符号。直接输出JSON，前后不要有空行或空格。", + "skill_md": "---\nname: city-info\ndescription: 城市信息\n---\n\n回答关于城市的问题。\n" + }, + "trainPassRate": 0.0, + "validationPassRate": 1.0, + "metricBreakdown": { + "final_response_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "GEPA accepted proposal (val_score=1.0000)", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 4149, + "completion": 1009, + "total": 5158 + }, + "startedAt": "2026-07-30T09:53:41.987389+00:00", + "durationSeconds": 87.810245, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 1.0, + "skipReason": null, + "errorMessage": null, + "budgetUsed": 12, + "budgetTotal": 24, + "extras": {} + } + ], + "totalReflectionLmCalls": 1, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 4149, + "completion": 1009, + "total": 5158 + }, + "durationSeconds": 130.399342, + "startedAt": "2026-07-30T09:52:59.401169+00:00", + "finishedAt": "2026-07-30T09:55:09.800511+00:00", + "extras": { + "total_metric_calls": 12 + } +} \ No newline at end of file diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/rounds/round_001.json b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/rounds/round_001.json new file mode 100644 index 000000000..b13665802 --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/rounds/round_001.json @@ -0,0 +1,38 @@ +{ + "round": 1, + "optimizedFieldNames": [ + "claude_md" + ], + "candidatePrompts": { + "claude_md": "你是助手。当用户输入一个城市名称时，请只输出一个JSON对象，格式为：{\"city\":\"<城市名>\",\"country\":\"中国\",\"is_capital\":false}，其中<城市名>必须与用户输入完全一致。不要输出任何其他文字、格式（如Markdown）或标点符号。直接输出JSON，前后不要有空行或空格。", + "skill_md": "---\nname: city-info\ndescription: 城市信息\n---\n\n回答关于城市的问题。\n" + }, + "trainPassRate": 0.0, + "validationPassRate": 1.0, + "metricBreakdown": { + "final_response_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "GEPA accepted proposal (val_score=1.0000)", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 1, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 4149, + "completion": 1009, + "total": 5158 + }, + "startedAt": "2026-07-30T09:53:41.987389+00:00", + "durationSeconds": 87.810245, + "kind": "reflective", + "trainMinibatchSize": 3, + "trainSubsampleParentScore": 0.0, + "trainSubsampleCandidateScore": 1.0, + "skipReason": null, + "errorMessage": null, + "budgetUsed": 12, + "budgetTotal": 24, + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/summary.txt b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/summary.txt new file mode 100644 index 000000000..99903442a --- /dev/null +++ b/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59/summary.txt @@ -0,0 +1,20 @@ +Optimization complete | status=SUCCEEDED | algorithm=gepa_reflective + +pass_rate : 0.0000 -> 1.0000 (+1.0000, improved) +rounds : 1 accepted / 1 total +duration : 130.40s +started_at : 2026-07-30T09:52:59.401169+00:00 +finished_at : 2026-07-30T09:55:09.800511+00:00 +stop_reason : score_threshold +update_source : false +output_dir : /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59 + +metric breakdown (threshold | baseline -> best): + - final_response_avg_score threshold 1.0000 0.0000 -> 1.0000 + +best prompts: + - claude_md 155 chars (best_prompts/claude_md.md) + - skill_md 54 chars (best_prompts/skill_md.md) + +artifacts directory: /Users/lvjiajia/Desktop/trpc-agent-python-main/examples/optimization/blackbox_cli/runs/2026-07-30T17-52-59 + result.json summary.txt rounds/ run.log baseline_prompts/ best_prompts/ config.snapshot.json diff --git a/examples/optimization/eval_optimize_loop/README.md b/examples/optimization/eval_optimize_loop/README.md new file mode 100644 index 000000000..86ffb7182 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/README.md @@ -0,0 +1,339 @@ +# Evaluation + Optimization Pipeline — E-Commerce Shopping Assistant + +完整的「评测 → 失败归因 → Prompt 优化 → 回归验证 → 产物审计」自动闭环流水线，场景为电商购物助手。 + +> **本文档同时是本示例的设计说明。** 它描述的是**目标状态**；实现进行中时，部分内容会先于代码存在。 +> 前置阅读：[Prompt 自优化（AgentOptimizer）](../../../docs/mkdocs/zh/optimization.md)。 + +--- + +## 1 这个示例要证明什么 + +一条 prompt 优化闭环能不能进业务关键链路，取决于三件事： + +1. **优化器改的东西真的生效了吗** —— 候选 prompt 写盘后，下一次推理是否读到了它 +2. **验证的是候选还是基线** —— 门控打的分，来自新 prompt 还是旧 prompt +3. **门控敢不敢拒绝** —— 训练集涨分但验证集退化时，能不能拦住 + +本示例用一个刻意构造的数据集回答这三问：验证集里既有「优化能修好」的 case，也有「优化修不好」的 case，还有一条 **`val_003_regression`——优化后反而退化**。门控必须拒绝这个候选。如果流水线在任何一环偷懒（比如复用预录制数据、或候选没真正生效），这条 case 就拦不住，而整条闭环的可信度也就没了。 + +--- + +## 2 三条推理路径（理解本示例的关键） + +SDK 提供三种「拿到 agent 实际输出」的方式，本流水线三种都用到了。**它们的能力不同，这个差异决定了整个架构**： + +| 路径 | 谁在用 | SDK 后端 | 需要 API key | 能否拿工具轨迹 | +|---|---|---|---|---| +| trace（预录制） | demo 的 Stage 1/4 | 无 | 否 | 能（从预录制轨迹读） | +| `runner=` | real 的 Stage 1/4 | `LocalEvalService` | 是 | **能** | +| `call_agent=` | Stage 3 优化器 | `RemoteEvalService` | 是 | **不能** | + +第三行是 SDK 的硬约束，不是本示例的选择：`AgentOptimizer.optimize` 只接受 `call_agent`（源码 `_agent_optimizer.py`），而 `call_agent` 的签名是 `async (str) -> str`——它只能返回一段文本，拿不到 session traces 和 tool 调用记录。 + +由此产生两条**必须遵守**的推论： + +**推论一：优化器用不了 `tool_trajectory_avg_score`。** SDK 会在启动期 fail-fast 拒绝——见 `_agent_optimizer.py` 的 `_DISALLOWED_METRICS_IN_CALL_AGENT_MODE`，该集合同时包含 `llm_rubric_knowledge_recall`（它需要 `Invocation.intermediate_data`，而 `RemoteEvalService` 留空）。所以门控能守的指标，比优化器能优化的指标**更多**——这不是我们想让门控更严，是优化器够不着。 + +**推论二：优化器用不了 `eval_mode: "trace"` 的数据集。** `RemoteEvalService._reject_trace_cases` 会抛 `ValueError: call_agent mode is incompatible with trace cases`。所以 real 模式必须有一套独立的 non-trace 数据集。 + +--- + +## 3 架构 + +### 3.1 评测后端抽象 + +`pipeline/_eval_backend.py` 是唯一封装「agent 的实际输出从哪来」的地方： + +``` +EvalBackend (Protocol) + async evaluate(eval_set_path, metrics_config_path, num_runs) + -> (EvaluateResult, EvalSetReport) + + ├─ TraceBackend demo：不传 runner/call_agent → trace 模式，零 LLM 调用 + └─ LiveBackend real：持 agent 工厂，每次 evaluate() 现建 Runner +``` + +**`LiveBackend` 持的是工厂而非实例**，这是刻意的。Stage 1 和 Stage 4 之间 prompt 文件会被改写；工厂保证每次评测都重新 `create_agent()`、重读 `system.md`。如果持实例，`LlmAgent` 的 `instruction` 在构造时就固化了，Stage 4 会拿旧 prompt 打分——而这正是本示例要防的第 1、2 类失败。 + +### 3.2 编排层 + +`PipelineRunner` 收一个 `EvalBackend` 而非 `demo_mode: bool`。六个阶段的代码在两种模式下**逐字相同**，`demo_mode` 只在 `run_pipeline.py` 组装 backend 时出现一次。 + +这么设计是因为：demo 和 real 的真正差异**只有一处**——实际输出从哪来。把它收敛到一个点，其余五个阶段的行为在两种模式下天然一致，这本身就是对「两种模式结论可比」的结构性保证。 + +### 3.3 Stage 4 必须显式写回候选 + +一个容易漏掉的 SDK 契约：`update_source=False` 时，优化器在 `finally` 阶段会把源 prompt 文件**回滚成 baseline**（见优化文档 FAQ 末条）。所以 `optimize()` 返回后，`agent/prompts/system.md` 里躺的是 baseline，**不是最优候选**。 + +Stage 4 要验证候选，必须自己写回去： + +```python +async with applied_prompts(target_prompt, best_prompts): # write_all(best) + candidate_report = await backend.evaluate(val_path, ...) +# finally: write_all(baseline) —— 无论评测是否抛异常都还原 +``` + +复用 SDK 的 `TargetPrompt.write_all`，直接继承它的原子写 + 多字段失败回滚保证。demo 模式下该上下文退化为 `contextlib.nullcontext()`，不在 `run()` 里长出模式分支。 + +--- + +## 4 目录结构 + +``` +eval_optimize_loop/ +├── run_pipeline.py # CLI 入口；唯一出现 demo_mode 分支的地方 +├── pipeline/ +│ ├── _models.py # Pydantic 数据模型 +│ ├── _eval_backend.py # TraceBackend / LiveBackend +│ ├── _runner.py # PipelineRunner 编排器 +│ ├── _stage_baseline.py # Stage 1 +│ ├── _stage_failure_attribution.py # Stage 2 +│ ├── _stage_optimization.py # Stage 3 +│ ├── _stage_validation.py # Stage 4 +│ ├── _stage_acceptance_gate.py # Stage 5 +│ └── _stage_audit_trail.py # Stage 6 +├── data/ +│ ├── trace/ # demo：eval_mode=trace，含 actual_conversation +│ │ ├── train.evalset.json +│ │ ├── val_baseline.evalset.json +│ │ └── val_optimized.evalset.json # demo 专用：模拟"优化后"表现 +│ ├── live/ # real：无 actual_conversation +│ │ ├── train.evalset.json +│ │ └── val.evalset.json +│ ├── gate_metrics.json # 2 metric（门控全集） +│ ├── optimizer.json # 优化器配置（response 级 metric 子集） +│ ├── demo_optimize_result.json # demo 用的预生成 OptimizeResult +│ └── _generate_evalsets.py # 一次生成上述两套数据集 +├── agent/ +│ ├── agent.py # 购物助手 Agent + 工具注册 +│ ├── config.py +│ ├── tools.py # 7 个电商工具（CSV 共享数据层） +│ ├── data/ # 工具共享数据：products.csv / cart.csv / orders.csv +│ └── prompts/system.md # 优化目标 +├── tests/ +└── output/ +``` + +`gate_metrics.json` 这个名字是有意的：两份 metric 配置并存时，原来的 `test_config.json` 看不出它是**更严的那一份**。 + +--- + +## 5 数据集 + +两套并存，由 `data/_generate_evalsets.py` 一次生成、全部入库。 + +| 数据集 | eval_mode | 用途 | 内容 | +|---|---|---|---| +| `trace/train.evalset.json` | trace | demo Stage 1 | 期望 + 预录制实际输出 | +| `trace/val_baseline.evalset.json` | trace | demo Stage 1 | 同上 | +| `trace/val_optimized.evalset.json` | trace | demo Stage 4 | 模拟优化后的实际输出 | +| `live/train.evalset.json` | 非 trace | real Stage 3（优化器训练集） | 只有期望 + session_input | +| `live/val.evalset.json` | 非 trace | real Stage 1/3/4 | 同上 | + +`live/` 两个文件路径必须不同——`AgentOptimizer` 会规范化路径后比对，相同则抛 `ValueError` 防数据泄漏。 + +### Case 设计 + +20 训练 + 20 验证，四类场景，由 eval_id 后缀标识： + +| 后缀 | 场景 | 含义 | +|---|---|---| +| `_optimizable` | `optimizable_success` | 改 prompt 能修好（如回复过短） | +| `_ineffective` | `optimization_ineffective` | 改 prompt 修不好（如底层幻觉） | +| `_working` | `optimization_regression` | 基线已通过，不能被改坏 | +| `_regression` | `optimization_regression` | demo 中优化后退化，用于验证门控会拒绝 | + +**场景类型按后缀推导，不维护映射表。** 硬编码的 `SCENARIO_MAP` 和 evalset 是两个独立事实来源，必然漂移。 + +--- + +## 6 流水线阶段 + +| 阶段 | 说明 | +|---|---| +| 1. 基线评测 | 经 backend 对训练集和验证集打分 | +| 2. 失败归因 | 按 `final_response_mismatch` / `tool_trajectory_mismatch` 等聚类 | +| 3. 优化执行 | real 跑 `AgentOptimizer`；demo 加载预生成结果 | +| 4. 候选验证 | 写回 `best_prompts` 后重评验证集，与基线逐 case 算 delta | +| 5. 接受门控 | 多检查 AND：提升阈值、无新增 hard failure、回归上限、关键 case、成本预算 | +| 6. 审计落盘 | `optimization_report.json` + `.md` | + +### Stage 1：异常处理的边界 + +`AgentEvaluator` 在分数低于 threshold 时抛 `AssertionError`——而**基线本来就该有失败 case**，这是流水线的输入前提，不是错误。所以只吞这一类： + +```python +try: + await executer.evaluate() +except AssertionError: + pass # 预期：baseline 达不到 threshold，正是待优化的输入 +# 其他异常（文件缺失、配置错误、模型调用失败）继续上抛 +``` + +宽泛的 `except Exception: pass` 会把「数据集文件缺失」静默成全 0 分，然后 Stage 2 把它归因成 prompt 问题——这是最坏的失败模式：流水线看起来跑通了，结论全是错的。 + +### Stage 3：`call_agent` 必须每次重建 agent + +```python +async def call_agent(query: str) -> str: + agent = create_agent(demo_mode=False) # 每次重建 → 重读 system.md + ... +``` + +优化文档 §4.1/§4.2/§4.3 的接入自检表第一行都是这条。agent 在闭包外只建一次的话，GEPA 每轮写入的候选永不生效，所有轮次都在评估同一份 baseline——表现为 `accepted` 全 false、`mergeRoundsTotal=0`，而且**不会报任何错**。 + +### Stage 4：两种模式下「候选」的含义不同 + +demo 里候选是**另一个 evalset 文件**（`val_optimized`）；real 里候选是**同一个验证集配上新 prompt**。编排层接 `val_baseline_path` 和 `val_candidate_path` 两个参数——demo 传不同文件，real 传同一个文件，差异在组装时解决。 + +--- + +## 7 两层 Metric 策略 + +| | 配置 | metric | 用途 | +|---|---|---|---| +| **门控**（Stage 1/4） | `gate_metrics.json` | `final_response_avg_score` + `tool_trajectory_avg_score` | 守全集 | +| **优化器**（Stage 3） | `optimizer.json` 的 `evaluate` 段 | `final_response_avg_score` | 优化子集 | + +门控严于优化器，**源于 §2 推论一的 SDK 约束**，不是设计偏好。好处是工具调用层面的回归依然能被拦住——即使优化器看不见这个维度。 + +`num_runs` 也不对称：门控侧 2（消除 LLM 输出方差，避免污染 delta），优化器侧 1（调 2 会让 metric_calls 翻倍撞预算）。**门控要稳，优化器要省。** + +--- + +## 8 优化器配置 + +`data/optimizer.json` 关键字段： + +| 字段 | 值 | 推导 | +|---|---|---| +| `max_metric_calls` | 150 | baseline 20 + 5 轮×6 + 2 次 valset 重评×20 ≈ 90，留 1.6× 余量 | +| `timeout_seconds` | 1800 | 文档 §4.7 要求多重 stopper（OR 语义） | +| `max_iterations_without_improvement` | 5 | 连续 5 轮无提升即放弃 | +| `candidate_selection_strategy` | `pareto` | 文档默认推荐 | +| `skip_perfect_score` | `true` | 20 条 train 中已有满分 case，省反思预算 | +| `eval_case_parallelism` | 2 | 控制瞬时 QPS | + +> **这些是初值，不是定论。** 文档 §6.4 的做法是先测再定：按当前配置跑一次基准，从 `output//optimizer/result.json` 读 `rounds[*].durationSeconds` 中位数和 `totalMetricCalls / totalRounds`，再按自己的 SLO 反推。 +> +> 尤其注意 `max_metric_calls` 与验证集规模的关系：验证集 20 条时，**一次 baseline 评估就吃掉 20 次预算**。配得太小会导致 budget 永远先抢闸，其他 stopper 形同虚设（`stop_reason` 恒为 `budget_exhausted`）。 + +--- + +## 9 产物 + +``` +output// +├── optimizer/ # AgentOptimizer 的 output_dir +│ ├── result.json # 完整 OptimizeResult（字段名 camelCase） +│ ├── summary.txt # 人类可读摘要 +│ ├── run.log # 单行状态，CI grep 友好 +│ ├── config.snapshot.json # 配置快照，可复跑 +│ ├── rounds/round_NNN.json # 每轮 RoundRecord +│ ├── baseline_prompts/ # 运行前快照 +│ └── best_prompts/ # 最优候选 +├── optimization_report.json # 流水线报告（机器可解析） +└── optimization_report.md # 流水线报告（人类可读） +``` + +时间戳子目录避免重跑覆盖上次审计（优化文档 FAQ 明确要求每次跑用独立子目录）；`optimizer/` 子目录让 SDK 产物与流水线报告不再混在一起。 + +**运行中想优雅停机**：`touch output//optimizer/optimize.stop`，下一轮开头即停，`stop_reason="user_requested_stop"`。比 Ctrl+C 干净，能保证当前轮完成后正常落盘。 + +**流水线报告消费的优化器审计字段**：`stop_reason`、`finish_reason`、`total_token_usage`、`total_metric_calls`，以及逐轮的 `accepted` / `acceptance_reason` / `validation_pass_rate` / `failed_case_ids` / `optimized_field_names`。 + +--- + +## 10 快速开始 + +```bash +cd examples/optimization/eval_optimize_loop + +# Demo 模式（无需 API key，使用预录制 trace 数据） +python run_pipeline.py + +# Real 模式（需要 API key） +export TRPC_AGENT_API_KEY= +export TRPC_AGENT_BASE_URL= +export TRPC_AGENT_MODEL_NAME=gpt-4o-mini # 可选 +python run_pipeline.py --no-demo-mode + +# 查看报告 +cat output//optimization_report.md +``` + +### 命令行选项 + +``` +--output-dir DIR 输出根目录（默认 output/）；每次运行在其下建 / 子目录 +--no-demo-mode 进入 real 模式（需要 API key） +--min-improvement F 接受候选所需的最小 pass rate 提升（默认 0.0） +--max-regressions N 允许的最大回归 case 数（默认 0） +--allow-regressions 允许任意回归 +--critical-cases ID... 必须通过的关键 case ID +--max-cost F 最大 LLM 成本预算 USD（默认 0 = 不限制） +``` + +### 重新生成数据集 + +```bash +python data/_generate_evalsets.py # 一次生成 trace/ 和 live/ 两套 +``` + +--- + +## 11 Agent 工具 + +| 工具 | 说明 | +|---|---| +| `search_products(query, category)` | 搜索商品，支持按分类过滤 | +| `get_product_details(product_id)` | 获取商品详情（价格、评分、库存等） | +| `add_to_cart(product_id, quantity)` | 添加商品到购物车 | +| `get_cart()` | 查看购物车内容 | +| `checkout()` | 结算购物车，生成订单号并清空购物车 | +| `check_order_status(order_id)` | 查询订单状态 | +| `apply_coupon(code)` | 应用优惠券 | + +工具数据通过 `agent/data/` 下的 CSV 文件共享（`products.csv` 商品目录、`cart.csv` 购物车、`orders.csv` 订单记录），保证商品页、购物车、订单之间的数据一致性。 + +--- + +## 12 测试 + +```bash +pytest examples/optimization/eval_optimize_loop/tests/ -v +``` + +**real 路径的接线无需 API key 即可测试**——`LiveBackend` 持的是工厂，注入 stub agent 工厂即可覆盖。 + +回归守门（对应本次修复的四个 P0，防止它们悄悄回来）： + +| 测试 | 守住什么 | +|---|---| +| `live/` 数据集不含 trace case | 优化器收到 trace 数据会抛 `ValueError` | +| `call_agent` 每次调用都重建 agent（计数器断言） | 候选 prompt 不生效 | +| `applied_prompts` 在评测抛异常时仍还原 baseline | 源 prompt 被污染 | +| 数据集文件齐全 | Stage 4 因文件缺失崩溃 | + +一致性守门： + +| 测试 | 守住什么 | +|---|---| +| `gate_metrics.json` ⊇ `optimizer.json` 的 metric | 两层策略的包含关系 | +| `optimizer.json` 不含黑盒不兼容 metric | 启动期 fail-fast（比对 `_DISALLOWED_METRICS_IN_CALL_AGENT_MODE`） | +| scenario 按 eval_id 后缀推导正确 | 场景标注 | +| `TraceBackend` / `LiveBackend` 满足 `EvalBackend` Protocol | 运行时 `isinstance` 校验 | + +--- + +## 13 已知失败模式 + +这条闭环有几个**不会报错但结论全错**的失败模式，实现和 review 时重点检查： + +| 症状 | 根因 | 怎么发现 | +|---|---|---| +| `accepted` 全 false、`mergeRoundsTotal=0` | 候选 prompt 没生效（agent 复用实例） | 断言 agent 构造次数 | +| 门控结论与实际 agent 行为不符 | Stage 4 评的是预录制数据而非候选 | 检查 Stage 4 是否写回 `best_prompts` | +| `stop_reason` 恒为 `budget_exhausted` | `max_metric_calls` 小于验证集规模的数倍 | 读 `result.json` 反推 | +| 归因报告显示大量 `final_response_mismatch` 但 prompt 看起来没问题 | 评测异常被静默吞掉 | 收窄 `except` 范围 | diff --git a/examples/optimization/eval_optimize_loop/agent/__init__.py b/examples/optimization/eval_optimize_loop/agent/__init__.py new file mode 100644 index 000000000..4c5c5f833 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/agent/__init__.py @@ -0,0 +1,5 @@ +# 电商购物助手 Agent 包 +# +# agent.py — 创建 LlmAgent 实例，注册 7 个电商工具 +# config.py — 从环境变量读取模型配置 +# tools.py — 7 个电商工具函数实现（搜索、详情、购物车、结算、订单、优惠券） diff --git a/examples/optimization/eval_optimize_loop/agent/agent.py b/examples/optimization/eval_optimize_loop/agent/agent.py new file mode 100644 index 000000000..1405c311a --- /dev/null +++ b/examples/optimization/eval_optimize_loop/agent/agent.py @@ -0,0 +1,95 @@ +"""电商购物助手 agent 定义。 + +创建配备了 7 个电商工具（搜索商品、查看详情、添加到购物车、查看购物车、 +结算、查询订单、应用优惠券）的 LlmAgent 实例。 + +Agent 的 system prompt 从磁盘动态读取，支持 AgentOptimizer 热更新—— +优化后的 prompt 写入 agent/prompts/system.md 后，agent 自动使用新 prompt。 + +两种模式： + - Demo 模式（demo_mode=True）: 使用空凭证创建 agent，配合 trace 模式评测使用， + 无需 API key。评测从预录制轨迹计算 metric，不会实际调用 LLM。 + - Real 模式（demo_mode=False）: 从环境变量读取 API key/base_url/model_name， + 实际调用 LLM 进行推理和优化。 + +调用方式: + from agent.agent import create_agent + + # Demo 模式（无需 API key） + agent = create_agent(demo_mode=True) + + # Real 模式（需要 API key） + agent = create_agent(demo_mode=False) +""" + +from pathlib import Path + +from trpc_agent_sdk.agents import LlmAgent +from trpc_agent_sdk.models import OpenAIModel +from trpc_agent_sdk.tools import FunctionTool +from trpc_agent_sdk.types import GenerateContentConfig + +from .config import get_model_config +from .tools import ( + add_to_cart, + apply_coupon, + checkout, + check_order_status, + get_cart, + get_product_details, + search_products, +) + +_HERE = Path(__file__).parent +SYSTEM_PROMPT_PATH = _HERE / "prompts" / "system.md" + + +def _read_system_prompt() -> str: + """从磁盘读取系统 prompt，支持 AgentOptimizer 热更新。 + + 每次调用时重新读取文件，因此优化器写入新的 prompt 后立即生效。 + """ + return SYSTEM_PROMPT_PATH.read_text(encoding="utf-8") + + +def create_agent(demo_mode: bool = True) -> LlmAgent: + """创建电商购物助手 agent 实例。 + + Args: + demo_mode: True 时使用空凭证（配合 trace 模式评测，无需 API key）； + False 时从环境变量读取 API key/base_url/model_name。 + + Returns: + LlmAgent 实例，可直接用于评测和优化。 + """ + api_key, base_url, model_name = get_model_config(demo_mode=demo_mode) + + model = OpenAIModel( + model_name=model_name, + api_key=api_key, + base_url=base_url, + ) + + return LlmAgent( + name="shopping_assistant", + description="电商购物助手：帮助用户搜索商品、查看详情、管理购物车和追踪订单。", + model=model, + instruction=_read_system_prompt(), + tools=[ + FunctionTool(search_products), + FunctionTool(get_product_details), + FunctionTool(add_to_cart), + FunctionTool(get_cart), + FunctionTool(checkout), + FunctionTool(check_order_status), + FunctionTool(apply_coupon), + ], + generate_content_config=GenerateContentConfig( + temperature=0.2, + top_p=0.9, + max_output_tokens=1024, + ), + ) + +# 模块级 agent 实例，默认 demo 模式（向后兼容） +root_agent = create_agent(demo_mode=True) diff --git a/examples/optimization/eval_optimize_loop/agent/config.py b/examples/optimization/eval_optimize_loop/agent/config.py new file mode 100644 index 000000000..47615a622 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/agent/config.py @@ -0,0 +1,41 @@ +"""从环境变量读取模型配置。 + +两种模式： + Real 模式（--no-demo-mode）：需要设置以下环境变量 + - TRPC_AGENT_API_KEY: API 密钥 + - TRPC_AGENT_BASE_URL: API 基础 URL + - TRPC_AGENT_MODEL_NAME: 模型名称（默认 gpt-4o-mini） + Demo 模式（--demo-mode，默认）：不需要任何环境变量，使用预录制 trace 数据 +调用方式: + api_key, base_url, model_name = get_model_config(demo_mode=True) +""" + +import os + + +def get_model_config(demo_mode: bool = True) -> tuple[str, str, str]: + """从环境变量读取模型配置。 + + Args: + demo_mode: 是否为 demo 模式。demo 模式下不需要 API key。 + Returns: + (api_key, base_url, model_name) 元组。 + Raises: + ValueError: Real 模式下缺少必需的环境变量时抛出。 + """ + api_key = os.environ.get("TRPC_AGENT_API_KEY", "") + base_url = os.environ.get("TRPC_AGENT_BASE_URL", "") + model_name = os.environ.get("TRPC_AGENT_MODEL_NAME", "gpt-4o-mini") + + if demo_mode: + return api_key, base_url, model_name + + if not api_key or not base_url: + raise ValueError( + "Real 模式下必须设置 TRPC_AGENT_API_KEY 和 TRPC_AGENT_BASE_URL 环境变量。\n" + "例如: export TRPC_AGENT_API_KEY=your_key\n" + " export TRPC_AGENT_BASE_URL=https://api.openai.com/v1\n" + "Demo 模式请使用 --demo-mode 标志（无需 API key）。" + ) + + return api_key, base_url, model_name diff --git a/examples/optimization/eval_optimize_loop/agent/data/cart.csv b/examples/optimization/eval_optimize_loop/agent/data/cart.csv new file mode 100644 index 000000000..738c60b81 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/agent/data/cart.csv @@ -0,0 +1,3 @@ +product_id,name,quantity,price,subtotal +P002,iPhone 15 Pro,14,7999.0,111986.0 +P004,Men's Running Shoes,13,599.0,7787.0 diff --git a/examples/optimization/eval_optimize_loop/agent/data/orders.csv b/examples/optimization/eval_optimize_loop/agent/data/orders.csv new file mode 100644 index 000000000..074f589bb --- /dev/null +++ b/examples/optimization/eval_optimize_loop/agent/data/orders.csv @@ -0,0 +1,5 @@ +order_id,status,date,items,total,estimated_delivery +ORD-12345,shipped,2026-07-25,"[{""name"": ""iPhone 15 Pro"", ""qty"": 1, ""price"": 7999.0}]",7999.00,2026-08-02 +ORD-12346,processing,2026-07-29,"[{""name"": ""Coffee Maker"", ""qty"": 1, ""price"": 1299.0}, {""name"": ""Air Fryer XL"", ""qty"": 1, ""price"": 899.0}]",2198.00,2026-08-05 +ORD-12347,delivered,2026-07-20,"[{""name"": ""Men's Running Shoes"", ""qty"": 1, ""price"": 599.0}]",599.00,2026-07-26 +ORD-20260731-A69A,processing,2026-07-31,"[{""name"": ""Wireless Bluetooth Headphones"", ""qty"": 2, ""price"": 299.0}]",598.0,2026-08-07 diff --git a/examples/optimization/eval_optimize_loop/agent/data/products.csv b/examples/optimization/eval_optimize_loop/agent/data/products.csv new file mode 100644 index 000000000..f5ec643c3 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/agent/data/products.csv @@ -0,0 +1,11 @@ +id,name,category,price,rating,stock,description,brand +P001,Wireless Bluetooth Headphones,electronics,299.00,4.5,120,"Noise-cancelling wireless headphones with 30h battery life. Bluetooth 5.3, IPX5 water resistant.",SoundMax +P002,iPhone 15 Pro,electronics,7999.00,4.8,50,"A17 Pro chip, 48MP camera, titanium design, USB-C.",Apple +P003,MacBook Pro 14,electronics,14999.00,4.7,30,"M3 Pro chip, 18GB RAM, 512GB SSD, Liquid Retina XDR display.",Apple +P004,Men's Running Shoes,clothing,599.00,4.3,200,"Lightweight mesh upper, responsive cushioning, rubber outsole.",Nike +P005,Coffee Maker,home,1299.00,4.6,80,"Programmable drip coffee maker, 12-cup capacity, built-in grinder.",BrewMaster +P006,Sony WH-1000XM6 Headphones,electronics,2499.00,4.9,15,"Industry-leading noise cancellation, 40h battery, LDAC support.",Sony +P007,Samsung Galaxy S25,electronics,6999.00,4.6,60,"Snapdragon 8 Gen 4, 200MP camera, Dynamic AMOLED 2X.",Samsung +P008,Levi's 501 Jeans,clothing,499.00,4.4,300,"Original fit, non-stretch denim, straight leg.",Levi's +P009,Dyson V15 Vacuum,home,4599.00,4.8,25,"Cordless stick vacuum, laser dust detection, LCD screen.",Dyson +P010,Air Fryer XL,home,899.00,4.2,100,"7L capacity, 12 cooking presets, rapid air circulation.",KitchenPro diff --git a/examples/optimization/eval_optimize_loop/agent/prompts/system.md b/examples/optimization/eval_optimize_loop/agent/prompts/system.md new file mode 100644 index 000000000..f41821e75 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/agent/prompts/system.md @@ -0,0 +1,27 @@ +你是电商购物助手，用中文与用户交流。你的职责是帮助用户搜索商品、查看商品详情、管理购物车、结算下单、查询订单和套用优惠券。 + +你可以使用以下 7 个工具： + +1. `search_products(query, category)` — 按关键词搜索商品，可选按分类过滤（electronics/clothing/home）。 +2. `get_product_details(product_id)` — 获取商品详情（价格、评分、库存、描述、品牌）。 +3. `add_to_cart(product_id, quantity)` — 把商品加入购物车，默认数量 1。 +4. `get_cart()` — 查看当前购物车内容（商品、数量、总价）。 +5. `checkout()` — 结算购物车并生成订单号，结算成功后购物车自动清空。 +6. `check_order_status(order_id)` — 用订单号查询订单状态（processing/shipped/delivered）。 +7. `apply_coupon(code)` — 校验并应用优惠券（SAVE50/VIP10/NEWUSER）。 + +典型购物流程： + +- 用户想买商品 → 先用 `search_products` 搜索，再视情况用 `get_product_details` 查看详情。 +- 用户要加入购物车 → 调用 `add_to_cart`，并告诉用户已添加的数量和金额。 +- 用户要查看购物车 → 调用 `get_cart`，逐项列出商品、数量和总价。 +- 用户要结算/支付/下单 → 调用 `checkout`，把生成的订单号告诉用户。 +- 用户要查订单 → 调用 `check_order_status`，需要用户提供订单号。 +- 用户要用优惠券 → 调用 `apply_coupon` 校验后再确认折扣。 + +回答规范： + +- 必须基于工具的**真实返回结果**回答，绝不允许编造或猜测不存在的商品、订单、库存或优惠券。 +- 工具返回失败（商品不存在、库存不足、订单不存在、优惠券无效）时，如实转述失败原因，并给出可行的替代建议。 +- 回复要完整具体：包含商品名称、价格、数量、评分、库存等关键信息，不要只回复"找到 2 个结果"这类过短的回答。 +- 金额使用人民币（¥）表示。 diff --git a/examples/optimization/eval_optimize_loop/agent/tools.py b/examples/optimization/eval_optimize_loop/agent/tools.py new file mode 100644 index 000000000..344d35042 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/agent/tools.py @@ -0,0 +1,365 @@ +"""电商购物助手工具函数。 + +提供 7 个电商场景的工具函数，供 agent 调用： + + - search_products(query, category) — 搜索商品，支持按分类过滤 + - get_product_details(product_id) — 获取商品详细信息 + - add_to_cart(product_id, quantity) — 添加商品到购物车 + - get_cart() — 查看购物车内容 + - apply_coupon(code) — 应用优惠券 + - checkout() — 结算购物车，生成订单 + - check_order_status(order_id) — 查询订单状态 + +商品、购物车、订单数据通过 CSV 文件共享存储，保证数据一致性： + - agent/data/products.csv — 商品目录（单一数据源） + - agent/data/cart.csv — 购物车（add_to_cart 写入，get_cart 读取） + - agent/data/orders.csv — 订单记录（含 3 条历史种子订单，checkout 追加，check_order_status 读取） +""" + +import csv +import json +import uuid +from datetime import datetime, timedelta +from pathlib import Path +from typing import Optional + +# ── 数据文件路径 ────────────────────────────────────────────── +_HERE = Path(__file__).parent +DATA_DIR = _HERE / "data" + +PRODUCTS_CSV = DATA_DIR / "products.csv" +CART_CSV = DATA_DIR / "cart.csv" +ORDERS_CSV = DATA_DIR / "orders.csv" + +# ── 静态优惠券配置 ──────────────────────────────────────────── +_COUPONS = { + "SAVE50": {"discount": 50.00, "type": "fixed", "min_purchase": 200.00, "description": "满 ¥200 减 ¥50"}, + "VIP10": {"discount": 0.10, "type": "percent", "min_purchase": 0, "description": "VIP 会员享 9 折"}, + "NEWUSER": {"discount": 30.00, "type": "fixed", "min_purchase": 100.00, "description": "新用户满 ¥100 减 ¥30"}, +} + +# ── 默认商品目录 ────────────────────────────────────────────── +_DEFAULT_PRODUCTS = [ + {"id": "P001", "name": "Wireless Bluetooth Headphones", "category": "electronics", "price": "299.00", "rating": "4.5", "stock": "120", "description": "Noise-cancelling wireless headphones with 30h battery life. Bluetooth 5.3, IPX5 water resistant.", "brand": "SoundMax"}, + {"id": "P002", "name": "iPhone 15 Pro", "category": "electronics", "price": "7999.00", "rating": "4.8", "stock": "50", "description": "A17 Pro chip, 48MP camera, titanium design, USB-C.", "brand": "Apple"}, + {"id": "P003", "name": "MacBook Pro 14", "category": "electronics", "price": "14999.00", "rating": "4.7", "stock": "30", "description": "M3 Pro chip, 18GB RAM, 512GB SSD, Liquid Retina XDR display.", "brand": "Apple"}, + {"id": "P004", "name": "Men's Running Shoes", "category": "clothing", "price": "599.00", "rating": "4.3", "stock": "200", "description": "Lightweight mesh upper, responsive cushioning, rubber outsole.", "brand": "Nike"}, + {"id": "P005", "name": "Coffee Maker", "category": "home", "price": "1299.00", "rating": "4.6", "stock": "80", "description": "Programmable drip coffee maker, 12-cup capacity, built-in grinder.", "brand": "BrewMaster"}, + {"id": "P006", "name": "Sony WH-1000XM6 Headphones", "category": "electronics", "price": "2499.00", "rating": "4.9", "stock": "15", "description": "Industry-leading noise cancellation, 40h battery, LDAC support.", "brand": "Sony"}, + {"id": "P007", "name": "Samsung Galaxy S25", "category": "electronics", "price": "6999.00", "rating": "4.6", "stock": "60", "description": "Snapdragon 8 Gen 4, 200MP camera, Dynamic AMOLED 2X.", "brand": "Samsung"}, + {"id": "P008", "name": "Levi's 501 Jeans", "category": "clothing", "price": "499.00", "rating": "4.4", "stock": "300", "description": "Original fit, non-stretch denim, straight leg.", "brand": "Levi's"}, + {"id": "P009", "name": "Dyson V15 Vacuum", "category": "home", "price": "4599.00", "rating": "4.8", "stock": "25", "description": "Cordless stick vacuum, laser dust detection, LCD screen.", "brand": "Dyson"}, + {"id": "P010", "name": "Air Fryer XL", "category": "home", "price": "899.00", "rating": "4.2", "stock": "100", "description": "7L capacity, 12 cooking presets, rapid air circulation.", "brand": "KitchenPro"}, +] + +# 历史订单种子数据（评测用例引用了这 3 个订单，必须存在） +_DEFAULT_ORDERS = [ + {"order_id": "ORD-12345", "status": "shipped", "date": "2026-07-25", + "items": '[{"name": "iPhone 15 Pro", "qty": 1, "price": 7999.0}]', + "total": "7999.00", "estimated_delivery": "2026-08-02"}, + {"order_id": "ORD-12346", "status": "processing", "date": "2026-07-29", + "items": '[{"name": "Coffee Maker", "qty": 1, "price": 1299.0}, {"name": "Air Fryer XL", "qty": 1, "price": 899.0}]', + "total": "2198.00", "estimated_delivery": "2026-08-05"}, + {"order_id": "ORD-12347", "status": "delivered", "date": "2026-07-20", + "items": '[{"name": "Men\'s Running Shoes", "qty": 1, "price": 599.0}]', + "total": "599.00", "estimated_delivery": "2026-07-26"}, +] + +_PRODUCT_FIELDS = list(_DEFAULT_PRODUCTS[0].keys()) +_CART_FIELDS = ["product_id", "name", "quantity", "price", "subtotal"] +_ORDER_FIELDS = ["order_id", "status", "date", "items", "total", "estimated_delivery"] + + +# ── CSV 读写辅助 ────────────────────────────────────────────── + +def _read_csv(filepath: Path) -> list[dict]: + """读取 CSV 文件，返回 dict 列表。文件不存在时返回空列表。""" + if not filepath.exists(): + return [] + with open(filepath, "r", newline="", encoding="utf-8") as f: + return list(csv.DictReader(f)) + + +def _write_csv(filepath: Path, fieldnames: list[str], rows: list[dict]) -> None: + """将 dict 列表写入 CSV 文件。""" + with open(filepath, "w", newline="", encoding="utf-8") as f: + writer = csv.DictWriter(f, fieldnames=fieldnames) + writer.writeheader() + writer.writerows(rows) + + +def _init_data() -> None: + """初始化数据目录和默认 CSV 文件（仅首次运行时创建）。 + + orders.csv 以历史订单种子初始化——评测用例依赖 ORD-12345/46/47。 + """ + DATA_DIR.mkdir(parents=True, exist_ok=True) + if not PRODUCTS_CSV.exists(): + _write_csv(PRODUCTS_CSV, _PRODUCT_FIELDS, _DEFAULT_PRODUCTS) + if not CART_CSV.exists(): + _write_csv(CART_CSV, _CART_FIELDS, []) + if not ORDERS_CSV.exists(): + _write_csv(ORDERS_CSV, _ORDER_FIELDS, _DEFAULT_ORDERS) + + +def reset_data() -> None: + """重置所有数据到初始状态（用于评测/测试间清理）。""" + _write_csv(PRODUCTS_CSV, _PRODUCT_FIELDS, _DEFAULT_PRODUCTS) + _write_csv(CART_CSV, _CART_FIELDS, []) + _write_csv(ORDERS_CSV, _ORDER_FIELDS, _DEFAULT_ORDERS) + + +# 模块加载时初始化数据 +_init_data() + + +# ── 工具函数 ────────────────────────────────────────────────── + +def search_products(query: str, category: Optional[str] = None) -> dict: + """搜索商品：按关键词和可选分类过滤。 + + Args: + query: 商品名称或描述的关键词（不区分大小写）。 + category: 可选的分类过滤（electronics, clothing, home）。 + + Returns: + {"query": ..., "category": ..., "total": N, "products": [...]} + """ + products = _read_csv(PRODUCTS_CSV) + query_lower = query.lower() + results = [] + for p in products: + if query_lower in p["name"].lower(): + if category is None or p["category"] == category: + results.append({ + "id": p["id"], + "name": p["name"], + "category": p["category"], + "price": float(p["price"]), + "rating": float(p["rating"]), + "stock": int(p["stock"]), + }) + return {"query": query, "category": category, "total": len(results), "products": results} + + +def get_product_details(product_id: str) -> dict: + """根据商品 ID 获取商品详细信息。 + + Args: + product_id: 商品 ID（如 P001）。 + + Returns: + {"found": True, "product": {...}} 或 {"found": False, "error": "..."} + """ + products = _read_csv(PRODUCTS_CSV) + for p in products: + if p["id"] == product_id: + return {"found": True, "product": { + "id": p["id"], + "name": p["name"], + "category": p["category"], + "price": float(p["price"]), + "rating": float(p["rating"]), + "stock": int(p["stock"]), + "description": p["description"], + "brand": p["brand"], + }} + return {"found": False, "error": f"商品 '{product_id}' 不存在。请检查商品 ID（有效范围: P001-P010）。"} + + +def add_to_cart(product_id: str, quantity: int = 1) -> dict: + """添加商品到购物车，检查商品是否存在及库存是否充足。 + + 同一商品重复添加时累加数量，不会创建重复条目。 + + Args: + product_id: 商品 ID（如 P001）。 + quantity: 添加数量（默认 1）。 + + Returns: + 成功: {"success": True, "message": "...", "item_total": ...} + 失败: {"success": False, "error": "..."} + """ + products = _read_csv(PRODUCTS_CSV) + product = None + for p in products: + if p["id"] == product_id: + product = p + break + + if product is None: + return {"success": False, "error": f"商品 '{product_id}' 不存在。请检查商品 ID。"} + + stock = int(product["stock"]) + if quantity > stock: + return {"success": False, "error": f"库存不足。{product['name']} 仅有 {stock} 件库存。"} + + price = float(product["price"]) + cart = _read_csv(CART_CSV) + + # 购物车已有该商品则累加数量 + for item in cart: + if item["product_id"] == product_id: + new_qty = int(item["quantity"]) + quantity + if new_qty > stock: + return {"success": False, "error": f"库存不足。购物车已有 {item['quantity']} 件，再加 {quantity} 件将超出 {product['name']} 的 {stock} 件库存。"} + item["quantity"] = str(new_qty) + item["subtotal"] = str(round(price * new_qty, 2)) + _write_csv(CART_CSV, _CART_FIELDS, cart) + return { + "success": True, + "message": f"已将 {quantity}x {product['name']} 添加到购物车（购物车现有 {new_qty} 件）。", + "item_total": round(price * quantity, 2), + } + + # 新商品加入购物车 + cart.append({ + "product_id": product_id, + "name": product["name"], + "quantity": str(quantity), + "price": str(price), + "subtotal": str(round(price * quantity, 2)), + }) + _write_csv(CART_CSV, _CART_FIELDS, cart) + + return { + "success": True, + "message": f"已将 {quantity}x {product['name']} 添加到购物车。", + "item_total": round(price * quantity, 2), + } + + +def get_cart() -> dict: + """查看当前购物车内容。 + + Returns: + {"cart": {"items": [...], "total_items": N, "total_price": ..., "currency": "CNY"}} + """ + cart = _read_csv(CART_CSV) + items = [] + total_items = 0 + total_price = 0.0 + for item in cart: + qty = int(item["quantity"]) + subtotal = float(item["subtotal"]) + items.append({ + "product_id": item["product_id"], + "name": item["name"], + "quantity": qty, + "price": float(item["price"]), + "subtotal": subtotal, + }) + total_items += qty + total_price += subtotal + + return { + "cart": { + "items": items, + "total_items": total_items, + "total_price": round(total_price, 2), + "currency": "CNY", + } + } + + +def checkout() -> dict: + """结算购物车，生成订单并清空购物车。 + + 将购物车中所有商品转为订单记录，生成唯一订单号， + 订单状态初始为 "processing"，预计 7 天后送达。 + 结算成功后购物车自动清空。 + + Returns: + 成功: {"success": True, "order_id": "ORD-...", "status": "processing", ...} + 失败: {"success": False, "error": "购物车为空，无法结算。"} + """ + cart = _read_csv(CART_CSV) + if not cart: + return {"success": False, "error": "购物车为空，无法结算。"} + + items = [] + total = 0.0 + for item in cart: + qty = int(item["quantity"]) + price = float(item["price"]) + subtotal = float(item["subtotal"]) + items.append({"name": item["name"], "qty": qty, "price": price}) + total += subtotal + + total = round(total, 2) + order_id = f"ORD-{datetime.now().strftime('%Y%m%d')}-{uuid.uuid4().hex[:4].upper()}" + date = datetime.now().strftime("%Y-%m-%d") + estimated_delivery = (datetime.now() + timedelta(days=7)).strftime("%Y-%m-%d") + + orders = _read_csv(ORDERS_CSV) + orders.append({ + "order_id": order_id, + "status": "processing", + "date": date, + "items": json.dumps(items, ensure_ascii=False), + "total": str(total), + "estimated_delivery": estimated_delivery, + }) + _write_csv(ORDERS_CSV, _ORDER_FIELDS, orders) + + # 结算成功，清空购物车 + _write_csv(CART_CSV, _CART_FIELDS, []) + + return { + "success": True, + "order_id": order_id, + "status": "processing", + "date": date, + "items": items, + "total": total, + "estimated_delivery": estimated_delivery, + "message": f"订单 {order_id} 已生成，总金额 ¥{total}，预计 {estimated_delivery} 送达。", + } + + +def check_order_status(order_id: str) -> dict: + """利用订单号查询订单状态。 + + 支持三种状态：shipped（已发货）、processing（处理中）、delivered（已交付）。 + + Args: + order_id: 订单 ID（如 ORD-20260731-A1B2），注意要用大写。 + + Returns: + 成功: {"found": True, "order_id": ..., "status": ..., ...} + 失败: {"found": False, "error": "..."} + """ + orders = _read_csv(ORDERS_CSV) + for order in orders: + if order["order_id"] == order_id: + return { + "found": True, + "order_id": order_id, + "status": order["status"], + "date": order["date"], + "items": json.loads(order["items"]), + "total": float(order["total"]), + "estimated_delivery": order["estimated_delivery"], + } + return {"found": False, "error": f"订单 '{order_id}' 不存在。请核实订单 ID。"} + + +def apply_coupon(code: str) -> dict: + """应用优惠券折扣。 + + 支持三种优惠券编号： + - SAVE50: 满 ¥200 减 ¥50 + - VIP10: VIP 会员 9 折 + - NEWUSER: 新用户满 ¥100 减 ¥30 + + Args: + code: 优惠券编号（区分大小写，全都用大写）。 + + Returns: + 有效: {"valid": True, "code": ..., "discount": ..., "type": ..., ...} + 无效: {"valid": False, "error": "..."} + """ + coupon = _COUPONS.get(code.upper()) + if coupon: + return {"valid": True, "code": code.upper(), **coupon} + return {"valid": False, "error": f"优惠券 '{code}' 无效或已过期。有效优惠券: SAVE50, VIP10, NEWUSER。"} diff --git a/examples/optimization/eval_optimize_loop/data/_generate_evalsets.py b/examples/optimization/eval_optimize_loop/data/_generate_evalsets.py new file mode 100644 index 000000000..d0a166352 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/_generate_evalsets.py @@ -0,0 +1,501 @@ +#!/usr/bin/env python3 +"""生成评测 evalset JSON 文件。 + +trace/ 子目录: 三件套 trace 数据（demo 模式回放用），每条 case 含 actual_conversation。 +live/ 子目录: 两件套非 trace 数据（real 模式现调 LLM 用），无 actual_conversation。 + +每个 case 包含 eval_id、conversation（期望行为）、session_input，scenario 类型 +由 eval_id 后缀推导（_optimizable/_ineffective/_working/_regression）。 + +所有用户提问与模型回复均为中文。 +""" + +from __future__ import annotations + +import json +import copy +import os +import re + +# ============================================================ +# 辅助函数 +# ============================================================ + + +def _expected_turn_body(user_text: str, tool_name: str, tool_args: dict, + expected_response: str, tool_id: str) -> dict: + """trace 与 live 共享的期望轮次体：user_content + intermediate_data + final_response.""" + return { + "user_content": {"parts": [{"text": user_text}], "role": "user"}, + "intermediate_data": { + "tool_uses": [{"id": tool_id, "name": tool_name, "args": tool_args}] + }, + "final_response": {"parts": [{"text": expected_response}], "role": "model"}, + } + + +def _with_invocation_id(turn: dict, invocation_id: str) -> dict: + """trace 专用：在 intermediate_data 与 final_response 之间插入 invocation_id.""" + return { + "user_content": turn["user_content"], + "intermediate_data": turn["intermediate_data"], + "invocation_id": invocation_id, + "final_response": turn["final_response"], + } + + +def _make_turn(inv_id: str, user_text: str, actual_response: str, expected_response: str, + tool_name: str, tool_args: dict) -> tuple[dict, dict]: + """创建一个评测轮次，返回 (实际轮次, 期望轮次)。""" + base = { + "user_content": {"parts": [{"text": user_text}], "role": "user"}, + "intermediate_data": { + "tool_uses": [ + {"id": inv_id, "name": tool_name, "args": tool_args} + ] + } + } + actual = copy.deepcopy(base) + actual["invocation_id"] = f"{inv_id}-act" + actual["final_response"] = {"parts": [{"text": actual_response}], "role": "model"} + + expected_body = _expected_turn_body(user_text, tool_name, tool_args, expected_response, inv_id) + expected = _with_invocation_id(expected_body, f"{inv_id}-exp") + + return actual, expected + + +def _make_case(eval_id: str, actual_turns: list[dict], expected_turns: list[dict]) -> dict: + """创建一个评测 case。""" + return { + "eval_id": eval_id, + "eval_mode": "trace", + "actual_conversation": actual_turns, + "conversation": expected_turns, + "session_input": {"app_name": "shopping_assistant", "user_id": "user", "state": {}} + } + + +# ============================================================ +# Case 定义：每个 case 包含 (eval_id, 用户问题, 工具名, 工具参数, +# 实际回复（差）, 期望回复（好）) +# +# 场景类型由 eval_id 后缀推导（_optimizable / _ineffective / _working / _regression）， +# 由后续任务负责。 +# ============================================================ + +TRAIN_CASES = [ + # ---- optimizable: 回复过短，可通过 prompt 要求详细输出修复 ---- + ("train_001_optimizable", "帮我搜索无线耳机。", "search_products", + {"query": "wireless headphones"}, + "找到 2 个结果。", + "找到 2 款无线耳机：Sony WH-1000XM6 售价 ¥2,499（评分 4.9，库存 15 件），Wireless Bluetooth Headphones 售价 ¥299（评分 4.5，库存 120 件）。"), + + # ---- ineffective: agent 忽略工具返回，幻觉出不存在的订单状态 ---- + ("train_002_ineffective", "查询我的订单 ORD-99999。", "check_order_status", + {"order_id": "ORD-99999"}, + "您的订单 ORD-99999 正在运输途中，明天送达。", + "订单 ORD-99999 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。"), + + # ---- working: 正常添加购物车 ---- + ("train_003_working", "往我的购物车里加 2 个 iPhone 15 Pro。", "add_to_cart", + {"product_id": "P002", "quantity": 2}, + "已将 2 个 iPhone 15 Pro 加入购物车，合计 ¥15,998.00。", + "已将 2 个 iPhone 15 Pro 加入购物车，合计 ¥15,998.00。"), + + # ---- optimizable: 获取商品详情回复过短 ---- + ("train_004_optimizable", "介绍一下 MacBook Pro 14。", "get_product_details", + {"product_id": "P003"}, + "它是一台笔记本电脑。", + "MacBook Pro 14（P003）苹果出品：M3 Pro 芯片，18GB 内存，512GB 固态硬盘，Liquid Retina XDR 显示屏。售价 ¥14,999，评分 4.7/5，库存 30 件。"), + + # ---- ineffective: 无效优惠券，agent 幻觉成功 ---- + ("train_005_ineffective", "帮我用优惠券 INVALID99。", "apply_coupon", + {"code": "INVALID99"}, + "优惠券 INVALID99 已使用！为您节省 ¥99。", + "优惠券 INVALID99 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。"), + + # ---- working: 查询存在且已发货的订单 ---- + ("train_006_working", "查询订单 ORD-12345 的状态。", "check_order_status", + {"order_id": "ORD-12345"}, + "订单 ORD-12345 已于 2026-07-25 发货，预计 2026-08-02 送达。总金额 ¥7,999。", + "订单 ORD-12345 已于 2026-07-25 发货，预计 2026-08-02 送达。总金额 ¥7,999。"), + + # ---- optimizable: 搜索跑鞋回复过短 ---- + ("train_007_optimizable", "搜索跑步鞋。", "search_products", + {"query": "running shoes"}, + "找到 1 个结果。", + "找到 Men's Running Shoes（P004）耐克出品，售价 ¥599，评分 4.3/5，库存 200 件。轻量网面鞋身，缓震回弹良好。"), + + # ---- optimizable: 搜索咖啡机回复过短 ---- + ("train_008_optimizable", "找找咖啡机。", "search_products", + {"query": "coffee maker"}, + "找到 1 件商品。", + "找到 Coffee Maker（P005）BrewMaster 出品，售价 ¥1,299，评分 4.6/5，库存 80 件。12 杯容量可编程滴漏咖啡机，内置研磨器。"), + + # ---- ineffective: 添加不存在的商品，agent 编造成功 ---- + ("train_009_ineffective", "把商品 P999 加入购物车。", "add_to_cart", + {"product_id": "P999", "quantity": 1}, + "商品 P999 已成功加入购物车。", + "商品 P999 不存在，请检查商品 ID。有效商品：P001 至 P010。"), + + # ---- working: 查看购物车 ---- + ("train_010_working", "我的购物车里有什么？", "get_cart", + {}, + "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。", + "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。"), + + # ---- optimizable: 搜索吸尘器回复过短 ---- + ("train_011_optimizable", "搜索吸尘器。", "search_products", + {"query": "vacuum"}, + "1 个结果。", + "找到 Dyson V15 Vacuum（P009）戴森出品，售价 ¥4,599，评分 4.8/5，库存 25 件。无线手持吸尘器，配备激光探测和 LCD 屏幕。"), + + # ---- ineffective: 查询另一个不存在的订单，agent 幻觉 ---- + ("train_012_ineffective", "我的订单 ORD-88888 到哪了？", "check_order_status", + {"order_id": "ORD-88888"}, + "您的订单 ORD-88888 正在处理中，很快就会发货。", + "订单 ORD-88888 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。"), + + # ---- working: 应用有效优惠券 SAVE50 ---- + ("train_013_working", "给我的订单用优惠券 SAVE50。", "apply_coupon", + {"code": "SAVE50"}, + "优惠券 SAVE50 已使用！订单满 ¥200 立减 ¥50.00。", + "优惠券 SAVE50 已使用！订单满 ¥200 立减 ¥50.00。"), + + # ---- optimizable: 获取三星手机详情过短 ---- + ("train_014_optimizable", "介绍一下 Samsung Galaxy S25。", "get_product_details", + {"product_id": "P007"}, + "它是三星的一款手机。", + "Samsung Galaxy S25（P007）三星出品：Snapdragon 8 Gen 4 芯片，200MP 摄像头，Dynamic AMOLED 2X 显示屏。售价 ¥6,999，评分 4.6/5，库存 60 件。"), + + # ---- optimizable: 获取索尼耳机详情过短 ---- + ("train_015_optimizable", "Sony WH-1000XM6 耳机的详情。", "get_product_details", + {"product_id": "P006"}, + "那些是耳机。", + "Sony WH-1000XM6（P006）索尼出品：业界领先的降噪技术，40 小时续航，支持 LDAC。售价 ¥2,499，评分 4.9/5，库存 15 件。"), + + # ---- ineffective: 超量添加，agent 忽略库存限制 ---- + ("train_016_ineffective", "往购物车加 200 个 MacBook Pro 14。", "add_to_cart", + {"product_id": "P003", "quantity": 200}, + "已将 200 个 MacBook Pro 14 加入购物车。", + "库存不足。MacBook Pro 14 仅有 30 件库存，无法添加 200 件。"), + + # ---- working: 查询已交付订单 ---- + ("train_017_working", "查询我的订单 ORD-12347 状态。", "check_order_status", + {"order_id": "ORD-12347"}, + "订单 ORD-12347 已于 2026-07-26 交付。商品：Men's Running Shoes x1，总金额 ¥599。", + "订单 ORD-12347 已于 2026-07-26 交付。商品：Men's Running Shoes x1，总金额 ¥599。"), + + # ---- optimizable: 获取戴森吸尘器详情过短 ---- + ("train_018_optimizable", "介绍一下 Dyson V15。", "get_product_details", + {"product_id": "P009"}, + "一台吸尘器。", + "Dyson V15 Vacuum（P009）戴森出品：无线手持吸尘器，配备激光探测和 LCD 屏幕。售价 ¥4,599，评分 4.8/5，库存 25 件。"), + + # ---- ineffective: 应用不存在的优惠券（变体），agent 幻觉 ---- + ("train_019_ineffective", "使用促销码 SUPER50。", "apply_coupon", + {"code": "SUPER50"}, + "促销码 SUPER50 已接受！为您打 5 折。", + "优惠券 SUPER50 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。"), + + # ---- optimizable: 搜索牛仔裤回复过短 ---- + ("train_020_optimizable", "搜索牛仔裤。", "search_products", + {"query": "jeans"}, + "找到 1 条。", + "找到 Levi's 501 Jeans（P008）Levi's 出品，售价 ¥499，评分 4.4/5，库存 300 件。原版剪裁，非弹力牛仔布，直筒裤型。"), +] + +# 验证集 cases：与训练集类似但使用不同的 eval_id 和查询 +VAL_CASES = [ + # ---- optimizable: 获取商品详情回复过短，可通过 prompt 修复 ---- + ("val_001_optimizable", "介绍一下 MacBook Pro 14。", "get_product_details", + {"product_id": "P003"}, + "它是一台笔记本电脑。", + "MacBook Pro 14（P003）苹果出品：M3 Pro 芯片，18GB 内存，512GB 固态硬盘，Liquid Retina XDR 显示屏。售价 ¥14,999，评分 4.7/5，库存 30 件。"), + + # ---- ineffective: 无效优惠券幻觉，prompt 无法修复 ---- + ("val_002_ineffective", "帮我用优惠券 INVALID99。", "apply_coupon", + {"code": "INVALID99"}, + "优惠券 INVALID99 已使用！为您节省 ¥99。", + "优惠券 INVALID99 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。"), + + # ---- regression risk: 当前 PASSED，优化后可能退化 ---- + ("val_003_regression", "我的购物车里有什么？", "get_cart", + {}, + "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。", + "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。"), + + # ---- optimizable: 搜索耳机回复过短 ---- + ("val_004_optimizable", "帮我找找无线耳机。", "search_products", + {"query": "wireless headphones"}, + "这是结果。", + "找到 2 款无线耳机：Sony WH-1000XM6 售价 ¥2,499（评分 4.9，库存 15 件），Wireless Bluetooth Headphones 售价 ¥299（评分 4.5，库存 120 件）。"), + + # ---- regression risk: 当前 PASSED，优化后可能出错 ---- + ("val_005_regression", "我的订单 ORD-12346 到哪了？", "check_order_status", + {"order_id": "ORD-12346"}, + "订单 ORD-12346 正在处理中。商品：Coffee Maker x1、Air Fryer XL x1，总金额 ¥2,198，预计 2026-08-05 送达。", + "订单 ORD-12346 正在处理中。商品：Coffee Maker x1、Air Fryer XL x1，总金额 ¥2,198，预计 2026-08-05 送达。"), + + # ---- optimizable: 获取 iPhone 详情过短 ---- + ("val_006_optimizable", "给我看看 iPhone 15 Pro 的详情。", "get_product_details", + {"product_id": "P002"}, + "iPhone。", + "iPhone 15 Pro（P002）苹果出品：A17 Pro 芯片，48MP 摄像头，钛金属设计，支持 USB-C。售价 ¥7,999，评分 4.8/5，库存 50 件。"), + + # ---- ineffective: 查询不存在的订单，agent 幻觉 ---- + ("val_007_ineffective", "帮我追踪订单 ORD-77777。", "check_order_status", + {"order_id": "ORD-77777"}, + "您的订单 ORD-77777 今天正在派送中。", + "订单 ORD-77777 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。"), + + # ---- optimizable: 搜索空气炸锅回复过短 ---- + ("val_008_optimizable", "找找空气炸锅。", "search_products", + {"query": "air fryer"}, + "找到了。", + "找到 Air Fryer XL（P010）KitchenPro 出品，售价 ¥899，评分 4.2/5，库存 100 件。7L 容量，12 种烹饪预设，快速热风循环。"), + + # ---- regression risk: 当前 PASSED，优化后可能画蛇添足 ---- + ("val_009_regression", "往我的购物车加一双跑步鞋。", "add_to_cart", + {"product_id": "P004", "quantity": 1}, + "已将 1 双 Men's Running Shoes 加入购物车，合计 ¥599.00。", + "已将 1 双 Men's Running Shoes 加入购物车，合计 ¥599.00。"), + + # ---- optimizable: 获取 Levi's 牛仔裤详情过短 ---- + ("val_010_optimizable", "介绍一下 Levi's 501 Jeans。", "get_product_details", + {"product_id": "P008"}, + "Levi's 的牛仔裤。", + "Levi's 501 Jeans（P008）Levi's 出品：原版剪裁，非弹力牛仔布，直筒裤型。售价 ¥499，评分 4.4/5，库存 300 件。"), + + # ---- ineffective: 无效优惠券（变体），agent 幻觉 ---- + ("val_011_ineffective", "兑换优惠券 FREEDELIVERY。", "apply_coupon", + {"code": "FREEDELIVERY"}, + "FREEDELIVERY 优惠券已激活！您的订单免运费。", + "优惠券 FREEDELIVERY 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。"), + + # ---- optimizable: 搜索电子产品回复过短 ---- + ("val_012_optimizable", "给我看看 ¥1000 以下的电子产品。", "search_products", + {"query": "electronics", "category": "electronics"}, + "没有结果。", + "找到 Wireless Bluetooth Headphones（P001）售价 ¥299，评分 4.5/5，库存 120 件。其他电子产品起价 ¥2,499 以上。"), + + # ---- regression risk: 当前 PASSED，优化后委托可能变化 ---- + ("val_013_regression", "使用 VIP10 优惠券。", "apply_coupon", + {"code": "VIP10"}, + "优惠券 VIP10 已使用！已为您的订单应用 VIP 会员 9 折优惠。", + "优惠券 VIP10 已使用！已为您的订单应用 VIP 会员 9 折优惠。"), + + # ---- optimizable: 获取咖啡机详情过短 ---- + ("val_014_optimizable", "给我 Coffee Maker 的详细信息。", "get_product_details", + {"product_id": "P005"}, + "一台咖啡机。", + "Coffee Maker（P005）BrewMaster 出品：可编程滴漏咖啡机，12 杯容量，内置研磨器。售价 ¥1,299，评分 4.6/5，库存 80 件。"), + + # ---- ineffective: 添加不存在商品，agent 编造 ---- + ("val_015_ineffective", "把商品 P888 加入购物车。", "add_to_cart", + {"product_id": "P888", "quantity": 1}, + "P888 已加入购物车。", + "商品 P888 不存在，请检查商品 ID。有效商品：P001 至 P010。"), + + # ---- optimizable: 获取索尼耳机详情过短 ---- + ("val_016_optimizable", "跟我说说 Sony WH-1000XM6。", "get_product_details", + {"product_id": "P006"}, + "耳机。", + "Sony WH-1000XM6（P006）索尼出品：业界领先的降噪技术，40 小时续航，支持 LDAC。售价 ¥2,499，评分 4.9/5，库存 15 件。"), + + # ---- optimizable: 搜索+详情两步操作，回复过短 ---- + ("val_017_optimizable", "我需要一部新手机，你们有什么？", "search_products", + {"query": "phone"}, + "有 2 款手机可选。", + "找到 2 款手机：iPhone 15 Pro（P002）售价 ¥7,999（评分 4.8），Samsung Galaxy S25（P007）售价 ¥6,999（评分 4.6）。两款都很不错，各有特色。"), + + # ---- ineffective: 查询不存在的订单（变体），agent 幻觉 ---- + ("val_018_ineffective", "查询订单 ORD-00001 的状态。", "check_order_status", + {"order_id": "ORD-00001"}, + "ORD-00001 等待发货中。", + "订单 ORD-00001 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。"), + + # ---- regression risk: 当前 PASSED，优化后优惠券可能被错误处理 ---- + ("val_019_regression", "使用我的 NEWUSER 优惠券。", "apply_coupon", + {"code": "NEWUSER"}, + "优惠券 NEWUSER 已使用！新用户满 ¥100 立减 ¥30。", + "优惠券 NEWUSER 已使用！新用户满 ¥100 立减 ¥30。"), + + # ---- optimizable: 获取 Air Fryer 详情过短 ---- + ("val_020_optimizable", "介绍一下 Air Fryer XL。", "get_product_details", + {"product_id": "P010"}, + "一件厨房电器。", + "Air Fryer XL（P010）KitchenPro 出品：7L 容量，12 种烹饪预设，快速热风循环。售价 ¥899，评分 4.2/5，库存 100 件。"), +] + + +def generate_evalset(eval_set_id: str, name: str, description: str, cases: list) -> dict: + """根据 case 定义列表生成 evalset JSON 结构。""" + eval_cases = [] + for case_def in cases: + eval_id, user_text, tool_name, tool_args, actual_resp, expected_resp = case_def[:6] + + actual_turn, expected_turn = _make_turn( + inv_id=eval_id.replace("train_", "t").replace("val_", "v"), + user_text=user_text, + actual_response=actual_resp, + expected_response=expected_resp, + tool_name=tool_name, + tool_args=tool_args, + ) + + eval_cases.append(_make_case( + eval_id=eval_id, + actual_turns=[actual_turn], + expected_turns=[expected_turn], + )) + + return { + "eval_set_id": eval_set_id, + "name": name, + "description": description, + "eval_cases": eval_cases, + } + + +def generate_optimized_evalset(val_cases: list) -> dict: + """生成优化后的验证集 evalset。 + + 规则： + - optimizable cases: actual_conversation 的 final_response 改为与 conversation 一致（表示优化成功） + - ineffective cases: 保持不变（优化无法修复） + - regression cases: actual_conversation 的 final_response 改为退化版本（表示过拟合） + """ + # 退化响应映射：eval_id → 退化后的实际响应 + regression_responses = { + "val_003_regression": "让我先搜索一些商品加到您的购物车。您想买点什么？", + "val_005_regression": "订单 ORD-12346 已发货，即将送达。", + "val_009_regression": "跑步鞋很不错！您还需要袜子或运动服吗？", + "val_013_regression": "VIP10 已应用。您的忠诚度积分增加了 10%。", + "val_019_regression": "NEWUSER 已应用。不过您可以试试 SAVE50，折扣更优惠。", + } + + eval_cases = [] + for case_def in val_cases: + eval_id, user_text, tool_name, tool_args, base_actual, expected_resp = case_def[:6] + if "_optimizable" in eval_id: + scenario = "optimizable_success" + elif "_ineffective" in eval_id: + scenario = "optimization_ineffective" + elif "_regression" in eval_id: + scenario = "optimization_regression" + else: + raise ValueError(f"Unknown scenario suffix in eval_id: {eval_id}") + + actual_turn, expected_turn = _make_turn( + inv_id=eval_id.replace("val_", "o"), + user_text=user_text, + actual_response=base_actual, + expected_response=expected_resp, + tool_name=tool_name, + tool_args=tool_args, + ) + + # 根据场景类型调整实际响应 + if scenario == "optimizable_success": + # 优化成功：actual 改为 expected（PASS） + actual_turn["final_response"] = {"parts": [{"text": expected_resp}], "role": "model"} + elif scenario == "optimization_regression": + # 过拟合退化：actual 改为退化版本（FAILED） + actual_turn["final_response"] = {"parts": [{"text": regression_responses[eval_id]}], "role": "model"} + # ineffective: 保持不变 + + eval_cases.append(_make_case( + eval_id=eval_id, + actual_turns=[actual_turn], + expected_turns=[expected_turn], + )) + + return { + "eval_set_id": "shopping_assistant_val_optimized", + "name": "购物助手验证集（优化后）", + "description": "20 个验证用例，反映提示词优化后的 agent 行为：可优化用例已通过，不可优化用例保持不变，回归用例体现过拟合。", + "eval_cases": eval_cases, + } + + +def _regex_facts(expected: str) -> str: + """把期望句子转成"关键事实 regex"模式（live 评测用）。 + + 按中文标点把期望句子拆成关键事实片段，段间用 [\\s\\S]* 连接， + 允许模型自由措辞，但关键事实必须按序出现（与 final_response_avg_score + 的 regex 判据配套）。评分里的 "/5" 属非必要事实，先去掉。 + """ + normalized = expected.replace("/5", "") + segments = [s.strip() for s in re.split(r"[，。：；！]", normalized) if s.strip()] + return r"[\s\S]*".join(re.escape(s) for s in segments) + + +def generate_live_evalset(eval_set_id: str, name: str, description: str, cases: list) -> dict: + """生成 live 数据集: 无 actual_conversation, 仅期望 + session_input.""" + eval_cases = [] + for case_def in cases: + eval_id, user_text, tool_name, tool_args, _actual, expected_resp = case_def[:6] + expected_turn = _expected_turn_body( + user_text=user_text, + tool_name=tool_name, + tool_args=tool_args, + expected_response=_regex_facts(expected_resp), + tool_id=eval_id.replace("train_", "lt").replace("val_", "lv"), + ) + eval_cases.append({ + "eval_id": eval_id, + "eval_mode": "non-trace", + "conversation": [expected_turn], + "session_input": {"app_name": "shopping_assistant", "user_id": "user", "state": {}}, + }) + return { + "eval_set_id": eval_set_id, + "name": name, + "description": description, + "eval_cases": eval_cases, + } + + +def main(): + data_dir = os.path.dirname(os.path.abspath(__file__)) + + # ---- trace 三件套 ---- + trace_dir = os.path.join(data_dir, "trace") + os.makedirs(trace_dir, exist_ok=True) + + train = generate_evalset("shopping_assistant_train", "训练集", "20 个训练用例。", TRAIN_CASES) + val_base = generate_evalset("shopping_assistant_val", "验证集基线", "20 个验证用例。", VAL_CASES) + val_opt = generate_optimized_evalset(VAL_CASES) + + for name, payload in [("train.evalset.json", train), ("val_baseline.evalset.json", val_base), ("val_optimized.evalset.json", val_opt)]: + p = os.path.join(trace_dir, name) + with open(p, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, ensure_ascii=False) + print(f"Generated {p} ({len(payload['eval_cases'])} cases)") + + # ---- live 两件套（不同路径! 否则 AgentOptimizer 抛 ValueError） ---- + live_dir = os.path.join(data_dir, "live") + os.makedirs(live_dir, exist_ok=True) + + train_live = generate_live_evalset("shopping_assistant_train_live", "训练集（live 模式）", "20 个训练用例（live 模式）。", TRAIN_CASES) + val_live = generate_live_evalset("shopping_assistant_val_live", "验证集（live 模式）", "20 个验证用例（live 模式）。", VAL_CASES) + + train_live_path = os.path.join(live_dir, "train.evalset.json") + val_live_path = os.path.join(live_dir, "val.evalset.json") + for path, payload in [(train_live_path, train_live), (val_live_path, val_live)]: + with open(path, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, ensure_ascii=False) + print(f"Generated {path} ({len(payload['eval_cases'])} cases)") + + # ---- 删除旧的扁平文件 ---- + for stale in ["train_baseline.evalset.json", "val_baseline.evalset.json", "val_optimized.evalset.json"]: + stale_path = os.path.join(data_dir, stale) + if os.path.exists(stale_path): + os.remove(stale_path) + print(f"Removed {stale_path}") + + +if __name__ == "__main__": + main() diff --git a/examples/optimization/eval_optimize_loop/data/demo_optimize_result.json b/examples/optimization/eval_optimize_loop/data/demo_optimize_result.json new file mode 100644 index 000000000..6669b974c --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/demo_optimize_result.json @@ -0,0 +1,161 @@ +{ + "schemaVersion": "v1", + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finishReason": "completed", + "stopReason": "required_metrics_passing", + "errorMessage": "", + "baselinePassRate": 0.25, + "bestPassRate": 0.75, + "passRateImprovement": 0.5, + "baselineMetricBreakdown": { + "final_response_avg_score": 0.38, + "tool_trajectory_avg_score": 1.0 + }, + "bestMetricBreakdown": { + "final_response_avg_score": 0.79, + "tool_trajectory_avg_score": 1.0 + }, + "metricThresholds": { + "final_response_avg_score": 0.6, + "tool_trajectory_avg_score": 0.75 + }, + "perMetricBestCandidates": {}, + "baselinePrompts": { + "system_prompt": "你是电商购物助手，用中文与用户交流。你的职责是帮助用户搜索商品、查看商品详情、管理购物车、结算下单、查询订单和套用优惠券。\n\n你可以使用以下 7 个工具：\n\n1. `search_products(query, category)` — 按关键词搜索商品，可选按分类过滤（electronics/clothing/home）。\n2. `get_product_details(product_id)` — 获取商品详情（价格、评分、库存、描述、品牌）。\n3. `add_to_cart(product_id, quantity)` — 把商品加入购物车，默认数量 1。\n4. `get_cart()` — 查看当前购物车内容（商品、数量、总价）。\n5. `checkout()` — 结算购物车并生成订单号，结算成功后购物车自动清空。\n6. `check_order_status(order_id)` — 用订单号查询订单状态（processing/shipped/delivered）。\n7. `apply_coupon(code)` — 校验并应用优惠券（SAVE50/VIP10/NEWUSER）。\n\n典型购物流程：\n\n- 用户想买商品 → 先用 `search_products` 搜索，再视情况用 `get_product_details` 查看详情。\n- 用户要加入购物车 → 调用 `add_to_cart`，并告诉用户已添加的数量和金额。\n- 用户要查看购物车 → 调用 `get_cart`，逐项列出商品、数量和总价。\n- 用户要结算/支付/下单 → 调用 `checkout`，把生成的订单号告诉用户。\n- 用户要查订单 → 调用 `check_order_status`，需要用户提供订单号。\n- 用户要用优惠券 → 调用 `apply_coupon` 校验后再确认折扣。\n\n回答规范：\n\n- 必须基于工具的**真实返回结果**回答，绝不允许编造或猜测不存在的商品、订单、库存或优惠券。\n- 工具返回失败（商品不存在、库存不足、订单不存在、优惠券无效）时，如实转述失败原因，并给出可行的替代建议。\n- 回复要完整具体：包含商品名称、价格、数量、评分、库存等关键信息，不要只回复\"找到 2 个结果\"这类过短的回答。\n- 金额使用人民币（¥）表示。\n" + }, + "bestPrompts": { + "system_prompt": "你是电商购物助手，用中文与用户交流。你必须基于工具的真实返回结果回答，绝不编造不存在的商品、订单、库存或优惠券。回复要完整具体：包含商品名称、价格、数量、评分、库存等关键信息，不要只回复\"找到 2 个结果\"这类过短回答。典型流程：搜索(search_products) → 详情(get_product_details) → 加购(add_to_cart) → 确认(get_cart) → 结算(checkout 生成订单号) → 查询订单(check_order_status)；优惠券(apply_coupon)先校验再确认。" + }, + "totalRounds": 3, + "rounds": [ + { + "round": 1, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": { + "system_prompt": "你是电商购物助手，用中文与用户交流。你必须基于工具的真实返回结果回答，绝不编造不存在的商品、订单、库存或优惠券。回复要完整具体：包含商品名称、价格、数量、评分、库存等关键信息，不要只回复\"找到 2 个结果\"这类过短回答。典型流程：搜索(search_products) → 详情(get_product_details) → 加购(add_to_cart) → 确认(get_cart) → 结算(checkout 生成订单号) → 查询订单(check_order_status)；优惠券(apply_coupon)先校验再确认。" + }, + "trainPassRate": 0.0, + "validationPassRate": 0.3, + "metricBreakdown": { + "final_response_avg_score": 0.42, + "tool_trajectory_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "prompt 更明确要求详细回答，多数 optimizable 用例通过", + "failedCaseIds": [ + "train_001_optimizable" + ], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 0, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-31T07:00:00+00:00", + "durationSeconds": 38.5, + "kind": "reflective", + "trainMinibatchSize": 0, + "trainSubsampleParentScore": null, + "trainSubsampleCandidateScore": null, + "skipReason": null, + "errorMessage": null, + "budgetUsed": null, + "budgetTotal": null, + "extras": {} + }, + { + "round": 2, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": { + "system_prompt": "你是电商购物助手，用中文与用户交流。你必须基于工具的真实返回结果回答，绝不编造不存在的商品、订单、库存或优惠券。回复要完整具体：包含商品名称、价格、数量、评分、库存等关键信息，不要只回复\"找到 2 个结果\"这类过短回答。典型流程：搜索(search_products) → 详情(get_product_details) → 加购(add_to_cart) → 确认(get_cart) → 结算(checkout 生成订单号) → 查询订单(check_order_status)；优惠券(apply_coupon)先校验再确认。" + }, + "trainPassRate": 0.0, + "validationPassRate": 0.55, + "metricBreakdown": { + "final_response_avg_score": 0.61, + "tool_trajectory_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "进一步强调工具数据真实性，减少幻觉", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 0, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-31T07:00:00+00:00", + "durationSeconds": 38.5, + "kind": "reflective", + "trainMinibatchSize": 0, + "trainSubsampleParentScore": null, + "trainSubsampleCandidateScore": null, + "skipReason": null, + "errorMessage": null, + "budgetUsed": null, + "budgetTotal": null, + "extras": {} + }, + { + "round": 3, + "optimizedFieldNames": [ + "system_prompt" + ], + "candidatePrompts": { + "system_prompt": "你是电商购物助手，用中文与用户交流。你必须基于工具的真实返回结果回答，绝不编造不存在的商品、订单、库存或优惠券。回复要完整具体：包含商品名称、价格、数量、评分、库存等关键信息，不要只回复\"找到 2 个结果\"这类过短回答。典型流程：搜索(search_products) → 详情(get_product_details) → 加购(add_to_cart) → 确认(get_cart) → 结算(checkout 生成订单号) → 查询订单(check_order_status)；优惠券(apply_coupon)先校验再确认。" + }, + "trainPassRate": 0.0, + "validationPassRate": 0.75, + "metricBreakdown": { + "final_response_avg_score": 0.79, + "tool_trajectory_avg_score": 1.0 + }, + "accepted": true, + "acceptanceReason": "要求完整呈现价格/库存等细节，全部 optimizable 用例通过", + "failedCaseIds": [], + "failedCasesTruncated": 0, + "perFieldDiagnosis": {}, + "reflectionLmCalls": 0, + "roundLlmCost": 0.0, + "roundTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "startedAt": "2026-07-31T07:00:00+00:00", + "durationSeconds": 38.5, + "kind": "reflective", + "trainMinibatchSize": 0, + "trainSubsampleParentScore": null, + "trainSubsampleCandidateScore": null, + "skipReason": null, + "errorMessage": null, + "budgetUsed": null, + "budgetTotal": null, + "extras": {} + } + ], + "totalReflectionLmCalls": 6, + "totalJudgeModelCalls": 0, + "totalLlmCost": 0.0, + "totalTokenUsage": { + "prompt": 0, + "completion": 0, + "total": 0 + }, + "durationSeconds": 120.0, + "startedAt": "2026-07-31T07:00:00+00:00", + "finishedAt": "2026-07-31T07:00:01+00:00", + "extras": {} +} \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/data/gate_metrics.json b/examples/optimization/eval_optimize_loop/data/gate_metrics.json new file mode 100644 index 000000000..e02a416f6 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/gate_metrics.json @@ -0,0 +1,79 @@ +{ + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 0.6, + "criterion": { + "final_response": { + "text": { + "match": "regex", + "case_insensitive": true + } + } + } + }, + { + "metric_name": "llm_rubric_response", + "threshold": 0.6, + "criterion": { + "llmJudge": { + "judge_model": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "api_key": "${TRPC_AGENT_API_KEY}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "generation_config": { + "max_tokens": 2048, + "temperature": 0 + } + }, + "rubrics": [ + { + "id": "fact_accuracy", + "description": "事实准确性", + "type": "FACT_ACCURACY", + "content": { + "text": "回答中提到的商品名称、价格、评分、库存数量、订单号等事实信息必须与工具返回结果一致，不得编造或臆测。只要出现一处关键事实错误或凭空捏造，本项判否。" + } + }, + { + "id": "completeness", + "description": "完整性", + "type": "COMPLETENESS", + "content": { + "text": "回答应覆盖任务要求的关键信息：商品搜索或推荐场景需包含商品名称与价格（若工具提供，还应包含评分、库存）；详情、购物车、结算、订单查询等场景需包含对应关键字段（价格、数量、评分、库存、订单号、状态等）。仅回复「已找到 N 个结果」这类缺少关键信息的过短回答，本项判否。" + } + }, + { + "id": "failure_handling", + "description": "失败处理", + "type": "FAILURE_HANDLING", + "content": { + "text": "若工具调用失败或结果为空（商品不存在、库存不足、订单不存在、优惠券无效等），回答必须如实转述失败原因，并给出可行的替代建议。若任务实际成功完成，本项直接判是；若回答隐瞒失败或编造成功结果，本项判否。" + } + } + ] + } + } + }, + { + "metric_name": "tool_trajectory_avg_score", + "threshold": 0.75, + "criterion": { + "tool_trajectory": { + "default": { + "name": { + "match": "exact", + "case_insensitive": false + }, + "arguments": { + "match": "exact" + } + }, + "order_sensitive": true, + "subset_matching": false + } + } + } + ], + "num_runs": 2 +} \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/data/live/train.evalset.json b/examples/optimization/eval_optimize_loop/data/live/train.evalset.json new file mode 100644 index 000000000..40f9177b1 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/live/train.evalset.json @@ -0,0 +1,808 @@ +{ + "eval_set_id": "shopping_assistant_train_live", + "name": "训练集（live 模式）", + "description": "20 个训练用例（live 模式）。", + "eval_cases": [ + { + "eval_id": "train_001_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我搜索无线耳机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt001_optimizable", + "name": "search_products", + "args": { + "query": "wireless headphones" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ 2\\ 款无线耳机[\\s\\S]*Sony\\ WH\\-1000XM6\\ 售价\\ ¥2,499（评分\\ 4\\.9[\\s\\S]*库存\\ 15\\ 件）[\\s\\S]*Wireless\\ Bluetooth\\ Headphones\\ 售价\\ ¥299（评分\\ 4\\.5[\\s\\S]*库存\\ 120\\ 件）" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_002_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询我的订单 ORD-99999。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt002_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-99999" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "订单\\ ORD\\-99999\\ 不存在[\\s\\S]*请核实订单\\ ID[\\s\\S]*有效订单[\\s\\S]*ORD\\-12345、ORD\\-12346、ORD\\-12347" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_003_working", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往我的购物车里加 2 个 iPhone 15 Pro。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt003_working", + "name": "add_to_cart", + "args": { + "product_id": "P002", + "quantity": 2 + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "已将\\ 2\\ 个\\ iPhone\\ 15\\ Pro\\ 加入购物车[\\s\\S]*合计\\ ¥15,998\\.00" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_004_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt004_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P003" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "MacBook\\ Pro\\ 14（P003）苹果出品[\\s\\S]*M3\\ Pro\\ 芯片[\\s\\S]*18GB\\ 内存[\\s\\S]*512GB\\ 固态硬盘[\\s\\S]*Liquid\\ Retina\\ XDR\\ 显示屏[\\s\\S]*售价\\ ¥14,999[\\s\\S]*评分\\ 4\\.7[\\s\\S]*库存\\ 30\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_005_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我用优惠券 INVALID99。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt005_ineffective", + "name": "apply_coupon", + "args": { + "code": "INVALID99" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "优惠券\\ INVALID99\\ 无效或已过期[\\s\\S]*有效优惠券[\\s\\S]*SAVE50（满\\ ¥200\\ 减\\ ¥50）、VIP10（9\\ 折）、NEWUSER（满\\ ¥100\\ 减\\ ¥30）" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_006_working", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询订单 ORD-12345 的状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt006_working", + "name": "check_order_status", + "args": { + "order_id": "ORD-12345" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "订单\\ ORD\\-12345\\ 已于\\ 2026\\-07\\-25\\ 发货[\\s\\S]*预计\\ 2026\\-08\\-02\\ 送达[\\s\\S]*总金额\\ ¥7,999" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_007_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索跑步鞋。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt007_optimizable", + "name": "search_products", + "args": { + "query": "running shoes" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ Men's\\ Running\\ Shoes（P004）耐克出品[\\s\\S]*售价\\ ¥599[\\s\\S]*评分\\ 4\\.3[\\s\\S]*库存\\ 200\\ 件[\\s\\S]*轻量网面鞋身[\\s\\S]*缓震回弹良好" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_008_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "找找咖啡机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt008_optimizable", + "name": "search_products", + "args": { + "query": "coffee maker" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ Coffee\\ Maker（P005）BrewMaster\\ 出品[\\s\\S]*售价\\ ¥1,299[\\s\\S]*评分\\ 4\\.6[\\s\\S]*库存\\ 80\\ 件[\\s\\S]*12\\ 杯容量可编程滴漏咖啡机[\\s\\S]*内置研磨器" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_009_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "把商品 P999 加入购物车。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt009_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P999", + "quantity": 1 + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "商品\\ P999\\ 不存在[\\s\\S]*请检查商品\\ ID[\\s\\S]*有效商品[\\s\\S]*P001\\ 至\\ P010" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_010_working", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的购物车里有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt010_working", + "name": "get_cart", + "args": {} + } + ] + }, + "final_response": { + "parts": [ + { + "text": "您的购物车有\\ 3\\ 件商品[\\s\\S]*iPhone\\ 15\\ Pro\\ x1（¥7,999）、Men's\\ Running\\ Shoes\\ x2（¥1,198）[\\s\\S]*合计\\ ¥9,197\\ CNY" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_011_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索吸尘器。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt011_optimizable", + "name": "search_products", + "args": { + "query": "vacuum" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ Dyson\\ V15\\ Vacuum（P009）戴森出品[\\s\\S]*售价\\ ¥4,599[\\s\\S]*评分\\ 4\\.8[\\s\\S]*库存\\ 25\\ 件[\\s\\S]*无线手持吸尘器[\\s\\S]*配备激光探测和\\ LCD\\ 屏幕" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_012_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的订单 ORD-88888 到哪了？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt012_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-88888" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "订单\\ ORD\\-88888\\ 不存在[\\s\\S]*请核实订单\\ ID[\\s\\S]*有效订单[\\s\\S]*ORD\\-12345、ORD\\-12346、ORD\\-12347" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_013_working", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我的订单用优惠券 SAVE50。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt013_working", + "name": "apply_coupon", + "args": { + "code": "SAVE50" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "优惠券\\ SAVE50\\ 已使用[\\s\\S]*订单满\\ ¥200\\ 立减\\ ¥50\\.00" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_014_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Samsung Galaxy S25。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt014_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P007" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "Samsung\\ Galaxy\\ S25（P007）三星出品[\\s\\S]*Snapdragon\\ 8\\ Gen\\ 4\\ 芯片[\\s\\S]*200MP\\ 摄像头[\\s\\S]*Dynamic\\ AMOLED\\ 2X\\ 显示屏[\\s\\S]*售价\\ ¥6,999[\\s\\S]*评分\\ 4\\.6[\\s\\S]*库存\\ 60\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_015_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "Sony WH-1000XM6 耳机的详情。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt015_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P006" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "Sony\\ WH\\-1000XM6（P006）索尼出品[\\s\\S]*业界领先的降噪技术[\\s\\S]*40\\ 小时续航[\\s\\S]*支持\\ LDAC[\\s\\S]*售价\\ ¥2,499[\\s\\S]*评分\\ 4\\.9[\\s\\S]*库存\\ 15\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_016_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往购物车加 200 个 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt016_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P003", + "quantity": 200 + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "库存不足[\\s\\S]*MacBook\\ Pro\\ 14\\ 仅有\\ 30\\ 件库存[\\s\\S]*无法添加\\ 200\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_017_working", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询我的订单 ORD-12347 状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt017_working", + "name": "check_order_status", + "args": { + "order_id": "ORD-12347" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "订单\\ ORD\\-12347\\ 已于\\ 2026\\-07\\-26\\ 交付[\\s\\S]*商品[\\s\\S]*Men's\\ Running\\ Shoes\\ x1[\\s\\S]*总金额\\ ¥599" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_018_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Dyson V15。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt018_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P009" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "Dyson\\ V15\\ Vacuum（P009）戴森出品[\\s\\S]*无线手持吸尘器[\\s\\S]*配备激光探测和\\ LCD\\ 屏幕[\\s\\S]*售价\\ ¥4,599[\\s\\S]*评分\\ 4\\.8[\\s\\S]*库存\\ 25\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_019_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用促销码 SUPER50。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt019_ineffective", + "name": "apply_coupon", + "args": { + "code": "SUPER50" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "优惠券\\ SUPER50\\ 无效或已过期[\\s\\S]*有效优惠券[\\s\\S]*SAVE50（满\\ ¥200\\ 减\\ ¥50）、VIP10（9\\ 折）、NEWUSER（满\\ ¥100\\ 减\\ ¥30）" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_020_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索牛仔裤。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lt020_optimizable", + "name": "search_products", + "args": { + "query": "jeans" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ Levi's\\ 501\\ Jeans（P008）Levi's\\ 出品[\\s\\S]*售价\\ ¥499[\\s\\S]*评分\\ 4\\.4[\\s\\S]*库存\\ 300\\ 件[\\s\\S]*原版剪裁[\\s\\S]*非弹力牛仔布[\\s\\S]*直筒裤型" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + } + ] +} \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/data/live/val.evalset.json b/examples/optimization/eval_optimize_loop/data/live/val.evalset.json new file mode 100644 index 000000000..2c81f88d4 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/live/val.evalset.json @@ -0,0 +1,808 @@ +{ + "eval_set_id": "shopping_assistant_val_live", + "name": "验证集（live 模式）", + "description": "20 个验证用例（live 模式）。", + "eval_cases": [ + { + "eval_id": "val_001_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv001_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P003" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "MacBook\\ Pro\\ 14（P003）苹果出品[\\s\\S]*M3\\ Pro\\ 芯片[\\s\\S]*18GB\\ 内存[\\s\\S]*512GB\\ 固态硬盘[\\s\\S]*Liquid\\ Retina\\ XDR\\ 显示屏[\\s\\S]*售价\\ ¥14,999[\\s\\S]*评分\\ 4\\.7[\\s\\S]*库存\\ 30\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_002_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我用优惠券 INVALID99。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv002_ineffective", + "name": "apply_coupon", + "args": { + "code": "INVALID99" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "优惠券\\ INVALID99\\ 无效或已过期[\\s\\S]*有效优惠券[\\s\\S]*SAVE50（满\\ ¥200\\ 减\\ ¥50）、VIP10（9\\ 折）、NEWUSER（满\\ ¥100\\ 减\\ ¥30）" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_003_regression", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的购物车里有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv003_regression", + "name": "get_cart", + "args": {} + } + ] + }, + "final_response": { + "parts": [ + { + "text": "您的购物车有\\ 3\\ 件商品[\\s\\S]*iPhone\\ 15\\ Pro\\ x1（¥7,999）、Men's\\ Running\\ Shoes\\ x2（¥1,198）[\\s\\S]*合计\\ ¥9,197\\ CNY" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_004_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我找找无线耳机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv004_optimizable", + "name": "search_products", + "args": { + "query": "wireless headphones" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ 2\\ 款无线耳机[\\s\\S]*Sony\\ WH\\-1000XM6\\ 售价\\ ¥2,499（评分\\ 4\\.9[\\s\\S]*库存\\ 15\\ 件）[\\s\\S]*Wireless\\ Bluetooth\\ Headphones\\ 售价\\ ¥299（评分\\ 4\\.5[\\s\\S]*库存\\ 120\\ 件）" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_005_regression", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的订单 ORD-12346 到哪了？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv005_regression", + "name": "check_order_status", + "args": { + "order_id": "ORD-12346" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "订单\\ ORD\\-12346\\ 正在处理中[\\s\\S]*商品[\\s\\S]*Coffee\\ Maker\\ x1、Air\\ Fryer\\ XL\\ x1[\\s\\S]*总金额\\ ¥2,198[\\s\\S]*预计\\ 2026\\-08\\-05\\ 送达" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_006_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 iPhone 15 Pro 的详情。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv006_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P002" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "iPhone\\ 15\\ Pro（P002）苹果出品[\\s\\S]*A17\\ Pro\\ 芯片[\\s\\S]*48MP\\ 摄像头[\\s\\S]*钛金属设计[\\s\\S]*支持\\ USB\\-C[\\s\\S]*售价\\ ¥7,999[\\s\\S]*评分\\ 4\\.8[\\s\\S]*库存\\ 50\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_007_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我追踪订单 ORD-77777。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv007_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-77777" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "订单\\ ORD\\-77777\\ 不存在[\\s\\S]*请核实订单\\ ID[\\s\\S]*有效订单[\\s\\S]*ORD\\-12345、ORD\\-12346、ORD\\-12347" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_008_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "找找空气炸锅。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv008_optimizable", + "name": "search_products", + "args": { + "query": "air fryer" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ Air\\ Fryer\\ XL（P010）KitchenPro\\ 出品[\\s\\S]*售价\\ ¥899[\\s\\S]*评分\\ 4\\.2[\\s\\S]*库存\\ 100\\ 件[\\s\\S]*7L\\ 容量[\\s\\S]*12\\ 种烹饪预设[\\s\\S]*快速热风循环" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_009_regression", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往我的购物车加一双跑步鞋。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv009_regression", + "name": "add_to_cart", + "args": { + "product_id": "P004", + "quantity": 1 + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "已将\\ 1\\ 双\\ Men's\\ Running\\ Shoes\\ 加入购物车[\\s\\S]*合计\\ ¥599\\.00" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_010_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Levi's 501 Jeans。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv010_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P008" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "Levi's\\ 501\\ Jeans（P008）Levi's\\ 出品[\\s\\S]*原版剪裁[\\s\\S]*非弹力牛仔布[\\s\\S]*直筒裤型[\\s\\S]*售价\\ ¥499[\\s\\S]*评分\\ 4\\.4[\\s\\S]*库存\\ 300\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_011_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "兑换优惠券 FREEDELIVERY。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv011_ineffective", + "name": "apply_coupon", + "args": { + "code": "FREEDELIVERY" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "优惠券\\ FREEDELIVERY\\ 无效或已过期[\\s\\S]*有效优惠券[\\s\\S]*SAVE50（满\\ ¥200\\ 减\\ ¥50）、VIP10（9\\ 折）、NEWUSER（满\\ ¥100\\ 减\\ ¥30）" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_012_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 ¥1000 以下的电子产品。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv012_optimizable", + "name": "search_products", + "args": { + "query": "electronics", + "category": "electronics" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ Wireless\\ Bluetooth\\ Headphones（P001）售价\\ ¥299[\\s\\S]*评分\\ 4\\.5[\\s\\S]*库存\\ 120\\ 件[\\s\\S]*其他电子产品起价\\ ¥2,499\\ 以上" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_013_regression", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用 VIP10 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv013_regression", + "name": "apply_coupon", + "args": { + "code": "VIP10" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "优惠券\\ VIP10\\ 已使用[\\s\\S]*已为您的订单应用\\ VIP\\ 会员\\ 9\\ 折优惠" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_014_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我 Coffee Maker 的详细信息。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv014_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P005" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "Coffee\\ Maker（P005）BrewMaster\\ 出品[\\s\\S]*可编程滴漏咖啡机[\\s\\S]*12\\ 杯容量[\\s\\S]*内置研磨器[\\s\\S]*售价\\ ¥1,299[\\s\\S]*评分\\ 4\\.6[\\s\\S]*库存\\ 80\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_015_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "把商品 P888 加入购物车。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv015_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P888", + "quantity": 1 + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "商品\\ P888\\ 不存在[\\s\\S]*请检查商品\\ ID[\\s\\S]*有效商品[\\s\\S]*P001\\ 至\\ P010" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_016_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "跟我说说 Sony WH-1000XM6。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv016_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P006" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "Sony\\ WH\\-1000XM6（P006）索尼出品[\\s\\S]*业界领先的降噪技术[\\s\\S]*40\\ 小时续航[\\s\\S]*支持\\ LDAC[\\s\\S]*售价\\ ¥2,499[\\s\\S]*评分\\ 4\\.9[\\s\\S]*库存\\ 15\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_017_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我需要一部新手机，你们有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv017_optimizable", + "name": "search_products", + "args": { + "query": "phone" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "找到\\ 2\\ 款手机[\\s\\S]*iPhone\\ 15\\ Pro（P002）售价\\ ¥7,999（评分\\ 4\\.8）[\\s\\S]*Samsung\\ Galaxy\\ S25（P007）售价\\ ¥6,999（评分\\ 4\\.6）[\\s\\S]*两款都很不错[\\s\\S]*各有特色" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_018_ineffective", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询订单 ORD-00001 的状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv018_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-00001" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "订单\\ ORD\\-00001\\ 不存在[\\s\\S]*请核实订单\\ ID[\\s\\S]*有效订单[\\s\\S]*ORD\\-12345、ORD\\-12346、ORD\\-12347" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_019_regression", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用我的 NEWUSER 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv019_regression", + "name": "apply_coupon", + "args": { + "code": "NEWUSER" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "优惠券\\ NEWUSER\\ 已使用[\\s\\S]*新用户满\\ ¥100\\ 立减\\ ¥30" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_020_optimizable", + "eval_mode": "non-trace", + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Air Fryer XL。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "lv020_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P010" + } + } + ] + }, + "final_response": { + "parts": [ + { + "text": "Air\\ Fryer\\ XL（P010）KitchenPro\\ 出品[\\s\\S]*7L\\ 容量[\\s\\S]*12\\ 种烹饪预设[\\s\\S]*快速热风循环[\\s\\S]*售价\\ ¥899[\\s\\S]*评分\\ 4\\.2[\\s\\S]*库存\\ 100\\ 件" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + } + ] +} \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/data/optimizer.json b/examples/optimization/eval_optimize_loop/data/optimizer.json new file mode 100644 index 000000000..e18aa05b2 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/optimizer.json @@ -0,0 +1,88 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 0.8, + "criterion": { + "final_response": { + "text": { + "match": "regex", + "case_insensitive": true + } + } + } + }, + { + "metric_name": "llm_rubric_response", + "threshold": 0.6, + "criterion": { + "llmJudge": { + "judge_model": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "api_key": "${TRPC_AGENT_API_KEY}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "generation_config": { + "max_tokens": 2048, + "temperature": 0 + } + }, + "rubrics": [ + { + "id": "fact_accuracy", + "description": "事实准确性", + "type": "FACT_ACCURACY", + "content": { + "text": "回答中提到的商品名称、价格、评分、库存数量、订单号等事实信息必须与工具返回结果一致，不得编造或臆测。只要出现一处关键事实错误或凭空捏造，本项判否。" + } + }, + { + "id": "completeness", + "description": "完整性", + "type": "COMPLETENESS", + "content": { + "text": "回答应覆盖任务要求的关键信息：商品搜索或推荐场景需包含商品名称与价格（若工具提供，还应包含评分、库存）；详情、购物车、结算、订单查询等场景需包含对应关键字段（价格、数量、评分、库存、订单号、状态等）。仅回复「已找到 N 个结果」这类缺少关键信息的过短回答，本项判否。" + } + }, + { + "id": "failure_handling", + "description": "失败处理", + "type": "FAILURE_HANDLING", + "content": { + "text": "若工具调用失败或结果为空（商品不存在、库存不足、订单不存在、优惠券无效等），回答必须如实转述失败原因，并给出可行的替代建议。若任务实际成功完成，本项直接判是；若回答隐瞒失败或编造成功结果，本项判否。" + } + } + ] + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "model_name": "${TRPC_AGENT_MODEL_NAME}", + "api_key": "${TRPC_AGENT_API_KEY}", + "base_url": "${TRPC_AGENT_BASE_URL}", + "generation_config": { + "max_tokens": 8192, + "temperature": 0.01 + } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "reflection_minibatch_size": 3, + "skip_perfect_score": true, + "max_metric_calls": 150, + "max_iterations_without_improvement": 5, + "timeout_seconds": 1800 + } + } +} \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/data/trace/train.evalset.json b/examples/optimization/eval_optimize_loop/data/trace/train.evalset.json new file mode 100644 index 000000000..8a3ef2d35 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/trace/train.evalset.json @@ -0,0 +1,1469 @@ +{ + "eval_set_id": "shopping_assistant_train", + "name": "训练集", + "description": "20 个训练用例。", + "eval_cases": [ + { + "eval_id": "train_001_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我搜索无线耳机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t001_optimizable", + "name": "search_products", + "args": { + "query": "wireless headphones" + } + } + ] + }, + "invocation_id": "t001_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到 2 个结果。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我搜索无线耳机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t001_optimizable", + "name": "search_products", + "args": { + "query": "wireless headphones" + } + } + ] + }, + "invocation_id": "t001_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 2 款无线耳机：Sony WH-1000XM6 售价 ¥2,499（评分 4.9，库存 15 件），Wireless Bluetooth Headphones 售价 ¥299（评分 4.5，库存 120 件）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_002_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询我的订单 ORD-99999。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t002_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-99999" + } + } + ] + }, + "invocation_id": "t002_ineffective-act", + "final_response": { + "parts": [ + { + "text": "您的订单 ORD-99999 正在运输途中，明天送达。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询我的订单 ORD-99999。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t002_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-99999" + } + } + ] + }, + "invocation_id": "t002_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-99999 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_003_working", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往我的购物车里加 2 个 iPhone 15 Pro。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t003_working", + "name": "add_to_cart", + "args": { + "product_id": "P002", + "quantity": 2 + } + } + ] + }, + "invocation_id": "t003_working-act", + "final_response": { + "parts": [ + { + "text": "已将 2 个 iPhone 15 Pro 加入购物车，合计 ¥15,998.00。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往我的购物车里加 2 个 iPhone 15 Pro。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t003_working", + "name": "add_to_cart", + "args": { + "product_id": "P002", + "quantity": 2 + } + } + ] + }, + "invocation_id": "t003_working-exp", + "final_response": { + "parts": [ + { + "text": "已将 2 个 iPhone 15 Pro 加入购物车，合计 ¥15,998.00。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_004_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t004_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P003" + } + } + ] + }, + "invocation_id": "t004_optimizable-act", + "final_response": { + "parts": [ + { + "text": "它是一台笔记本电脑。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t004_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P003" + } + } + ] + }, + "invocation_id": "t004_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "MacBook Pro 14（P003）苹果出品：M3 Pro 芯片，18GB 内存，512GB 固态硬盘，Liquid Retina XDR 显示屏。售价 ¥14,999，评分 4.7/5，库存 30 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_005_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我用优惠券 INVALID99。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t005_ineffective", + "name": "apply_coupon", + "args": { + "code": "INVALID99" + } + } + ] + }, + "invocation_id": "t005_ineffective-act", + "final_response": { + "parts": [ + { + "text": "优惠券 INVALID99 已使用！为您节省 ¥99。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我用优惠券 INVALID99。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t005_ineffective", + "name": "apply_coupon", + "args": { + "code": "INVALID99" + } + } + ] + }, + "invocation_id": "t005_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 INVALID99 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_006_working", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询订单 ORD-12345 的状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t006_working", + "name": "check_order_status", + "args": { + "order_id": "ORD-12345" + } + } + ] + }, + "invocation_id": "t006_working-act", + "final_response": { + "parts": [ + { + "text": "订单 ORD-12345 已于 2026-07-25 发货，预计 2026-08-02 送达。总金额 ¥7,999。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询订单 ORD-12345 的状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t006_working", + "name": "check_order_status", + "args": { + "order_id": "ORD-12345" + } + } + ] + }, + "invocation_id": "t006_working-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-12345 已于 2026-07-25 发货，预计 2026-08-02 送达。总金额 ¥7,999。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_007_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索跑步鞋。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t007_optimizable", + "name": "search_products", + "args": { + "query": "running shoes" + } + } + ] + }, + "invocation_id": "t007_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到 1 个结果。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索跑步鞋。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t007_optimizable", + "name": "search_products", + "args": { + "query": "running shoes" + } + } + ] + }, + "invocation_id": "t007_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 Men's Running Shoes（P004）耐克出品，售价 ¥599，评分 4.3/5，库存 200 件。轻量网面鞋身，缓震回弹良好。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_008_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "找找咖啡机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t008_optimizable", + "name": "search_products", + "args": { + "query": "coffee maker" + } + } + ] + }, + "invocation_id": "t008_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到 1 件商品。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "找找咖啡机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t008_optimizable", + "name": "search_products", + "args": { + "query": "coffee maker" + } + } + ] + }, + "invocation_id": "t008_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 Coffee Maker（P005）BrewMaster 出品，售价 ¥1,299，评分 4.6/5，库存 80 件。12 杯容量可编程滴漏咖啡机，内置研磨器。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_009_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "把商品 P999 加入购物车。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t009_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P999", + "quantity": 1 + } + } + ] + }, + "invocation_id": "t009_ineffective-act", + "final_response": { + "parts": [ + { + "text": "商品 P999 已成功加入购物车。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "把商品 P999 加入购物车。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t009_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P999", + "quantity": 1 + } + } + ] + }, + "invocation_id": "t009_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "商品 P999 不存在，请检查商品 ID。有效商品：P001 至 P010。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_010_working", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的购物车里有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t010_working", + "name": "get_cart", + "args": {} + } + ] + }, + "invocation_id": "t010_working-act", + "final_response": { + "parts": [ + { + "text": "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的购物车里有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t010_working", + "name": "get_cart", + "args": {} + } + ] + }, + "invocation_id": "t010_working-exp", + "final_response": { + "parts": [ + { + "text": "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_011_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索吸尘器。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t011_optimizable", + "name": "search_products", + "args": { + "query": "vacuum" + } + } + ] + }, + "invocation_id": "t011_optimizable-act", + "final_response": { + "parts": [ + { + "text": "1 个结果。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索吸尘器。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t011_optimizable", + "name": "search_products", + "args": { + "query": "vacuum" + } + } + ] + }, + "invocation_id": "t011_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 Dyson V15 Vacuum（P009）戴森出品，售价 ¥4,599，评分 4.8/5，库存 25 件。无线手持吸尘器，配备激光探测和 LCD 屏幕。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_012_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的订单 ORD-88888 到哪了？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t012_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-88888" + } + } + ] + }, + "invocation_id": "t012_ineffective-act", + "final_response": { + "parts": [ + { + "text": "您的订单 ORD-88888 正在处理中，很快就会发货。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的订单 ORD-88888 到哪了？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t012_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-88888" + } + } + ] + }, + "invocation_id": "t012_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-88888 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_013_working", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我的订单用优惠券 SAVE50。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t013_working", + "name": "apply_coupon", + "args": { + "code": "SAVE50" + } + } + ] + }, + "invocation_id": "t013_working-act", + "final_response": { + "parts": [ + { + "text": "优惠券 SAVE50 已使用！订单满 ¥200 立减 ¥50.00。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我的订单用优惠券 SAVE50。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t013_working", + "name": "apply_coupon", + "args": { + "code": "SAVE50" + } + } + ] + }, + "invocation_id": "t013_working-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 SAVE50 已使用！订单满 ¥200 立减 ¥50.00。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_014_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Samsung Galaxy S25。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t014_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P007" + } + } + ] + }, + "invocation_id": "t014_optimizable-act", + "final_response": { + "parts": [ + { + "text": "它是三星的一款手机。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Samsung Galaxy S25。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t014_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P007" + } + } + ] + }, + "invocation_id": "t014_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Samsung Galaxy S25（P007）三星出品：Snapdragon 8 Gen 4 芯片，200MP 摄像头，Dynamic AMOLED 2X 显示屏。售价 ¥6,999，评分 4.6/5，库存 60 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_015_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "Sony WH-1000XM6 耳机的详情。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t015_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P006" + } + } + ] + }, + "invocation_id": "t015_optimizable-act", + "final_response": { + "parts": [ + { + "text": "那些是耳机。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "Sony WH-1000XM6 耳机的详情。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t015_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P006" + } + } + ] + }, + "invocation_id": "t015_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Sony WH-1000XM6（P006）索尼出品：业界领先的降噪技术，40 小时续航，支持 LDAC。售价 ¥2,499，评分 4.9/5，库存 15 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_016_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往购物车加 200 个 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t016_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P003", + "quantity": 200 + } + } + ] + }, + "invocation_id": "t016_ineffective-act", + "final_response": { + "parts": [ + { + "text": "已将 200 个 MacBook Pro 14 加入购物车。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往购物车加 200 个 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t016_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P003", + "quantity": 200 + } + } + ] + }, + "invocation_id": "t016_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "库存不足。MacBook Pro 14 仅有 30 件库存，无法添加 200 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_017_working", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询我的订单 ORD-12347 状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t017_working", + "name": "check_order_status", + "args": { + "order_id": "ORD-12347" + } + } + ] + }, + "invocation_id": "t017_working-act", + "final_response": { + "parts": [ + { + "text": "订单 ORD-12347 已于 2026-07-26 交付。商品：Men's Running Shoes x1，总金额 ¥599。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询我的订单 ORD-12347 状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t017_working", + "name": "check_order_status", + "args": { + "order_id": "ORD-12347" + } + } + ] + }, + "invocation_id": "t017_working-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-12347 已于 2026-07-26 交付。商品：Men's Running Shoes x1，总金额 ¥599。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_018_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Dyson V15。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t018_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P009" + } + } + ] + }, + "invocation_id": "t018_optimizable-act", + "final_response": { + "parts": [ + { + "text": "一台吸尘器。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Dyson V15。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t018_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P009" + } + } + ] + }, + "invocation_id": "t018_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Dyson V15 Vacuum（P009）戴森出品：无线手持吸尘器，配备激光探测和 LCD 屏幕。售价 ¥4,599，评分 4.8/5，库存 25 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_019_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用促销码 SUPER50。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t019_ineffective", + "name": "apply_coupon", + "args": { + "code": "SUPER50" + } + } + ] + }, + "invocation_id": "t019_ineffective-act", + "final_response": { + "parts": [ + { + "text": "促销码 SUPER50 已接受！为您打 5 折。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用促销码 SUPER50。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t019_ineffective", + "name": "apply_coupon", + "args": { + "code": "SUPER50" + } + } + ] + }, + "invocation_id": "t019_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 SUPER50 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "train_020_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索牛仔裤。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t020_optimizable", + "name": "search_products", + "args": { + "query": "jeans" + } + } + ] + }, + "invocation_id": "t020_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到 1 条。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "搜索牛仔裤。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "t020_optimizable", + "name": "search_products", + "args": { + "query": "jeans" + } + } + ] + }, + "invocation_id": "t020_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 Levi's 501 Jeans（P008）Levi's 出品，售价 ¥499，评分 4.4/5，库存 300 件。原版剪裁，非弹力牛仔布，直筒裤型。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + } + ] +} \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/data/trace/val_baseline.evalset.json b/examples/optimization/eval_optimize_loop/data/trace/val_baseline.evalset.json new file mode 100644 index 000000000..47906bff8 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/trace/val_baseline.evalset.json @@ -0,0 +1,1469 @@ +{ + "eval_set_id": "shopping_assistant_val", + "name": "验证集基线", + "description": "20 个验证用例。", + "eval_cases": [ + { + "eval_id": "val_001_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v001_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P003" + } + } + ] + }, + "invocation_id": "v001_optimizable-act", + "final_response": { + "parts": [ + { + "text": "它是一台笔记本电脑。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v001_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P003" + } + } + ] + }, + "invocation_id": "v001_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "MacBook Pro 14（P003）苹果出品：M3 Pro 芯片，18GB 内存，512GB 固态硬盘，Liquid Retina XDR 显示屏。售价 ¥14,999，评分 4.7/5，库存 30 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_002_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我用优惠券 INVALID99。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v002_ineffective", + "name": "apply_coupon", + "args": { + "code": "INVALID99" + } + } + ] + }, + "invocation_id": "v002_ineffective-act", + "final_response": { + "parts": [ + { + "text": "优惠券 INVALID99 已使用！为您节省 ¥99。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我用优惠券 INVALID99。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v002_ineffective", + "name": "apply_coupon", + "args": { + "code": "INVALID99" + } + } + ] + }, + "invocation_id": "v002_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 INVALID99 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_003_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的购物车里有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v003_regression", + "name": "get_cart", + "args": {} + } + ] + }, + "invocation_id": "v003_regression-act", + "final_response": { + "parts": [ + { + "text": "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的购物车里有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v003_regression", + "name": "get_cart", + "args": {} + } + ] + }, + "invocation_id": "v003_regression-exp", + "final_response": { + "parts": [ + { + "text": "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_004_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我找找无线耳机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v004_optimizable", + "name": "search_products", + "args": { + "query": "wireless headphones" + } + } + ] + }, + "invocation_id": "v004_optimizable-act", + "final_response": { + "parts": [ + { + "text": "这是结果。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我找找无线耳机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v004_optimizable", + "name": "search_products", + "args": { + "query": "wireless headphones" + } + } + ] + }, + "invocation_id": "v004_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 2 款无线耳机：Sony WH-1000XM6 售价 ¥2,499（评分 4.9，库存 15 件），Wireless Bluetooth Headphones 售价 ¥299（评分 4.5，库存 120 件）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_005_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的订单 ORD-12346 到哪了？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v005_regression", + "name": "check_order_status", + "args": { + "order_id": "ORD-12346" + } + } + ] + }, + "invocation_id": "v005_regression-act", + "final_response": { + "parts": [ + { + "text": "订单 ORD-12346 正在处理中。商品：Coffee Maker x1、Air Fryer XL x1，总金额 ¥2,198，预计 2026-08-05 送达。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的订单 ORD-12346 到哪了？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v005_regression", + "name": "check_order_status", + "args": { + "order_id": "ORD-12346" + } + } + ] + }, + "invocation_id": "v005_regression-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-12346 正在处理中。商品：Coffee Maker x1、Air Fryer XL x1，总金额 ¥2,198，预计 2026-08-05 送达。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_006_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 iPhone 15 Pro 的详情。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v006_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P002" + } + } + ] + }, + "invocation_id": "v006_optimizable-act", + "final_response": { + "parts": [ + { + "text": "iPhone。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 iPhone 15 Pro 的详情。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v006_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P002" + } + } + ] + }, + "invocation_id": "v006_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "iPhone 15 Pro（P002）苹果出品：A17 Pro 芯片，48MP 摄像头，钛金属设计，支持 USB-C。售价 ¥7,999，评分 4.8/5，库存 50 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_007_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我追踪订单 ORD-77777。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v007_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-77777" + } + } + ] + }, + "invocation_id": "v007_ineffective-act", + "final_response": { + "parts": [ + { + "text": "您的订单 ORD-77777 今天正在派送中。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我追踪订单 ORD-77777。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v007_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-77777" + } + } + ] + }, + "invocation_id": "v007_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-77777 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_008_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "找找空气炸锅。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v008_optimizable", + "name": "search_products", + "args": { + "query": "air fryer" + } + } + ] + }, + "invocation_id": "v008_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到了。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "找找空气炸锅。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v008_optimizable", + "name": "search_products", + "args": { + "query": "air fryer" + } + } + ] + }, + "invocation_id": "v008_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 Air Fryer XL（P010）KitchenPro 出品，售价 ¥899，评分 4.2/5，库存 100 件。7L 容量，12 种烹饪预设，快速热风循环。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_009_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往我的购物车加一双跑步鞋。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v009_regression", + "name": "add_to_cart", + "args": { + "product_id": "P004", + "quantity": 1 + } + } + ] + }, + "invocation_id": "v009_regression-act", + "final_response": { + "parts": [ + { + "text": "已将 1 双 Men's Running Shoes 加入购物车，合计 ¥599.00。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往我的购物车加一双跑步鞋。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v009_regression", + "name": "add_to_cart", + "args": { + "product_id": "P004", + "quantity": 1 + } + } + ] + }, + "invocation_id": "v009_regression-exp", + "final_response": { + "parts": [ + { + "text": "已将 1 双 Men's Running Shoes 加入购物车，合计 ¥599.00。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_010_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Levi's 501 Jeans。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v010_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P008" + } + } + ] + }, + "invocation_id": "v010_optimizable-act", + "final_response": { + "parts": [ + { + "text": "Levi's 的牛仔裤。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Levi's 501 Jeans。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v010_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P008" + } + } + ] + }, + "invocation_id": "v010_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Levi's 501 Jeans（P008）Levi's 出品：原版剪裁，非弹力牛仔布，直筒裤型。售价 ¥499，评分 4.4/5，库存 300 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_011_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "兑换优惠券 FREEDELIVERY。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v011_ineffective", + "name": "apply_coupon", + "args": { + "code": "FREEDELIVERY" + } + } + ] + }, + "invocation_id": "v011_ineffective-act", + "final_response": { + "parts": [ + { + "text": "FREEDELIVERY 优惠券已激活！您的订单免运费。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "兑换优惠券 FREEDELIVERY。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v011_ineffective", + "name": "apply_coupon", + "args": { + "code": "FREEDELIVERY" + } + } + ] + }, + "invocation_id": "v011_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 FREEDELIVERY 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_012_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 ¥1000 以下的电子产品。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v012_optimizable", + "name": "search_products", + "args": { + "query": "electronics", + "category": "electronics" + } + } + ] + }, + "invocation_id": "v012_optimizable-act", + "final_response": { + "parts": [ + { + "text": "没有结果。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 ¥1000 以下的电子产品。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v012_optimizable", + "name": "search_products", + "args": { + "query": "electronics", + "category": "electronics" + } + } + ] + }, + "invocation_id": "v012_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 Wireless Bluetooth Headphones（P001）售价 ¥299，评分 4.5/5，库存 120 件。其他电子产品起价 ¥2,499 以上。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_013_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用 VIP10 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v013_regression", + "name": "apply_coupon", + "args": { + "code": "VIP10" + } + } + ] + }, + "invocation_id": "v013_regression-act", + "final_response": { + "parts": [ + { + "text": "优惠券 VIP10 已使用！已为您的订单应用 VIP 会员 9 折优惠。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用 VIP10 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v013_regression", + "name": "apply_coupon", + "args": { + "code": "VIP10" + } + } + ] + }, + "invocation_id": "v013_regression-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 VIP10 已使用！已为您的订单应用 VIP 会员 9 折优惠。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_014_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我 Coffee Maker 的详细信息。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v014_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P005" + } + } + ] + }, + "invocation_id": "v014_optimizable-act", + "final_response": { + "parts": [ + { + "text": "一台咖啡机。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我 Coffee Maker 的详细信息。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v014_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P005" + } + } + ] + }, + "invocation_id": "v014_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Coffee Maker（P005）BrewMaster 出品：可编程滴漏咖啡机，12 杯容量，内置研磨器。售价 ¥1,299，评分 4.6/5，库存 80 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_015_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "把商品 P888 加入购物车。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v015_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P888", + "quantity": 1 + } + } + ] + }, + "invocation_id": "v015_ineffective-act", + "final_response": { + "parts": [ + { + "text": "P888 已加入购物车。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "把商品 P888 加入购物车。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v015_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P888", + "quantity": 1 + } + } + ] + }, + "invocation_id": "v015_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "商品 P888 不存在，请检查商品 ID。有效商品：P001 至 P010。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_016_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "跟我说说 Sony WH-1000XM6。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v016_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P006" + } + } + ] + }, + "invocation_id": "v016_optimizable-act", + "final_response": { + "parts": [ + { + "text": "耳机。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "跟我说说 Sony WH-1000XM6。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v016_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P006" + } + } + ] + }, + "invocation_id": "v016_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Sony WH-1000XM6（P006）索尼出品：业界领先的降噪技术，40 小时续航，支持 LDAC。售价 ¥2,499，评分 4.9/5，库存 15 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_017_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我需要一部新手机，你们有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v017_optimizable", + "name": "search_products", + "args": { + "query": "phone" + } + } + ] + }, + "invocation_id": "v017_optimizable-act", + "final_response": { + "parts": [ + { + "text": "有 2 款手机可选。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我需要一部新手机，你们有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v017_optimizable", + "name": "search_products", + "args": { + "query": "phone" + } + } + ] + }, + "invocation_id": "v017_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 2 款手机：iPhone 15 Pro（P002）售价 ¥7,999（评分 4.8），Samsung Galaxy S25（P007）售价 ¥6,999（评分 4.6）。两款都很不错，各有特色。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_018_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询订单 ORD-00001 的状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v018_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-00001" + } + } + ] + }, + "invocation_id": "v018_ineffective-act", + "final_response": { + "parts": [ + { + "text": "ORD-00001 等待发货中。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询订单 ORD-00001 的状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v018_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-00001" + } + } + ] + }, + "invocation_id": "v018_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-00001 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_019_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用我的 NEWUSER 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v019_regression", + "name": "apply_coupon", + "args": { + "code": "NEWUSER" + } + } + ] + }, + "invocation_id": "v019_regression-act", + "final_response": { + "parts": [ + { + "text": "优惠券 NEWUSER 已使用！新用户满 ¥100 立减 ¥30。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用我的 NEWUSER 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v019_regression", + "name": "apply_coupon", + "args": { + "code": "NEWUSER" + } + } + ] + }, + "invocation_id": "v019_regression-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 NEWUSER 已使用！新用户满 ¥100 立减 ¥30。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_020_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Air Fryer XL。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v020_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P010" + } + } + ] + }, + "invocation_id": "v020_optimizable-act", + "final_response": { + "parts": [ + { + "text": "一件厨房电器。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Air Fryer XL。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "v020_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P010" + } + } + ] + }, + "invocation_id": "v020_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Air Fryer XL（P010）KitchenPro 出品：7L 容量，12 种烹饪预设，快速热风循环。售价 ¥899，评分 4.2/5，库存 100 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + } + ] +} \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/data/trace/val_optimized.evalset.json b/examples/optimization/eval_optimize_loop/data/trace/val_optimized.evalset.json new file mode 100644 index 000000000..3a82cd6d9 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/trace/val_optimized.evalset.json @@ -0,0 +1,1469 @@ +{ + "eval_set_id": "shopping_assistant_val_optimized", + "name": "购物助手验证集（优化后）", + "description": "20 个验证用例，反映提示词优化后的 agent 行为：可优化用例已通过，不可优化用例保持不变，回归用例体现过拟合。", + "eval_cases": [ + { + "eval_id": "val_001_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o001_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P003" + } + } + ] + }, + "invocation_id": "o001_optimizable-act", + "final_response": { + "parts": [ + { + "text": "MacBook Pro 14（P003）苹果出品：M3 Pro 芯片，18GB 内存，512GB 固态硬盘，Liquid Retina XDR 显示屏。售价 ¥14,999，评分 4.7/5，库存 30 件。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 MacBook Pro 14。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o001_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P003" + } + } + ] + }, + "invocation_id": "o001_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "MacBook Pro 14（P003）苹果出品：M3 Pro 芯片，18GB 内存，512GB 固态硬盘，Liquid Retina XDR 显示屏。售价 ¥14,999，评分 4.7/5，库存 30 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_002_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我用优惠券 INVALID99。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o002_ineffective", + "name": "apply_coupon", + "args": { + "code": "INVALID99" + } + } + ] + }, + "invocation_id": "o002_ineffective-act", + "final_response": { + "parts": [ + { + "text": "优惠券 INVALID99 已使用！为您节省 ¥99。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我用优惠券 INVALID99。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o002_ineffective", + "name": "apply_coupon", + "args": { + "code": "INVALID99" + } + } + ] + }, + "invocation_id": "o002_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 INVALID99 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_003_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的购物车里有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o003_regression", + "name": "get_cart", + "args": {} + } + ] + }, + "invocation_id": "o003_regression-act", + "final_response": { + "parts": [ + { + "text": "让我先搜索一些商品加到您的购物车。您想买点什么？" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的购物车里有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o003_regression", + "name": "get_cart", + "args": {} + } + ] + }, + "invocation_id": "o003_regression-exp", + "final_response": { + "parts": [ + { + "text": "您的购物车有 3 件商品：iPhone 15 Pro x1（¥7,999）、Men's Running Shoes x2（¥1,198），合计 ¥9,197 CNY。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_004_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我找找无线耳机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o004_optimizable", + "name": "search_products", + "args": { + "query": "wireless headphones" + } + } + ] + }, + "invocation_id": "o004_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到 2 款无线耳机：Sony WH-1000XM6 售价 ¥2,499（评分 4.9，库存 15 件），Wireless Bluetooth Headphones 售价 ¥299（评分 4.5，库存 120 件）。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我找找无线耳机。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o004_optimizable", + "name": "search_products", + "args": { + "query": "wireless headphones" + } + } + ] + }, + "invocation_id": "o004_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 2 款无线耳机：Sony WH-1000XM6 售价 ¥2,499（评分 4.9，库存 15 件），Wireless Bluetooth Headphones 售价 ¥299（评分 4.5，库存 120 件）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_005_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的订单 ORD-12346 到哪了？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o005_regression", + "name": "check_order_status", + "args": { + "order_id": "ORD-12346" + } + } + ] + }, + "invocation_id": "o005_regression-act", + "final_response": { + "parts": [ + { + "text": "订单 ORD-12346 已发货，即将送达。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我的订单 ORD-12346 到哪了？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o005_regression", + "name": "check_order_status", + "args": { + "order_id": "ORD-12346" + } + } + ] + }, + "invocation_id": "o005_regression-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-12346 正在处理中。商品：Coffee Maker x1、Air Fryer XL x1，总金额 ¥2,198，预计 2026-08-05 送达。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_006_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 iPhone 15 Pro 的详情。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o006_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P002" + } + } + ] + }, + "invocation_id": "o006_optimizable-act", + "final_response": { + "parts": [ + { + "text": "iPhone 15 Pro（P002）苹果出品：A17 Pro 芯片，48MP 摄像头，钛金属设计，支持 USB-C。售价 ¥7,999，评分 4.8/5，库存 50 件。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 iPhone 15 Pro 的详情。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o006_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P002" + } + } + ] + }, + "invocation_id": "o006_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "iPhone 15 Pro（P002）苹果出品：A17 Pro 芯片，48MP 摄像头，钛金属设计，支持 USB-C。售价 ¥7,999，评分 4.8/5，库存 50 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_007_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我追踪订单 ORD-77777。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o007_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-77777" + } + } + ] + }, + "invocation_id": "o007_ineffective-act", + "final_response": { + "parts": [ + { + "text": "您的订单 ORD-77777 今天正在派送中。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "帮我追踪订单 ORD-77777。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o007_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-77777" + } + } + ] + }, + "invocation_id": "o007_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-77777 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_008_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "找找空气炸锅。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o008_optimizable", + "name": "search_products", + "args": { + "query": "air fryer" + } + } + ] + }, + "invocation_id": "o008_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到 Air Fryer XL（P010）KitchenPro 出品，售价 ¥899，评分 4.2/5，库存 100 件。7L 容量，12 种烹饪预设，快速热风循环。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "找找空气炸锅。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o008_optimizable", + "name": "search_products", + "args": { + "query": "air fryer" + } + } + ] + }, + "invocation_id": "o008_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 Air Fryer XL（P010）KitchenPro 出品，售价 ¥899，评分 4.2/5，库存 100 件。7L 容量，12 种烹饪预设，快速热风循环。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_009_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往我的购物车加一双跑步鞋。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o009_regression", + "name": "add_to_cart", + "args": { + "product_id": "P004", + "quantity": 1 + } + } + ] + }, + "invocation_id": "o009_regression-act", + "final_response": { + "parts": [ + { + "text": "跑步鞋很不错！您还需要袜子或运动服吗？" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "往我的购物车加一双跑步鞋。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o009_regression", + "name": "add_to_cart", + "args": { + "product_id": "P004", + "quantity": 1 + } + } + ] + }, + "invocation_id": "o009_regression-exp", + "final_response": { + "parts": [ + { + "text": "已将 1 双 Men's Running Shoes 加入购物车，合计 ¥599.00。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_010_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Levi's 501 Jeans。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o010_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P008" + } + } + ] + }, + "invocation_id": "o010_optimizable-act", + "final_response": { + "parts": [ + { + "text": "Levi's 501 Jeans（P008）Levi's 出品：原版剪裁，非弹力牛仔布，直筒裤型。售价 ¥499，评分 4.4/5，库存 300 件。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Levi's 501 Jeans。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o010_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P008" + } + } + ] + }, + "invocation_id": "o010_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Levi's 501 Jeans（P008）Levi's 出品：原版剪裁，非弹力牛仔布，直筒裤型。售价 ¥499，评分 4.4/5，库存 300 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_011_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "兑换优惠券 FREEDELIVERY。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o011_ineffective", + "name": "apply_coupon", + "args": { + "code": "FREEDELIVERY" + } + } + ] + }, + "invocation_id": "o011_ineffective-act", + "final_response": { + "parts": [ + { + "text": "FREEDELIVERY 优惠券已激活！您的订单免运费。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "兑换优惠券 FREEDELIVERY。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o011_ineffective", + "name": "apply_coupon", + "args": { + "code": "FREEDELIVERY" + } + } + ] + }, + "invocation_id": "o011_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 FREEDELIVERY 无效或已过期。有效优惠券：SAVE50（满 ¥200 减 ¥50）、VIP10（9 折）、NEWUSER（满 ¥100 减 ¥30）。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_012_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 ¥1000 以下的电子产品。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o012_optimizable", + "name": "search_products", + "args": { + "query": "electronics", + "category": "electronics" + } + } + ] + }, + "invocation_id": "o012_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到 Wireless Bluetooth Headphones（P001）售价 ¥299，评分 4.5/5，库存 120 件。其他电子产品起价 ¥2,499 以上。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我看看 ¥1000 以下的电子产品。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o012_optimizable", + "name": "search_products", + "args": { + "query": "electronics", + "category": "electronics" + } + } + ] + }, + "invocation_id": "o012_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 Wireless Bluetooth Headphones（P001）售价 ¥299，评分 4.5/5，库存 120 件。其他电子产品起价 ¥2,499 以上。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_013_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用 VIP10 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o013_regression", + "name": "apply_coupon", + "args": { + "code": "VIP10" + } + } + ] + }, + "invocation_id": "o013_regression-act", + "final_response": { + "parts": [ + { + "text": "VIP10 已应用。您的忠诚度积分增加了 10%。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用 VIP10 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o013_regression", + "name": "apply_coupon", + "args": { + "code": "VIP10" + } + } + ] + }, + "invocation_id": "o013_regression-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 VIP10 已使用！已为您的订单应用 VIP 会员 9 折优惠。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_014_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我 Coffee Maker 的详细信息。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o014_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P005" + } + } + ] + }, + "invocation_id": "o014_optimizable-act", + "final_response": { + "parts": [ + { + "text": "Coffee Maker（P005）BrewMaster 出品：可编程滴漏咖啡机，12 杯容量，内置研磨器。售价 ¥1,299，评分 4.6/5，库存 80 件。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "给我 Coffee Maker 的详细信息。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o014_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P005" + } + } + ] + }, + "invocation_id": "o014_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Coffee Maker（P005）BrewMaster 出品：可编程滴漏咖啡机，12 杯容量，内置研磨器。售价 ¥1,299，评分 4.6/5，库存 80 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_015_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "把商品 P888 加入购物车。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o015_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P888", + "quantity": 1 + } + } + ] + }, + "invocation_id": "o015_ineffective-act", + "final_response": { + "parts": [ + { + "text": "P888 已加入购物车。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "把商品 P888 加入购物车。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o015_ineffective", + "name": "add_to_cart", + "args": { + "product_id": "P888", + "quantity": 1 + } + } + ] + }, + "invocation_id": "o015_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "商品 P888 不存在，请检查商品 ID。有效商品：P001 至 P010。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_016_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "跟我说说 Sony WH-1000XM6。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o016_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P006" + } + } + ] + }, + "invocation_id": "o016_optimizable-act", + "final_response": { + "parts": [ + { + "text": "Sony WH-1000XM6（P006）索尼出品：业界领先的降噪技术，40 小时续航，支持 LDAC。售价 ¥2,499，评分 4.9/5，库存 15 件。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "跟我说说 Sony WH-1000XM6。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o016_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P006" + } + } + ] + }, + "invocation_id": "o016_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Sony WH-1000XM6（P006）索尼出品：业界领先的降噪技术，40 小时续航，支持 LDAC。售价 ¥2,499，评分 4.9/5，库存 15 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_017_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我需要一部新手机，你们有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o017_optimizable", + "name": "search_products", + "args": { + "query": "phone" + } + } + ] + }, + "invocation_id": "o017_optimizable-act", + "final_response": { + "parts": [ + { + "text": "找到 2 款手机：iPhone 15 Pro（P002）售价 ¥7,999（评分 4.8），Samsung Galaxy S25（P007）售价 ¥6,999（评分 4.6）。两款都很不错，各有特色。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "我需要一部新手机，你们有什么？" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o017_optimizable", + "name": "search_products", + "args": { + "query": "phone" + } + } + ] + }, + "invocation_id": "o017_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "找到 2 款手机：iPhone 15 Pro（P002）售价 ¥7,999（评分 4.8），Samsung Galaxy S25（P007）售价 ¥6,999（评分 4.6）。两款都很不错，各有特色。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_018_ineffective", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询订单 ORD-00001 的状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o018_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-00001" + } + } + ] + }, + "invocation_id": "o018_ineffective-act", + "final_response": { + "parts": [ + { + "text": "ORD-00001 等待发货中。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "查询订单 ORD-00001 的状态。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o018_ineffective", + "name": "check_order_status", + "args": { + "order_id": "ORD-00001" + } + } + ] + }, + "invocation_id": "o018_ineffective-exp", + "final_response": { + "parts": [ + { + "text": "订单 ORD-00001 不存在，请核实订单 ID。有效订单：ORD-12345、ORD-12346、ORD-12347。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_019_regression", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用我的 NEWUSER 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o019_regression", + "name": "apply_coupon", + "args": { + "code": "NEWUSER" + } + } + ] + }, + "invocation_id": "o019_regression-act", + "final_response": { + "parts": [ + { + "text": "NEWUSER 已应用。不过您可以试试 SAVE50，折扣更优惠。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "使用我的 NEWUSER 优惠券。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o019_regression", + "name": "apply_coupon", + "args": { + "code": "NEWUSER" + } + } + ] + }, + "invocation_id": "o019_regression-exp", + "final_response": { + "parts": [ + { + "text": "优惠券 NEWUSER 已使用！新用户满 ¥100 立减 ¥30。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + }, + { + "eval_id": "val_020_optimizable", + "eval_mode": "trace", + "actual_conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Air Fryer XL。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o020_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P010" + } + } + ] + }, + "invocation_id": "o020_optimizable-act", + "final_response": { + "parts": [ + { + "text": "Air Fryer XL（P010）KitchenPro 出品：7L 容量，12 种烹饪预设，快速热风循环。售价 ¥899，评分 4.2/5，库存 100 件。" + } + ], + "role": "model" + } + } + ], + "conversation": [ + { + "user_content": { + "parts": [ + { + "text": "介绍一下 Air Fryer XL。" + } + ], + "role": "user" + }, + "intermediate_data": { + "tool_uses": [ + { + "id": "o020_optimizable", + "name": "get_product_details", + "args": { + "product_id": "P010" + } + } + ] + }, + "invocation_id": "o020_optimizable-exp", + "final_response": { + "parts": [ + { + "text": "Air Fryer XL（P010）KitchenPro 出品：7L 容量，12 种烹饪预设，快速热风循环。售价 ¥899，评分 4.2/5，库存 100 件。" + } + ], + "role": "model" + } + } + ], + "session_input": { + "app_name": "shopping_assistant", + "user_id": "user", + "state": {} + } + } + ] +} \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/design.md b/examples/optimization/eval_optimize_loop/design.md new file mode 100644 index 000000000..d01cfd195 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/design.md @@ -0,0 +1,17 @@ +# 方案设计说明 + +## 失败归因方法 + +失败归因系统基于 `EvalCaseResult.overall_eval_metric_results` 逐 metric 检查失败原因。对确定性 metric（`final_response_avg_score`、`tool_trajectory_avg_score`），归因为 `final_response_mismatch`（回复文本不含期望内容）和 `tool_trajectory_mismatch`（工具调用与期望不一致）。当两个 metric 同时失败时归因为 `both_metrics_failed`，表示行为层面出现根本偏差。对 LLM 评分 metric（`llm_rubric_response`、`llm_rubric_knowledge_recall`），额外支持 `llm_rubric_fail` 和 `knowledge_recall_insufficient` 分类。所有失败 case 按类别聚类，生成 `FailureAttributionReport`，标识主要失败模式（如 70% 失败为 final_response_mismatch），为优化方向提供数据支持。每个失败 case 至少给出一个可解释原因，通过 metric 名称和阈值对比自动生成。 + +## 接受策略 + +接受门控采用多检查 AND 逻辑——任一检查失败即拒绝候选 prompt。检查项包括：(1) 验证集 pass rate 提升 ≥ 指定阈值；(2) 无新增 hard failure（baseline 通过的 case 候选不能失败）；(3) 回归数量不超过上限；(4) 关键 case 必须通过；(5) 总成本不超过预算。此设计优先保证安全性：一个在训练集上大幅提升但导致验证集关键 case 退化的候选会被拒绝。 + +## 防过拟合策略 + +防过拟合采用三层机制：首先，优化器使用训练集进行 prompt 搜索，但接受门控基于验证集独立评估，两者严格分离。其次，`no_new_hard_failures` 规则直接检测过拟合信号——训练集提升但验证集新增失败。第三，逐 case delta 报告透明展示每个 case 的分数变化，使得"训练集提升但验证集上部分 case 退化"的典型过拟合模式一目了然。示例数据中 train_001 和 val_001 优化后通过、val_003 意外退化的场景，正是一个被门控正确拒绝的过拟合案例。 + +## 产物审计 + +每次运行生成两份报告：`optimization_report.json`（机器可解析，适合 diff 和自动化分析）和 `optimization_report.md`（人类可读，适合非技术干系人）。审计轨迹包含：baseline 逐 case 逐 metric 评分、失败归因聚类、优化轮次记录与候选 prompt、候选验证评分与逐 case delta、门控决策及所有检查结果、运行耗时和成本。JSON 报告使用确定性键序和一致结构，支持跨运行 diff 对比。所有数据落盘到指定 output 目录，确保每次运行完全可复现。 diff --git a/examples/optimization/eval_optimize_loop/pipeline/__init__.py b/examples/optimization/eval_optimize_loop/pipeline/__init__.py new file mode 100644 index 000000000..b1f6b818d --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/__init__.py @@ -0,0 +1,12 @@ +# 评测 + 优化流水线（Evaluation + Optimization Pipeline） +# +# 本包实现了完整的 6 阶段自动闭环流水线： +# 1. _stage_baseline.py — 基线评测 +# 2. _stage_failure_attribution.py — 失败归因 +# 3. _stage_optimization.py — 优化执行 +# 4. _stage_validation.py — 候选验证 +# 5. _stage_acceptance_gate.py — 接受门控 +# 6. _stage_audit_trail.py — 审计轨迹 +# +# _models.py — 所有阶段的 Pydantic 数据模型 +# _runner.py — PipelineRunner 编排器 diff --git a/examples/optimization/eval_optimize_loop/pipeline/_eval_backend.py b/examples/optimization/eval_optimize_loop/pipeline/_eval_backend.py new file mode 100644 index 000000000..48a36e1ca --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_eval_backend.py @@ -0,0 +1,217 @@ +"""EvalBackend: 封装「agent 的实际输出从哪来」这一唯一差异。""" + +from __future__ import annotations + +import json +import os +import tempfile +from contextlib import asynccontextmanager +from typing import AsyncIterator, Protocol, runtime_checkable + +from trpc_agent_sdk.evaluation._agent_evaluator import AgentEvaluator +from trpc_agent_sdk.evaluation._agent_evaluator import _EvaluationCasesFailed +from trpc_agent_sdk.evaluation._eval_config import EvalConfig +from trpc_agent_sdk.evaluation._eval_metrics import EvalStatus +from trpc_agent_sdk.evaluation._eval_result import EvaluateResult +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt +from trpc_agent_sdk.runners import Runner +from trpc_agent_sdk.sessions import InMemorySessionService + +from agent.agent import create_agent +from pipeline._models import EvalSetReport, PerCaseScore + + +# App name used for the live runner. Eval cases store this in session_input.app_name +# when they want to override the default; otherwise the runner's app_name is used. +LIVE_RUNNER_APP_NAME = "shopping_assistant" + + +@runtime_checkable +class EvalBackend(Protocol): + async def evaluate( + self, + *, + eval_set_path: str, + metrics_config_path: str, + num_runs: int = 1, + ) -> tuple[EvaluateResult, EvalSetReport]: ... + + +class TraceBackend: + """trace 模式 backend: 不调用 agent, 直接从预录制轨迹计算 metric.""" + + async def evaluate( + self, + *, + eval_set_path: str, + metrics_config_path: str, + num_runs: int = 1, + ) -> tuple[EvaluateResult, EvalSetReport]: + with open(metrics_config_path, "r") as f: + config_data = json.load(f) + config_data = _strip_llm_metrics(config_data) + EvalConfig(**config_data) # 校验 + + # demo/trace 模式不能调用 LLM judge: 把剔除 llm_* 后的配置写到临时 + # 文件再传路径（get_executer 只接受路径），用后即删。 + tmp_metrics_path: str | None = None + try: + with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as tf: + tmp_metrics_path = tf.name + json.dump(config_data, tf, ensure_ascii=False, indent=2) + executer = AgentEvaluator.get_executer( + eval_dataset_file_path_or_dir=eval_set_path, + eval_metrics_file_path_or_dir=tmp_metrics_path, + num_runs=num_runs, + ) + # 只吞 _EvaluationCasesFailed（case 失败是基线场景的预期信号）; + # 无关 AssertionError（如第三方库断言）必须传播，否则半成品结果 + # 会被 _build_report 当成真实报告送入门控。 + try: + await executer.evaluate() + except _EvaluationCasesFailed: + pass + + raw: EvaluateResult | None = executer.get_result() + if raw is None: + raise RuntimeError(f"评测失败 ({eval_set_path}): 无返回结果") + return raw, _build_report(raw) + finally: + if tmp_metrics_path is not None: + os.unlink(tmp_metrics_path) + + +class LiveBackend: + """real 模式 backend: 每次 evaluate() 现建 Runner（→ 新 agent → 重读 system.md）。 + + Why a factory, not an instance: ``LlmAgent`` freezes ``instruction`` at + construction time. Holding a long-lived agent would let Stage 4 pick up the + *old* prompt. ``optimization.md`` §4.1-§4.3 require a fresh agent per + evaluation, so the new ``system.md`` (written by the optimizer) is read on + every call. + """ + + def __init__(self, agent_factory=create_agent) -> None: + self._agent_factory = agent_factory + self._agents_built: int = 0 # 供 Task 16.3 回归测试断言 + + def agents_built(self) -> int: + return self._agents_built + + async def evaluate( + self, + *, + eval_set_path: str, + metrics_config_path: str, + num_runs: int = 1, + ) -> tuple[EvaluateResult, EvalSetReport]: + # 1. 现建 agent — 触发 _read_system_prompt() 重新读 system.md + agent = self._agent_factory(demo_mode=False) + self._agents_built += 1 + + # 2. 包成 Runner 注入到 AgentEvaluator.get_executer(..., runner=...) + # SDK 签名: get_executer(..., runner: Optional[Runner] = None) 见 + # _agent_evaluator.py:373 + runner = Runner( + app_name=LIVE_RUNNER_APP_NAME, + agent=agent, + session_service=InMemorySessionService(), + ) + + executer = AgentEvaluator.get_executer( + eval_dataset_file_path_or_dir=eval_set_path, + eval_metrics_file_path_or_dir=metrics_config_path, + num_runs=num_runs, + runner=runner, + ) + # 只吞 _EvaluationCasesFailed — 基线本来就该有失败 case + try: + await executer.evaluate() + except _EvaluationCasesFailed: + pass + + raw: EvaluateResult | None = executer.get_result() + if raw is None: + raise RuntimeError(f"评测失败 ({eval_set_path}): 无返回结果") + return raw, _build_report(raw) + + +@asynccontextmanager +async def applied_prompts( + target_prompt: TargetPrompt, + prompts: dict[str, str], +) -> AsyncIterator[None]: + """写入候选 prompt, 退出时还原 baseline. 异常路径也保证还原. + + Why this exists: ``AgentOptimizer.optimize(update_source=False)`` 在 ``finally`` + 把源文件回滚成 baseline (optimization.md §3.3 FAQ)。Stage 4 若不显式写回候选, + 评的就是 baseline 而非候选。 + + 选型理由: 使用 SDK 主 API ``read_all`` / ``write_all`` + (``trpc_agent_sdk/evaluation/_target_prompt.py:128`` / ``:135``) —— 两者均为 + async, 覆盖全部已注册字段 (path-backed 与 callback-backed), 且 ``write_all`` + 自带 tmp + os.replace 原子写与多字段部分失败回滚。不使用单字段 + ``read`` 逐个读写, 因为那样无法获得跨字段的原子性。 + """ + baseline = await target_prompt.read_all() + try: + await target_prompt.write_all(prompts) + yield + finally: + await target_prompt.write_all(baseline) + + +def _strip_llm_metrics(config_data: dict) -> dict: + """剔除 llm_* 指标（如 llm_rubric_response）。 + + trace 模式用录制好的轨迹重放，无法调用 LLM judge；保留这些指标会让 + demo 流水线去请求外部模型。live 模式不经过这里，指标原样生效。 + """ + metrics = config_data.get("metrics") + if not isinstance(metrics, list): + return config_data + kept = [m for m in metrics if not str(m.get("metric_name", "")).startswith("llm_")] + return {**config_data, "metrics": kept} + + +def _build_report(raw: EvaluateResult) -> EvalSetReport: + """Build EvalSetReport from raw SDK EvaluateResult. (Move from _stage_baseline.)""" + per_case: list[PerCaseScore] = [] + total = passed = failed = 0 + metric_scores: dict[str, list[float]] = {} + captured_set_id = "unknown" + + for eval_set_id, aggregate in raw.results_by_eval_set_id.items(): + captured_set_id = eval_set_id + for eval_id, runs in aggregate.eval_results_by_eval_id.items(): + total += 1 + case_result = runs[0] + if case_result.final_eval_status == EvalStatus.PASSED: + passed += 1 + elif case_result.final_eval_status == EvalStatus.FAILED: + failed += 1 + + scores: dict[str, float] = {} + statuses: dict[str, str] = {} + for m in case_result.overall_eval_metric_results: + scores[m.metric_name] = m.score or 0.0 + statuses[m.metric_name] = str(m.eval_status.name) if m.eval_status else "NOT_EVALUATED" + metric_scores.setdefault(m.metric_name, []).append(m.score or 0.0) + + per_case.append(PerCaseScore( + eval_id=eval_id, + overall_status=str(case_result.final_eval_status.name) if case_result.final_eval_status else "NOT_EVALUATED", + metric_scores=scores, + metric_statuses=statuses, + )) + + breakdown = {n: sum(s) / len(s) if s else 0.0 for n, s in metric_scores.items()} + return EvalSetReport( + eval_set_id=captured_set_id, + num_cases=total, + num_passed=passed, + num_failed=failed, + pass_rate=passed / total if total > 0 else 0.0, + metric_breakdown=breakdown, + per_case=per_case, + ) diff --git a/examples/optimization/eval_optimize_loop/pipeline/_models.py b/examples/optimization/eval_optimize_loop/pipeline/_models.py new file mode 100644 index 000000000..102024ef8 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_models.py @@ -0,0 +1,210 @@ +"""评测+优化流水线的 Pydantic 数据模型定义。 + +本模块定义了流水线中所有阶段使用的数据结构，包括： +- 门控配置和决策模型 +- 评测集报告和逐 case 评分 +- 失败归因报告 +- 优化执行报告 +- 顶层流水线报告 + +所有模型继承自 EvalBaseModel，支持 JSON 序列化/反序列化。 +""" + +from __future__ import annotations + +from typing import Any, Literal, Optional + +from pydantic import Field + +from trpc_agent_sdk.evaluation._common import EvalBaseModel + +# ---- 枚举类型定义 ---- + +FailureCategory = Literal[ + "final_response_mismatch", # 最终回复文本与期望内容不匹配 + "tool_trajectory_mismatch", # 工具调用轨迹与期望不一致 + "both_metrics_failed", # 回复和工具两个维度均失败 + "llm_rubric_fail", # LLM 评判 rubric 未通过 + "knowledge_recall_insufficient", # 知识库召回不足 + "unknown", # 未知失败原因 +] + +ScenarioType = Literal[ + "optimizable_success", # 当前失败，通过优化 prompt 可修复 + "optimization_ineffective", # 当前失败，仅优化 prompt 无法修复（需改代码/工具） + "optimization_regression", # 当前通过，但优化后存在退化风险 +] + +TransitionType = Literal[ + "PASSED->PASSED", # 基线通过，候选也通过（无变化） + "FAILED->PASSED", # 基线失败，候选通过（优化成功） + "PASSED->FAILED", # 基线通过，候选失败（退化/过拟合） + "FAILED->FAILED", # 基线失败，候选仍失败（优化无效） +] + + +class CriticalCaseConfig(EvalBaseModel): + """关键 case 配置：指定候选评估中必须通过的 case。 + + 用于接受门控中的关键 case 检查——如果某个 case 被标记为关键， + 则候选 prompt 必须在该 case 上通过，否则整个候选被拒绝。 + """ + + eval_id: str = Field(description="必须通过的评测 case ID。") + metric_name: Optional[str] = Field( + default=None, + description="如果指定，则该特定 metric 必须通过。为 None 时要求 overall_status 为 PASSED。", + ) + + +class AcceptanceGateConfig(EvalBaseModel): + """第 5 阶段接受门控的配置。 + + 门控采用多检查 AND 逻辑——所有检查项必须通过才接受候选 prompt。 + 各检查项包括：pass rate 提升阈值、无新增 hard failure、回归数量限制、 + 关键 case 必须通过、成本预算上限。 + """ + + min_improvement_threshold: float = Field( + default=0.0, + ge=0.0, + le=1.0, + description="接受候选所需的最小 pass_rate 提升。0 表示任何提升均可接受。", + ) + no_new_hard_failures: bool = Field( + default=True, + description="如果为 True，则任何基线 PASSED 但候选 FAILED 的 case 都会导致拒绝。", + ) + max_regressions_allowed: int = Field( + default=0, + ge=0, + description="允许的最大回归 case 数量。当 > 0 时覆盖 no_new_hard_failures 的严格限制。", + ) + critical_case_ids: list[str] = Field( + default_factory=list, + description="候选评估中必须通过的 case ID 列表。", + ) + max_cost_budget: float = Field( + default=0.0, + ge=0.0, + description="允许的最大 LLM 调用成本（USD）。0 表示不限制预算。", + ) + + +class GateCheckResult(EvalBaseModel): + """单个门控检查项的结果。""" + + check_name: str # 检查项名称（如 improvement_threshold, regression_check） + passed: bool # 是否通过 + detail: str # 详细说明（含具体数值） + + +class GateDecision(EvalBaseModel): + """所有门控检查后的最终决策。""" + + accepted: bool # 是否接受候选 prompt + reason: str # 接受/拒绝的原因描述 + checks: list[GateCheckResult] # 所有检查项的结果列表 + baseline_pass_rate: float # 基线 pass rate + candidate_pass_rate: float # 候选 pass rate + improvement: float # pass rate 变化量 + regressed_case_ids: list[str] # 发生退化的 case ID 列表 + + +class PerCaseScore(EvalBaseModel): + """单个评测 case 的评分。""" + + eval_id: str # case 唯一标识 + overall_status: str # 整体状态（PASSED/FAILED/NOT_EVALUATED） + metric_scores: dict[str, float] = Field(default_factory=dict) # 各 metric 得分 + metric_statuses: dict[str, str] = Field(default_factory=dict) # 各 metric 状态 + + +class EvalSetReport(EvalBaseModel): + """单次评测运行的完整报告。""" + + eval_set_id: str # 评测集标识 + num_cases: int # case 总数 + num_passed: int # 通过数 + num_failed: int # 失败数 + pass_rate: float # 通过率 + metric_breakdown: dict[str, float] = Field(default_factory=dict) # 各 metric 平均分 + per_case: list[PerCaseScore] = Field(default_factory=list) # 逐 case 评分 + + +class PerCaseDelta(EvalBaseModel): + """单个评测 case 的基线 vs 候选对比。 + + 用于 Stage 4 候选验证，展示每个 case 在优化前后的变化。 + """ + + eval_id: str # case 唯一标识 + scenario: str # 场景类型（optimizable_success 等） + baseline_status: str # 基线状态 + candidate_status: str # 候选状态 + baseline_scores: dict[str, float] = Field(default_factory=dict) # 基线各 metric 得分 + candidate_scores: dict[str, float] = Field(default_factory=dict) # 候选各 metric 得分 + score_delta: dict[str, float] = Field(default_factory=dict) # 分数变化量 + transition: str # 状态转换（如 FAILED->PASSED） + + +class FailureAttributionReport(EvalBaseModel): + """失败归因阶段（Stage 2）的输出。 + + 对失败的 case 按失败原因分类并聚类，帮助识别主要失败模式。 + """ + + total_cases_evaluated: int # 评估的 case 总数 + total_failed: int # 失败的 case 数 + clusters: dict[str, list[str]] = Field(default_factory=dict) # 类别 → case ID 列表 + per_case_categories: dict[str, list[str]] = Field(default_factory=dict) # case ID → 失败类别列表 + summary: str = "" # 人类可读的摘要 + + +class OptimizationExecutionReport(EvalBaseModel): + """优化执行阶段（Stage 3）的输出。 + + 记录 AgentOptimizer 的运行结果，包括算法、轮次、pass rate 变化和成本。 + """ + + algorithm: str # 使用的优化算法（如 gepa_reflective） + status: str # 执行状态（SUCCEEDED/FAILED） + total_rounds: int # 优化总轮次 + baseline_pass_rate: float # 优化前 pass rate + best_pass_rate: float # 最优 pass rate + pass_rate_improvement: float # pass rate 提升量 + duration_seconds: float # 优化耗时（秒） + total_llm_cost: float # LLM 调用总成本（USD） + best_prompts: dict[str, str] = Field(default_factory=dict) # 最优 prompt 内容 + + # 审计字段 (Stage 6 报告消费) + stop_reason: str = "" # SDK: result.stop_reason + finish_reason: str = "" # SDK: result.finish_reason + total_token_usage: dict[str, int] = Field(default_factory=dict) # SDK: result.total_token_usage + total_metric_calls: int = 0 # SDK: result.total_metric_calls + rounds: list[dict] = Field(default_factory=list) # SDK: result.rounds + + +class PipelineReport(EvalBaseModel): + """顶层流水线报告（对应 optimization_report.json）。 + + 聚合了所有 6 个阶段的输出结果，是流水线的最终产物。 + """ + + pipeline_version: str = "1.0.0" # 流水线版本号 + timestamp: str = "" # 运行时间戳（ISO 8601） + pipeline_duration_seconds: float = 0.0 # 流水线总耗时 + demo_mode: bool = False # 是否为 demo 模式 + + # 各阶段输出 + baseline_train: Optional[EvalSetReport] = None # Stage 1: 训练集基线评测 + baseline_val: Optional[EvalSetReport] = None # Stage 1: 验证集基线评测 + failure_attribution: Optional[FailureAttributionReport] = None # Stage 2: 失败归因 + optimization_execution: Optional[OptimizationExecutionReport] = None # Stage 3: 优化执行 + candidate_validation: Optional[EvalSetReport] = None # Stage 4: 候选验证 + case_deltas: list[PerCaseDelta] = Field(default_factory=list) # Stage 4: 逐 case delta + gate_decision: Optional[GateDecision] = None # Stage 5: 门控决策 + + # 汇总 + overall_pass_rate_change: float = 0.0 # 整体 pass rate 变化 + overall_verdict: str = "" # 最终判决（ACCEPTED/REJECTED） diff --git a/examples/optimization/eval_optimize_loop/pipeline/_runner.py b/examples/optimization/eval_optimize_loop/pipeline/_runner.py new file mode 100644 index 000000000..2064ac49f --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_runner.py @@ -0,0 +1,201 @@ +"""PipelineRunner — 编排 6 阶段流水线, 通过 EvalBackend 收敛 demo/real 差异.""" + +from __future__ import annotations + +import time +import uuid +from datetime import datetime, timezone +from pathlib import Path +from typing import Optional + +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt + +from pipeline._eval_backend import EvalBackend +from pipeline._models import ( + AcceptanceGateConfig, + PerCaseScore, + PipelineReport, +) +from pipeline._stage_acceptance_gate import AcceptanceGate +from pipeline._stage_audit_trail import ReportGenerator +from pipeline._stage_baseline import BaselineEvaluator +from pipeline._stage_failure_attribution import FailureAttributor +from pipeline._stage_optimization import OptimizationExecutor +from pipeline._stage_validation import ValidationComparator + + +class PipelineRunner: + def __init__( + self, + *, + train_eval_path: str, + val_baseline_eval_path: str, + val_candidate_eval_path: str, + gate_metrics_config_path: str, + optimizer_config_path: str, + prompt_source_path: str, + prompt_field_name: str, + gate_config: AcceptanceGateConfig, + backend: EvalBackend, + demo_mode: bool, + output_dir: str = "output", + scenario_map: Optional[dict[str, str]] = None, + demo_optimize_result_path: Optional[str] = None, # 仅 demo_mode=True 时有意义 + train_eval_path_real: Optional[str] = None, # 仅 demo_mode=False 时使用 + ) -> None: + self._train_path = train_eval_path + self._val_baseline_path = val_baseline_eval_path + self._val_candidate_path = val_candidate_eval_path + self._train_path_real = train_eval_path_real + self._gate_metrics_path = gate_metrics_config_path + self._optimizer_config_path = optimizer_config_path + self._prompt_source_path = prompt_source_path + self._prompt_field_name = prompt_field_name + self._gate_config = gate_config + self._backend = backend + self._demo_mode = demo_mode + self._output_dir = output_dir + self._scenario_map = scenario_map or {} + self._demo_result_path = demo_optimize_result_path + self.run_dir: Path | None = None # 在 run() 中按 UTC 时间戳创建并赋值. + + def set_scenario_map(self, scenario_map: dict[str, str]) -> None: + """Public setter for scenario_map — Task 14 uses this to avoid direct + private-attribute mutation.""" + self._scenario_map = scenario_map + + async def run(self) -> PipelineReport: + start = time.time() + run_ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + run_dir = Path(self._output_dir) / run_ts + run_dir.mkdir(parents=True, exist_ok=True) + self.run_dir = run_dir + + report = PipelineReport( + pipeline_version="1.0.0", + timestamp=datetime.now(timezone.utc).isoformat(), + demo_mode=self._demo_mode, + ) + + # ---- Stage 1: 基线 ---- + print("[Stage 1/6] 基线评测...") + _, train_report = await BaselineEvaluator.evaluate( + eval_set_path=self._train_path, + metrics_config_path=self._gate_metrics_path, + backend=self._backend, + num_runs=2, + ) + _, val_report = await BaselineEvaluator.evaluate( + eval_set_path=self._val_baseline_path, + metrics_config_path=self._gate_metrics_path, + backend=self._backend, + num_runs=2, + ) + report.baseline_train = train_report + report.baseline_val = val_report + + # ---- Stage 2: 失败归因 ---- + print("[Stage 2/6] 失败归因...") + # 合并 train + val 的 per_case 为 dict[eval_id, PerCaseScore] + combined: dict[str, PerCaseScore] = {} + for c in train_report.per_case: + combined[c.eval_id] = c + for c in val_report.per_case: + combined[c.eval_id] = c + attribution = FailureAttributor.cluster_from_per_case(combined) + report.failure_attribution = attribution + + # ---- Stage 3: 优化 ---- + print("[Stage 3/6] 优化执行...") + if self._demo_mode and self._demo_result_path: + opt_report = OptimizationExecutor.run_demo(self._demo_result_path) + else: + from agent.agent import create_agent + + target_prompt = TargetPrompt().add_path(self._prompt_field_name, self._prompt_source_path) + + async def call_agent(input_text: str) -> str: + from trpc_agent_sdk.context import ( + InvocationContext, + create_agent_context, + new_invocation_context_id, + ) + from trpc_agent_sdk.sessions import InMemorySessionService + from trpc_agent_sdk.types import Content, Part + # agent 每次重建以重读 system.md + agent = create_agent(demo_mode=False) + user_content = Content(role="user", parts=[Part.from_text(text=input_text)]) + agent_context = create_agent_context() + session_service = InMemorySessionService() + session = await session_service.create_session( + app_name="optimizer", + user_id="optimize_model", + session_id=str(uuid.uuid4()), + agent_context=agent_context, + ) + ctx = InvocationContext( + session_service=session_service, + invocation_id=new_invocation_context_id(), + agent=agent, + session=session, + agent_context=agent_context, + user_content=user_content, + override_messages=[user_content], + ) + last_text = "" + async for event in agent.run_async(ctx): + if event.is_final_response(): + if event.content and event.content.parts: + for p in event.content.parts: + if p.text and not getattr(p, "thought", False): + last_text += p.text + return last_text.strip() + + opt_report = await OptimizationExecutor.run_real( + config_path=self._optimizer_config_path, + call_agent=call_agent, + target_prompt=target_prompt, + train_dataset_path=self._train_path_real or self._train_path, + validation_dataset_path=self._val_baseline_path, + output_dir=str(run_dir), + ) + report.optimization_execution = opt_report + + # ---- Stage 4: 候选验证 ---- + print("[Stage 4/6] 候选验证...") + target_prompt_obj = TargetPrompt().add_path(self._prompt_field_name, self._prompt_source_path) + # demo 的候选是另一份预录制 evalset，重放时读不到 prompt；写回只会污染源文件， + # 故传空 best_prompts 让 applied_prompts 退化为 no-op。 + candidate_prompts = {} if self._demo_mode else opt_report.best_prompts + candidate_report, deltas = await ValidationComparator.evaluate_and_compare( + backend=self._backend, + val_eval_path=self._val_candidate_path, + metrics_config_path=self._gate_metrics_path, + target_prompt=target_prompt_obj, + best_prompts=candidate_prompts, + baseline_report=val_report, + scenario_map=self._scenario_map, + ) + report.candidate_validation = candidate_report + report.case_deltas = deltas + + # ---- Stage 5: 门控 ---- + print("[Stage 5/6] 接受门控...") + gate = AcceptanceGate(self._gate_config) + decision = gate.evaluate( + baseline_pass_rate=val_report.pass_rate, + candidate_pass_rate=candidate_report.pass_rate, + baseline_case_statuses={c.eval_id: c.overall_status for c in val_report.per_case}, + candidate_case_statuses={c.eval_id: c.overall_status for c in candidate_report.per_case}, + total_cost=opt_report.total_llm_cost, + ) + report.gate_decision = decision + report.overall_pass_rate_change = candidate_report.pass_rate - val_report.pass_rate + report.overall_verdict = "ACCEPTED" if decision.accepted else "REJECTED" + + # ---- Stage 6: 报告 ---- + print("[Stage 6/6] 审计落盘...") + report.pipeline_duration_seconds = time.time() - start + ReportGenerator.generate_json(report, str(run_dir)) + ReportGenerator.generate_markdown(report, str(run_dir)) + return report \ No newline at end of file diff --git a/examples/optimization/eval_optimize_loop/pipeline/_stage_acceptance_gate.py b/examples/optimization/eval_optimize_loop/pipeline/_stage_acceptance_gate.py new file mode 100644 index 000000000..095a0789a --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_stage_acceptance_gate.py @@ -0,0 +1,249 @@ +"""Stage 5: 接受门控 — 评估候选 prompt 是否满足接受条件。 + +本阶段实现安全优先的候选 prompt 接受策略，采用多检查 AND 逻辑—— +任一检查失败则拒绝候选。检查项包括： + + 1. 提升阈值检查（improvement_threshold）: + 验证集 pass rate 提升必须 ≥ 配置阈值 + + 2. 回归检查（regression_check / no_new_hard_failures）: + 基线通过的 case 候选不能失败（no_new_hard_failures） + 或回归数量不超过上限（max_regressions_allowed） + + 3. 关键 case 检查（critical_cases）: + 指定的关键 case 必须在候选评估中通过 + + 4. 成本预算检查（cost_budget）: + LLM 调用总成本不超过预算上限 + +设计理念：安全优先——即使候选在训练集上大幅提升，如果导致验证集 +关键 case 退化，也会被拒绝。 + +调用方式： + gate = AcceptanceGate(config) + decision = gate.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + baseline_case_statuses={"val_001": "FAILED", "val_002": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED"}, + total_cost=0.01, + ) +""" + +from __future__ import annotations + +from pipeline._models import AcceptanceGateConfig, GateCheckResult, GateDecision + + +class AcceptanceGate: + """接受门控：按可配置标准评估候选 prompt 是否可接受。 + + 通过 evaluate() 方法运行所有门控检查，返回 GateDecision。 + 每个检查生成一个 GateCheckResult，包含检查名称、是否通过和详细说明。 + """ + + def __init__(self, config: AcceptanceGateConfig) -> None: + """初始化接受门控。 + + Args: + config: AcceptanceGateConfig 实例，包含所有检查项的配置参数。 + """ + self._config = config + + def evaluate( + self, + *, + baseline_pass_rate: float, + candidate_pass_rate: float, + baseline_case_statuses: dict[str, str], + candidate_case_statuses: dict[str, str], + total_cost: float = 0.0, + ) -> GateDecision: + """运行所有门控检查并返回决策。 + + Args: + baseline_pass_rate: 基线验证集 pass rate。 + candidate_pass_rate: 候选验证集 pass rate。 + baseline_case_statuses: eval_id → 基线状态的映射。 + candidate_case_statuses: eval_id → 候选状态的映射。 + total_cost: LLM 调用总成本（USD）。 + + Returns: + GateDecision: 包含接受/拒绝决定、原因和所有检查结果。 + """ + checks: list[GateCheckResult] = [] + improvement = candidate_pass_rate - baseline_pass_rate + + # 找出发生退化的 case（PASSED → FAILED） + actual_regressed = [] + for eval_id in baseline_case_statuses: + if (baseline_case_statuses[eval_id] == "PASSED" + and candidate_case_statuses.get(eval_id) == "FAILED"): + actual_regressed.append(eval_id) + + # 执行各项检查 + self._check_improvement(improvement, checks) + self._check_regressions(actual_regressed, checks) + self._check_critical_cases(candidate_case_statuses, checks) + self._check_cost_budget(total_cost, checks) + + # 所有检查都必须通过 + all_passed = all(c.passed for c in checks) + reason = self._build_reason(all_passed, checks, improvement, actual_regressed) + + return GateDecision( + accepted=all_passed, + reason=reason, + checks=checks, + baseline_pass_rate=baseline_pass_rate, + candidate_pass_rate=candidate_pass_rate, + improvement=improvement, + regressed_case_ids=actual_regressed, + ) + + def _check_improvement( + self, improvement: float, checks: list[GateCheckResult] + ) -> None: + """检查 pass rate 提升是否达到阈值。 + + 提升 > 阈值 → 通过；提升 ≤ 阈值 → 失败。 + """ + threshold = self._config.min_improvement_threshold + if improvement > threshold: + checks.append(GateCheckResult( + check_name="improvement_threshold", + passed=True, + detail=f"提升 {improvement:.4f} > 阈值 {threshold}", + )) + else: + checks.append(GateCheckResult( + check_name="improvement_threshold", + passed=False, + detail=f"提升 {improvement:.4f} ≤ 阈值 {threshold}", + )) + + def _check_regressions( + self, + regressed_ids: list[str], + checks: list[GateCheckResult], + ) -> None: + """检查回归数量是否在允许范围内。 + + 逻辑（max_regressions_allowed > 0 时覆盖 no_new_hard_failures 的 + 严格限制，见 _models.py 字段契约）： + - 回归数超过 max_regressions_allowed → 失败 + - 严格模式（no_new_hard_failures=True 且 max_regressions_allowed==0） + 下存在任何回归 → 失败 + - 否则 → 通过 + """ + num_regressions = len(regressed_ids) + + if num_regressions > self._config.max_regressions_allowed: + if self._config.max_regressions_allowed == 0 and self._config.no_new_hard_failures: + # 严格模式（上限 0 = 未配置放宽）: 报告为 no_new_hard_failures + checks.append(GateCheckResult( + check_name="no_new_hard_failures", + passed=False, + detail=f"{num_regressions} 个 case 退化: {', '.join(regressed_ids)}", + )) + else: + checks.append(GateCheckResult( + check_name="regression_limit", + passed=False, + detail=f"{num_regressions} 个退化超过上限 {self._config.max_regressions_allowed}", + )) + else: + checks.append(GateCheckResult( + check_name="regression_check", + passed=True, + detail=f"{num_regressions} 个退化，在上限 {self._config.max_regressions_allowed} 以内", + )) + + def _check_critical_cases( + self, + candidate_statuses: dict[str, str], + checks: list[GateCheckResult], + ) -> None: + """验证所有关键 case 是否通过。 + + 如果未配置关键 case，该检查自动通过。 + """ + critical_ids = self._config.critical_case_ids + if not critical_ids: + checks.append(GateCheckResult( + check_name="critical_cases", + passed=True, + detail="未配置关键 case。", + )) + return + + failed_critical = [ + cid for cid in critical_ids + if candidate_statuses.get(cid) != "PASSED" + ] + if failed_critical: + checks.append(GateCheckResult( + check_name="critical_cases", + passed=False, + detail=f"关键 case 失败: {', '.join(failed_critical)}", + )) + else: + checks.append(GateCheckResult( + check_name="critical_cases", + passed=True, + detail=f"全部 {len(critical_ids)} 个关键 case 通过。", + )) + + def _check_cost_budget( + self, total_cost: float, checks: list[GateCheckResult] + ) -> None: + """检查 LLM 调用成本是否在预算内。 + + 如果 max_cost_budget=0（不限制），该检查自动通过。 + """ + budget = self._config.max_cost_budget + if budget <= 0.0: + checks.append(GateCheckResult( + check_name="cost_budget", + passed=True, + detail="未配置成本预算。", + )) + elif total_cost <= budget: + checks.append(GateCheckResult( + check_name="cost_budget", + passed=True, + detail=f"成本 ${total_cost:.2f} 在预算 ${budget:.2f} 以内", + )) + else: + checks.append(GateCheckResult( + check_name="cost_budget", + passed=False, + detail=f"成本 ${total_cost:.2f} 超出预算 ${budget:.2f}", + )) + + @staticmethod + def _build_reason( + all_passed: bool, + checks: list[GateCheckResult], + improvement: float, + regressed_ids: list[str], + ) -> str: + """生成人类可读的决策原因字符串。 + + Args: + all_passed: 所有检查是否都通过。 + checks: 所有检查结果列表。 + improvement: pass rate 变化。 + regressed_ids: 退化的 case ID 列表。 + + Returns: + 原因描述字符串。 + """ + if all_passed: + return ( + f"接受: 全部 {len(checks)} 项检查通过。" + f"改善: {improvement:+.4f}。" + ) + failed = [c for c in checks if not c.passed] + reasons = [f"{c.check_name}: {c.detail}" for c in failed] + return f"拒绝: {'; '.join(reasons)}" diff --git a/examples/optimization/eval_optimize_loop/pipeline/_stage_audit_trail.py b/examples/optimization/eval_optimize_loop/pipeline/_stage_audit_trail.py new file mode 100644 index 000000000..5be078b61 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_stage_audit_trail.py @@ -0,0 +1,321 @@ +"""Stage 6: 审计轨迹 — 生成 optimization_report.json 和 optimization_report.md。 + +本阶段是流水线的终点，将 PipelineReport 导出为两种格式： + - JSON 格式：机器可解析，适合 diff 对比和自动化分析 + - Markdown 格式：人类可读，适合非技术干系人查看 + +两份报告包含相同的完整信息： + - 基线评测结果（训练集 + 验证集，逐 case 逐 metric） + - 失败归因聚类 + - 优化执行记录（算法、轮次、候选 prompt） + - 候选验证评分和逐 case delta + - 门控决策及所有检查项 + - 运行时间和成本统计 + +调用方式： + json_path = ReportGenerator.generate_json(report, "output/") + md_path = ReportGenerator.generate_markdown(report, "output/") +""" + +from __future__ import annotations + +import json +import os +from datetime import datetime, timezone + +from pipeline._models import PipelineReport + + +class ReportGenerator: + """报告生成器：将 PipelineReport 导出为 JSON 和 Markdown 文件。 + + 所有报告写入指定的 output_dir 目录，确保每次运行完全可复现。 + """ + + @staticmethod + def generate_json(report: PipelineReport, output_dir: str) -> str: + """生成 optimization_report.json。 + + Args: + report: 完整的 PipelineReport。 + output_dir: 输出目录路径。 + + Returns: + 写入的 JSON 文件完整路径。 + """ + path = os.path.join(output_dir, "optimization_report.json") + with open(path, "w", encoding="utf-8") as f: + f.write(report.model_dump_json(indent=2)) + return path + + @staticmethod + def generate_markdown(report: PipelineReport, output_dir: str) -> str: + """生成 optimization_report.md。 + + Args: + report: 完整的 PipelineReport。 + output_dir: 输出目录路径。 + + Returns: + 写入的 Markdown 文件完整路径。 + """ + path = os.path.join(output_dir, "optimization_report.md") + content = ReportGenerator._render_markdown(report) + with open(path, "w", encoding="utf-8") as f: + f.write(content) + return path + + @staticmethod + def _render_markdown(report: PipelineReport) -> str: + """渲染完整的 Markdown 报告内容。 + + 按 6 个阶段组织内容，使用表格和列表呈现数据。 + + Args: + report: 完整的 PipelineReport。 + + Returns: + Markdown 格式的报告字符串。 + """ + lines = [ + "# 优化报告", + "", + f"**生成时间**: {report.timestamp}", + f"**流水线版本**: {report.pipeline_version}", + f"**模式**: {'Demo (trace 模式)' if report.demo_mode else 'Real (LLM)'}", + f"**耗时**: {report.pipeline_duration_seconds:.2f}s", + "", + "---", + "", + "## 总体判决", + "", + ] + + # 总体判决（ACCEPTED / REJECTED） + verdict = report.overall_verdict + if verdict == "ACCEPTED": + lines.append(f"**判决: 接受 (ACCEPTED)** — 候选 prompt 建议用于生产环境。") + elif verdict == "REJECTED": + lines.append(f"**判决: 拒绝 (REJECTED)** — 候选 prompt 不应采用。") + else: + lines.append(f"**判决: {verdict}**") + + if report.gate_decision: + lines.append(f"**原因**: {report.gate_decision.reason}") + lines.append(f"**Pass Rate 变化**: {report.overall_pass_rate_change:+.4f}") + + lines.extend([ + "", + "---", + "", + "## 1. 基线评测", + "", + ]) + + # Stage 1: 基线评测结果 + if report.baseline_train: + lines.extend(ReportGenerator._render_eval_set("训练集", report.baseline_train)) + if report.baseline_val: + lines.extend(ReportGenerator._render_eval_set("验证集", report.baseline_val)) + + lines.extend([ + "---", + "", + "## 2. 失败归因", + "", + ]) + + # Stage 2: 失败归因 + if report.failure_attribution: + fa = report.failure_attribution + lines.append(f"**总 Case 数**: {fa.total_cases_evaluated}") + lines.append(f"**失败 Case 数**: {fa.total_failed}") + lines.append(f"**摘要**: {fa.summary}") + lines.append("") + if fa.clusters: + lines.append("| 失败类别 | 数量 | Case ID |") + lines.append("|---------|------|--------|") + for category, case_ids in sorted(fa.clusters.items()): + lines.append(f"| {category} | {len(case_ids)} | {', '.join(case_ids)} |") + lines.append("") + + if fa.per_case_categories: + lines.append("### 逐 Case 归因") + lines.append("") + for eval_id, categories in sorted(fa.per_case_categories.items()): + if categories: + lines.append(f"- **{eval_id}**: {', '.join(categories)}") + lines.append("") + + lines.extend([ + "---", + "", + "## 3. 优化执行", + "", + ]) + + # Stage 3: 优化执行 + if report.optimization_execution: + oe = report.optimization_execution + lines.append(f"| 字段 | 值 |") + lines.append(f"|------|----|") + lines.append(f"| 算法 | {oe.algorithm} |") + lines.append(f"| 状态 | {oe.status} |") + lines.append(f"| 总轮次 | {oe.total_rounds} |") + lines.append(f"| 基线 Pass Rate | {oe.baseline_pass_rate:.4f} |") + lines.append(f"| 最优 Pass Rate | {oe.best_pass_rate:.4f} |") + lines.append(f"| 提升 | {oe.pass_rate_improvement:+.4f} |") + lines.append(f"| 耗时 | {oe.duration_seconds:.2f}s |") + lines.append(f"| 成本 | ${oe.total_llm_cost:.4f} |") + if oe.stop_reason: + lines.append(f"| Stop Reason | {oe.stop_reason} |") + if oe.finish_reason: + lines.append(f"| Finish Reason | {oe.finish_reason} |") + if oe.total_metric_calls: + lines.append(f"| Total Metric Calls | {oe.total_metric_calls} |") + if oe.total_token_usage: + lines.append(f"| Token Usage | {oe.total_token_usage} |") + lines.append("") + + if oe.best_prompts: + lines.append("### 优化后 Prompt") + lines.append("") + for name, content in oe.best_prompts.items(): + lines.append(f"**{name}**:") + lines.append("```") + lines.append(content.strip()) + lines.append("```") + lines.append("") + + if oe.rounds: + lines.append("### 逐轮记录") + lines.append("") + lines.append("| Round | Train Pass Rate | Val Pass Rate | Accepted | Reason | Failed Cases |") + lines.append("|-------|----------------|---------------|----------|--------|--------------|") + for r in oe.rounds: + failed = r.get("failedCaseIds") or [] + lines.append( + f"| {r.get('round', '')} | {r.get('trainPassRate', 0):.4f} | " + f"{r.get('validationPassRate', 0):.4f} | " + f"{'Y' if r.get('accepted') else 'N'} | " + f"{r.get('acceptanceReason', '')} | " + f"{', '.join(failed)} |" + ) + lines.append("") + + lines.extend([ + "---", + "", + "## 4. 候选验证", + "", + ]) + + # Stage 4: 候选验证 + if report.candidate_validation: + lines.extend(ReportGenerator._render_eval_set("候选验证", report.candidate_validation)) + + lines.extend([ + "---", + "", + "## 5. 逐 Case Delta", + "", + ]) + + # Stage 4: 逐 case delta 表 + if report.case_deltas: + lines.append("| Eval ID | 场景 | 基线 | 候选 | 转换 | 分数变化 |") + lines.append("|---------|------|------|------|------|---------|") + for delta in report.case_deltas: + score_str = ", ".join( + f"{k}: {v:+.2f}" for k, v in delta.score_delta.items() + ) if delta.score_delta else "-" + lines.append( + f"| {delta.eval_id} | {delta.scenario} | {delta.baseline_status} " + f"| {delta.candidate_status} | {delta.transition} | {score_str} |" + ) + lines.append("") + + lines.extend([ + "---", + "", + "## 6. 接受门控决策", + "", + ]) + + # Stage 5: 门控决策 + if report.gate_decision: + gd = report.gate_decision + lines.append(f"**决定**: {'接受 (ACCEPTED)' if gd.accepted else '拒绝 (REJECTED)'}") + lines.append(f"**原因**: {gd.reason}") + lines.append(f"**基线 Pass Rate**: {gd.baseline_pass_rate:.4f}") + lines.append(f"**候选 Pass Rate**: {gd.candidate_pass_rate:.4f}") + lines.append(f"**改善**: {gd.improvement:+.4f}") + lines.append("") + + lines.append("### 门控检查项") + lines.append("") + lines.append("| 检查项 | 结果 | 详情 |") + lines.append("|--------|------|------|") + for check in gd.checks: + icon = "通过 (PASS)" if check.passed else "失败 (FAIL)" + lines.append(f"| {check.check_name} | {icon} | {check.detail} |") + lines.append("") + + if gd.regressed_case_ids: + lines.append(f"**退化的 Case**: {', '.join(gd.regressed_case_ids)}") + lines.append("") + + lines.extend([ + "---", + "", + "## 附录: 可复现性", + "", + f"- **时间戳**: {report.timestamp}", + f"- **流水线版本**: {report.pipeline_version}", + f"- **Demo 模式**: {report.demo_mode}", + ]) + + return "\n".join(lines) + "\n" + + @staticmethod + def _render_eval_set(title: str, report) -> list[str]: + """渲染单个评测集报告为 Markdown 表格。 + + 包含评测集概览、metric 平均分表格和逐 case 评分表格。 + + Args: + title: 评测集标题（如 "训练集"）。 + report: EvalSetReport 实例。 + + Returns: + Markdown 行列表。 + """ + from pipeline._models import EvalSetReport + + lines = [ + f"### {title}", + "", + f"- **评测集**: {report.eval_set_id}", + f"- **Case 数**: {report.num_cases} 总计（{report.num_passed} 通过, {report.num_failed} 失败）", + f"- **Pass Rate**: {report.pass_rate:.4f}", + "", + ] + + if report.metric_breakdown: + lines.append("| Metric | 平均分 |") + lines.append("|--------|--------|") + for name, score in report.metric_breakdown.items(): + lines.append(f"| {name} | {score:.4f} |") + lines.append("") + + if report.per_case: + lines.append("| Eval ID | 状态 | 分数 |") + lines.append("|---------|------|------|") + for case in report.per_case: + score_str = ", ".join( + f"{k}: {v:.2f}" for k, v in case.metric_scores.items() + ) if case.metric_scores else "-" + lines.append(f"| {case.eval_id} | {case.overall_status} | {score_str} |") + lines.append("") + + return lines diff --git a/examples/optimization/eval_optimize_loop/pipeline/_stage_baseline.py b/examples/optimization/eval_optimize_loop/pipeline/_stage_baseline.py new file mode 100644 index 000000000..91d7524a5 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_stage_baseline.py @@ -0,0 +1,26 @@ +"""Stage 1: 基线评测 — 对训练集/验证集运行 AgentEvaluator.""" + +from __future__ import annotations + +from trpc_agent_sdk.evaluation._eval_result import EvaluateResult + +from pipeline._eval_backend import EvalBackend +from pipeline._models import EvalSetReport + + +class BaselineEvaluator: + """基线评测执行器 — 委托给 EvalBackend.""" + + @staticmethod + async def evaluate( + *, + eval_set_path: str, + metrics_config_path: str, + backend: EvalBackend, + num_runs: int = 1, + ) -> tuple[EvaluateResult, EvalSetReport]: + return await backend.evaluate( + eval_set_path=eval_set_path, + metrics_config_path=metrics_config_path, + num_runs=num_runs, + ) diff --git a/examples/optimization/eval_optimize_loop/pipeline/_stage_failure_attribution.py b/examples/optimization/eval_optimize_loop/pipeline/_stage_failure_attribution.py new file mode 100644 index 000000000..beb34dacf --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_stage_failure_attribution.py @@ -0,0 +1,248 @@ +"""Stage 2: 失败归因 — 对评测失败的 case 进行分类和聚类。 + +本阶段分析 Stage 1 基线评测的失败 case，通过检查各 metric 的失败状态 +将失败归类为以下类型之一： + - final_response_mismatch: 最终回复文本与期望不匹配 + - tool_trajectory_mismatch: 工具调用轨迹与期望不一致 + - both_metrics_failed: 回复和工具两个维度均失败（根本性偏差） + - llm_rubric_fail: LLM 评判 rubric 未通过 + - knowledge_recall_insufficient: 知识库召回不足 + - unknown: 无法归类的失败 + +同一 case 可能归入多个类别（如同时存在回复不匹配和 rubric 失败）。 +聚类结果用于识别主要失败模式，为 Stage 3 优化提供数据支持。 + +调用方式： + # 推荐：直接从 Stage 1 报告的 per_case 派生（不依赖 SDK 内部结构） + attribution_report = FailureAttributor.cluster_from_per_case(per_case) + + # 兼容旧路径：从 SDK EvalCaseResult 派生 + attribution_report = FailureAttributor.cluster(results_by_eval_id) +""" + +from __future__ import annotations + +from collections import defaultdict + +from trpc_agent_sdk.evaluation._eval_metrics import EvalStatus +from trpc_agent_sdk.evaluation._eval_result import EvalCaseResult + +from pipeline._models import FailureAttributionReport, PerCaseScore + + +class FailureAttributor: + """失败归因器：分类并聚类评测中的失败 case。 + + 核心方法： + - cluster_from_per_case(): 从 Stage 1 报告的 per_case 聚类（推荐，无 SDK 依赖） + - classify(): 对单个 EvalCaseResult 按 metric 状态分类 + - cluster(): 对一批 case 按类别聚类，生成 FailureAttributionReport + + 分类逻辑： + 1. 检查 case 是否 FAILED（通过的 case 不归因） + 2. 检查 final_response_avg_score 和 tool_trajectory_avg_score + - 两者同时失败 → both_metrics_failed + - 仅一个失败 → 对应单独类别 + 3. 检查 LLM rubric metric（llm_rubric_response / llm_rubric_knowledge_recall） + 4. 无任何已知 metric 失败 → unknown + """ + + @staticmethod + def cluster_from_per_case( + per_case: dict[str, PerCaseScore], + ) -> FailureAttributionReport: + """从 Stage 1 报告的 per_case 派生失败归因（不依赖 SDK EvalCaseResult）。 + + 与 cluster() 的分类逻辑等价，但输入是 Stage 1 已产出的 PerCaseScore， + 使 Stage 2 完全自给，不再耦合 SDK 内部数据结构。 + + Args: + per_case: eval_id → PerCaseScore 的映射（来自 EvalSetReport.per_case）。 + + Returns: + FailureAttributionReport: 包含聚类结果、逐 case 类别和摘要。 + """ + clusters: dict[str, list[str]] = defaultdict(list) + per_case_cats: dict[str, list[str]] = {} + total = len(per_case) + total_failed = 0 + + for eval_id, score in per_case.items(): + if score.overall_status != "FAILED": + # 通过 / 未评测的 case 不需要归因 + per_case_cats[eval_id] = [] + continue + + total_failed += 1 + categories: list[str] = [] + statuses = score.metric_statuses + + # 确定性 metric：回复匹配与工具轨迹匹配 + final_resp_failed = ( + statuses.get("final_response_avg_score") == "FAILED" + ) + tool_traj_failed = ( + statuses.get("tool_trajectory_avg_score") == "FAILED" + ) + + if final_resp_failed and tool_traj_failed: + # 两个核心 metric 同时失败，说明行为层面出现根本偏差 + categories.append("both_metrics_failed") + else: + if final_resp_failed: + categories.append("final_response_mismatch") + if tool_traj_failed: + categories.append("tool_trajectory_mismatch") + + # LLM rubric metric（软性评估） + if statuses.get("llm_rubric_response") == "FAILED": + categories.append("llm_rubric_fail") + + if statuses.get("llm_rubric_knowledge_recall") == "FAILED": + categories.append("knowledge_recall_insufficient") + + # 兜底：无已知 metric 失败 + if not categories: + categories.append("unknown") + + for category in categories: + clusters[category].append(eval_id) + per_case_cats[eval_id] = categories + + summary = FailureAttributor._build_summary( + total, total_failed, dict(clusters) + ) + + return FailureAttributionReport( + total_cases_evaluated=total, + total_failed=total_failed, + clusters=dict(clusters), + per_case_categories=per_case_cats, + summary=summary, + ) + + @staticmethod + def classify(case_result: EvalCaseResult) -> list[str]: + """对单个评测结果进行失败分类。 + + Args: + case_result: 单个 case 的评测结果（EvalCaseResult 实例）。 + + Returns: + 失败类别字符串列表。如果 case 通过，返回空列表。 + 可能返回多个类别（如同时命中 final_response_mismatch 和 llm_rubric_fail）。 + """ + # 通过的 case 不需要归因 + if case_result.final_eval_status != EvalStatus.FAILED: + return [] + + metric_results = case_result.overall_eval_metric_results + if not metric_results: + return ["unknown"] + + # 构建 metric_name → eval_status 映射 + metric_status_map = { + m.metric_name: m.eval_status + for m in metric_results + } + + categories: list[str] = [] + + # 检查确定性 metric（回复匹配和工具轨迹匹配） + final_resp_failed = metric_status_map.get("final_response_avg_score") == EvalStatus.FAILED + tool_traj_failed = metric_status_map.get("tool_trajectory_avg_score") == EvalStatus.FAILED + + if final_resp_failed and tool_traj_failed: + # 两个核心 metric 同时失败，说明行为层面出现根本偏差 + categories.append("both_metrics_failed") + else: + if final_resp_failed: + categories.append("final_response_mismatch") + if tool_traj_failed: + categories.append("tool_trajectory_mismatch") + + # 检查 LLM rubric metric（软性评估） + if metric_status_map.get("llm_rubric_response") == EvalStatus.FAILED: + categories.append("llm_rubric_fail") + + if metric_status_map.get("llm_rubric_knowledge_recall") == EvalStatus.FAILED: + categories.append("knowledge_recall_insufficient") + + # 兜底：无已知 metric 失败 + if not categories: + categories.append("unknown") + + return categories + + @staticmethod + def cluster( + results_by_eval_id: dict[str, list[EvalCaseResult]], + ) -> FailureAttributionReport: + """对一批 case 按失败类别聚类。 + + Args: + results_by_eval_id: eval_id → EvalCaseResult 列表的映射 + （列表长度取决于 num_runs，通常为 1）。 + + Returns: + FailureAttributionReport: 包含聚类结果、逐 case 类别和摘要。 + """ + clusters: dict[str, list[str]] = defaultdict(list) + per_case: dict[str, list[str]] = {} + total = 0 + total_failed = 0 + + for eval_id, runs in results_by_eval_id.items(): + total += 1 + case_result = runs[0] # 取第一次运行的分类结果 + + if case_result.final_eval_status == EvalStatus.FAILED: + total_failed += 1 + categories = FailureAttributor.classify(case_result) + per_case[eval_id] = categories + # 将 case 添加到对应的类别聚类中 + for category in categories: + clusters[category].append(eval_id) + else: + per_case[eval_id] = [] + + # 生成人类可读摘要 + summary = FailureAttributor._build_summary( + total, total_failed, dict(clusters) + ) + + return FailureAttributionReport( + total_cases_evaluated=total, + total_failed=total_failed, + clusters=dict(clusters), + per_case_categories=per_case, + summary=summary, + ) + + @staticmethod + def _build_summary( + total: int, + total_failed: int, + clusters: dict[str, list[str]], + ) -> str: + """生成人类可读的失败归因摘要。 + + Args: + total: case 总数。 + total_failed: 失败 case 数。 + clusters: 类别 → case ID 列表的聚类结果。 + + Returns: + 摘要字符串（如 "4 out of 10 cases failed. Main issues: ..."）。 + """ + if total_failed == 0: + return f"全部 {total} 个 case 通过。无失败需要归因。" + + lines = [f"{total_failed}/{total} 个 case 失败。"] + + if clusters: + lines.append("失败分类:") + for category, case_ids in sorted(clusters.items()): + lines.append(f" - {category}: {len(case_ids)} 个 case") + + return " ".join(lines) diff --git a/examples/optimization/eval_optimize_loop/pipeline/_stage_optimization.py b/examples/optimization/eval_optimize_loop/pipeline/_stage_optimization.py new file mode 100644 index 000000000..2ed48b51b --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_stage_optimization.py @@ -0,0 +1,168 @@ +"""Stage 3: 优化执行 — 运行 AgentOptimizer 或加载 demo 结果。 + +本阶段负责执行 prompt 优化，支持两种模式： + - Real 模式: 调用 AgentOptimizer.optimize()，通过真实 LLM 进行多轮优化搜索 + - Demo 模式: 从预生成的 OptimizeResult JSON 加载结果（无需 API key） + +优化器使用 gepa_reflective 算法，通过反思机制迭代改进 prompt。 +优化过程产出最优 prompt 候选、pass rate 变化轨迹和成本统计。 + +调用方式： + # Demo 模式 + report = OptimizationExecutor.run_demo("data/demo_optimize_result.json") + + # Real 模式 + report = await OptimizationExecutor.run_real( + config_path="data/optimizer.json", + call_agent=my_agent_callable, + target_prompt=target_prompt, + train_dataset_path="data/train_baseline.evalset.json", + validation_dataset_path="data/val_baseline.evalset.json", + output_dir="output/20260730_120000/", # SDK 产物落在 .../optimizer/ + ) +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from trpc_agent_sdk.evaluation._optimize_result import OptimizeResult + +from pipeline._models import OptimizationExecutionReport + + +class OptimizationExecutor: + """优化执行器：支持真实优化和 demo 模式。 + + Demo 模式适合快速验证流水线行为，无需 LLM 调用。 + Real 模式通过 AgentOptimizer 实际运行多轮优化搜索。 + """ + + @staticmethod + async def run_real( + *, + config_path: str, + call_agent, + target_prompt, + train_dataset_path: str, + validation_dataset_path: str, + output_dir: str, + ) -> OptimizationExecutionReport: + """使用 AgentOptimizer 执行真实优化。 + + 通过 gepa_reflective 算法进行多轮迭代： + 每轮执行 → 失败归因 → 反思 → 生成候选 prompt → 重新评测 → 接受/拒绝。 + + Args: + config_path: optimizer.json 配置文件路径。 + call_agent: agent 推理的异步可调用对象。 + target_prompt: TargetPrompt 注册表，指定优化目标字段。 + train_dataset_path: 训练集 evalset 路径。 + validation_dataset_path: 验证集 evalset 路径。 + output_dir: 本次运行的时间戳目录；SDK 产物会落到其下的 + ``optimizer/`` 子目录，避免与流水线报告混在一起。 + + Returns: + OptimizationExecutionReport: 包含算法、轮次、pass rate 和成本信息。 + """ + from trpc_agent_sdk.evaluation._agent_optimizer import AgentOptimizer + + optimizer_subdir = Path(output_dir) / "optimizer" + optimizer_subdir.mkdir(parents=True, exist_ok=True) + + result: OptimizeResult = await AgentOptimizer.optimize( + config_path=config_path, + call_agent=call_agent, + target_prompt=target_prompt, + train_dataset_path=train_dataset_path, + validation_dataset_path=validation_dataset_path, + output_dir=str(optimizer_subdir), + update_source=False, + verbose=1, # 实时打印每轮进度，避免长时间盲等。 + ) + + return OptimizationExecutor._from_optimize_result(result) + + @staticmethod + def run_demo(demo_result_path: str) -> OptimizationExecutionReport: + """从预生成的 OptimizeResult JSON 加载 demo 优化结果。 + + 适用于快速演示场景，无需 API key。 + + Args: + demo_result_path: demo_optimize_result.json 文件路径。 + + Returns: + OptimizationExecutionReport: 从 JSON 反序列化的优化报告。 + """ + result = OptimizeResult.from_file(demo_result_path) + return OptimizationExecutor._from_optimize_result(result) + + @staticmethod + def _from_optimize_result(result: OptimizeResult) -> OptimizationExecutionReport: + """将 SDK OptimizeResult 转换为流水线内部的报告格式。 + + 同时透传审计字段（stop_reason / finish_reason / token 用量 / + metric 调用次数 / 逐轮记录），供 Stage 6 审计轨迹消费。 + + Args: + result: SDK 返回的 OptimizeResult 实例。 + + Returns: + OptimizationExecutionReport: 流水线内部格式的优化报告。 + """ + return OptimizationExecutionReport( + algorithm=result.algorithm, + status=result.status, + total_rounds=result.total_rounds, + baseline_pass_rate=result.baseline_pass_rate, + best_pass_rate=result.best_pass_rate, + pass_rate_improvement=result.pass_rate_improvement, + duration_seconds=result.duration_seconds, + total_llm_cost=result.total_llm_cost, + best_prompts=dict(result.best_prompts), + stop_reason=getattr(result, "stop_reason", "") or "", + finish_reason=getattr(result, "finish_reason", "") or "", + total_token_usage=OptimizationExecutor._as_dict( + getattr(result, "total_token_usage", None) + ), + total_metric_calls=OptimizationExecutor._metric_calls(result), + rounds=[ + OptimizationExecutor._as_dict(r) + for r in (getattr(result, "rounds", None) or []) + ], + ) + + @staticmethod + def _as_dict(value: Any) -> dict[str, Any]: + """把 SDK 对象（Pydantic 模型 / dict / 其它）统一转成 dict。 + + Pydantic 模型使用 ``by_alias=True`` 输出 camelCase 键，与 SDK + JSON 产物保持一致；非模型对象退化为 JSON 可序列化形式。 + """ + if value is None: + return {} + if hasattr(value, "model_dump"): + return value.model_dump(by_alias=True) + if isinstance(value, dict): + return dict(value) + if hasattr(value, "__dict__"): + return dict(vars(value)) + return json.loads(json.dumps(value, default=str)) + + @staticmethod + def _metric_calls(result: OptimizeResult) -> int: + """读取 metric 调用总次数。 + + 新版 SDK 直接暴露 ``total_metric_calls``；当前版本把它放在 + ``extras`` 里（由 gepa 回填），再退化到 judge model 调用次数。 + """ + direct = getattr(result, "total_metric_calls", None) + if direct is not None: + return int(direct) + extras = getattr(result, "extras", None) or {} + if isinstance(extras, dict) and extras.get("total_metric_calls") is not None: + return int(extras["total_metric_calls"]) + return int(getattr(result, "total_judge_model_calls", 0) or 0) diff --git a/examples/optimization/eval_optimize_loop/pipeline/_stage_validation.py b/examples/optimization/eval_optimize_loop/pipeline/_stage_validation.py new file mode 100644 index 000000000..3a46a232f --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline/_stage_validation.py @@ -0,0 +1,123 @@ +"""Stage 4: 候选验证 — 写回 best_prompts 后重评验证集，与基线逐 case 对比. + +Why write-back is mandatory: ``AgentOptimizer.optimize(update_source=False)`` 在 +``finally`` 把源 prompt 回滚成 baseline（optimization.md §3.3 FAQ）。Stage 4 若不 +显式写回候选 prompt，评的就是 baseline —— 候选报告会与基线完全相同，门控形同虚设。 +因此 real 模式下必须在 ``applied_prompts`` 上下文内重评**同一份**验证集，退出时 +（含异常路径）自动还原 baseline。 + +demo 模式下 ``best_prompts`` 为空 dict —— 无 prompt 可写回，直接评测（TraceBackend +从预录制轨迹算分，与 prompt 无关）。 + +调用方式： + # CLI 入口由 run_pipeline.derive_scenario(eval_id) 按 eval_id 后缀推导场景, + # 在调用本方法前构造好 scenario_map dict 并传入. + from run_pipeline import derive_scenario + scenario_map = {eid: derive_scenario(eid) for eid in val_ids} + candidate_report, deltas = await ValidationComparator.evaluate_and_compare( + backend=backend, + val_eval_path="data/val.evalset.json", + metrics_config_path="data/gate_metrics.json", + target_prompt=target_prompt, + best_prompts=opt_report.best_prompts, + baseline_report=val_report, + scenario_map=scenario_map, + ) +""" + +from __future__ import annotations + +import contextlib + +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt + +from pipeline._eval_backend import EvalBackend, applied_prompts +from pipeline._models import EvalSetReport, PerCaseDelta + + +class ValidationComparator: + """候选验证对比器：写回候选 prompt 重评验证集，并计算与基线的差异。""" + + @staticmethod + async def evaluate_and_compare( + *, + backend: EvalBackend, + val_eval_path: str, + metrics_config_path: str, + target_prompt: TargetPrompt, + best_prompts: dict[str, str], + baseline_report: EvalSetReport, + scenario_map: dict[str, str] | None = None, + num_runs: int = 2, + ) -> tuple[EvalSetReport, list[PerCaseDelta]]: + """写回 best_prompts 后重评验证集，返回候选报告与逐 case delta。 + + Args: + backend: 评测后端（TraceBackend / LiveBackend）。 + val_eval_path: 验证集 evalset 文件路径 —— 与 Stage 1 基线**同一份**。 + metrics_config_path: 门控 metric 配置文件路径。 + target_prompt: prompt 写回目标（real 模式指向 agent/system.md）。 + best_prompts: Stage 3 产出的最优 prompt；demo 模式为空 dict。 + baseline_report: Stage 1 验证集基线的 EvalSetReport。 + scenario_map: eval_id → 场景类型的可选映射（用于标注 delta 的场景）。 + num_runs: 重复评测次数，用于抵消 LLM 非确定性（README §7）。 + + Returns: + (候选 EvalSetReport, 逐 case PerCaseDelta 列表)。 + """ + if scenario_map is None: + scenario_map = {} + + # demo 模式下 best_prompts 为空 → 不写回（nullcontext） + ctx = ( + applied_prompts(target_prompt, best_prompts) + if best_prompts + else contextlib.nullcontext() + ) + async with ctx: + _, candidate_report = await backend.evaluate( + eval_set_path=val_eval_path, + metrics_config_path=metrics_config_path, + num_runs=num_runs, + ) + + return candidate_report, _compute_deltas( + baseline_report=baseline_report, + candidate_report=candidate_report, + scenario_map=scenario_map, + ) + + +def _compute_deltas( + *, + baseline_report: EvalSetReport, + candidate_report: EvalSetReport, + scenario_map: dict[str, str], +) -> list[PerCaseDelta]: + """逐 case 对比基线与候选：状态转换 + 各 metric 分数变化。""" + base = {c.eval_id: c for c in baseline_report.per_case} + cand = {c.eval_id: c for c in candidate_report.per_case} + deltas: list[PerCaseDelta] = [] + + for eval_id in sorted(set(base) | set(cand)): + b = base.get(eval_id) + c = cand.get(eval_id) + b_status = b.overall_status if b else "MISSING" + c_status = c.overall_status if c else "MISSING" + b_scores = dict(b.metric_scores) if b else {} + c_scores = dict(c.metric_scores) if c else {} + delta = { + m: c_scores.get(m, 0.0) - b_scores.get(m, 0.0) + for m in (set(b_scores) | set(c_scores)) + } + deltas.append(PerCaseDelta( + eval_id=eval_id, + scenario=scenario_map.get(eval_id, "unknown"), + baseline_status=b_status, + candidate_status=c_status, + baseline_scores=b_scores, + candidate_scores=c_scores, + score_delta=delta, + transition=f"{b_status}->{c_status}", + )) + return deltas diff --git a/examples/optimization/eval_optimize_loop/run_pipeline.py b/examples/optimization/eval_optimize_loop/run_pipeline.py new file mode 100644 index 000000000..aa0577632 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/run_pipeline.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""评测 + 优化流水线 CLI 入口。 + +本脚本是完整的"评测 → 失败归因 → Prompt 优化 → 回归验证 → 产物审计"自动闭环 +流水线的命令行入口，场景为电商购物助手。 + +两种运行模式: + Demo 模式（默认，无需 API key）: 使用预录制 trace 数据运行流水线， + trace 模式下评测从预录制轨迹计算 metric，不实际调用 LLM。 + + Real 模式（需要 API key）: 实际调用 LLM 进行推理和优化，需先设置环境变量: + export TRPC_AGENT_API_KEY=your_api_key + export TRPC_AGENT_BASE_URL=https://api.openai.com/v1 + export TRPC_AGENT_MODEL_NAME=gpt-4o-mini # 可选，默认 gpt-4o-mini + +用法: + # Demo 模式（无需 API key，使用预录制 trace 数据） + python run_pipeline.py + + # Demo 模式显式指定 + python run_pipeline.py --demo-mode + + # Real 模式（需要 API key） + python run_pipeline.py --no-demo-mode + + # 自定义输出目录 + python run_pipeline.py --output-dir my_runs/experiment_1 + + # Real 模式 + 自定义门控配置 + python run_pipeline.py --no-demo-mode --max-regressions 1 + +流水线执行 6 个阶段: + 1. 基线评测（Baseline Evaluation） — 在训练集和验证集上运行 AgentEvaluator + 2. 失败归因（Failure Attribution） — 对失败 case 分类并聚类 + 3. 优化执行（Optimization Execution） — 加载 demo 或运行真实 AgentOptimizer + 4. 候选验证（Candidate Validation） — 重新评估验证集，计算逐 case delta + 5. 接受门控（Acceptance Gate） — 按可配置标准评估候选 prompt + 6. 审计轨迹（Audit Trail） — 生成 optimization_report.json + .md +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import sys +from pathlib import Path + +from pipeline._eval_backend import LiveBackend, TraceBackend +from pipeline._models import AcceptanceGateConfig +from pipeline._runner import PipelineRunner + +# ---- 路径常量 ---- +_HERE = Path(__file__).parent +DATA_DIR = _HERE / "data" # 数据目录（evalset、配置等） +OUTPUT_DIR = _HERE / "output" # 默认输出目录 +PROMPT_PATH = _HERE / "agent" / "prompts" / "system.md" # 系统 prompt 文件 + + +def derive_scenario(eval_id: str) -> str: + """按 eval_id 后缀推导场景. 不维护映射表. + + - ``_optimizable`` → ``optimizable_success`` + - ``_ineffective`` → ``optimization_ineffective`` + - ``_working`` / ``_regression`` → ``optimization_regression`` + - 未知后缀 → ``"unknown"`` (调用方应显式处理) + """ + if eval_id.endswith("_optimizable"): + return "optimizable_success" + if eval_id.endswith("_ineffective"): + return "optimization_ineffective" + if eval_id.endswith("_working") or eval_id.endswith("_regression"): + return "optimization_regression" + return "unknown" + + +def build_gate_config( + min_improvement: float = 0.0, + no_new_hard_failures: bool = True, + max_regressions: int = 0, + critical_case_ids: list[str] | None = None, + max_cost_budget: float = 0.0, +) -> AcceptanceGateConfig: + """构建接受门控配置。 + + Args: + min_improvement: 接受候选所需的最小 pass rate 提升（0~1）。 + no_new_hard_failures: 是否禁止新增 hard failure（基线通过→候选失败）。 + max_regressions: 允许的最大回归 case 数。 + critical_case_ids: 必须通过的关键 case ID 列表。 + max_cost_budget: 最大 LLM 成本预算（USD，0=不限制）。 + + Returns: + AcceptanceGateConfig 实例。 + """ + return AcceptanceGateConfig( + min_improvement_threshold=min_improvement, + no_new_hard_failures=no_new_hard_failures, + max_regressions_allowed=max_regressions, + critical_case_ids=critical_case_ids or [], + max_cost_budget=max_cost_budget, + ) + + +def _build_scenario_map(val_eval_path: Path) -> dict[str, str]: + """读取验证集 evalset.json, 推导出每个 case 的场景.""" + cases = json.loads(val_eval_path.read_text())["eval_cases"] + return {c["eval_id"]: derive_scenario(c["eval_id"]) for c in cases} + + +async def main() -> None: + """CLI 主入口：解析命令行参数并运行流水线。""" + parser = argparse.ArgumentParser( + description="评测 + 优化流水线 — 电商购物助手场景", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=__doc__, + ) + # ---- 命令行参数 ---- + parser.add_argument( + "--output-dir", + type=str, + default=str(OUTPUT_DIR), + help=f"报告输出目录（默认: {OUTPUT_DIR}）", + ) + parser.add_argument( + "--min-improvement", + type=float, + default=0.0, + help="接受候选所需的最小 pass rate 提升（默认: 0.0）", + ) + parser.add_argument( + "--max-regressions", + type=int, + default=0, + help="允许的最大回归 case 数（默认: 0）", + ) + parser.add_argument( + "--allow-regressions", + action="store_true", + help="允许任意数量的回归（设置 no_new_hard_failures=False, max_regressions=999）", + ) + parser.add_argument( + "--critical-cases", + type=str, + nargs="*", + default=[], + help="必须通过的关键 case ID（如 --critical-cases val_003_regression）", + ) + parser.add_argument( + "--max-cost", + type=float, + default=0.0, + help="最大 LLM 成本预算（USD，默认: 0 = 不限制）", + ) + parser.add_argument( + "--demo-mode", + action=argparse.BooleanOptionalAction, + default=True, + help="以 demo 模式运行（使用预录制 trace 数据，无需 API key，默认: True）。" + "使用 --no-demo-mode 进入 real 模式，需要设置 TRPC_AGENT_API_KEY 等环境变量。", + ) + + args = parser.parse_args() + + # Real 模式下检查环境变量 + if not args.demo_mode: + missing = [] + if not os.environ.get("TRPC_AGENT_API_KEY"): + missing.append("TRPC_AGENT_API_KEY") + if not os.environ.get("TRPC_AGENT_BASE_URL"): + missing.append("TRPC_AGENT_BASE_URL") + if missing: + print(f"[错误] Real 模式需要设置以下环境变量: {', '.join(missing)}") + print("示例:") + print(" export TRPC_AGENT_API_KEY=your_api_key") + print(" export TRPC_AGENT_BASE_URL=https://api.openai.com/v1") + print(" export TRPC_AGENT_MODEL_NAME=gpt-4o-mini # 可选") + print() + print("或使用 Demo 模式（无需 API key）:") + print(" python run_pipeline.py --demo-mode") + sys.exit(1) + model_name = os.environ.get("TRPC_AGENT_MODEL_NAME", "gpt-4o-mini") + print(f"[Real 模式] API Key: sk-{'*' * 10}... | Base URL: {os.environ['TRPC_AGENT_BASE_URL']} | Model: {model_name}") + else: + print("[Demo 模式] 使用预录制 trace 数据，无需 API key") + + # 处理 --allow-regressions 标志 + if args.allow_regressions: + no_new_hard = False + max_reg = 999 + else: + no_new_hard = True + max_reg = args.max_regressions + + # 构建门控配置 + gate_config = build_gate_config( + min_improvement=args.min_improvement, + no_new_hard_failures=no_new_hard, + max_regressions=max_reg, + critical_case_ids=args.critical_cases, + max_cost_budget=args.max_cost, + ) + + # 组装 backend 与路径 — demo/real 模式的唯一差异收敛于此. + if args.demo_mode: + val_baseline_eval_path = DATA_DIR / "trace" / "val_baseline.evalset.json" + val_candidate_eval_path = DATA_DIR / "trace" / "val_optimized.evalset.json" + train_eval_path = str(DATA_DIR / "trace" / "train.evalset.json") + backend = TraceBackend() + demo_optimize_result_path = str(DATA_DIR / "demo_optimize_result.json") + train_eval_path_real = None + else: + val_baseline_eval_path = DATA_DIR / "live" / "val.evalset.json" + val_candidate_eval_path = DATA_DIR / "live" / "val.evalset.json" + train_eval_path = str(DATA_DIR / "live" / "train.evalset.json") + # LiveBackend 默认 agent_factory=create_agent; 每次评估重建 agent 以重读 system.md. + backend = LiveBackend() + demo_optimize_result_path = None + train_eval_path_real = str(DATA_DIR / "live" / "train.evalset.json") + + scenario_map = _build_scenario_map(val_baseline_eval_path) + + runner = PipelineRunner( + train_eval_path=train_eval_path, + val_baseline_eval_path=str(val_baseline_eval_path), + val_candidate_eval_path=str(val_candidate_eval_path), + gate_metrics_config_path=str(DATA_DIR / "gate_metrics.json"), + optimizer_config_path=str(DATA_DIR / "optimizer.json"), + prompt_source_path=str(PROMPT_PATH), + prompt_field_name="system_prompt", + gate_config=gate_config, + backend=backend, + demo_mode=args.demo_mode, + output_dir=args.output_dir, + scenario_map=scenario_map, + demo_optimize_result_path=demo_optimize_result_path, + train_eval_path_real=train_eval_path_real, + ) + + report = await runner.run() + + # ---- 打印摘要 ---- + print("\n" + "=" * 60) + print(f"流水线结果: {report.overall_verdict}") + print(f"基线 Pass Rate: {report.baseline_val.pass_rate:.4f}" if report.baseline_val else "基线: N/A") + print(f"候选 Pass Rate: {report.candidate_validation.pass_rate:.4f}" if report.candidate_validation else "候选: N/A") + print(f"改善: {report.overall_pass_rate_change:+.4f}") + + if report.gate_decision: + for check in report.gate_decision.checks: + status = "通过" if check.passed else "失败" + print(f" [{status}] {check.check_name}: {check.detail}") + + # runner 在 output_dir 下创建 / 子目录; PipelineRunner 暴露 + # 该路径以便此处准确告知用户报告位置. + print(f"\n报告已写入: {runner.run_dir}") + print("=" * 60) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/optimization/eval_optimize_loop/tests/__init__.py b/examples/optimization/eval_optimize_loop/tests/__init__.py new file mode 100644 index 000000000..277ce8dc5 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/__init__.py @@ -0,0 +1,7 @@ +# eval_optimize_loop 测试包 +# +# test_models.py — 数据模型单元测试 +# test_failure_attribution.py — 失败归因逻辑测试 +# test_acceptance_gate.py — 接受门控逻辑测试 +# test_pipeline_fake.py — 全流水线集成测试（demo 模式） +# conftest.py — 共享 fixtures diff --git a/examples/optimization/eval_optimize_loop/tests/conftest.py b/examples/optimization/eval_optimize_loop/tests/conftest.py new file mode 100644 index 000000000..df82f5e39 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/conftest.py @@ -0,0 +1,39 @@ +"""eval_optimize_loop 测试包的共享 fixtures。 + +提供以下 fixtures: + - temp_dir: 临时目录，用于测试输出 + - data_dir: data/ 目录路径 + - sample_gate_config: 默认接受门控配置字典 +""" + +import json +import os +import tempfile +from pathlib import Path + +import pytest + + +@pytest.fixture +def temp_dir(): + """为测试输出创建临时目录，测试结束后自动清理。""" + with tempfile.TemporaryDirectory() as tmp: + yield tmp + + +@pytest.fixture +def data_dir(): + """返回 data/ 目录的绝对路径。""" + return Path(__file__).parent.parent / "data" + + +@pytest.fixture +def sample_gate_config(): + """返回测试用的默认接受门控配置。""" + return { + "min_improvement_threshold": 0.0, + "no_new_hard_failures": True, + "max_regressions_allowed": 0, + "critical_case_ids": [], + "max_cost_budget": 0.0, + } diff --git a/examples/optimization/eval_optimize_loop/tests/test_acceptance_gate.py b/examples/optimization/eval_optimize_loop/tests/test_acceptance_gate.py new file mode 100644 index 000000000..5f5f4c478 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_acceptance_gate.py @@ -0,0 +1,267 @@ +"""接受门控（Stage 5）单元测试。 + +测试 AcceptanceGate 的各项检查逻辑，覆盖： + - 提升阈值检查（通过/失败） + - 回归检查（禁止回归/允许限制数量） + - 关键 case 检查 + - 成本预算检查 + - 多检查组合场景 + +门控核心理念：安全优先——任一检查失败则拒绝候选。 +""" + +import pytest + +from pipeline._models import AcceptanceGateConfig, GateCheckResult, GateDecision + + +class TestGateCheckResult: + """GateCheckResult 模型的单元测试。""" + + def test_passed_check(self): + """通过检查的 GateCheckResult 应正确记录信息。""" + check = GateCheckResult(check_name="improvement", passed=True, detail="+0.2 提升") + assert check.passed is True + assert check.check_name == "improvement" + + def test_failed_check(self): + """失败检查的 GateCheckResult 应包含退化 case 信息。""" + check = GateCheckResult(check_name="no_regressions", passed=False, detail="1 个退化: val_003") + assert check.passed is False + assert "val_003" in check.detail + + +class TestAcceptanceGate: + """AcceptanceGate 类的单元测试。 + + 测试所有门控检查项在不同配置下的行为。 + """ + + @pytest.fixture + def gate(self): + """导入 AcceptanceGate 类作为 fixture。""" + from pipeline._stage_acceptance_gate import AcceptanceGate + return AcceptanceGate + + def build_gate_config(self, **overrides): + """构建门控配置的辅助方法，支持覆盖默认值。 + + Args: + **overrides: 要覆盖的配置项（如 min_improvement_threshold=0.1）。 + + Returns: + AcceptanceGateConfig 实例。 + """ + defaults = { + "min_improvement_threshold": 0.0, + "no_new_hard_failures": True, + "max_regressions_allowed": 0, + "critical_case_ids": [], + "max_cost_budget": 0.0, + } + defaults.update(overrides) + return AcceptanceGateConfig(**defaults) + + def test_accept_when_improvement_above_threshold(self, gate): + """提升超过阈值时，候选应被接受。""" + config = self.build_gate_config(min_improvement_threshold=0.1) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED", "val_003": "PASSED"}, + total_cost=0.0, + ) + + assert decision.accepted is True + assert decision.improvement == pytest.approx(0.33) + + def test_reject_when_improvement_below_threshold(self, gate): + """提升低于阈值时，候选应被拒绝。""" + config = self.build_gate_config(min_improvement_threshold=0.2) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.5, # 提升 0.17，低于阈值 0.2 + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED", "val_003": "PASSED"}, + total_cost=0.0, + ) + + assert decision.accepted is False + assert "improvement" in decision.reason.lower() + + def test_reject_when_new_hard_failures_present(self, gate): + """存在新增 hard failure（基线通过→候选失败）时应被拒绝。""" + config = self.build_gate_config(no_new_hard_failures=True) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.5, + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED", "val_003": "FAILED"}, # val_003 退化 + total_cost=0.0, + ) + + assert decision.accepted is False + assert "val_003" in decision.regressed_case_ids + assert "regressed" in decision.reason.lower() or "退化" in decision.reason + + def test_accept_when_regressions_within_limit(self, gate): + """回归数量在允许范围内时，候选应被接受。""" + config = self.build_gate_config(no_new_hard_failures=False, max_regressions_allowed=1) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED", "val_003": "FAILED"}, # 1 个回归 + total_cost=0.0, + ) + + # 1 个回归（val_003），在允许范围内 + assert decision.accepted is True + + def test_accept_when_max_regressions_overrides_no_new_hard_failures(self, gate): + """max_regressions_allowed > 0 时应覆盖 no_new_hard_failures 的严格限制（_models.py 契约）。""" + config = self.build_gate_config(no_new_hard_failures=True, max_regressions_allowed=2) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED", "val_004": "PASSED", "val_005": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED", "val_003": "FAILED", "val_004": "PASSED", "val_005": "PASSED"}, # 1 个回归 + total_cost=0.0, + ) + + # 1 个回归 ≤ 上限 2，即使 no_new_hard_failures=True 也应接受 + assert decision.accepted is True + + def test_reject_when_regressions_exceed_overridden_limit(self, gate): + """max_regressions_allowed > 0 时，回归数超过上限仍应拒绝。""" + config = self.build_gate_config(no_new_hard_failures=True, max_regressions_allowed=2) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + baseline_case_statuses={"val_001": "FAILED", "val_002": "PASSED", "val_003": "PASSED", "val_004": "PASSED", "val_005": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED", "val_003": "FAILED", "val_004": "FAILED", "val_005": "PASSED"}, # 3 个回归 + total_cost=0.0, + ) + + assert decision.accepted is False + assert len(decision.regressed_case_ids) == 3 + + def test_reject_when_regressions_exceed_limit(self, gate): + """回归数量超过上限时，候选应被拒绝。""" + config = self.build_gate_config(no_new_hard_failures=False, max_regressions_allowed=0) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.5, + candidate_pass_rate=0.66, + baseline_case_statuses={"val_001": "FAILED", "val_002": "PASSED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED", "val_003": "PASSED"}, + total_cost=0.0, + ) + + assert decision.accepted is False + assert len(decision.regressed_case_ids) == 1 + + def test_reject_when_critical_case_fails(self, gate): + """关键 case 失败时，即使有改善也应拒绝。""" + config = self.build_gate_config( + min_improvement_threshold=0.0, + critical_case_ids=["val_003"], + ) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "PASSED", "val_003": "FAILED"}, # 关键 case 失败 + total_cost=0.0, + ) + + assert decision.accepted is False + assert "critical" in decision.reason.lower() + + def test_reject_when_cost_exceeds_budget(self, gate): + """成本超出预算时，候选应被拒绝。""" + config = self.build_gate_config(max_cost_budget=5.0) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "PASSED", "val_003": "PASSED"}, + total_cost=10.0, # 超出 5.0 预算 + ) + + assert decision.accepted is False + assert "cost" in decision.reason.lower() or "budget" in decision.reason.lower() or "成本" in decision.reason or "预算" in decision.reason + + def test_all_checks_passing(self, gate): + """所有检查通过时，候选应被接受并包含完整检查列表。""" + config = self.build_gate_config( + min_improvement_threshold=0.0, + no_new_hard_failures=True, + critical_case_ids=["val_003"], + max_cost_budget=10.0, + ) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=1.0, + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "PASSED", "val_003": "PASSED"}, + total_cost=5.0, + ) + + assert decision.accepted is True + assert len(decision.checks) >= 4 # improvement, regressions, critical, cost + + def test_no_improvement_no_regressions(self, gate): + """无提升但无退化时：threshold 为 0 时仍被拒绝（提升未超过阈值）。 + + 注意：提升必须严格大于阈值才能通过。 + """ + config = self.build_gate_config(min_improvement_threshold=0.0) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.33, # 无变化 + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + total_cost=0.0, + ) + + assert decision.accepted is False # 无提升（0 ≤ 0 不通过） + assert decision.improvement == 0.0 + + def test_accept_with_tiny_improvement(self, gate): + """阈值为 0 且有微小提升时，应接受候选。""" + config = self.build_gate_config(min_improvement_threshold=0.0) + gate_instance = gate(config) + + decision = gate_instance.evaluate( + baseline_pass_rate=0.33, + candidate_pass_rate=0.34, # 微小提升 + baseline_case_statuses={"val_001": "FAILED", "val_002": "FAILED", "val_003": "PASSED"}, + candidate_case_statuses={"val_001": "PASSED", "val_002": "FAILED", "val_003": "PASSED"}, + total_cost=0.0, + ) + + assert decision.accepted is True + assert decision.improvement > 0 diff --git a/examples/optimization/eval_optimize_loop/tests/test_applied_prompts.py b/examples/optimization/eval_optimize_loop/tests/test_applied_prompts.py new file mode 100644 index 000000000..1a8c48e2e --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_applied_prompts.py @@ -0,0 +1,50 @@ +"""applied_prompts 在 yield 内抛异常时仍还原 baseline. + +把最佳候选 prompt 写入磁盘是 Stage 4 的正确性前提 (optimization.md §3.3 FAQ): +``AgentOptimizer.optimize(update_source=False)`` 在 finally 块把源文件回滚成 baseline, +若不显式写回候选, Stage 4 评的就是 baseline 而非候选。 +""" + +import asyncio +from pathlib import Path +import tempfile + +from trpc_agent_sdk.evaluation._target_prompt import TargetPrompt + +from pipeline._eval_backend import applied_prompts + + +def test_applied_prompts_restores_baseline_on_exception(): + with tempfile.TemporaryDirectory() as tmp: + prompt_path = Path(tmp) / "system.md" + prompt_path.write_text("BASELINE", encoding="utf-8") + + target = TargetPrompt().add_path("system_prompt", str(prompt_path)) + + async def _run(): + async with applied_prompts(target, {"system_prompt": "CANDIDATE"}): + assert prompt_path.read_text(encoding="utf-8") == "CANDIDATE" + raise RuntimeError("simulated evaluate failure") + + try: + asyncio.run(_run()) + except RuntimeError: + pass + + assert prompt_path.read_text(encoding="utf-8") == "BASELINE" + + +def test_applied_prompts_restores_on_success(): + """正常退出也应还原 baseline, 不留下候选污染后续运行.""" + with tempfile.TemporaryDirectory() as tmp: + prompt_path = Path(tmp) / "system.md" + prompt_path.write_text("BASELINE", encoding="utf-8") + + target = TargetPrompt().add_path("system_prompt", str(prompt_path)) + + async def _run(): + async with applied_prompts(target, {"system_prompt": "CANDIDATE"}): + assert prompt_path.read_text(encoding="utf-8") == "CANDIDATE" + + asyncio.run(_run()) + assert prompt_path.read_text(encoding="utf-8") == "BASELINE" diff --git a/examples/optimization/eval_optimize_loop/tests/test_call_agent_rebuild.py b/examples/optimization/eval_optimize_loop/tests/test_call_agent_rebuild.py new file mode 100644 index 000000000..0ab2460d1 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_call_agent_rebuild.py @@ -0,0 +1,37 @@ +"""call_agent 闭包在 real 模式下应每次重建 agent, 否则候选 prompt 不生效. + +PipelineRunner.run() 中 call_agent 的定义: +``create_agent(demo_mode=False)`` 在每次调用时执行 (见 ``_runner.py:119``), +这保证优化器的每轮评测都从磁盘重读 ``system.md``. +""" + +from unittest.mock import MagicMock, patch + + +def test_call_agent_rebuilds_agent_per_invocation(): + """验证 agent 每次调用都重建——两次调用触发两次 create_agent.""" + build_count = {"n": 0} + + def fake_create_agent(*, demo_mode=False): + build_count["n"] += 1 + agent = MagicMock() + agent.demo_mode = demo_mode + return agent + + # 模拟 call_agent 闭包的重建模式 + async def call_agent(_input_text: str) -> str: + fake_create_agent(demo_mode=False) + return "ok" + + import asyncio + asyncio.run(call_agent("a")) + asyncio.run(call_agent("b")) + + assert build_count["n"] == 2, f"expected 2 builds, got {build_count['n']}" + + +def test_create_agent_importable(): + """create_agent 在 demo_mode 下可安全导入 (不请求 API key).""" + from agent.agent import create_agent + agent = create_agent(demo_mode=True) + assert agent is not None diff --git a/examples/optimization/eval_optimize_loop/tests/test_datasets_complete.py b/examples/optimization/eval_optimize_loop/tests/test_datasets_complete.py new file mode 100644 index 000000000..7889ef4e2 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_datasets_complete.py @@ -0,0 +1,20 @@ +"""保护: 所有数据集文件存在且非空.""" + +from pathlib import Path + +DATA_DIR = Path(__file__).parent.parent / "data" + +REQUIRED = [ + "trace/train.evalset.json", + "trace/val_baseline.evalset.json", + "trace/val_optimized.evalset.json", + "live/train.evalset.json", + "live/val.evalset.json", +] + + +def test_all_datasets_present_and_nonempty(): + for rel in REQUIRED: + path = DATA_DIR / rel + assert path.exists(), f"missing: {rel}" + assert path.stat().st_size > 0, f"empty: {rel}" diff --git a/examples/optimization/eval_optimize_loop/tests/test_eval_backend.py b/examples/optimization/eval_optimize_loop/tests/test_eval_backend.py new file mode 100644 index 000000000..bf5364f17 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_eval_backend.py @@ -0,0 +1,106 @@ +"""TraceBackend / LiveBackend 满足 EvalBackend Protocol，及异常处理行为。 + +TraceBackend.evaluate 只应吞掉 SDK 的 _EvaluationCasesFailed（评测 case 有 +失败是基线场景的预期信号），其他 AssertionError（如第三方库断言）必须向上 +传播——否则半成品结果会被当成真实报告送入门控。 +""" + +import json +import tempfile +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from pipeline._eval_backend import EvalBackend, TraceBackend, LiveBackend +from trpc_agent_sdk.evaluation._agent_evaluator import _EvaluationCasesFailed +from trpc_agent_sdk.evaluation._eval_result import EvaluateResult + + +class _FakeExecuter: + """替代 AgentEvaluator.get_executer 返回值的测试替身。""" + + def __init__(self, result=None, error=None): + self._result = result + self._error = error + + async def evaluate(self): + if self._error is not None: + raise self._error + + def get_result(self): + return self._result + + +def _write_metrics(tmp_path: Path, content: str) -> str: + p = tmp_path / "metrics.json" + p.write_text(content) + return str(p) + + +def _patch_executer(monkeypatch, executer): + monkeypatch.setattr( + "pipeline._eval_backend.AgentEvaluator.get_executer", + lambda **kwargs: executer, + ) + + +def test_trace_backend_is_eval_backend(): + assert isinstance(TraceBackend(), EvalBackend) + + +def test_live_backend_is_eval_backend(): + assert isinstance(LiveBackend(), EvalBackend) + + +async def test_trace_backend_propagates_unrelated_assertion_error(monkeypatch, tmp_path): + """评测过程中与 case 失败无关的 AssertionError 必须向上传播，不得吞掉。""" + executer = _FakeExecuter(error=AssertionError("internal invariant broken")) + _patch_executer(monkeypatch, executer) + + with pytest.raises(AssertionError): + await TraceBackend().evaluate( + eval_set_path=str(tmp_path / "eval.json"), + metrics_config_path=_write_metrics(tmp_path, '{"metrics": []}'), + ) + + +async def test_trace_backend_swallows_evaluation_cases_failed(monkeypatch, tmp_path): + """_EvaluationCasesFailed（case 失败信号）应被吞掉，结果仍返回给门控。""" + result = EvaluateResult() + executer = _FakeExecuter(result=result, error=_EvaluationCasesFailed("2 cases failed")) + _patch_executer(monkeypatch, executer) + + raw, report = await TraceBackend().evaluate( + eval_set_path=str(tmp_path / "eval.json"), + metrics_config_path=_write_metrics(tmp_path, '{"metrics": []}'), + ) + + assert raw is result + assert report is not None + + +async def test_trace_backend_raises_runtime_error_when_no_result(monkeypatch, tmp_path): + """评测未产出任何结果时应报错，而不是把空结果当报告。""" + executer = _FakeExecuter(result=None) + _patch_executer(monkeypatch, executer) + + with pytest.raises(RuntimeError): + await TraceBackend().evaluate( + eval_set_path=str(tmp_path / "eval.json"), + metrics_config_path=_write_metrics(tmp_path, '{"metrics": []}'), + ) + + +async def test_trace_backend_invalid_metrics_config_leaves_no_temp_file(monkeypatch, tmp_path): + """非法 metrics 配置应让 ValidationError 向上传播，且不残留临时文件。""" + before = set(Path(tempfile.gettempdir()).glob("tmp*.json")) + + with pytest.raises(ValidationError): + await TraceBackend().evaluate( + eval_set_path=str(tmp_path / "eval.json"), + metrics_config_path=_write_metrics(tmp_path, '{"metrics": "not-a-list"}'), + ) + + after = set(Path(tempfile.gettempdir()).glob("tmp*.json")) + assert after == before diff --git a/examples/optimization/eval_optimize_loop/tests/test_failure_attribution.py b/examples/optimization/eval_optimize_loop/tests/test_failure_attribution.py new file mode 100644 index 000000000..ccbf8b5a0 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_failure_attribution.py @@ -0,0 +1,245 @@ +"""失败归因（Stage 2）单元测试。 + +测试 FailureAttributor 的分类和聚类逻辑，覆盖： + - 通过 case（空分类） + - 单一 metric 失败（final_response / tool_trajectory） + - 双重 metric 失败（both_metrics_failed） + - LLM rubric 失败（llm_rubric_fail / knowledge_recall_insufficient） + - 未知失败（无 metric 数据） + - 批量聚类和摘要生成 +""" + +import pytest + +from trpc_agent_sdk.evaluation._eval_metrics import EvalMetric, EvalStatus +from trpc_agent_sdk.evaluation._eval_result import EvalCaseResult, EvalMetricResult + + +def make_case_result(eval_id, eval_status, metric_results): + """创建测试用 EvalCaseResult 的辅助函数。 + + Args: + eval_id: case ID。 + eval_status: 最终状态（EvalStatus.PASSED 或 FAILED）。 + metric_results: EvalMetricResult 列表。 + + Returns: + EvalCaseResult 实例。 + """ + return EvalCaseResult( + eval_set_id="test_set", + eval_id=eval_id, + final_eval_status=eval_status, + overall_eval_metric_results=metric_results, + eval_metric_result_per_invocation=[], + session_id=f"session_{eval_id}", + ) + + +def make_metric_result(metric_name, eval_status, score=0.0, threshold=1.0): + """创建测试用 EvalMetricResult 的辅助函数。 + + Args: + metric_name: metric 名称。 + eval_status: 评估状态。 + score: 得分。 + threshold: 阈值。 + + Returns: + EvalMetricResult 实例。 + """ + metric = EvalMetric(metric_name=metric_name, threshold=threshold) + result = EvalMetricResult( + metric_name=metric.metric_name, + threshold=metric.threshold, + score=score, + eval_status=eval_status, + ) + return result + + +class TestFailureAttribution: + """FailureAttributor 的单元测试。 + + 验证失败分类逻辑的正确性和聚类报告的结构完整性。 + """ + + @pytest.fixture + def attributor(self): + """导入 FailureAttributor 类作为 fixture。""" + from pipeline._stage_failure_attribution import FailureAttributor + return FailureAttributor + + def test_classify_passing_case(self, attributor): + """通过的 case 应返回空失败类别列表。""" + result = make_case_result( + "case_001", + EvalStatus.PASSED, + [ + make_metric_result("final_response_avg_score", EvalStatus.PASSED, score=1.0), + make_metric_result("tool_trajectory_avg_score", EvalStatus.PASSED, score=1.0), + ], + ) + categories = attributor.classify(result) + assert categories == [] + + def test_classify_final_response_failure(self, attributor): + """仅 final_response_avg_score 失败时，应返回 'final_response_mismatch'。""" + result = make_case_result( + "case_001", + EvalStatus.FAILED, + [ + make_metric_result("final_response_avg_score", EvalStatus.FAILED, score=0.0), + make_metric_result("tool_trajectory_avg_score", EvalStatus.PASSED, score=1.0), + ], + ) + categories = attributor.classify(result) + assert "final_response_mismatch" in categories + + def test_classify_tool_trajectory_failure(self, attributor): + """仅 tool_trajectory_avg_score 失败时，应返回 'tool_trajectory_mismatch'。""" + result = make_case_result( + "case_002", + EvalStatus.FAILED, + [ + make_metric_result("final_response_avg_score", EvalStatus.PASSED, score=1.0), + make_metric_result("tool_trajectory_avg_score", EvalStatus.FAILED, score=0.0), + ], + ) + categories = attributor.classify(result) + assert "tool_trajectory_mismatch" in categories + + def test_classify_both_failed(self, attributor): + """两个 metric 同时失败时，应返回 'both_metrics_failed' 而非单独类别。""" + result = make_case_result( + "case_003", + EvalStatus.FAILED, + [ + make_metric_result("final_response_avg_score", EvalStatus.FAILED, score=0.0), + make_metric_result("tool_trajectory_avg_score", EvalStatus.FAILED, score=0.0), + ], + ) + categories = attributor.classify(result) + assert categories == ["both_metrics_failed"] + + def test_classify_unknown(self, attributor): + """无 metric 数据的失败 case 应返回 'unknown'。""" + result = make_case_result("case_004", EvalStatus.FAILED, []) + categories = attributor.classify(result) + assert categories == ["unknown"] + + def test_cluster_multiple_cases(self, attributor): + """多个失败 case 应按类别正确聚类。 + + 创建 3 个失败 case（2 个回复不匹配 + 1 个工具轨迹不匹配） + 和 1 个通过 case，验证聚类结果。 + """ + case1 = make_case_result( + "c1", EvalStatus.FAILED, + [make_metric_result("final_response_avg_score", EvalStatus.FAILED, score=0.0)], + ) + case2 = make_case_result( + "c2", EvalStatus.FAILED, + [make_metric_result("final_response_avg_score", EvalStatus.FAILED, score=0.0)], + ) + case3 = make_case_result( + "c3", EvalStatus.FAILED, + [make_metric_result("tool_trajectory_avg_score", EvalStatus.FAILED, score=0.0)], + ) + case4 = make_case_result( + "c4", EvalStatus.PASSED, + [make_metric_result("final_response_avg_score", EvalStatus.PASSED, score=1.0)], + ) + + results_by_id = {"c1": [case1], "c2": [case2], "c3": [case3], "c4": [case4]} + report = attributor.cluster(results_by_id) + + assert report.total_cases_evaluated == 4 + assert report.total_failed == 3 + assert "final_response_mismatch" in report.clusters + assert "tool_trajectory_mismatch" in report.clusters + assert len(report.clusters["final_response_mismatch"]) == 2 + assert len(report.clusters["tool_trajectory_mismatch"]) == 1 + + def test_cluster_empty(self, attributor): + """空结果集应产生空报告。""" + report = attributor.cluster({}) + assert report.total_cases_evaluated == 0 + assert report.total_failed == 0 + assert report.clusters == {} + + def test_cluster_all_passing(self, attributor): + """全部通过的 case 应产生空聚类报告。""" + case1 = make_case_result( + "c1", EvalStatus.PASSED, + [make_metric_result("final_response_avg_score", EvalStatus.PASSED, score=1.0)], + ) + case2 = make_case_result( + "c2", EvalStatus.PASSED, + [make_metric_result("final_response_avg_score", EvalStatus.PASSED, score=1.0)], + ) + results_by_id = {"c1": [case1], "c2": [case2]} + report = attributor.cluster(results_by_id) + + assert report.total_cases_evaluated == 2 + assert report.total_failed == 0 + assert report.clusters == {} + + def test_classify_llm_rubric_failure(self, attributor): + """llm_rubric_response 失败时应返回 'llm_rubric_fail'。""" + result = make_case_result( + "case_005", + EvalStatus.FAILED, + [ + make_metric_result("final_response_avg_score", EvalStatus.PASSED, score=1.0), + make_metric_result("llm_rubric_response", EvalStatus.FAILED, score=0.3, threshold=0.66), + ], + ) + categories = attributor.classify(result) + assert "llm_rubric_fail" in categories + + def test_classify_knowledge_recall_failure(self, attributor): + """llm_rubric_knowledge_recall 失败时应返回 'knowledge_recall_insufficient'。""" + result = make_case_result( + "case_006", + EvalStatus.FAILED, + [ + make_metric_result("final_response_avg_score", EvalStatus.PASSED, score=1.0), + make_metric_result("llm_rubric_knowledge_recall", EvalStatus.FAILED, score=0.2, threshold=0.5), + ], + ) + categories = attributor.classify(result) + assert "knowledge_recall_insufficient" in categories + + def test_per_case_categories_populated(self, attributor): + """per_case_categories 映射应包含每个失败 case 的归因。""" + case1 = make_case_result( + "train_001", EvalStatus.FAILED, + [make_metric_result("final_response_avg_score", EvalStatus.FAILED, score=0.0)], + ) + case2 = make_case_result( + "train_002", EvalStatus.FAILED, + [ + make_metric_result("final_response_avg_score", EvalStatus.FAILED, score=0.0), + make_metric_result("tool_trajectory_avg_score", EvalStatus.FAILED, score=0.0), + ], + ) + results_by_id = {"train_001": [case1], "train_002": [case2]} + report = attributor.cluster(results_by_id) + + assert "train_001" in report.per_case_categories + assert "train_002" in report.per_case_categories + assert report.per_case_categories["train_001"] == ["final_response_mismatch"] + assert report.per_case_categories["train_002"] == ["both_metrics_failed"] + + def test_summary_generated(self, attributor): + """报告应包含人类可读摘要，提及失败数量。""" + case1 = make_case_result( + "c1", EvalStatus.FAILED, + [make_metric_result("final_response_avg_score", EvalStatus.FAILED, score=0.0)], + ) + results_by_id = {"c1": [case1]} + report = attributor.cluster(results_by_id) + + assert len(report.summary) > 0 + assert "1" in report.summary # 提及失败数量 diff --git a/examples/optimization/eval_optimize_loop/tests/test_gate_superset_of_optimizer.py b/examples/optimization/eval_optimize_loop/tests/test_gate_superset_of_optimizer.py new file mode 100644 index 000000000..61f7ed267 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_gate_superset_of_optimizer.py @@ -0,0 +1,12 @@ +"""两层 metric 包含关系: gate_metrics 必须 ⊇ optimizer.metrics.""" + +import json +from pathlib import Path + +DATA_DIR = Path(__file__).parent.parent / "data" + + +def test_gate_metrics_is_superset_of_optimizer_metrics(): + gate = {m["metric_name"] for m in json.loads((DATA_DIR / "gate_metrics.json").read_text())["metrics"]} + opt = {m["metric_name"] for m in json.loads((DATA_DIR / "optimizer.json").read_text())["evaluate"]["metrics"]} + assert opt <= gate, f"optimizer metric {opt - gate} not in gate_metrics" diff --git a/examples/optimization/eval_optimize_loop/tests/test_live_datasets_no_trace.py b/examples/optimization/eval_optimize_loop/tests/test_live_datasets_no_trace.py new file mode 100644 index 000000000..14f0cc35c --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_live_datasets_no_trace.py @@ -0,0 +1,17 @@ +"""保护: live 数据集不能含 trace cases (RemoteEvalService 会拒绝).""" + +import json +from pathlib import Path + +DATA_DIR = Path(__file__).parent.parent / "data" + + +def test_live_datasets_have_no_trace_cases(): + for name in ["train.evalset.json", "val.evalset.json"]: + path = DATA_DIR / "live" / name + data = json.loads(path.read_text(encoding="utf-8")) + for case in data["eval_cases"]: + assert case.get("eval_mode") != "trace", \ + f"{name} contains trace case {case['eval_id']}" + assert "actual_conversation" not in case, \ + f"{name}/{case['eval_id']} has actual_conversation (trace-only field)" diff --git a/examples/optimization/eval_optimize_loop/tests/test_models.py b/examples/optimization/eval_optimize_loop/tests/test_models.py new file mode 100644 index 000000000..d84075ad4 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_models.py @@ -0,0 +1,304 @@ +"""流水线 Pydantic 数据模型单元测试。 + +测试 _models.py 中所有数据模型的： + - 默认值 + - 字段验证（范围、类型约束） + - JSON 序列化/反序列化往返 +""" + +import json + +import pytest +from pydantic import ValidationError + +from pipeline._models import ( + AcceptanceGateConfig, + CriticalCaseConfig, + FailureAttributionReport, + GateCheckResult, + GateDecision, + OptimizationExecutionReport, + PerCaseDelta, + PerCaseScore, + PipelineReport, +) + + +class TestAcceptanceGateConfig: + """AcceptanceGateConfig 模型测试。""" + + def test_default_config(self): + """默认配置应有合理的安全默认值。""" + config = AcceptanceGateConfig() + assert config.min_improvement_threshold == 0.0 + assert config.no_new_hard_failures is True # 默认禁止退化 + assert config.max_regressions_allowed == 0 + assert config.critical_case_ids == [] + assert config.max_cost_budget == 0.0 # 默认不限制预算 + + def test_full_config(self): + """完整配置应正确存储所有字段。""" + config = AcceptanceGateConfig( + min_improvement_threshold=0.1, + no_new_hard_failures=False, + max_regressions_allowed=2, + critical_case_ids=["case_001", "case_002"], + max_cost_budget=5.0, + ) + assert config.min_improvement_threshold == 0.1 + assert config.no_new_hard_failures is False + assert config.max_regressions_allowed == 2 + assert config.critical_case_ids == ["case_001", "case_002"] + assert config.max_cost_budget == 5.0 + + def test_invalid_threshold_negative(self): + """负阈值应触发验证错误（ge=0.0 约束）。""" + with pytest.raises(ValidationError): + AcceptanceGateConfig(min_improvement_threshold=-0.1) + + def test_invalid_threshold_above_one(self): + """超过 1.0 的阈值应触发验证错误（le=1.0 约束）。""" + with pytest.raises(ValidationError): + AcceptanceGateConfig(min_improvement_threshold=1.5) + + def test_invalid_negative_regressions(self): + """负回归数应触发验证错误（ge=0 约束）。""" + with pytest.raises(ValidationError): + AcceptanceGateConfig(max_regressions_allowed=-1) + + def test_invalid_negative_budget(self): + """负预算应触发验证错误（ge=0.0 约束）。""" + with pytest.raises(ValidationError): + AcceptanceGateConfig(max_cost_budget=-1.0) + + def test_roundtrip_json(self): + """JSON 序列化→反序列化应保持数据不变。""" + config = AcceptanceGateConfig( + min_improvement_threshold=0.05, + no_new_hard_failures=True, + max_regressions_allowed=1, + critical_case_ids=["val_003"], + max_cost_budget=10.0, + ) + data = config.model_dump() + restored = AcceptanceGateConfig(**data) + assert restored.min_improvement_threshold == 0.05 + assert restored.no_new_hard_failures is True + assert restored.critical_case_ids == ["val_003"] + + +class TestGateDecision: + """GateDecision 模型测试。""" + + def test_accept_decision(self): + """接受决策应完整记录所有检查结果。""" + decision = GateDecision( + accepted=True, + reason="全部检查通过", + checks=[ + GateCheckResult(check_name="improvement", passed=True, detail="+0.2"), + GateCheckResult(check_name="no_regressions", passed=True, detail="0 个退化"), + ], + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + improvement=0.33, + regressed_case_ids=[], + ) + assert decision.accepted is True + assert len(decision.checks) == 2 + + def test_reject_decision(self): + """拒绝决策应记录退化 case 信息。""" + decision = GateDecision( + accepted=False, + reason="检测到 val_003 退化", + checks=[ + GateCheckResult(check_name="improvement", passed=True, detail="+0.33"), + GateCheckResult( + check_name="no_regressions", + passed=False, + detail="1 个 case 退化: val_003", + ), + ], + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + improvement=0.33, + regressed_case_ids=["val_003"], + ) + assert decision.accepted is False + assert decision.regressed_case_ids == ["val_003"] + + def test_roundtrip_json(self): + """JSON 序列化/反序列化往返测试。""" + decision = GateDecision( + accepted=True, + reason="ok", + checks=[GateCheckResult(check_name="test", passed=True, detail="ok")], + baseline_pass_rate=0.5, + candidate_pass_rate=1.0, + improvement=0.5, + regressed_case_ids=[], + ) + json_str = decision.model_dump_json() + restored = GateDecision.model_validate_json(json_str) + assert restored.accepted == decision.accepted + assert restored.improvement == 0.5 + + +class TestPerCaseDelta: + """PerCaseDelta 模型测试。""" + + def test_improvement_delta(self): + """FAILED→PASSED 的 delta 应正确记录分数变化。""" + delta = PerCaseDelta( + eval_id="val_001", + scenario="optimizable_success", + baseline_status="FAILED", + candidate_status="PASSED", + baseline_scores={"final_response_avg_score": 0.0}, + candidate_scores={"final_response_avg_score": 1.0}, + score_delta={"final_response_avg_score": 1.0}, + transition="FAILED->PASSED", + ) + assert delta.transition == "FAILED->PASSED" + assert delta.score_delta["final_response_avg_score"] == 1.0 + + def test_regression_delta(self): + """PASSED→FAILED 的 delta 应正确记录负向变化。""" + delta = PerCaseDelta( + eval_id="val_003", + scenario="optimization_regression", + baseline_status="PASSED", + candidate_status="FAILED", + baseline_scores={"final_response_avg_score": 1.0}, + candidate_scores={"final_response_avg_score": 0.0}, + score_delta={"final_response_avg_score": -1.0}, + transition="PASSED->FAILED", + ) + assert delta.transition == "PASSED->FAILED" + + +class TestPipelineReport: + """PipelineReport 顶层模型测试。""" + + def test_empty_report(self): + """空报告应有正确的默认值。""" + report = PipelineReport() + assert report.pipeline_version == "1.0.0" + assert report.overall_verdict == "" + assert report.case_deltas == [] + + def test_full_report_roundtrip(self): + """完整报告的 JSON 序列化/反序列化往返测试。 + + 验证包含所有阶段输出的报告可以正确序列化和恢复。 + """ + report = PipelineReport( + pipeline_version="1.0.0", + timestamp="2026-01-01T00:00:00", + pipeline_duration_seconds=30.0, + demo_mode=True, + baseline_train=None, + baseline_val=None, + failure_attribution=FailureAttributionReport( + total_cases_evaluated=6, + total_failed=4, + clusters={"final_response_mismatch": ["train_001", "val_001"]}, + per_case_categories={"train_001": ["final_response_mismatch"]}, + summary="4/6 个 case 失败。", + ), + optimization_execution=OptimizationExecutionReport( + algorithm="demo", + status="SUCCEEDED", + total_rounds=1, + baseline_pass_rate=0.33, + best_pass_rate=0.66, + pass_rate_improvement=0.33, + duration_seconds=10.0, + total_llm_cost=0.0, + best_prompts={"system_prompt": "优化后的 prompt 内容。"}, + ), + candidate_validation=None, + case_deltas=[ + PerCaseDelta( + eval_id="val_001", + scenario="optimizable_success", + baseline_status="FAILED", + candidate_status="PASSED", + baseline_scores={}, + candidate_scores={}, + score_delta={}, + transition="FAILED->PASSED", + ), + ], + gate_decision=GateDecision( + accepted=True, + reason="全部检查通过", + checks=[GateCheckResult(check_name="test", passed=True, detail="ok")], + baseline_pass_rate=0.33, + candidate_pass_rate=0.66, + improvement=0.33, + regressed_case_ids=[], + ), + overall_pass_rate_change=0.33, + overall_verdict="ACCEPTED", + ) + json_str = report.model_dump_json() + restored = PipelineReport.model_validate_json(json_str) + assert restored.pipeline_version == "1.0.0" + assert restored.overall_verdict == "ACCEPTED" + assert len(restored.case_deltas) == 1 + assert restored.gate_decision.accepted is True + + +class TestCriticalCaseConfig: + """CriticalCaseConfig 模型测试。""" + + def test_basic(self): + """基本配置只需 eval_id。""" + cfg = CriticalCaseConfig(eval_id="val_003") + assert cfg.eval_id == "val_003" + assert cfg.metric_name is None + + def test_with_metric(self): + """可指定特定 metric 而非整体状态。""" + cfg = CriticalCaseConfig(eval_id="val_003", metric_name="final_response_avg_score") + assert cfg.eval_id == "val_003" + assert cfg.metric_name == "final_response_avg_score" + + def test_missing_eval_id(self): + """缺少 eval_id 应触发验证错误。""" + with pytest.raises(ValidationError): + CriticalCaseConfig() + + +class TestFailureAttributionReport: + """FailureAttributionReport 模型测试。""" + + def test_empty_report(self): + """空报告应有默认空值。""" + report = FailureAttributionReport(total_cases_evaluated=0, total_failed=0) + assert report.clusters == {} + assert report.per_case_categories == {} + assert report.summary == "" + + def test_with_failures(self): + """包含失败数据的报告应正确存储聚类信息。""" + report = FailureAttributionReport( + total_cases_evaluated=6, + total_failed=4, + clusters={ + "final_response_mismatch": ["train_001", "train_002", "val_001"], + "tool_trajectory_mismatch": ["val_002"], + }, + per_case_categories={ + "train_001": ["final_response_mismatch"], + "train_002": ["final_response_mismatch"], + "val_001": ["final_response_mismatch"], + "val_002": ["tool_trajectory_mismatch"], + }, + summary="4 个 case 失败。主要问题: 回复不匹配 (3 个 case)。", + ) + assert len(report.clusters) == 2 + assert len(report.clusters["final_response_mismatch"]) == 3 + assert report.total_failed == 4 diff --git a/examples/optimization/eval_optimize_loop/tests/test_optimizer_config.py b/examples/optimization/eval_optimize_loop/tests/test_optimizer_config.py new file mode 100644 index 000000000..787683f6a --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_optimizer_config.py @@ -0,0 +1,43 @@ +"""optimizer.json / gate_metrics.json 两层 metric 策略校验。 + +Stage 3 的 real 模式通过 ``AgentOptimizer.optimize(call_agent=...)`` 运行， +属于黑盒模式：call_agent 只返回最终文本，SDK 拿不到工具调用轨迹和 +``intermediate_data``。因此 optimizer.json 的 evaluate.metrics 不能包含 +依赖 trace 的 metric，否则 SDK 在启动时 fail-fast 抛 ValueError。 + +而 gate_metrics.json 走 trace 模式（Stage 1/4），预录制的 evalset 自带轨迹， +所以可以同时使用 final_response_avg_score 与 tool_trajectory_avg_score。 + +两层关系：gate_metrics 是 optimizer_metrics 的超集 —— gate 能看到 optimizer +看不到的退化信号（工具调用序列）。 +""" + +import json +from pathlib import Path + +from trpc_agent_sdk.evaluation._agent_optimizer import ( + _DISALLOWED_METRICS_IN_CALL_AGENT_MODE, +) + +DATA_DIR = Path(__file__).parent.parent / "data" + + +def test_optimizer_metrics_compatible_with_call_agent_mode(): + """optimizer.json 不得配置黑盒模式无法采集的 metric。""" + config = json.loads((DATA_DIR / "optimizer.json").read_text(encoding="utf-8")) + names = {m["metric_name"] for m in config["evaluate"]["metrics"]} + + assert not (names & _DISALLOWED_METRICS_IN_CALL_AGENT_MODE) + assert names, "optimizer.json 至少需要保留一个可用于优化的 metric" + + +def test_gate_metrics_keeps_trajectory_metric(): + """gate_metrics.json 走 trace 模式，应保留 tool_trajectory_avg_score。 + + 两层 metric 策略的核心断言：门控配置可以（也应该）保留 optimizer + 不允许的 metric —— 因为门控跑的是 trace 模式，SDK 能拿到工具调用轨迹。 + """ + config = json.loads((DATA_DIR / "gate_metrics.json").read_text(encoding="utf-8")) + names = {m["metric_name"] for m in config["metrics"]} + + assert "tool_trajectory_avg_score" in names diff --git a/examples/optimization/eval_optimize_loop/tests/test_pipeline_fake.py b/examples/optimization/eval_optimize_loop/tests/test_pipeline_fake.py new file mode 100644 index 000000000..5825f263f --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_pipeline_fake.py @@ -0,0 +1,272 @@ +"""全流水线集成测试（demo/fake 模式）。 + +本测试模块在 demo 模式下运行完整的 6 阶段流水线,验证: + 1. 流水线能否正常完成 + 2. 各阶段产出是否符合预期 + 3. 门控决策是否正确拒绝退化 + 4. 报告文件是否正确写入 + +所有测试使用预录制的 trace evalset 数据,无需 API key。 +""" + +import json +import os +import time +from pathlib import Path + +import pytest + +from pipeline._eval_backend import TraceBackend +from pipeline._models import AcceptanceGateConfig, PipelineReport +from pipeline._runner import PipelineRunner +from run_pipeline import derive_scenario + +# ---- 测试数据路径 ---- +DATA_DIR = Path(__file__).parent.parent / "data" +TRACE_DIR = DATA_DIR / "trace" +PROMPT_PATH = Path(__file__).parent.parent / "agent" / "prompts" / "system.md" + + +def _build_scenario_map_from_eval(eval_path: Path) -> dict[str, str]: + """读取 evalset.json, 用真实 derive_scenario() 推导每个 case 的场景。 + + 替代旧的本地 40 条 SCENARIO_MAP 副本 — 现在它直接跟踪 run_pipeline 的实现, + 派生规则变化时测试随之更新。 + """ + cases = json.loads(eval_path.read_text(encoding="utf-8"))["eval_cases"] + return {c["eval_id"]: derive_scenario(c["eval_id"]) for c in cases} + + +@pytest.fixture +def trace_scenario_map() -> dict[str, str]: + """用真实 derive_scenario() 从 val_baseline.evalset.json 推导的场景映射。""" + return _build_scenario_map_from_eval(TRACE_DIR / "val_baseline.evalset.json") + + +@pytest.fixture +def gate_config(): + """默认门控配置:禁止回归,提升阈值=0。""" + return AcceptanceGateConfig( + min_improvement_threshold=0.0, + no_new_hard_failures=True, + max_regressions_allowed=0, + critical_case_ids=[], + max_cost_budget=0.0, + ) + + +@pytest.fixture +def pipeline_runner(tmp_path, gate_config, trace_scenario_map): + """创建 demo 模式的 PipelineRunner 实例。 + + 使用 tmp_path 作为输出目录,测试结束后自动清理。 + """ + return PipelineRunner( + train_eval_path=str(TRACE_DIR / "train.evalset.json"), + val_baseline_eval_path=str(TRACE_DIR / "val_baseline.evalset.json"), + val_candidate_eval_path=str(TRACE_DIR / "val_optimized.evalset.json"), + gate_metrics_config_path=str(DATA_DIR / "gate_metrics.json"), + optimizer_config_path=str(DATA_DIR / "optimizer.json"), + prompt_source_path=str(PROMPT_PATH), + prompt_field_name="system_prompt", + gate_config=gate_config, + backend=TraceBackend(), + demo_mode=True, + output_dir=str(tmp_path), + scenario_map=trace_scenario_map, + demo_optimize_result_path=str(DATA_DIR / "demo_optimize_result.json"), + ) + + +class TestPipelineFakeMode: + """全流水线集成测试(demo 模式)。 + + 验证完整的 6 阶段流水线在 demo 模式下正确运行, + 各阶段产出符合预期,门控决策正确。 + """ + + async def test_full_pipeline_completes(self, pipeline_runner): + """流水线应在 2 分钟内无错误完成。""" + start = time.time() + report = await pipeline_runner.run() + elapsed = time.time() - start + + assert isinstance(report, PipelineReport) + assert report.demo_mode is True + assert elapsed < 120 # 2 分钟内完成 + + async def test_baseline_generated(self, pipeline_runner): + """基线评测应为训练集和验证集生成结果。 + + 预期: + - 训练集 20 cases: 15 失败(9 optimizable + 6 ineffective),5 通过 + - 验证集 20 cases: 15 失败(10 optimizable + 5 ineffective),5 通过 + """ + report = await pipeline_runner.run() + + assert report.baseline_train is not None + assert report.baseline_val is not None + assert report.baseline_train.num_cases == 20 + assert report.baseline_val.num_cases == 20 + # 训练集: 9 optimizable + 6 ineffective = 15 失败, 5 working = 5 通过 + assert report.baseline_train.num_failed == 15 + assert report.baseline_train.num_passed == 5 + # 验证集: 10 optimizable + 5 ineffective = 15 失败, 5 regression/working = 5 通过 + assert report.baseline_val.num_failed == 15 + assert report.baseline_val.num_passed == 5 + + async def test_failure_attribution_generated(self, pipeline_runner): + """失败归因应正确识别失败类别并聚类。 + + 预期:40 个 case 中 30 个失败,分布在多个类别中。 + """ + report = await pipeline_runner.run() + + assert report.failure_attribution is not None + # 训练集20 + 验证集20 = 40 个 case + assert report.failure_attribution.total_cases_evaluated == 40 + # 训练集15失败 + 验证集15失败 = 30 失败 + assert report.failure_attribution.total_failed == 30 + assert len(report.failure_attribution.clusters) > 0 + assert len(report.failure_attribution.summary) > 0 + + async def test_optimization_execution_generated(self, pipeline_runner): + """优化执行应从 demo JSON 加载预生成结果。""" + report = await pipeline_runner.run() + + assert report.optimization_execution is not None + assert report.optimization_execution.algorithm == "gepa_reflective" + assert report.optimization_execution.status == "SUCCEEDED" + assert report.optimization_execution.total_rounds > 0 + assert len(report.optimization_execution.best_prompts) > 0 + + async def test_candidate_validation_with_deltas(self, pipeline_runner): + """候选验证应生成逐 case delta 对比。 + + 预期: + - 20 个 case delta + - 10 个 FAILED->PASSED(optimizable 场景) + - 5 个 FAILED->FAILED(ineffective 场景) + - 5 个 PASSED->FAILED(regression 场景) + """ + report = await pipeline_runner.run() + + assert report.candidate_validation is not None + # 验证集 20 条 case 都有 delta + assert len(report.case_deltas) == 20 + + # val_001 优化成功: FAILED -> PASSED + val_001 = next(d for d in report.case_deltas if d.eval_id == "val_001_optimizable") + assert val_001.transition == "FAILED->PASSED" + assert val_001.scenario == "optimizable_success" + + # val_002 优化无效: FAILED -> FAILED + val_002 = next(d for d in report.case_deltas if d.eval_id == "val_002_ineffective") + assert val_002.transition == "FAILED->FAILED" + assert val_002.scenario == "optimization_ineffective" + + # val_003 过拟合退化: PASSED -> FAILED + val_003 = next(d for d in report.case_deltas if d.eval_id == "val_003_regression") + assert val_003.transition == "PASSED->FAILED" + assert val_003.scenario == "optimization_regression" + + # 统计各类 transition 数量 + transitions = [d.transition for d in report.case_deltas] + assert transitions.count("FAILED->PASSED") == 10 # 10 optimizable cases + assert transitions.count("FAILED->FAILED") == 5 # 5 ineffective cases + assert transitions.count("PASSED->FAILED") == 5 # 5 regression cases + + async def test_gate_rejects_regression(self, pipeline_runner): + """门控应因检测到回归而拒绝候选 prompt。 + + 由于 val_003_regression 等回归 case 的存在,候选应被拒绝。 + """ + report = await pipeline_runner.run() + + assert report.gate_decision is not None + assert report.gate_decision.accepted is False + assert report.overall_verdict == "REJECTED" + assert "val_003_regression" in report.gate_decision.regressed_case_ids + + async def test_gate_checks_present(self, pipeline_runner): + """门控决策应包含所有配置的检查项。""" + report = await pipeline_runner.run() + + assert report.gate_decision is not None + assert len(report.gate_decision.checks) >= 4 + + check_names = {c.check_name for c in report.gate_decision.checks} + assert "improvement_threshold" in check_names + assert "regression_check" in check_names or "no_new_hard_failures" in check_names + assert "critical_cases" in check_names + assert "cost_budget" in check_names + + async def test_reports_written_to_output(self, pipeline_runner, tmp_path): + """JSON 和 Markdown 报告应写入输出目录。""" + await pipeline_runner.run() + + # PipelineRunner.run() 在 output_dir 下创建 / 子目录, + # 报告位于 runner.run_dir。 + assert pipeline_runner.run_dir is not None + run_dir = str(pipeline_runner.run_dir) + json_path = os.path.join(run_dir, "optimization_report.json") + md_path = os.path.join(run_dir, "optimization_report.md") + + assert os.path.exists(json_path) + assert os.path.exists(md_path) + + # 验证 JSON 内容结构 + with open(json_path, "r") as f: + data = json.load(f) + assert "pipelineVersion" in data or "pipeline_version" in data + assert "overallVerdict" in data or "overall_verdict" in data + + # 验证 Markdown 内容结构 + with open(md_path, "r") as f: + content = f.read() + assert "# 优化报告" in content + assert "## 总体判决" in content + + # 兼容原始约定:旧版本把报告直接写在 tmp_path 下。 + # 这里仅作健全性检查(子目录结构需要存在)。 + assert str(tmp_path) in run_dir + + async def test_gate_accepts_when_allow_regressions( + self, pipeline_runner, tmp_path, trace_scenario_map + ): + """当允许回归时,回归检查应通过。 + + 创建独立的 runner 实例,关闭 no_new_hard_failures 并允许大量回归。 + 注意:由于 demo 数据中 pass rate 无改善,整体仍可能被拒绝, + 但回归检查本身应通过。 + """ + runner = PipelineRunner( + train_eval_path=str(TRACE_DIR / "train.evalset.json"), + val_baseline_eval_path=str(TRACE_DIR / "val_baseline.evalset.json"), + val_candidate_eval_path=str(TRACE_DIR / "val_optimized.evalset.json"), + gate_metrics_config_path=str(DATA_DIR / "gate_metrics.json"), + optimizer_config_path=str(DATA_DIR / "optimizer.json"), + prompt_source_path=str(PROMPT_PATH), + prompt_field_name="system_prompt", + gate_config=AcceptanceGateConfig( + min_improvement_threshold=0.0, + no_new_hard_failures=False, # 关闭新增失败禁止 + max_regressions_allowed=999, # 允许大量回归 + critical_case_ids=[], + max_cost_budget=0.0, + ), + backend=TraceBackend(), + demo_mode=True, + output_dir=str(tmp_path), + scenario_map=trace_scenario_map, + demo_optimize_result_path=str(DATA_DIR / "demo_optimize_result.json"), + ) + + report = await runner.run() + + # 回归检查应通过(允许回归) + regression_check = next( + c for c in report.gate_decision.checks + if c.check_name == "regression_check" + ) + assert regression_check.passed is True diff --git a/examples/optimization/eval_optimize_loop/tests/test_scenario_derivation.py b/examples/optimization/eval_optimize_loop/tests/test_scenario_derivation.py new file mode 100644 index 000000000..27167dbda --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_scenario_derivation.py @@ -0,0 +1,11 @@ +"""scenario 由 eval_id 后缀推导, 不维护映射表.""" + +from run_pipeline import derive_scenario + + +def test_derive_scenario_from_suffix(): + assert derive_scenario("val_001_optimizable") == "optimizable_success" + assert derive_scenario("train_002_ineffective") == "optimization_ineffective" + assert derive_scenario("val_003_regression") == "optimization_regression" + assert derive_scenario("train_003_working") == "optimization_regression" + assert derive_scenario("foo_bar") == "unknown" diff --git a/tests/evaluation/test_eval_criterion_ext.py b/tests/evaluation/test_eval_criterion_ext.py index c52f6f1c4..d8157e3db 100644 --- a/tests/evaluation/test_eval_criterion_ext.py +++ b/tests/evaluation/test_eval_criterion_ext.py @@ -36,6 +36,13 @@ def test_regex_case_insensitive(self): c = TextCriterion(match="regex", case_insensitive=True) assert c.matches("Hello World", r"hello") is True + def test_regex_case_insensitive_keeps_character_classes(self): + r"""case_insensitive must not mangle \S in the pattern ([\s\S]* -> [\s\s]*).""" + c = TextCriterion(match="regex", case_insensitive=True) + pat = r"found\ 2\ headphones[\s\S]*Sony\ WH\-1000XM6\ 售价\ ¥2,499" + assert c.matches("Found 2 headphones：Sony WH-1000XM6 售价 ¥2,499", pat) is True + assert c.matches("found 2 headphones, no brands", pat) is False + class TestTextCriterionCompare: """Test suite for TextCriterion custom compare.""" diff --git a/trpc_agent_sdk/evaluation/_agent_evaluator.py b/trpc_agent_sdk/evaluation/_agent_evaluator.py index a2e47009d..b00b7d470 100644 --- a/trpc_agent_sdk/evaluation/_agent_evaluator.py +++ b/trpc_agent_sdk/evaluation/_agent_evaluator.py @@ -523,10 +523,10 @@ async def evaluate_eval_set( raise ValueError("call_agent is mutually exclusive with runner.") trace_only = AgentEvaluator._is_trace_only(eval_set) - if call_agent is None and agent_module is None and not trace_only: + if call_agent is None and agent_module is None and runner is None and not trace_only: non_trace_ids = [case.eval_id for case in eval_set.eval_cases if case.eval_mode != EvalModeTrace] - raise ValueError("`agent_module` is required unless every case in eval_set uses " - "eval_mode='trace'. Non-trace case ids: " + raise ValueError("`agent_module`, `call_agent`, or `runner` is required unless every " + "case in eval_set uses eval_mode='trace'. Non-trace case ids: " f"{non_trace_ids}") agent_for_eval: Optional[BaseAgent] = None diff --git a/trpc_agent_sdk/evaluation/_eval_criterion.py b/trpc_agent_sdk/evaluation/_eval_criterion.py index 6e4cbc24d..48ef22cda 100644 --- a/trpc_agent_sdk/evaluation/_eval_criterion.py +++ b/trpc_agent_sdk/evaluation/_eval_criterion.py @@ -75,18 +75,23 @@ def matches(self, actual: str, expected: str) -> bool: e = expected if expected is not None else "" if self.compare is not None: return self.compare(a, e) + if self.match == "regex": + import re + # Use IGNORECASE instead of lower()ing the pattern: lower() breaks + # character-class escapes, e.g. [\s\S]* becomes [\s\s]* so only + # whitespace can be skipped and punctuation-separated key facts + # never match. + flags = re.IGNORECASE if self.case_insensitive else 0 + try: + return bool(re.search(e, a, flags)) + except re.error: + return False if self.case_insensitive: a, e = a.lower(), e.lower() if self.match == "exact": return a == e if self.match == "contains": return e in a - if self.match == "regex": - import re - try: - return bool(re.search(e, a)) - except re.error: - return False return a == e @classmethod diff --git a/trpc_agent_sdk/evaluation/_local_eval_service.py b/trpc_agent_sdk/evaluation/_local_eval_service.py index 2ea1a8d76..b87760d71 100644 --- a/trpc_agent_sdk/evaluation/_local_eval_service.py +++ b/trpc_agent_sdk/evaluation/_local_eval_service.py @@ -635,10 +635,10 @@ async def _perform_inference_single_eval_item( raise ValueError( f"inference eval case (eval_case_id={eval_case.eval_id}, session_id={session_id}): " "actual_conversation is only supported in trace mode") - if root_agent is None: + if root_agent is None and self._runner is None: raise ValueError(f"inference eval case (eval_case_id={eval_case.eval_id}, " - f"session_id={session_id}): a root_agent is required for " - f"standard (non-trace) eval_mode; got root_agent=None") + f"session_id={session_id}): a root_agent or runner is required for " + f"standard (non-trace) eval_mode; got both root_agent=None and runner=None") inferences = await self._generate_inferences_from_agent( agent=root_agent, eval_case=eval_case,