构建 Evaluation + Optimization 的自动回归与提示词优化闭环 - #274

Open

rhljj wants to merge 26 commits into

trpc-group:mainfrom

rhljj:main

Commits on Jul 31, 2026

feat(pipeline): plumb audit fields and isolate optimizer output to subdir
rhljj
committed
feat(pipeline): failure attribution accepts per_case dict
rhljj
committed
refactor(pipeline): runner adopts EvalBackend and timestamp subdir
rhljj
committed
refactor(pipeline): drop dead counter and tighten per_case typing in runner
rhljj
committed
feat(pipeline): audit trail renders stop_reason, rounds, and token usage
rhljj
committed
refactor(data): split evalsets into trace/ and live/ subdirs
rhljj
committed
refactor(data): dedupe expected-turn builders and tighten regressions
rhljj
committed
fix(pipeline): stage 4 evaluates the candidate evalset, not the baseline
rhljj
committed
refactor(data): rename test_config.json to gate_metrics.json (gate layer)
rhljj
committed
feat(data): update optimizer config (pareto, max_metric_calls=150, timeout, skip_perfect)
rhljj
committed
refactor(cli): assemble backend, derive scenario from suffix
rhljj
committed
fix(cli): drop eager factory wrapper, expose run_dir, update stale docstring
rhljj
committed
test: update pipeline fake test paths to trace/ and gate_metrics

rhljj
and
claude
committed
test: update optimizer config test to reference gate_metrics.json

rhljj
and
claude
committed
test: add regression-gate test suite (7 new test files)

rhljj
and
claude
committed
docs: add EvalBackend protocol test to README consistency guards

rhljj
and
claude
committed
feat: update eval_optimize_loop example and evaluation module

rhljj
and
claude
committed
merge: combine local eval_optimize_loop work with upstream main
rhljj
committed
fix(pipeline): honor max_regressions_allowed override, narrow assertion swallow

rhljj
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

构建 Evaluation + Optimization 的自动回归与提示词优化闭环 - #274

构建 Evaluation + Optimization 的自动回归与提示词优化闭环#274
rhljj wants to merge 26 commits into
trpc-group:mainfrom
rhljj:main

Commits on Jul 30, 2026

snapshot: pre-SDD baseline (working state with broken real mode)

feat(pipeline): add EvalBackend Protocol with TraceBackend

feat(pipeline): add LiveBackend for real-mode evaluation

refactor(pipeline): narrow exception handling and delegate to EvalBackend

feat(pipeline): add applied_prompts context manager for candidate write-back

refactor(pipeline): stage 4 writes best_prompts via applied_prompts

feat(pipeline): extend OptimizationExecutionReport with audit fields

Commits on Jul 31, 2026

feat(pipeline): plumb audit fields and isolate optimizer output to subdir

feat(pipeline): failure attribution accepts per_case dict

refactor(pipeline): runner adopts EvalBackend and timestamp subdir

refactor(pipeline): drop dead counter and tighten per_case typing in runner

feat(pipeline): audit trail renders stop_reason, rounds, and token usage

refactor(data): split evalsets into trace/ and live/ subdirs

refactor(data): dedupe expected-turn builders and tighten regressions

fix(pipeline): stage 4 evaluates the candidate evalset, not the baseline

refactor(data): rename test_config.json to gate_metrics.json (gate layer)

feat(data): update optimizer config (pareto, max_metric_calls=150, timeout, skip_perfect)

refactor(cli): assemble backend, derive scenario from suffix

fix(cli): drop eager factory wrapper, expose run_dir, update stale docstring

test: update pipeline fake test paths to trace/ and gate_metrics

test: update optimizer config test to reference gate_metrics.json

test: add regression-gate test suite (7 new test files)

docs: add EvalBackend protocol test to README consistency guards

feat: update eval_optimize_loop example and evaluation module

merge: combine local eval_optimize_loop work with upstream main

fix(pipeline): honor max_regressions_allowed override, narrow assertion swallow