feat(project): add `project add evaluator code-based` - #2144

Draft

jariy17 wants to merge 18 commits into

feat/project-add-evaluator-code-based

Commits on Aug 27, 2026

feat(eval): ondemand simulate handler (batch pattern: ingestion-wait-ms, failures/sessions)
jariy17
committed
749f8a8
test(eval): ondemand simulate edge tests + register under ondemand
jariy17
committed
452d430
test(eval): ondemand simulate fixture golden + clock seam; strip handler comments
jariy17
committed
c29ad31

Commits on Aug 28, 2026

feat(eval): add ab-test config-bundle run (create)
jariy17
authored
95a1b67
fix(eval): address ab-test config-bundle run review
jariy17
authored
ac398b8
feat(eval): use --enable-on-create for ab-test config-bundle run
jariy17
authored
653dc9f
refactor(eval): reuse retryWhileRolePropagates for ab-test create
jariy17
authored
0ac6754
test(eval): golden fixture for ab-test config-bundle run
jariy17
authored
f325a1a
test(eval): consolidate ab-test command-flow + unhappy paths into one file
jariy17
authored
f09cea0
test(eval): fold config-bundle run golden into ab-test.fixture.test.tsx
jariy17
authored
0f31b84
feat(eval): add ab-test target-based run (create)
jariy17
authored
2c9274e
Merge remote-tracking branch 'origin/feat/eval-ondemand-simulate' into refactor
jariy17
committed
045e081
Merge remote-tracking branch 'origin/feat/eval-ab-test-config-bundle-run' into refactor
jariy17
committed
12c078c
Merge remote-tracking branch 'origin/feat/eval-ab-test-target-based' into refactor
jariy17
committed
7c158df
feat(project): add `project add evaluator code-based`
jariy17
committed
268305a
fix(project): guard against app/<name> collisions when scaffolding evaluators
jariy17
committed
d8eb522
fix(project): validate --metric class and require a Bedrock --model for code-based evaluators
jariy17
committed
821b337
fix(project): echo inferred mode caveats for code-based evaluators
jariy17
committed
303c8db