feat(eval): ondemand simulate — replay a dataset, evaluate synchronously - #2071

Open

jariy17 wants to merge 6 commits into

feat/eval-ondemand-simulate

Commits on Aug 27, 2026

feat(eval): ondemand simulate handler (batch pattern: ingestion-wait-ms, failures/sessions)
jariy17
committed
749f8a8
test(eval): ondemand simulate edge tests + register under ondemand
jariy17
committed
452d430
test(eval): ondemand simulate fixture golden + clock seam; strip handler comments
jariy17
committed
c29ad31

Commits on Aug 28, 2026

fix(eval): ondemand simulate — use withUserCancellation; forward turn expectedResponse as a reference input
jariy17
committed
a2617b7
fix(eval): ondemand simulate — expectedResponse is trace-level (correlate turn→traceId)
jariy17
committed
ac5d4e1
test(eval): dedupe golden session ids via uniquePerRecording helper (fix stale-trace pollution)
jariy17
committed
1b0ca37