EvalPort interop: PairedScore/EvalRecord → TestCase/ResultSet

[adhabnr-ux]

EvalPort interop: PairedScore/EvalRecord → EvalPort TestCase/ResultSet

Hi — I maintain EvalPort, an Apache-2.0 open
spec (with Python/TS SDKs) for portable LLM eval test cases, graders, suites, and
results. I read through src/evalshift/evaluators/base.py and the README, and wanted
to flag a concrete interop path rather than a vague "you should support X".

What I'm not proposing: EvalPort has no native notion of a paired source/target
comparison today — its ResultSet is the output of one suite run against one model.
So I'm not claiming PairedScore already "maps onto" something that exists. What I
think does map cleanly, without inventing anything, is the golden-suite side:

• EvalShift's golden examples (prompt_id, example_id, input vars, expected output,
  and — for the agent-migration path — the recorded toolset) line up almost 1:1 with
  EvalPort's TestCase schema:
  id, input, expected_output, tools_called, expected_tools. That last pair
  exists specifically for agent/tool-call evaluation, which is the same thing your
  README calls out as the killer scenario (an agent silently dropping
  notify_security_team after a migration, text-eval green, tool-call eval CRITICAL).

• On the output side, each half of a pair — the source-model run and the
  target-model run — is a normal EvalPort
  ResultSet:
  results[].grader_results[] already carries grader_id, type, score, passed,
  reason. PairedScore.source_score / PairedScore.target_score are just two of
  those, one per model run, keyed by the same test_case_id. PairedScore.delta and
  EvalRecord.blocking are genuinely EvalShift-specific — they belong to your
  analysis layer (Cohen's d, paired tests, BH correction across prompt×evaluator×slice),
  not to a generic result format, and I wouldn't want to force them into the spec.

Rough sketch, using your real field names, of what an adapter would look like (this
is illustrative, not a PR):

# evalshift's PairedScore/EvalRecord -> one EvalPort grader_result per side of the pair
from openeval.validate import validate_suite  # evalport-sdk (Python)

def to_grader_result(rec: "EvalRecord", side: str) -> dict:
    score = rec.source_score if side == "source" else rec.target_score
    return {
        "grader_id": rec.evaluator_name,
        "type": rec.kind or "custom",   # e.g. "semantic_similarity", "llm_judge"
        "score": score,
        "passed": rec.error is None,
        "reason": rec.explanation,
        "metadata": {**rec.metadata, "evalshift.run_id": rec.run_id, "evalshift.blocking": rec.blocking},
    }

# rec.delta / rec.blocking stay EvalShift-side — they're analysis output,
# not part of the portable result. Each model's ResultSet is standard EvalPort;
# the migration verdict is computed by diffing two of them.

Why this might be worth 20 minutes of thought rather than nothing: if EvalShift's
capture sync / golden suites spoke TestCase, and evalshift run/report could
optionally emit a ResultSet per model alongside your own scores.jsonl, the same
captured golden suite would become runnable through anything else that already
consumes EvalPort — for a concrete, verified sense of what "anything else" means
today, two of the 37 merged adapters are
ragas-openeval-adapter
(RAG-focused paired-style metrics, closest in spirit to your semantic-similarity
evaluator) and
mlflow-openeval-adapter
(run-artifact based, closest in spirit to your .evalshift/runs/<run-id>/ layout).
None of that requires EvalShift to change its own analysis or statistics — it's
purely an additional output format at the edges (capture sync in, report/bundle
out), so it wouldn't touch the AGPL-licensed statistical core.

If this isn't a direction you're interested in, no worries at all — I mostly wanted
to make sure the specific field-level claim was accurate rather than hand-wavy before
raising it. Happy to sketch a real converter if there's interest; I'd rather do that
as a follow-up than guess at scope here.

— Sahi, independent contributor (not affiliated with this project)

[babaliauskas]

Hi Sahi, thanks for taking the time to read the actual models rather than pitching from the README. The field names you cite on EvalRecord are all correct, and the agent scenario is the one we care most about.

I'm going to pass on adding an EvalPort emitter or importer to EvalShift itself for now. Two reasons.

1. The per-side scores are not portable on their own. You correctly keep delta and blocking out of the spec, but the raw source_score / target_score are also relative for most of our evaluators, not independent measurements of each model:

• Semantic similarity pins the source at 1.0 and gives the target its cosine similarity to the source. A standalone "source ResultSet" would be a file of constant 1.0s.
• LLM-as-judge is pairwise: (0, 1) means "target won", (0.5, 0.5) is a tie. A consumer reading only the source file would see "source scored 0" when it actually lost a head-to-head.
• Tool-call evaluators scored against captured expectations also pin the source at 1.0 by construction (see the provenance docstring on ExpectedToolCall).

Only the structural evaluators score each side independently. So "each half of a pair is a normal ResultSet" holds for a minority of rows, and the rest would read as a perfect source run over a comparison that never happened. We've been bitten by exactly that class of fabricated score before, which is why PairedScore is the unit and not a per-model score. The pair itself is the thing that would need a home in the spec, and you've said EvalPort doesn't model that today.

passed: rec.error is None has a similar problem: error means the measurement broke, and there's no per-side pass/fail in EvalShift; verdicts come out of the paired statistics.

2. The suite side is lossier than it looks. TestCase requires per-case graders, which we configure globally in evalshift.yaml. Everything specific to the agent path (expected_tool_rounds, tool_result_fixtures, per-example toolsets, history, generation_config) would end up in metadata where no other consumer can act on it. And an import path doesn't fit the model: a TestCase has no source-model run to compare against.

Where an adapter could live. Every run writes scores.jsonl and report.json under .evalshift/runs/<run-id>/, and those are the stable machine-readable outputs. If you want to build an evalshift-openeval-adapter on your side the way you've done for the others, that needs no changes here, and I'd suggest restricting it to structural evaluator rows (kind on each record) or emitting a single ResultSet that carries both sides plus delta in metadata, so downstream readers aren't handed a synthetic source run.

If EvalPort picks up independent consumers I'm happy to revisit a native emitter. Thanks again for the careful write-up.

[adhabnr-ux]

Thanks for the genuinely detailed read, Lukas — this is exactly the kind of review that's more useful than a yes.

You're right on both counts, and I don't think there's a version of this that holds up. The pairwise-relative scoring (semantic similarity pinning source at 1.0, LLM-as-judge's (0,1)/(0.5,0.5) shape, tool-call provenance pinning) means a "source ResultSet" would misrepresent what actually happened for most of your evaluators, not just lose some metadata — that's a correctness problem, not a portability inconvenience, and I'd rather EvalPort not be the reason a false "source scored 0" ends up in someone's dashboard. Same read on TestCase.graders being per-case where EvalShift's is global — that's a real modeling mismatch, not just a rough edge.

Your suggestion is the right shape for an external adapter if I build one: read scores.jsonl/report.json, scope it to the structural evaluator rows where independent per-side scores are actually meaningful, and for anything else emit a single ResultSet carrying both sides plus delta in metadata rather than manufacturing a synthetic source run. I'll follow that if I get to it rather than reopening the emitter/importer question.

Appreciate you engaging with the actual code instead of waving it off — and the offer to revisit if independent consumers show up. No further ask from me here.