Portable EvalPort output alongside compute_scores.py (TestCase/Result mapping for prediction files)

[adhabnr-ux]

Portable EvalPort output alongside router_evaluation/compute_scores.py

I maintain EvalPort, an open interchange format for eval data (TestCase/Grader/Result/ResultSet/GraderResult). Looked through router_inference/predictions/*.json, llm_evaluation/run.py, and router_evaluation/compute_scores.py — RouterArena already produces exactly the per-query record EvalPort's Result type is built for, it's just shaped as a router-specific dict:

{"global index": ..., "prompt": ..., "prediction": "gpt-4o-mini",
 "generated_result": {"success": true, "generated_answer": ..., "token_usage": {...}, "model_used": ...},
 "accuracy": 0.0-1.0, "cost": <float>, "for_optimality": false}

That maps cleanly onto EvalPort's schema (using the real field names from sdk/python/openeval/types.py, not a hypothetical mapping):

RouterArena field	EvalPort field
global index	Result.test_case_id / TestCase.id
prompt	TestCase.input
generated_result.generated_answer	Result.actual_output
accuracy (computed by the dataset's scorer in eval_reasoning.py)	GraderResult.score
accuracy >= 1.0	Result.passed / GraderResult.passed
cost	metadata.openeval.cost.estimated_cost_usd — this is a reserved key already in the spec specifically for per-result cost tracking, so it's not even a stretch
prediction (selected model), for_optimality	Result.metadata.routerarena.* (namespaced, non-normative)

Since accuracy/cost are dataset-specific (eval_reasoning.get_scorers_for_dataset), the natural grader shape is EvalPort's custom type with a params.handler pointing back at RouterArena's own scorer — the same escape hatch the shipped ragas-openeval-adapter uses for Ragas's per-metric scores it doesn't want to re-implement.

Rough sketch of what a routerarena-openeval-adapter converter would look like (using evalport-sdk's actual dataclasses):

from openeval.types import OPENEVAL_VERSION, GraderResult, Result, ResultSet

def to_openeval(predictions: list[dict], router_name: str, split: str) -> ResultSet:
    """Convert a router_inference/predictions/<router>.json (post llm_evaluation/run.py)
    into an EvalPort ResultSet. Optimality entries are excluded — same rule
    llm_evaluation/run.py's compute_router_metrics() already applies."""
    results = []
    for p in predictions:
        if p.get("for_optimality"):
            continue
        idx = p.get("global index") or p.get("global_index")
        accuracy = p.get("accuracy")
        generated = p.get("generated_result") or {}
        passed = bool(accuracy is not None and accuracy >= 1.0)
        results.append(Result(
            test_case_id=str(idx),
            passed=passed,
            grader_results=[GraderResult(
                grader_id="gr_accuracy", type="custom", score=accuracy, passed=passed,
                metadata={"handler": "routerarena:accuracy"},
            )],
            actual_output=generated.get("generated_answer"),
            metadata={
                "openeval": {"cost": {"estimated_cost_usd": p.get("cost")}},
                "routerarena": {
                    "selected_model": p.get("prediction"),
                    "model_used": generated.get("model_used"),
                },
            },
        ))
    return ResultSet(
        version=OPENEVAL_VERSION, suite_id=f"routerarena_{split}",
        run_id=router_name, started_at="...", results=results,
    )

Why this might be worth having: a portable ResultSet means anyone comparing routers can point a generic EvalPort viewer/aggregator at RouterArena output next to results from DeepEval, Inspect AI, etc., without writing bespoke parsers — same motivation as the leaderboard's own compute_scores.py, just interoperable beyond this repo.

Happy to build this as a standalone routerarena-openeval-adapter package (same pattern as the Ragas/AutoGen/CrewAI adapters — lives outside this repo, reads your public prediction-file shape, doesn't touch RouterArena's own code) if it's useful, or just leave this as a reference for anyone who wants to bolt on EvalPort export themselves. Let me know either way.

— Sahi, independent contributor (not affiliated with this project)