Exporting trace validation results as portable EvalPort ResultSets

[adhabnr-ux]

Hi AgentOps team — I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open JSON-Schema spec for portable LLM eval test cases, graders, and result sets, with Python/TS SDKs that validate against the schemas.

I was reading agentops/validation.py and noticed validate_trace_spans() already does most of the work an eval result needs: it pulls a trace's spans, calls get_trace_metrics() for token/cost data, runs check_llm_spans() to detect LLM activity, and returns a structured dict (trace_id, span_count, has_llm_spans, llm_span_names, metrics, ...). That's close in shape to a single EvalPort ResultSet entry — it's mostly missing an explicit pass/fail grader wrapper around what's already collected.

Idea: a small agentops.evalport helper that wraps validate_trace_spans() output (or any custom scorer) into an EvalPort-conformant ResultSet (schema: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json). Roughly:

{
  "version": "1.0.0",
  "suite_id": "agentops-regression",
  "run_id": trace_id,
  "started_at": "...",
  "results": [{
    "test_case_id": trace_id,
    "grader_results": [{
      "grader_id": "has_llm_activity",
      "type": "code",
      "score": 1.0 if result["has_llm_spans"] else 0.0,
      "passed": result["has_llm_spans"]
    }],
    "passed": result["has_llm_spans"],
    "metadata": {
      "total_tokens": result["metrics"]["total_tokens"],
      "total_cost": result["metrics"]["total_cost"]
    }
  }]
}

That would give teams already using AgentOps for monitoring a portable, diffable regression artifact per run — importable by any EvalPort-compatible tool — without AgentOps needing to change how it tracks sessions internally.

Full spec for reference: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

Just flagging this as an idea since the pieces already line up so well in validation.py — no pressure at all if it's not a priority right now. Happy to sketch a draft PR if there's interest.