Proposal: EvalPort import/export helper under lib/integrations

[adhabnr-ux]

Hi Openlayer team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema plus a validator, so a dataset or a graded run can move between eval tools without a bespoke converter per pair). Opening this as an issue first rather than jumping straight to a PR, since it's a new integration surface.

I installed openlayer (from PyPI) and read the actual generated types rather than guessing at the shape. openlayer.types.test_list_results_response maps onto EvalPort's Suite/ResultSet split more cleanly than I expected:

class ItemGoal(BaseModel):
    id: str
    name: str
    subtype: Literal["llmRubricThresholdV2", "stringValidation", "containsPii",
                      "customMetricThreshold", "metricThreshold", ...]   # ~40 variants
    type: Literal["integrity", "consistency", "performance"]
    thresholds: List[ItemGoalThreshold]
    ...

class ItemGoalThreshold(BaseModel):
    insight_name: Optional[str]
    measurement: Optional[str]
    operator: Optional[Literal["is", ">", ">=", "<", "<=", "!="]]
    value: Union[float, bool, str, List[str], None]

class Item(BaseModel):          # a goal's evaluation result
    id: str
    status: Literal["running", "passing", "failing", "skipped", "error"]
    status_message: Optional[str]
    goal: Optional[ItemGoal]
    expected_values: Optional[List[ItemExpectedValue]]
    rows_body: Optional[ItemRowsBody]

An Openlayer Goal is close to an EvalPort grader — subtype is the grader type, thresholds (operator + value) is the grader's params/threshold config. An Openlayer Item (a goal's run result, with its status of passing/failing/skipped/error) maps directly onto an EvalPort Result entry. So:

• to_openeval(goal: ItemGoal) -> dict — build an EvalPort grader definition from a Goal's subtype/thresholds.
• from_openeval(result_set: dict) -> list[Item-shaped dicts] — take a spec-valid EvalPort ResultSet and produce Openlayer-shaped result records (status mapped from EvalPort's pass/fail/error/skip, goal_id cross-referenced).

I noticed CONTRIBUTING.md already carves out src/openlayer/lib/ as generator-exempt, and openlayer.lib.integrations already holds hand-written per-tool integrations (trace_openai, trace_portkey, etc.) alongside the generated client — so a new openlayer.lib.integrations.openeval module with to_openeval()/from_openeval() seems like the natural, low-risk landing spot: it wouldn't touch anything generated, and it'd sit next to the existing integrations rather than inventing a new pattern.

If that's a reasonable direction, I'm happy to build it — tests would validate real output against EvalPort's actual JSON Schema (spec/schemas/testcase.json/resultset.json) via the evalport-sdk validator, not a mock, matching the bar the rest of the EvalPort ecosystem holds to. Also happy to build it as a fully standalone openlayer-openeval-adapter package living in the EvalPort repo instead (depending on openlayer as a normal dependency) if you'd rather keep this SDK's surface area untouched — no strong preference on my end, whichever is less overhead for you to maintain.

Let me know which you'd prefer, or if this isn't useful for where Openlayer is headed — either way, thanks for taking a look.