Proposal: an EvalPort/OpenEval adapter for LettuceDetect's span-level output

[adhabnr-ux]

Hi — I maintain EvalPort, an open, portable schema for LLM eval/grading results (openeval.validate.validate_suite() / validate_result_set()). It's meant to be a boring, tool-agnostic JSON shape so a result produced by one eval/grading tool can be read or compared by another.

Before opening this I read through lettucedetect/detectors/base.py and lettucedetect_api/models.py to check LettuceDetect actually has a real structured result, not just a score float:

• BaseDetector.predict() / predict_prompt() return a "spans"-format list of span dicts (start, end, text, confidence), filterable via min_confidence (_filter_spans_by_confidence in base.py).
• lettucedetect_api/models.py formalizes this as pydantic models — SpanDetectionItem (start: int, end: int, text: str, hallucination_score: float) and TokenDetectionItem, wrapped in SpanDetectionResponse/TokenDetectionResponse.

That's a genuine span-level grounding-verification result object — exactly the shape an OpenEval adapter is meant to sit on top of (a span/claim-level annotation with a confidence score, rather than a single aggregate number).

What I'm proposing: a small adapter (could live in lettucedetect/integrations/ alongside the existing LangChain callback integration, or in EvalPort itself) that converts a batch of SpanDetectionItem/span-dict results into an OpenEval ResultSet that passes validate_result_set() — mapping each detected span to a graded item with its confidence as the score. I'd be happy to draft a first-cut PR if that's useful, or just help spec the mapping if someone on your side wants to build it — this is meant as a low-effort invitation, not a request.

Repo/spec: https://github.com/adhabnr-ux/evalport

[adaamko]

Thanks for doing the homework before opening this — you read the result shape correctly. Two additions to it: the Python path also carries category/subcategory on each span when a typed detector runs (the taxonomy cascade or the generative model), and the HTTP models currently drop those fields — #75 tracks fixing that, so an adapter should map the optional typed fields too, not only the confidence.

On where it lives: I'd start the adapter in EvalPort, consuming our documented span format (start, end, text, confidence, optional category/subcategory). Our integrations directory holds things that change how LettuceDetect runs inside another system; a result-format adapter is the other direction and belongs with the schema, where it evolves with the schema. We're glad to link it from the integrations README once it exists, and if it turns out people actually route span results through EvalPort, hosting it here later is a cheap decision.

One mapping question worth settling in the spec: a clean answer produces zero spans, which is the most important result for eval purposes — make sure the ResultSet represents "checked, nothing flagged" per answer rather than only emitting graded items for flagged spans. Happy to review a draft.