Skip to content

docs(ragas): link evaluation scores to application spans - #1078

Merged
nik13 merged 2 commits into
mainfrom
docs/ragas-evaluation-recipe
Nov 9, 2026
Merged

nik13 merged 2 commits into
mainfrom
docs/ragas-evaluation-recipe

Conversation

@nik13

@nik13 nik13 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add a Ragas evaluation guide using the published futureagi Scores SDK to attach numeric results to the original application span. Application tracing stays separate from evaluator execution: this is a Scores SDK recipe, not a new Ragas tracing instrumentor.

  • Use value={"value": value} for numeric annotations, preserve valid zero scores, and reject NaN/infinite results instead of converting failures to zero.
  • Use the local ragas.metrics.collections.ExactMatch metric without model-provider credentials. The request contains the span/label IDs and score, not the response or reference text. Ragas telemetry is disabled in the setup.
  • Pin langchain-community==0.4.1 alongside Ragas 0.4.3 because 0.4.2 removed a module that Ragas imports.
  • Merge current main 970056be6443bff2051fbee1101f5b99de4cda12, retaining its catalog entries and adding only the Ragas entry in the alphabetical framework section. No neighboring entries were reformatted.

Validation on head 443549beeefbe830dc8b7f33c1e0d92d344b41bf

All installs, tests and builds ran remotely in E2B or GitHub Actions; none ran locally.

Published packages and exact page recipe (E2B)

The page's installation, environment exports (with synthetic credential/ID values), Python block and python score_ragas.py command were extracted from the committed MDX rather than maintained as a second example. The documented install block ran verbatim on Python 3.10 and 3.13:

python -m venv .venv
source .venv/bin/activate
python -m pip install "ragas==0.4.3" "langchain-community==0.4.1" "futureagi==0.6.13"
python -I scripts/test_ragas_recipe.py -v

The Python 3.10 environment additionally exercised the supported Ragas lower bound:

python -m pip install "ragas==0.4.0" "langchain-community==0.4.1" "futureagi==0.6.13"
python -I scripts/test_ragas_recipe.py -v

Results:

Python Ragas Recipe tests
3.10 0.4.3 2 passed
3.10 0.4.0 2 passed
3.13 0.4.3 2 passed

The exact python score_ragas.py command also passed twice, on Python 3.10/Ragas 0.4.0 and Python 3.13/Ragas 0.4.3, each producing one numeric-score POST to the loopback Scores endpoint. Tests cover original-span attachment, zero preservation, NaN/positive infinity/negative infinity rejection, three invalid span IDs, API rejection propagation, and omission of answer text. The committed test denies non-loopback sockets.

The missing-page negative control reproduced main's absent-guide condition: the targeted test failed with Ragas evaluation recipe is missing when the page was temporarily removed in E2B. The committed head passed. This is a negative control, not a claim that the whole base branch was executed.

E2B receipt: qw-docs-1078-443549be, exit 0; sandbox destruction confirmed.

Docs checks

npm run test:integrations
npm run audit-links
npm run check:integrations
npm run build
npm run test:integrations:browser
  • E2B integration suite: 283 passed, 0 failed, 0 skipped.
  • E2B link audit: 858 pages, 823 navigation entries, 0 broken navigation links, 0 broken content links. The audit also reports 31 orphan pages; this is not a claim of zero warnings.
  • PR Checks: integration/link checks, catalog type check, docs build, Pagefind and browser regression checks; both inherited Helicone conformance jobs also pass.
  • Ragas matrix: Ragas 0.4.0/Python 3.10 and Ragas 0.4.3/Python 3.13 pass.
  • All 10 GitHub checks on this head finished successfully, including inherited FastMCP and Temporal Go jobs.

Scope and review

Cloud score persistence was not exercised. Loopback validation proves the published SDK's request/response shape and recipe behavior only; it does not prove Cloud storage or UI persistence. No traceAI package, backend, public API, migration or release-version changes. Privacy defaults are unchanged.

Internal-standards self-review found no merge-blocking defects. Product E2E coverage gate: not applicable (this repository has no e2e/ harness); docs browser checks and the executable recipe test provide the applicable coverage. The PR is non-draft; required human approval remains separate from this technical qualification. No merge or approval was performed.

🤖 Generated with Claude Code

nik13 and others added 2 commits October 31, 2026 20:41
Co-Authored-By: Claude Code <noreply@anthropic.com>
Retain all catalog integrations and place Ragas beside the alphabetical framework entries. Preserve the score recipe and compatibility pin unchanged.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13
nik13 merged commit a38cb42 into main Nov 9, 2026
10 checks passed
@nik13
nik13 deleted the docs/ragas-evaluation-recipe branch November 9, 2026 23:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant