Repository navigation
docs(ragas): link evaluation scores to application spans - #1078
Merged
Merged
Conversation
Co-Authored-By: Claude Code <noreply@anthropic.com>
Retain all catalog integrations and place Ragas beside the alphabetical framework entries. Preserve the score recipe and compatibility pin unchanged. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a Ragas evaluation guide using the published
futureagiScores SDK to attach numeric results to the original application span. Application tracing stays separate from evaluator execution: this is a Scores SDK recipe, not a new Ragas tracing instrumentor.value={"value": value}for numeric annotations, preserve valid zero scores, and reject NaN/infinite results instead of converting failures to zero.ragas.metrics.collections.ExactMatchmetric without model-provider credentials. The request contains the span/label IDs and score, not the response or reference text. Ragas telemetry is disabled in the setup.langchain-community==0.4.1alongside Ragas 0.4.3 because 0.4.2 removed a module that Ragas imports.970056be6443bff2051fbee1101f5b99de4cda12, retaining its catalog entries and adding only the Ragas entry in the alphabetical framework section. No neighboring entries were reformatted.Validation on head
443549beeefbe830dc8b7f33c1e0d92d344b41bfAll installs, tests and builds ran remotely in E2B or GitHub Actions; none ran locally.
Published packages and exact page recipe (E2B)
The page's installation, environment exports (with synthetic credential/ID values), Python block and
python score_ragas.pycommand were extracted from the committed MDX rather than maintained as a second example. The documented install block ran verbatim on Python 3.10 and 3.13:The Python 3.10 environment additionally exercised the supported Ragas lower bound:
Results:
The exact
python score_ragas.pycommand also passed twice, on Python 3.10/Ragas 0.4.0 and Python 3.13/Ragas 0.4.3, each producing one numeric-score POST to the loopback Scores endpoint. Tests cover original-span attachment, zero preservation, NaN/positive infinity/negative infinity rejection, three invalid span IDs, API rejection propagation, and omission of answer text. The committed test denies non-loopback sockets.The missing-page negative control reproduced main's absent-guide condition: the targeted test failed with
Ragas evaluation recipe is missingwhen the page was temporarily removed in E2B. The committed head passed. This is a negative control, not a claim that the whole base branch was executed.E2B receipt:
qw-docs-1078-443549be, exit 0; sandbox destruction confirmed.Docs checks
Scope and review
Cloud score persistence was not exercised. Loopback validation proves the published SDK's request/response shape and recipe behavior only; it does not prove Cloud storage or UI persistence. No traceAI package, backend, public API, migration or release-version changes. Privacy defaults are unchanged.
Internal-standards self-review found no merge-blocking defects. Product E2E coverage gate: not applicable (this repository has no
e2e/harness); docs browser checks and the executable recipe test provide the applicable coverage. The PR is non-draft; required human approval remains separate from this technical qualification. No merge or approval was performed.🤖 Generated with Claude Code