diff --git a/docs/platforms/javascript/common/agent-tracing/manual-instrumentation.mdx b/docs/platforms/javascript/common/agent-tracing/manual-instrumentation.mdx index 11b352bf24bd6..6caaf2eb83882 100644 --- a/docs/platforms/javascript/common/agent-tracing/manual-instrumentation.mdx +++ b/docs/platforms/javascript/common/agent-tracing/manual-instrumentation.mdx @@ -287,11 +287,11 @@ Marking failed tools with an error status populates the Tool Errors widget. -## Evaluation Span +## Decision Span -This span represents an evaluation of generated content against a set of questions, such as grading a model's reply for accuracy or tone. Models like TypeSafe Jev produce these spans. +This span represents a decision on generated content against a set of questions, such as grading a model's reply for accuracy or tone. Models like TypeSafe Jev produce these spans. -Sentry identifies an evaluation by `gen_ai.operation.name: "evaluate"`. Evaluations share the `ai_client` operation type with ordinary LLM calls, so the operation name is what sets them apart. +Sentry identifies a decision by `gen_ai.operation.name: "decide"`. Decisions share the `ai_client` operation type with ordinary LLM calls, so the operation name is what sets them apart. ```javascript const questions = { @@ -327,10 +327,10 @@ const evaluationInput = [ await Sentry.startSpan( { - op: "gen_ai.evaluate", - name: "evaluate typesafe/jev-1.13", + op: "gen_ai.decide", + name: "decide typesafe/jev-1.13", attributes: { - "gen_ai.operation.name": "evaluate", + "gen_ai.operation.name": "decide", "gen_ai.request.model": "typesafe/jev-1.13", "gen_ai.provider.name": "typesafe", "gen_ai.agent.name": "Support Agent", // when this evaluation runs under an agent @@ -379,7 +379,7 @@ If you call Jev through the + diff --git a/docs/platforms/php/guides/laravel/agent-tracing/index.mdx b/docs/platforms/php/guides/laravel/agent-tracing/index.mdx index 8d09e3b7777bc..0da4dd3858e9d 100644 --- a/docs/platforms/php/guides/laravel/agent-tracing/index.mdx +++ b/docs/platforms/php/guides/laravel/agent-tracing/index.mdx @@ -120,7 +120,7 @@ $response = Classification::of('I was charged twice. Please refund the extra cha ->classify('typesafe'); ``` -Each classification creates a `gen_ai.evaluate` span named `evaluate `, with the provider, the response model (for example, `jev-1.13.0`), latency, and token usage. If the provider returns an error, the span is marked as failed. When recording is enabled, Sentry captures the classified text and the questions as input messages and the answers as output messages. Sentry shows these spans as evaluation spans. +Each classification creates a `gen_ai.evaluate` span named `evaluate `, with the provider, the response model (for example, `jev-1.13.0`), latency, and token usage. If the provider returns an error, the span is marked as failed. When recording is enabled, Sentry captures the classified text and the questions as input messages and the answers as output messages. Sentry shows these spans as decision spans. Classified text and answers can contain sensitive data, so Sentry only records them if `send_default_pii` is enabled. With the `data_collection` option, use the `gen_ai` key's `inputs` and `outputs` instead. diff --git a/docs/platforms/python/agent-tracing/manual-instrumentation.mdx b/docs/platforms/python/agent-tracing/manual-instrumentation.mdx index 4466f740534b5..fc68eaa05dfcf 100644 --- a/docs/platforms/python/agent-tracing/manual-instrumentation.mdx +++ b/docs/platforms/python/agent-tracing/manual-instrumentation.mdx @@ -261,11 +261,11 @@ Marking failed tools with an error status populates the Tool Errors widget. -## Evaluation Span +## Decision Span -This span represents an evaluation of generated content against a set of questions, such as grading a model's reply for accuracy or tone. Models like TypeSafe Jev produce these spans. +This span represents a decision on generated content against a set of questions, such as grading a model's reply for accuracy or tone. Models like TypeSafe Jev produce these spans. -Sentry identifies an evaluation by `gen_ai.operation.name: "evaluate"`. Evaluations share the `ai_client` operation type with ordinary LLM calls, so the operation name is what sets them apart. +Sentry identifies a decision by `gen_ai.operation.name: "decide"`. Decisions share the `ai_client` operation type with ordinary LLM calls, so the operation name is what sets them apart. ```python import json @@ -303,10 +303,10 @@ evaluation_input = [ ] with sentry_sdk.start_span( - op="gen_ai.evaluate", - name="evaluate typesafe/jev-1.13", + op="gen_ai.decide", + name="decide typesafe/jev-1.13", attributes={ - "gen_ai.operation.name": "evaluate", + "gen_ai.operation.name": "decide", "gen_ai.request.model": "typesafe/jev-1.13", "gen_ai.provider.name": "typesafe", "gen_ai.agent.name": "Support Agent", # when this evaluation runs under an agent @@ -345,7 +345,7 @@ If you call Jev through the TypeSafe - + diff --git a/docs/product/agents/conversations/img/evaluation-span-detail-input.png b/docs/product/agents/conversations/img/evaluation-span-detail-input.png index 32f465951ca7e..c789684fa21e0 100644 Binary files a/docs/product/agents/conversations/img/evaluation-span-detail-input.png and b/docs/product/agents/conversations/img/evaluation-span-detail-input.png differ diff --git a/docs/product/agents/conversations/img/evaluation-span-detail-output.png b/docs/product/agents/conversations/img/evaluation-span-detail-output.png index 39a53c1a63793..4176ee6b967b0 100644 Binary files a/docs/product/agents/conversations/img/evaluation-span-detail-output.png and b/docs/product/agents/conversations/img/evaluation-span-detail-output.png differ diff --git a/docs/product/agents/conversations/index.mdx b/docs/product/agents/conversations/index.mdx index b27d72df0f4e4..f3ee14c670869 100644 --- a/docs/product/agents/conversations/index.mdx +++ b/docs/product/agents/conversations/index.mdx @@ -144,35 +144,35 @@ Use the two tabs to inspect the conversation: The error count links to the conversation's failed spans, while **Trace** or **Traces** links connect the conversation to the surrounding distributed trace data. -## Evaluation Spans +## Decision Spans -Some conversations include evaluation spans: LLM-as-judge checks that grade generated content against a set of questions, such as scoring a model's reply for accuracy or classifying a support message by department. Models like TypeSafe Jev produce these spans. +Some conversations include decision spans: LLM-as-judge checks that grade generated content against a set of questions, such as scoring a model's reply for accuracy or classifying a support message by department. Models like TypeSafe Jev produce these spans. -Sentry recognizes an evaluation by its `gen_ai.operation.name` attribute (`evaluate`). Evaluations share the `ai_client` operation type with ordinary LLM calls, so this attribute is what tells them apart. +Sentry recognizes a decision by its `gen_ai.operation.name` attribute (`decide`). Decisions share the `ai_client` operation type with ordinary LLM calls, so this attribute is what tells them apart. ### In the Transcript and Timeline -In the **Transcript**, an evaluation appears as a row like a tool call, showing the evaluator, a one-line result (for example `authIssue: Yes, wantsRefund: Yes, urgency: high (1.6)`), and its duration. In the **Timeline**, the same one-line result shows as the span's secondary text. **Copy Transcript** includes these results in the Markdown. +In the **Transcript**, a decision appears as a row like a tool call, showing the evaluator, a one-line result (for example `wants_refund: Yes, department: billing, urgency: high (1.96)`), and its duration. In the **Timeline**, the same one-line result shows as the span's secondary text. **Copy Transcript** includes these results in the Markdown. -![Evaluation in the conversation transcript](./img/evaluation-transcript-row.png) +![Decision in the conversation transcript](./img/evaluation-transcript-row.png) -### Evaluation Detail +### Decision Detail -Select an evaluation — from the transcript, the timeline, or the trace view — to open its span detail. Unlike an LLM call, an evaluation stores the evaluated content, its questions, and its answers rather than chat messages, so the detail is laid out around those: +Select a decision — from the transcript, the timeline, or the trace view — to open its span detail. Unlike an LLM call, a decision stores the evaluated content, its questions, and its answers rather than chat messages, so the detail is laid out around those: - **Input** shows the evaluated text, followed by one card per question with its instructions, type, and allowed answers. A `score` question lists its scale (for example `0 · low`, `1 · medium`, `2 · high`); a `choice` question lists each option with its description. -- **Output** shows one card per answer with a short result, and, where the evaluator reports them, a probability bar per option or score value and a confidence percentage. Answers come in a few types: `boolean` (Yes/No), `score` (a value, named after the question's scale), and `choice` (the selected option). +- **Output** shows one card per answer with a short result, and, where the evaluator reports them, a probability bar per option or score value and a confidence percentage. Answers come in a few types: `predicate` and `boolean` (Yes/No), `score` (a value, named after the question's scale), and `choice` (the selected option). Both tabs have a **Pretty/Raw** toggle. Questions or answers that don't match the expected format are shown as their raw value and tagged **Unrecognized**.
-![Evaluation span detail, Input tab](./img/evaluation-span-detail-input.png) +![Decision span detail, Input tab](./img/evaluation-span-detail-input.png) -![Evaluation span detail, Output tab](./img/evaluation-span-detail-output.png) +![Decision span detail, Output tab](./img/evaluation-span-detail-output.png)
### Setup -Evaluation spans come from your SDK instrumentation. If you call an evaluator through a supported integration (for example the `@typesafe-ai/sdk` package or the Vercel AI SDK), these spans are created automatically. To record them by hand, see the Evaluation Span section of the manual instrumentation guide for [JavaScript](/platforms/javascript/guides/node/agent-tracing/manual-instrumentation/#evaluation-span) or [Python](/platforms/python/agent-tracing/manual-instrumentation/#evaluation-span). +Decision spans come from your SDK instrumentation. If you call an evaluator through a supported integration (for example the `@typesafe-ai/sdk` package or the Vercel AI SDK), these spans are created automatically. To record them by hand, see the Decision Span section of the manual instrumentation guide for [JavaScript](/platforms/javascript/guides/node/agent-tracing/manual-instrumentation/#decision-span) or [Python](/platforms/python/agent-tracing/manual-instrumentation/#decision-span). diff --git a/docs/product/agents/index.mdx b/docs/product/agents/index.mdx index 97fd846b1b88d..fa1cd21f25b1c 100644 --- a/docs/product/agents/index.mdx +++ b/docs/product/agents/index.mdx @@ -18,6 +18,7 @@ keywords: - agent tracing - AI tracing - evaluations + - decisions - LLM-as-judge og_image: /og-images/ai-monitoring-agents.png --- diff --git a/includes/tracing/ai-agents-module/evaluation-span.mdx b/includes/tracing/ai-agents-module/evaluation-span.mdx index a6de8c5d2b2b3..a753d202d041c 100644 --- a/includes/tracing/ai-agents-module/evaluation-span.mdx +++ b/includes/tracing/ai-agents-module/evaluation-span.mdx @@ -1,8 +1,8 @@ -Describes an evaluation of generated content against a set of questions (for example a TypeSafe Jev evaluation). +Describes a decision on generated content against a set of questions (for example a TypeSafe Jev evaluation). -- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"gen_ai.evaluate"`. -- The span `name` SHOULD be `"evaluate {gen_ai.request.model}"`. (e.g. `"evaluate typesafe/jev-1.13"`) -- The `gen_ai.operation.name` attribute MUST be `"evaluate"`. Sentry uses this attribute, not the span `op`, to tell evaluations apart from ordinary LLM calls (both share the `ai_client` operation type). +- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"gen_ai.decide"`. +- The span `name` SHOULD be `"decide {gen_ai.request.model}"`. (e.g. `"decide typesafe/jev-1.13"`) +- The `gen_ai.operation.name` attribute MUST be `"decide"`. Sentry uses this attribute, not the span `op`, to tell decisions apart from ordinary LLM calls (both share the `ai_client` operation type). - The `gen_ai.request.model` attribute SHOULD be the requested evaluation model. (e.g. `"typesafe/jev-1.13"`) - The `gen_ai.response.model` attribute SHOULD be the concrete model that responded. (e.g. `"typesafe/jev-1.13-20260917"`) - The `gen_ai.provider.name` attribute SHOULD be set to the provider. (e.g. `"typesafe"`)