Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -287,11 +287,11 @@ Marking failed tools with an error status populates the Tool Errors widget.
<Include name="tracing/ai-agents-module/execute-tool-span" />
</Expandable>

## Evaluation Span
## Decision Span

This span represents an evaluation of generated content against a set of questions, such as grading a model's reply for accuracy or tone. Models like <PlatformLink to="/configuration/integrations/typesafe/">TypeSafe Jev</PlatformLink> produce these spans.
This span represents a decision on generated content against a set of questions, such as grading a model's reply for accuracy or tone. Models like <PlatformLink to="/configuration/integrations/typesafe/">TypeSafe Jev</PlatformLink> produce these spans.

Sentry identifies an evaluation by `gen_ai.operation.name: "evaluate"`. Evaluations share the `ai_client` operation type with ordinary LLM calls, so the operation name is what sets them apart.
Sentry identifies a decision by `gen_ai.operation.name: "decide"`. Decisions share the `ai_client` operation type with ordinary LLM calls, so the operation name is what sets them apart.

```javascript
const questions = {
Expand Down Expand Up @@ -327,10 +327,10 @@ const evaluationInput = [

await Sentry.startSpan(
{
op: "gen_ai.evaluate",
name: "evaluate typesafe/jev-1.13",
op: "gen_ai.decide",
name: "decide typesafe/jev-1.13",
attributes: {
"gen_ai.operation.name": "evaluate",
"gen_ai.operation.name": "decide",
"gen_ai.request.model": "typesafe/jev-1.13",
"gen_ai.provider.name": "typesafe",
"gen_ai.agent.name": "Support Agent", // when this evaluation runs under an agent
Expand Down Expand Up @@ -379,7 +379,7 @@ If you call Jev through the <PlatformLink to="/configuration/integrations/typesa

</Alert>

<Expandable title="Evaluation span attributes">
<Expandable title="Decision span attributes">
<Include name="tracing/ai-agents-module/evaluation-span" />
</Expandable>

Expand Down
2 changes: 1 addition & 1 deletion docs/platforms/php/guides/laravel/agent-tracing/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ $response = Classification::of('I was charged twice. Please refund the extra cha
->classify('typesafe');
```

Each classification creates a `gen_ai.evaluate` span named `evaluate <model>`, with the provider, the response model (for example, `jev-1.13.0`), latency, and token usage. If the provider returns an error, the span is marked as failed. When recording is enabled, Sentry captures the classified text and the questions as input messages and the answers as output messages. Sentry shows these spans as <Link to="/product/agents/conversations/#evaluation-spans">evaluation spans</Link>.
Each classification creates a `gen_ai.evaluate` span named `evaluate <model>`, with the provider, the response model (for example, `jev-1.13.0`), latency, and token usage. If the provider returns an error, the span is marked as failed. When recording is enabled, Sentry captures the classified text and the questions as input messages and the answers as output messages. Sentry shows these spans as <Link to="/product/agents/conversations/#decision-spans">decision spans</Link>.

Classified text and answers can contain sensitive data, so Sentry only records them if `send_default_pii` is enabled. With the <PlatformLink to="/configuration/options/#data_collection">`data_collection` option</PlatformLink>, use the `gen_ai` key's `inputs` and `outputs` instead.

Expand Down
14 changes: 7 additions & 7 deletions docs/platforms/python/agent-tracing/manual-instrumentation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -261,11 +261,11 @@ Marking failed tools with an error status populates the Tool Errors widget.
<Include name="tracing/ai-agents-module/execute-tool-span" />
</Expandable>

## Evaluation Span
## Decision Span

This span represents an evaluation of generated content against a set of questions, such as grading a model's reply for accuracy or tone. Models like <PlatformLink to="/agent-tracing/typesafe/">TypeSafe Jev</PlatformLink> produce these spans.
This span represents a decision on generated content against a set of questions, such as grading a model's reply for accuracy or tone. Models like <PlatformLink to="/agent-tracing/typesafe/">TypeSafe Jev</PlatformLink> produce these spans.

Sentry identifies an evaluation by `gen_ai.operation.name: "evaluate"`. Evaluations share the `ai_client` operation type with ordinary LLM calls, so the operation name is what sets them apart.
Sentry identifies a decision by `gen_ai.operation.name: "decide"`. Decisions share the `ai_client` operation type with ordinary LLM calls, so the operation name is what sets them apart.

```python
import json
Expand Down Expand Up @@ -303,10 +303,10 @@ evaluation_input = [
]

with sentry_sdk.start_span(
op="gen_ai.evaluate",
name="evaluate typesafe/jev-1.13",
op="gen_ai.decide",
name="decide typesafe/jev-1.13",
attributes={
"gen_ai.operation.name": "evaluate",
"gen_ai.operation.name": "decide",
"gen_ai.request.model": "typesafe/jev-1.13",
"gen_ai.provider.name": "typesafe",
"gen_ai.agent.name": "Support Agent", # when this evaluation runs under an agent
Expand Down Expand Up @@ -345,7 +345,7 @@ If you call Jev through the <PlatformLink to="/agent-tracing/typesafe/">TypeSafe

</Alert>

<Expandable title="Evaluation span attributes">
<Expandable title="Decision span attributes">
<Include name="tracing/ai-agents-module/evaluation-span" />
</Expandable>

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
22 changes: 11 additions & 11 deletions docs/product/agents/conversations/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -144,35 +144,35 @@ Use the two tabs to inspect the conversation:

The error count links to the conversation's failed spans, while **Trace** or **Traces** links connect the conversation to the surrounding distributed trace data.

## Evaluation Spans
## Decision Spans

Some conversations include evaluation spans: LLM-as-judge checks that grade generated content against a set of questions, such as scoring a model's reply for accuracy or classifying a support message by department. Models like TypeSafe Jev produce these spans.
Some conversations include decision spans: LLM-as-judge checks that grade generated content against a set of questions, such as scoring a model's reply for accuracy or classifying a support message by department. Models like TypeSafe Jev produce these spans.

Sentry recognizes an evaluation by its `gen_ai.operation.name` attribute (`evaluate`). Evaluations share the `ai_client` operation type with ordinary LLM calls, so this attribute is what tells them apart.
Sentry recognizes a decision by its `gen_ai.operation.name` attribute (`decide`). Decisions share the `ai_client` operation type with ordinary LLM calls, so this attribute is what tells them apart.

### In the Transcript and Timeline

In the **Transcript**, an evaluation appears as a row like a tool call, showing the evaluator, a one-line result (for example `authIssue: Yes, wantsRefund: Yes, urgency: high (1.6)`), and its duration. In the **Timeline**, the same one-line result shows as the span's secondary text. **Copy Transcript** includes these results in the Markdown.
In the **Transcript**, a decision appears as a row like a tool call, showing the evaluator, a one-line result (for example `wants_refund: Yes, department: billing, urgency: high (1.96)`), and its duration. In the **Timeline**, the same one-line result shows as the span's secondary text. **Copy Transcript** includes these results in the Markdown.

![Evaluation in the conversation transcript](./img/evaluation-transcript-row.png)
![Decision in the conversation transcript](./img/evaluation-transcript-row.png)

### Evaluation Detail
### Decision Detail

Select an evaluation — from the transcript, the timeline, or the trace view — to open its span detail. Unlike an LLM call, an evaluation stores the evaluated content, its questions, and its answers rather than chat messages, so the detail is laid out around those:
Select a decision — from the transcript, the timeline, or the trace view — to open its span detail. Unlike an LLM call, a decision stores the evaluated content, its questions, and its answers rather than chat messages, so the detail is laid out around those:

- **Input** shows the evaluated text, followed by one card per question with its instructions, type, and allowed answers. A `score` question lists its scale (for example `0 · low`, `1 · medium`, `2 · high`); a `choice` question lists each option with its description.
- **Output** shows one card per answer with a short result, and, where the evaluator reports them, a probability bar per option or score value and a confidence percentage. Answers come in a few types: `boolean` (Yes/No), `score` (a value, named after the question's scale), and `choice` (the selected option).
- **Output** shows one card per answer with a short result, and, where the evaluator reports them, a probability bar per option or score value and a confidence percentage. Answers come in a few types: `predicate` and `boolean` (Yes/No), `score` (a value, named after the question's scale), and `choice` (the selected option).

Both tabs have a **Pretty/Raw** toggle. Questions or answers that don't match the expected format are shown as their raw value and tagged **Unrecognized**.

<div style={{display: 'grid', gridTemplateColumns: 'repeat(auto-fit, minmax(min(100%, 320px), 1fr))', columnGap: '16px', alignItems: 'start'}}>

![Evaluation span detail, Input tab](./img/evaluation-span-detail-input.png)
![Decision span detail, Input tab](./img/evaluation-span-detail-input.png)

![Evaluation span detail, Output tab](./img/evaluation-span-detail-output.png)
![Decision span detail, Output tab](./img/evaluation-span-detail-output.png)

</div>

### Setup

Evaluation spans come from your SDK instrumentation. If you call an evaluator through a supported integration (for example the `@typesafe-ai/sdk` package or the Vercel AI SDK), these spans are created automatically. To record them by hand, see the Evaluation Span section of the manual instrumentation guide for [JavaScript](/platforms/javascript/guides/node/agent-tracing/manual-instrumentation/#evaluation-span) or [Python](/platforms/python/agent-tracing/manual-instrumentation/#evaluation-span).
Decision spans come from your SDK instrumentation. If you call an evaluator through a supported integration (for example the `@typesafe-ai/sdk` package or the Vercel AI SDK), these spans are created automatically. To record them by hand, see the Decision Span section of the manual instrumentation guide for [JavaScript](/platforms/javascript/guides/node/agent-tracing/manual-instrumentation/#decision-span) or [Python](/platforms/python/agent-tracing/manual-instrumentation/#decision-span).
1 change: 1 addition & 0 deletions docs/product/agents/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ keywords:
- agent tracing
- AI tracing
- evaluations
- decisions
- LLM-as-judge
og_image: /og-images/ai-monitoring-agents.png
---
Expand Down
8 changes: 4 additions & 4 deletions includes/tracing/ai-agents-module/evaluation-span.mdx
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
Describes an evaluation of generated content against a set of questions (for example a TypeSafe Jev evaluation).
Describes a decision on generated content against a set of questions (for example a TypeSafe Jev evaluation).

- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"gen_ai.evaluate"`.
- The span `name` SHOULD be `"evaluate {gen_ai.request.model}"`. (e.g. `"evaluate typesafe/jev-1.13"`)
- The `gen_ai.operation.name` attribute MUST be `"evaluate"`. Sentry uses this attribute, not the span `op`, to tell evaluations apart from ordinary LLM calls (both share the `ai_client` operation type).
- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"gen_ai.decide"`.
- The span `name` SHOULD be `"decide {gen_ai.request.model}"`. (e.g. `"decide typesafe/jev-1.13"`)
- The `gen_ai.operation.name` attribute MUST be `"decide"`. Sentry uses this attribute, not the span `op`, to tell decisions apart from ordinary LLM calls (both share the `ai_client` operation type).
- The `gen_ai.request.model` attribute SHOULD be the requested evaluation model. (e.g. `"typesafe/jev-1.13"`)
- The `gen_ai.response.model` attribute SHOULD be the concrete model that responded. (e.g. `"typesafe/jev-1.13-20260917"`)
- The `gen_ai.provider.name` attribute SHOULD be set to the provider. (e.g. `"typesafe"`)
Expand Down
Loading