Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,7 @@ Pass the raw provider model string in `gen_ai.request.model` (and `gen_ai.respon

<Alert>

If you call Jev through the <PlatformLink to="/configuration/integrations/typesafe/">`@typesafe-ai/sdk`</PlatformLink> package or Vercel AI SDK's <PlatformLink to="/configuration/integrations/vercelai/#trace-jev-evaluations">`experimental_evaluate()`</PlatformLink>, these spans are created automatically, so you don't need to instrument them by hand.
If you call Jev through the <PlatformLink to="/configuration/integrations/typesafe/">`@typesafe-ai/sdk`</PlatformLink> package or Vercel AI SDK's <PlatformLink to="/configuration/integrations/vercelai/#trace-jev-decisions">`experimental_decide()`</PlatformLink>, these spans are created automatically, so you don't need to instrument them by hand.

</Alert>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ The integration captures `gen_ai.embeddings` spans for `embedQuery()` and `embed

## Trace Classifier Evaluations

<AvailableSince version="11.6.0" />
<AvailableSince version="11.7.0" />

[`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe)'s `TypeSafeClassifier` asks TypeSafe's Jev model typed questions about your input. The `langChainIntegration` traces these calls for you, whether you call the classifier yourself, from a <PlatformLink to="/configuration/integrations/langgraph/#trace-classifier-evaluations">LangGraph</PlatformLink> node, or through the `modelRouterMiddleware` and `autoModeMiddleware` of `@langchain/typesafe/middleware` inside a `createAgent` run. No additional integration or setup is needed.

Expand All @@ -84,11 +84,11 @@ const result = await classifier.invoke(
console.log(result.choices.department.choice);
```

Each `invoke()` call creates a `gen_ai.evaluate` span named `evaluate <model>`, with the model, provider, latency, and token usage. Inside an agent, the span also records the agent name. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures the `state` and `questions` sent to Jev as input messages and `answers` as output messages, including probabilities returned by the model. LangChain messages in the input are recorded as transcript lines, the same way the classifier sends them.
Each `invoke()` call creates a `gen_ai.decide` span named `decide <model>`, with the model, provider, latency, and token usage. Inside an agent, the span also records the agent name. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures the `state` and `questions` sent to Jev as input messages and `answers` as output messages, including probabilities returned by the model. LangChain messages in the input are recorded as transcript lines, the same way the classifier sends them.

Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other LangChain calls. Disable these options if evaluation state or answers contain sensitive data.

If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the <PlatformLink to="/configuration/integrations/typesafe/">TypeSafe</PlatformLink> or <PlatformLink to="/configuration/integrations/vercelai/#trace-jev-evaluations">Vercel AI</PlatformLink> integration.
If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_decide()` instead, see the <PlatformLink to="/configuration/integrations/typesafe/">TypeSafe</PlatformLink> or <PlatformLink to="/configuration/integrations/vercelai/#trace-jev-decisions">Vercel AI</PlatformLink> integration.

## Supported Versions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,13 @@ By default, tracing support is added to the following LangGraph operations:

- **Agent Creation** (`gen_ai.create_agent`) - Captures spans when compiling a StateGraph into an executable agent
- **Agent Invocation** (`gen_ai.invoke_agent`) - Captures spans for agent execution via `invoke()`
- **Classifier Evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` calls in graph nodes
- **Classifier Evaluations** (`gen_ai.decide`) - Captures spans for `TypeSafeClassifier` calls in graph nodes

## Trace Classifier Evaluations

<AvailableSince version="11.6.0" />
<AvailableSince version="11.7.0" />

When a graph node calls a `TypeSafeClassifier` from [`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe), the <PlatformLink to="/configuration/integrations/langchain/#trace-classifier-evaluations">LangChain integration</PlatformLink> records the call as a `gen_ai.evaluate` span. The span is a child of the graph's `invoke_agent` span and records the graph name as the agent name. No additional setup is needed.
When a graph node calls a `TypeSafeClassifier` from [`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe), the <PlatformLink to="/configuration/integrations/langchain/#trace-classifier-evaluations">LangChain integration</PlatformLink> records the call as a `gen_ai.decide` span. The span is a child of the graph's `invoke_agent` span and records the graph name as the agent name. No additional setup is needed.

```javascript
import { Annotation, END, START, StateGraph } from "@langchain/langgraph";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,17 @@ notSupported:

<Alert>

Requires Sentry SDK version `11.1.0` or higher.
Requires Sentry SDK version `11.7.0` or higher.

</Alert>

_Import name: `Sentry.typesafeIntegration`_

This integration is enabled by default when <PlatformLink to="/tracing/">tracing</PlatformLink> is enabled. To customize it, see [Modifying Default Integrations](./../#modifying-default-integrations).

The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture <PlatformLink to="/agent-tracing/">agent tracing</PlatformLink> `gen_ai.evaluate` spans, recording model, token usage, latency, and (when enabled) evaluation inputs and answers.
The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture <PlatformLink to="/agent-tracing/">agent tracing</PlatformLink> `gen_ai.decide` spans, recording model, token usage, latency, and (when enabled) decision inputs and answers.

If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-and-clef-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra Classifier Evaluations](/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations/).
If you call Jev through Vercel AI SDK's `experimental_decide()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-decisions) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-and-clef-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra Classifier Evaluations](/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations/).

If you call Jev through LangChain's `TypeSafeClassifier`, including from LangGraph nodes and the `@langchain/typesafe` middleware, see the [LangChain integration](/platforms/javascript/guides/node/configuration/integrations/langchain/#trace-classifier-evaluations).

Expand Down Expand Up @@ -70,7 +70,7 @@ For unbundled applications started with `bun run`, use [Manual Instrumentation](

## Supported Operations

Tracing support is added to `TypeSafeClient.systemOne()` calls to evaluate state against a set of questions using Jev. Each call creates a `gen_ai.evaluate` span. Recorded answers include probabilities and confidence values when returned by TypeSafe.
Tracing support is added to `TypeSafeClient.systemOne()` calls to evaluate state against a set of questions using Jev. Each call creates a `gen_ai.decide` span named `decide <model>`. Recorded answers include probabilities and confidence values when returned by TypeSafe.

## Supported Versions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,20 @@ By default, tracing support is added to the following `ai` functions:
- `embed()`
- `embedMany()`
- `rerank()`
- `experimental_evaluate()` — requires Sentry SDK `11.1.0` or higher and AI SDK 7
- `experimental_decide()` — requires Sentry SDK `11.7.0` or higher and AI SDK `7.0.128` or higher

Plus `generate()` and `stream()` on [`ToolLoopAgent`](https://ai-sdk.dev/docs/agents/overview#toolloopagent-class).

## Trace Jev Evaluations
## Trace Jev Decisions

With Sentry SDK `11.1.0` or higher and AI SDK 7, `vercelAIIntegration` automatically traces `experimental_evaluate()` calls, including calls to TypeSafe's Jev model through Vercel AI Gateway. Keep your existing Sentry setup with tracing enabled; no additional integration or telemetry registration is needed.
With Sentry SDK `11.7.0` or higher and AI SDK `7.0.128` or higher, `vercelAIIntegration` automatically traces `experimental_decide()` calls, including calls to TypeSafe's Jev model through Vercel AI Gateway. Keep your existing Sentry setup with tracing enabled; no additional integration or telemetry registration is needed.

Configure [AI Gateway authentication](https://ai-sdk.dev/docs/ai-sdk-core/evaluation#default-provider-strings) with `AI_GATEWAY_API_KEY` or Vercel OIDC, then call `experimental_evaluate()`:
Configure [AI Gateway authentication](https://ai-sdk.dev/docs/ai-sdk-core/decisions#default-provider-strings) with `AI_GATEWAY_API_KEY` or Vercel OIDC, then call `experimental_decide()`:

```javascript
import { experimental_evaluate } from "ai";
import { experimental_decide } from "ai";

const result = await experimental_evaluate({
const result = await experimental_decide({
model: "typesafe-ai/jev",
state: "I was charged twice. Please refund the extra charge.",
questions: {
Expand All @@ -85,11 +85,11 @@ const result = await experimental_evaluate({
console.log(result.answers.department);
```

Each call creates a `gen_ai.evaluate` span with model, latency, and token usage when available. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages. For TypeSafe answers, Sentry also includes confidence values from `providerMetadata.typesafe.confidence` when present.
Each call creates a `gen_ai.decide` span named `decide <model>`, with model, latency, and token usage when available. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages. For TypeSafe answers, Sentry also includes confidence values from `providerMetadata.typesafe.confidence` when present.

Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other Vercel AI calls. Disable these options if evaluation state or answers contain sensitive data.
Decision inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other Vercel AI calls. Disable these options if decision state or answers contain sensitive data.

The AI SDK's [evaluation API](https://ai-sdk.dev/docs/ai-sdk-core/evaluation) is experimental and may change in patch releases. If you use `TypeSafeClient.systemOne()` directly instead, see the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/).
The AI SDK's [decision API](https://ai-sdk.dev/docs/ai-sdk-core/decisions) is experimental and may change in patch releases. If you use `TypeSafeClient.systemOne()` directly instead, see the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/).

## Supported Versions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ You can browse the full list of available models in the [Cloudflare Workers AI m

## Trace Jev and Clef Evaluations

<AvailableSince version="11.3.0" />
<AvailableSince version="11.7.0" />

Calls to TypeSafe's [Jev model](https://developers.cloudflare.com/ai/models/typesafe/jev/) (`typesafe/jev`) through `env.AI.run()` are traced automatically. No additional integration or setup is needed:

Expand Down Expand Up @@ -71,9 +71,9 @@ const result = await env.AI.run("@cf/cloudflare/clef", {
});
```

Each call creates a `gen_ai.evaluate` span named `evaluate <model>`, with the response model (for example, `jev-1.13.0` or `clef`), latency, and token usage. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities and confidence values returned by the model. These spans have the same attributes as spans from the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/) on other runtimes.
Each call creates a `gen_ai.decide` span named `decide <model>`, with the response model (for example, `jev-1.13.0` or `clef`), latency, and token usage. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities and confidence values returned by the model. These spans have the same attributes as spans from the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/) on other runtimes.

Evaluation state and answers can contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the <PlatformLink to="/configuration/options/#dataCollection">`dataCollection` option</PlatformLink>.
Decision state and answers can contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the <PlatformLink to="/configuration/options/#dataCollection">`dataCollection` option</PlatformLink>.

## Next Steps

Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: Mastra Classifier Evaluations
sidebar_title: Classifier Evaluations
description: "Learn how Sentry traces Mastra Classifier evaluations, such as calls to TypeSafe's Jev, as gen_ai.evaluate spans."
description: "Learn how Sentry traces Mastra Classifier evaluations, such as calls to TypeSafe's Jev, as gen_ai.decide spans."
---

<AvailableSince version="11.6.0" />
<AvailableSince version="11.7.0" />

Mastra's [`Classifier`](https://mastra.ai/reference/classifier/classifier) asks a decision model, such as TypeSafe's Jev, typed questions about your application state. These calls don't go through the Vercel AI SDK, so the Mastra integration traces them for you. This includes classifiers that your processors and scorers call. If you've followed the [Mastra Quick Start](/platforms/javascript/guides/mastra/), no additional integration or setup is needed.

Expand Down Expand Up @@ -36,17 +36,17 @@ console.log(result.answers.department.choice);

## What Gets Captured

Each `evaluate()` call creates a `gen_ai.evaluate` span named `evaluate <model>`, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model.
Each `evaluate()` call creates a `gen_ai.decide` span named `decide <model>`, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model.

## Control Recorded Data

Sentry runs this data through your Mastra span output processors before recording it, so Mastra's default `SensitiveDataFilter` redacts sensitive keys such as `apiKey`. Evaluation state and answers can still contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the <PlatformLink to="/configuration/options/#dataCollection">`dataCollection` option</PlatformLink>.

## Supported Versions

- `@sentry/node`: `>=11.6.0`
- `@sentry/node`: `>=11.7.0`
- `@mastra/core`: `>=1.69.0`

## Related Integrations

If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) integration.
If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_decide()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-decisions) integration.
2 changes: 1 addition & 1 deletion includes/mastra/what-gets-captured.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Sentry's Mastra integration maps Mastra's span types to Sentry operations for th
| `PROVIDER_TOOL_CALL` | `gen_ai.execute_tool` |
| `CLIENT_TOOL_CALL` | `gen_ai.execute_tool` |
| `RAG_EMBEDDING` | `gen_ai.embeddings` |
| `CLASSIFIER_EVALUATION` | `gen_ai.evaluate` |
| `CLASSIFIER_EVALUATION` | `gen_ai.decide` |

Spans are tagged with `sentry.origin: auto.ai.mastra`.

Expand Down
Loading