diff --git a/docs/platforms/javascript/common/agent-tracing/manual-instrumentation.mdx b/docs/platforms/javascript/common/agent-tracing/manual-instrumentation.mdx index 07ae99ef4f2a1..adb9aa7222349 100644 --- a/docs/platforms/javascript/common/agent-tracing/manual-instrumentation.mdx +++ b/docs/platforms/javascript/common/agent-tracing/manual-instrumentation.mdx @@ -375,7 +375,7 @@ Pass the raw provider model string in `gen_ai.request.model` (and `gen_ai.respon -If you call Jev through the `@typesafe-ai/sdk` package or Vercel AI SDK's `experimental_evaluate()`, these spans are created automatically, so you don't need to instrument them by hand. +If you call Jev through the `@typesafe-ai/sdk` package or Vercel AI SDK's `experimental_decide()`, these spans are created automatically, so you don't need to instrument them by hand. diff --git a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx index 6f7f6d5cf5163..1f45f0711ce0d 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx @@ -57,7 +57,7 @@ The integration captures `gen_ai.embeddings` spans for `embedQuery()` and `embed ## Trace Classifier Evaluations - + [`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe)'s `TypeSafeClassifier` asks TypeSafe's Jev model typed questions about your input. The `langChainIntegration` traces these calls for you, whether you call the classifier yourself, from a LangGraph node, or through the `modelRouterMiddleware` and `autoModeMiddleware` of `@langchain/typesafe/middleware` inside a `createAgent` run. No additional integration or setup is needed. @@ -84,11 +84,11 @@ const result = await classifier.invoke( console.log(result.choices.department.choice); ``` -Each `invoke()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. Inside an agent, the span also records the agent name. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures the `state` and `questions` sent to Jev as input messages and `answers` as output messages, including probabilities returned by the model. LangChain messages in the input are recorded as transcript lines, the same way the classifier sends them. +Each `invoke()` call creates a `gen_ai.decide` span named `decide `, with the model, provider, latency, and token usage. Inside an agent, the span also records the agent name. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures the `state` and `questions` sent to Jev as input messages and `answers` as output messages, including probabilities returned by the model. LangChain messages in the input are recorded as transcript lines, the same way the classifier sends them. Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other LangChain calls. Disable these options if evaluation state or answers contain sensitive data. -If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the TypeSafe or Vercel AI integration. +If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_decide()` instead, see the TypeSafe or Vercel AI integration. ## Supported Versions diff --git a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx index d7ebd920a1c91..d64592409327b 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx @@ -33,13 +33,13 @@ By default, tracing support is added to the following LangGraph operations: - **Agent Creation** (`gen_ai.create_agent`) - Captures spans when compiling a StateGraph into an executable agent - **Agent Invocation** (`gen_ai.invoke_agent`) - Captures spans for agent execution via `invoke()` -- **Classifier Evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` calls in graph nodes +- **Classifier Evaluations** (`gen_ai.decide`) - Captures spans for `TypeSafeClassifier` calls in graph nodes ## Trace Classifier Evaluations - + -When a graph node calls a `TypeSafeClassifier` from [`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe), the LangChain integration records the call as a `gen_ai.evaluate` span. The span is a child of the graph's `invoke_agent` span and records the graph name as the agent name. No additional setup is needed. +When a graph node calls a `TypeSafeClassifier` from [`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe), the LangChain integration records the call as a `gen_ai.decide` span. The span is a child of the graph's `invoke_agent` span and records the graph name as the agent name. No additional setup is needed. ```javascript import { Annotation, END, START, StateGraph } from "@langchain/langgraph"; diff --git a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx index 372cfd23222e2..9fa53685351d4 100644 --- a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx @@ -10,7 +10,7 @@ notSupported: -Requires Sentry SDK version `11.1.0` or higher. +Requires Sentry SDK version `11.7.0` or higher. @@ -18,9 +18,9 @@ _Import name: `Sentry.typesafeIntegration`_ This integration is enabled by default when tracing is enabled. To customize it, see [Modifying Default Integrations](./../#modifying-default-integrations). -The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture agent tracing `gen_ai.evaluate` spans, recording model, token usage, latency, and (when enabled) evaluation inputs and answers. +The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture agent tracing `gen_ai.decide` spans, recording model, token usage, latency, and (when enabled) decision inputs and answers. -If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-and-clef-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra Classifier Evaluations](/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations/). +If you call Jev through Vercel AI SDK's `experimental_decide()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-decisions) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-and-clef-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra Classifier Evaluations](/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations/). If you call Jev through LangChain's `TypeSafeClassifier`, including from LangGraph nodes and the `@langchain/typesafe` middleware, see the [LangChain integration](/platforms/javascript/guides/node/configuration/integrations/langchain/#trace-classifier-evaluations). @@ -70,7 +70,7 @@ For unbundled applications started with `bun run`, use [Manual Instrumentation]( ## Supported Operations -Tracing support is added to `TypeSafeClient.systemOne()` calls to evaluate state against a set of questions using Jev. Each call creates a `gen_ai.evaluate` span. Recorded answers include probabilities and confidence values when returned by TypeSafe. +Tracing support is added to `TypeSafeClient.systemOne()` calls to evaluate state against a set of questions using Jev. Each call creates a `gen_ai.decide` span named `decide `. Recorded answers include probabilities and confidence values when returned by TypeSafe. ## Supported Versions diff --git a/docs/platforms/javascript/common/configuration/integrations/vercelai.mdx b/docs/platforms/javascript/common/configuration/integrations/vercelai.mdx index f6092ec3f8d80..f12e105eb1342 100644 --- a/docs/platforms/javascript/common/configuration/integrations/vercelai.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/vercelai.mdx @@ -54,20 +54,20 @@ By default, tracing support is added to the following `ai` functions: - `embed()` - `embedMany()` - `rerank()` -- `experimental_evaluate()` — requires Sentry SDK `11.1.0` or higher and AI SDK 7 +- `experimental_decide()` — requires Sentry SDK `11.7.0` or higher and AI SDK `7.0.128` or higher Plus `generate()` and `stream()` on [`ToolLoopAgent`](https://ai-sdk.dev/docs/agents/overview#toolloopagent-class). -## Trace Jev Evaluations +## Trace Jev Decisions -With Sentry SDK `11.1.0` or higher and AI SDK 7, `vercelAIIntegration` automatically traces `experimental_evaluate()` calls, including calls to TypeSafe's Jev model through Vercel AI Gateway. Keep your existing Sentry setup with tracing enabled; no additional integration or telemetry registration is needed. +With Sentry SDK `11.7.0` or higher and AI SDK `7.0.128` or higher, `vercelAIIntegration` automatically traces `experimental_decide()` calls, including calls to TypeSafe's Jev model through Vercel AI Gateway. Keep your existing Sentry setup with tracing enabled; no additional integration or telemetry registration is needed. -Configure [AI Gateway authentication](https://ai-sdk.dev/docs/ai-sdk-core/evaluation#default-provider-strings) with `AI_GATEWAY_API_KEY` or Vercel OIDC, then call `experimental_evaluate()`: +Configure [AI Gateway authentication](https://ai-sdk.dev/docs/ai-sdk-core/decisions#default-provider-strings) with `AI_GATEWAY_API_KEY` or Vercel OIDC, then call `experimental_decide()`: ```javascript -import { experimental_evaluate } from "ai"; +import { experimental_decide } from "ai"; -const result = await experimental_evaluate({ +const result = await experimental_decide({ model: "typesafe-ai/jev", state: "I was charged twice. Please refund the extra charge.", questions: { @@ -85,11 +85,11 @@ const result = await experimental_evaluate({ console.log(result.answers.department); ``` -Each call creates a `gen_ai.evaluate` span with model, latency, and token usage when available. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages. For TypeSafe answers, Sentry also includes confidence values from `providerMetadata.typesafe.confidence` when present. +Each call creates a `gen_ai.decide` span named `decide `, with model, latency, and token usage when available. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages. For TypeSafe answers, Sentry also includes confidence values from `providerMetadata.typesafe.confidence` when present. -Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other Vercel AI calls. Disable these options if evaluation state or answers contain sensitive data. +Decision inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other Vercel AI calls. Disable these options if decision state or answers contain sensitive data. -The AI SDK's [evaluation API](https://ai-sdk.dev/docs/ai-sdk-core/evaluation) is experimental and may change in patch releases. If you use `TypeSafeClient.systemOne()` directly instead, see the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/). +The AI SDK's [decision API](https://ai-sdk.dev/docs/ai-sdk-core/decisions) is experimental and may change in patch releases. If you use `TypeSafeClient.systemOne()` directly instead, see the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/). ## Supported Versions diff --git a/docs/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai.mdx b/docs/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai.mdx index 636c5a6a2a693..86066ca101a89 100644 --- a/docs/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai.mdx +++ b/docs/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai.mdx @@ -32,7 +32,7 @@ You can browse the full list of available models in the [Cloudflare Workers AI m ## Trace Jev and Clef Evaluations - + Calls to TypeSafe's [Jev model](https://developers.cloudflare.com/ai/models/typesafe/jev/) (`typesafe/jev`) through `env.AI.run()` are traced automatically. No additional integration or setup is needed: @@ -71,9 +71,9 @@ const result = await env.AI.run("@cf/cloudflare/clef", { }); ``` -Each call creates a `gen_ai.evaluate` span named `evaluate `, with the response model (for example, `jev-1.13.0` or `clef`), latency, and token usage. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities and confidence values returned by the model. These spans have the same attributes as spans from the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/) on other runtimes. +Each call creates a `gen_ai.decide` span named `decide `, with the response model (for example, `jev-1.13.0` or `clef`), latency, and token usage. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities and confidence values returned by the model. These spans have the same attributes as spans from the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/) on other runtimes. -Evaluation state and answers can contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the `dataCollection` option. +Decision state and answers can contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the `dataCollection` option. ## Next Steps diff --git a/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx b/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx index f1a543e4be709..061356e26ddd5 100644 --- a/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx +++ b/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx @@ -1,10 +1,10 @@ --- title: Mastra Classifier Evaluations sidebar_title: Classifier Evaluations -description: "Learn how Sentry traces Mastra Classifier evaluations, such as calls to TypeSafe's Jev, as gen_ai.evaluate spans." +description: "Learn how Sentry traces Mastra Classifier evaluations, such as calls to TypeSafe's Jev, as gen_ai.decide spans." --- - + Mastra's [`Classifier`](https://mastra.ai/reference/classifier/classifier) asks a decision model, such as TypeSafe's Jev, typed questions about your application state. These calls don't go through the Vercel AI SDK, so the Mastra integration traces them for you. This includes classifiers that your processors and scorers call. If you've followed the [Mastra Quick Start](/platforms/javascript/guides/mastra/), no additional integration or setup is needed. @@ -36,7 +36,7 @@ console.log(result.answers.department.choice); ## What Gets Captured -Each `evaluate()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model. +Each `evaluate()` call creates a `gen_ai.decide` span named `decide `, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model. ## Control Recorded Data @@ -44,9 +44,9 @@ Sentry runs this data through your Mastra span output processors before recordin ## Supported Versions -- `@sentry/node`: `>=11.6.0` +- `@sentry/node`: `>=11.7.0` - `@mastra/core`: `>=1.69.0` ## Related Integrations -If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) integration. +If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_decide()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-decisions) integration. diff --git a/includes/mastra/what-gets-captured.mdx b/includes/mastra/what-gets-captured.mdx index 00ec9f6d2208a..fa347750aca51 100644 --- a/includes/mastra/what-gets-captured.mdx +++ b/includes/mastra/what-gets-captured.mdx @@ -12,7 +12,7 @@ Sentry's Mastra integration maps Mastra's span types to Sentry operations for th | `PROVIDER_TOOL_CALL` | `gen_ai.execute_tool` | | `CLIENT_TOOL_CALL` | `gen_ai.execute_tool` | | `RAG_EMBEDDING` | `gen_ai.embeddings` | -| `CLASSIFIER_EVALUATION` | `gen_ai.evaluate` | +| `CLASSIFIER_EVALUATION` | `gen_ai.decide` | Spans are tagged with `sentry.origin: auto.ai.mastra`.