
@@ -61,7 +61,7 @@ npm install openai
Set your Agent Command Center key in your terminal:
```bash
-export AGENTCC_API_KEY="sk-agentcc-your-api-key-here"
+export AGENTCC_API_KEY="your-gateway-api-key"
```
## 5. Point the SDK at the gateway
@@ -76,7 +76,7 @@ from openai import OpenAI
client = OpenAI(
base_url="https://gateway.futureagi.com/v1", # 1. point at the gateway
- api_key=os.environ["AGENTCC_API_KEY"], # 2. use your sk-agentcc- key
+ api_key=os.environ["AGENTCC_API_KEY"], # 2. use your gateway key
)
response = client.chat.completions.create(
@@ -91,7 +91,7 @@ import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.futureagi.com/v1", // 1. point at the gateway
- apiKey: process.env.AGENTCC_API_KEY, // 2. use your sk-agentcc- key
+ apiKey: process.env.AGENTCC_API_KEY, // 2. use your gateway key
});
const response = await client.chat.completions.create({
@@ -133,7 +133,7 @@ Congratulations! You've successfully routed your first request!🎉
Not seeing a response? Try checking these:
-- **Authentication error**: confirm your key starts with `sk-agentcc-` (Settings → API Keys)
+- **Authentication error**: confirm you are using the gateway key created at **AI Gateway → API keys**, not the personal credentials from **Build → Keys**. Open **AI Gateway** from the product switcher to manage gateway keys
- **Model or provider error**: make sure that provider is configured in Agent Command Center → Providers
- **Wrong base URL**: it must be `https://gateway.futureagi.com/v1`
diff --git a/src/pages/docs/get-started/send-your-first-trace.mdx b/src/pages/docs/get-started/send-your-first-trace.mdx
index a30cb80f3..f0d7b891e 100644
--- a/src/pages/docs/get-started/send-your-first-trace.mdx
+++ b/src/pages/docs/get-started/send-your-first-trace.mdx
@@ -12,12 +12,12 @@ You can add [custom spans](/docs/cookbook/quickstart/manual-tracing), too.
## Prerequisites
- A Future AGI account and your **`FI_API_KEY`** and **`FI_SECRET_KEY`** (Dashboard → Build → Keys)
-- Python 3.10+
+- Python 3.10–3.13 for the Python example (`traceAI-openai` requires Python below 3.14)
- An OpenAI API key
-Find both keys at **Dashboard → Build → Keys**. Copy the **API Key** (`FI_API_KEY`) and **Secret Key** (`FI_SECRET_KEY`):
+Open **Dashboard → Build → Keys → Add API Key** and create a personal key. Copy both the **API Key** (`FI_API_KEY`) and **Secret Key** (`FI_SECRET_KEY`) from the one-time creation dialog before closing it. Existing entries in the Keys list cannot reveal these values again.
-
+
## 1. Install traceAI
@@ -74,7 +74,6 @@ print(response.choices[0].message.content)
```typescript JS/TS
import { register, ProjectType } from "@traceai/fi-core";
import { OpenAIInstrumentation } from "@traceai/openai";
-import { registerInstrumentations } from "@opentelemetry/instrumentation";
import OpenAI from "openai";
// Connect to Future AGI and create (or reuse) a project
@@ -83,11 +82,10 @@ const tracerProvider = register({
projectName: "my-llm-app",
});
-// Auto-instrument OpenAI: every call is now traced
-registerInstrumentations({
- instrumentations: [new OpenAIInstrumentation({})],
- tracerProvider,
-});
+// Explicitly instrument the imported SDK for ESM/TypeScript
+const instrumentation = new OpenAIInstrumentation({});
+instrumentation.setTracerProvider(tracerProvider);
+instrumentation.manuallyInstrument(OpenAI);
// Use OpenAI exactly as you normally would
const client = new OpenAI();
@@ -142,7 +140,7 @@ Congratulations! You've successfully sent your first trace!🎉
Not seeing your traces? Try checking these:
-- **Authentication error**: re-check `FI_API_KEY` and `FI_SECRET_KEY` against Dashboard → Build → Keys
+- **Authentication error**: check that `FI_API_KEY` and `FI_SECRET_KEY` are the pair saved from the creation dialog. If you did not save both values, create a new personal key at **Dashboard → Build → Keys → Add API Key** and copy both before closing
- **No trace in Tracing**: let the script finish. Traces flush as the program exits, so don't stop it early
- **Wrong place**: confirm you're on the **Tracing** tab of platform and in the `my-llm-app` project
diff --git a/src/pages/docs/integrations/coding-agents/gemini-cli.mdx b/src/pages/docs/integrations/coding-agents/gemini-cli.mdx
index c9dcd2e9a..503057fc8 100644
--- a/src/pages/docs/integrations/coding-agents/gemini-cli.mdx
+++ b/src/pages/docs/integrations/coding-agents/gemini-cli.mdx
@@ -139,7 +139,7 @@ Model messages and tool arguments/results then appear as input and output in Fut
**Port 4318 is already in use.** Set `GEMINI_OTLP_LISTEN=127.0.0.1:24318` before starting the collector and change Gemini CLI's `otlpEndpoint` to `http://127.0.0.1:24318`.
-**The collector reports an export error.** For `401` or `403`, check your Future AGI keys. For `400`, keep `compression: none` in the exporter: the Future AGI endpoint requires uncompressed OTLP requests for this setup.
+**The collector reports an export error.** For `401` or `403`, check your Future AGI keys. For `400`, check the response and that the content type matches the decoded OTLP body. HTTP accepts uncompressed bodies or a single `gzip`/`x-gzip` layer with a matching `Content-Encoding` header, so `compression: none` is optional. For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it).
**Model spans have no token counts.** Set `telemetry.traces` to `true` and restart Gemini CLI.
diff --git a/src/pages/docs/integrations/frameworks/tanstack-ai.mdx b/src/pages/docs/integrations/frameworks/tanstack-ai.mdx
index ddc25ae5b..084c91d76 100644
--- a/src/pages/docs/integrations/frameworks/tanstack-ai.mdx
+++ b/src/pages/docs/integrations/frameworks/tanstack-ai.mdx
@@ -63,9 +63,9 @@ export const GEN_AI_SPAN_KIND = "gen_ai.span.kind";
* sets the project_name and project_type resource attributes the collector
* requires. A batch without project_name is dropped.
*
- * `batch: true` is not passed: with @traceai/fi-core 1.0.0 on
- * @opentelemetry/sdk-trace(-base) 2.11.0 it does not attach the batch
- * processor, so spans go through register()'s default SimpleSpanProcessor.
+ * This recipe leaves `batch` unset and uses the default SimpleSpanProcessor.
+ * @traceai/fi-core 1.2.3 supports `batch: true`; if you upgrade and enable
+ * batching, await forceFlush() before returning from a short-lived invocation.
*
* `setGlobalTracerProvider: false`: the recipe passes this provider's tracer
* to otelMiddleware, so it never needs the global provider, and leaving the
@@ -520,7 +520,7 @@ TanStack starts its chat span under whatever span is active. If your app has its
- **Tokens are counted once.** TanStack also writes the sum of every model call's usage onto the chat span. Future AGI sums `gen_ai.usage.*` over every span of a trace for the trace view, sessions, dashboards and cost, so the recipe moves the chat span's copy to `tanstack.ai.root_usage.*` whenever the model-call spans carry usage. If no model call reported usage, the chat span keeps it.
- **Sessions.** With `threadIdAsSession: true` and a `threadId` on `chat()`, every span carries `session.id` and Future AGI groups the conversation. Without a `threadId`, TanStack generates `thread-
-` per call, so the recipe sets no session.
- **Exporter down.** With nothing listening at `FI_BASE_URL`, `chat()` and the route still answer; `flushTraces()` logs `[futureagi] span export failed` and never throws. A collector that accepts but never answers costs the request at most the 2 s flush bound.
-- **Batching.** `register({ batch: true })` has no effect with `@traceai/fi-core` 1.0.0 on OpenTelemetry SDK 2.x, so the recipe uses the default per-span export. With a very slow collector the OTLP exporter's limit of 30 exports in flight can drop spans (about 8 concurrent chats of 4 spans).
+- **Batching.** This recipe leaves `batch` unset, using the default per-span export (`batch: false`). In `@traceai/fi-core` 1.2.3, `register({ batch: true })` creates an OpenTelemetry `BatchSpanProcessor`; no separate provider is needed. If you upgrade and enable batching, await `tracerProvider.forceFlush()` after active spans end and before returning or exiting, so queued or in-flight spans are not cut off. **Historical compatibility:** the recipe's pinned 1.0.0 package did not attach a batch processor on OpenTelemetry SDK 2.x. With that per-span export and a very slow collector, the OTLP exporter's limit of 30 exports in flight can drop spans (about 8 concurrent chats of 4 spans).
- **Global provider untouched.** See [Apps that already run OpenTelemetry](#apps-that-already-run-opentelemetry).
## Limitations
diff --git a/src/pages/docs/integrations/traceai/bedrock-agentcore.mdx b/src/pages/docs/integrations/traceai/bedrock-agentcore.mdx
index ee1baab14..1f111ee8d 100644
--- a/src/pages/docs/integrations/traceai/bedrock-agentcore.mdx
+++ b/src/pages/docs/integrations/traceai/bedrock-agentcore.mdx
@@ -150,7 +150,7 @@ export OTEL_SERVICE_NAME=order-agent
export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,gen_ai_span_attributes_only,gen_ai_unredacted_attributes=
```
-The trace-specific endpoint is the **full URL**, not an origin. Remove any conflicting `OTEL_EXPORTER_OTLP_TRACES_HEADERS` override. Use uncompressed HTTP/protobuf; do not enable gzip. Plain OTLP exporters do not read `FI_*` authentication variables.
+The trace-specific endpoint is the **full URL**, not an origin. Remove any conflicting `OTEL_EXPORTER_OTLP_TRACES_HEADERS` override. Use HTTP/protobuf, either uncompressed or with a single `gzip` or `x-gzip` layer and a matching `Content-Encoding` header. Both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it). Plain OTLP exporters do not read `FI_*` authentication variables.
Strands' `gen_ai_unredacted_attributes=` setting redacts messages and tool arguments/results, **not exception messages or stack traces**. Those can contain sensitive data. The `FI_HIDE_*` settings do not apply to Strands' native spans. If you need error-detail hiding, use the LangChain/traceAI path above or a separately configured trusted collector that enforces your policy before export.
diff --git a/src/pages/docs/integrations/traceai/go/a2a.mdx b/src/pages/docs/integrations/traceai/go/a2a.mdx
index 7baf9262a..df1d90e30 100644
--- a/src/pages/docs/integrations/traceai/go/a2a.mdx
+++ b/src/pages/docs/integrations/traceai/go/a2a.mdx
@@ -41,7 +41,7 @@ For self-hosting, set `FI_BASE_URL` to your Future AGI API base URL. The default
## 3. Configure OpenTelemetry
-The package uses your tracer provider. Configure the standard OTLP/HTTP exporter with both authentication headers, the `project_name` resource attribute and no compression.
+The package uses your tracer provider. Configure the standard OTLP/HTTP exporter with both authentication headers and the `project_name` resource attribute. This example sends uncompressed bodies; HTTP also accepts a single `gzip` or `x-gzip` layer. OTLP/gRPC export remains uncompressed-only.
```go
import (
@@ -223,6 +223,6 @@ Task IDs, states, agent URLs, card metadata, session IDs and user IDs remain vis
## Troubleshooting
- **No traces:** check both Future AGI keys, `FI_PROJECT_NAME`, and provider shutdown. The exporter URL must end in `/tracer/v1/traces`, and `project_name` must be a resource attribute, not a span attribute.
-- **Export body errors:** use `otlptracehttp.NoCompression`; the collector does not accept gzip.
+- **Export body errors:** check that the content type matches the decoded OTLP body and that gzip bodies declare `Content-Encoding: gzip` (or `x-gzip`). For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it).
- **Missing stream spans:** call the returned traceAI client rather than `.Client`, use `WrapHandler` on the server, and set a deadline for blocked reads.
- **Separate client and server traces:** install the server interceptor and preserve W3C trace-context headers or gRPC metadata across proxies. An uninstrumented remote agent contributes no child spans.
diff --git a/src/pages/docs/integrations/traceai/go/eino.mdx b/src/pages/docs/integrations/traceai/go/eino.mdx
index b54f1427b..c28742668 100644
--- a/src/pages/docs/integrations/traceai/go/eino.mdx
+++ b/src/pages/docs/integrations/traceai/go/eino.mdx
@@ -63,7 +63,7 @@ export FI_PROJECT_NAME="my-service"
## 3. Create the tracer provider
-The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with the three things Future AGI needs: both auth headers, the `project_name` resource attribute, and an uncompressed body (the collector rejects gzip).
+The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with both auth headers and the `project_name` resource attribute. This example sends uncompressed bodies; HTTP also accepts a single `gzip` or `x-gzip` layer. OTLP/gRPC export remains uncompressed-only.
```go
import (
@@ -315,7 +315,7 @@ The embedding callback mapping omits embedding vectors. The module adds no resou
## Troubleshooting
- **No traces.** Check that the process calls `h.Shutdown` and `tp.Shutdown` before it exits, that the handler was registered before the first run, and that `FI_API_KEY`, `FI_SECRET_KEY` and `FI_PROJECT_NAME` are set. The exporter URL must be the full traces URL, `{FI_BASE_URL}/tracer/v1/traces`.
-- **Export errors about the body.** Keep `otlptracehttp.WithCompression(otlptracehttp.NoCompression)`; the collector rejects gzip.
+- **Export errors about the body.** Check that the content type matches the decoded OTLP body and that gzip bodies declare `Content-Encoding: gzip` (or `x-gzip`). For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it).
- **Spans in the wrong project or none at all.** `project_name` must be a resource attribute on the tracer provider, not a span attribute.
- **Two spans for every node.** The handler is registered twice: `AppendGlobalHandlers` ran twice, or the handler is global and also passed with `compose.WithCallbacks` / `adk.WithCallbacks`.
- **No model name or token counts on a model span.** The chat model fires no callbacks itself, so the graph reports only its messages. The eino-ext models report both; for your own model, fire `callbacks.OnStart` / `OnEnd` with `*model.CallbackInput` / `*model.CallbackOutput`.
diff --git a/src/pages/docs/integrations/traceai/go/gemini.mdx b/src/pages/docs/integrations/traceai/go/gemini.mdx
index 7e305434f..546626b0d 100644
--- a/src/pages/docs/integrations/traceai/go/gemini.mdx
+++ b/src/pages/docs/integrations/traceai/go/gemini.mdx
@@ -48,7 +48,7 @@ export GEMINI_API_KEY="your-gemini-api-key"
## 3. Create the tracer provider
-The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with the three things Future AGI needs: both auth headers, the `project_name` resource attribute, and an uncompressed body (the collector rejects gzip).
+The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with both auth headers and the `project_name` resource attribute. This example sends uncompressed bodies; HTTP also accepts a single `gzip` or `x-gzip` layer. OTLP/gRPC export remains uncompressed-only.
```go
import (
@@ -267,7 +267,7 @@ With `FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true` the model, token counts, functio
## Troubleshooting
- **No traces.** Check that the process calls `shutdown(tp)` (or `tp.Shutdown`) before it exits, and that `FI_API_KEY`, `FI_SECRET_KEY` and `FI_PROJECT_NAME` are set. The exporter URL must be the full traces URL, `{FI_BASE_URL}/tracer/v1/traces`.
-- **Export errors about the body.** Keep `otlptracehttp.WithCompression(otlptracehttp.NoCompression)`; the collector rejects gzip.
+- **Export errors about the body.** Check that the content type matches the decoded OTLP body and that gzip bodies declare `Content-Encoding: gzip` (or `x-gzip`). For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it).
- **Spans in the wrong project or none at all.** `project_name` must be a resource attribute on the tracer provider, not a span attribute.
- **No span for a call.** Check that `HTTPClient` on the `genai.ClientConfig` is the traced client. Only generation and embedding calls are spanned.
- **ERROR spans with `RESOURCE_EXHAUSTED` (429) or `UNAVAILABLE` (503).** Those are Gemini quota or capacity errors, recorded as they happened. On the free tier, space your requests or use another model.
diff --git a/src/pages/docs/integrations/traceai/go/openai.mdx b/src/pages/docs/integrations/traceai/go/openai.mdx
index d74dc28a0..fcd306bea 100644
--- a/src/pages/docs/integrations/traceai/go/openai.mdx
+++ b/src/pages/docs/integrations/traceai/go/openai.mdx
@@ -36,7 +36,7 @@ export OPENAI_API_KEY="your-openai-api-key"
## 3. Create the tracer provider
-The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with the three things Future AGI needs: both auth headers, the `project_name` resource attribute, and an uncompressed body (the collector rejects gzip).
+The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with both auth headers and the `project_name` resource attribute. This example sends uncompressed bodies; HTTP also accepts a single `gzip` or `x-gzip` layer. OTLP/gRPC export remains uncompressed-only.
```go
import (
@@ -238,7 +238,7 @@ A boolean switch counts only when it is `true` (any case). With both broad switc
## Troubleshooting
- **No traces.** Check that the process calls `shutdown(tp)` (or `tp.Shutdown`) before it exits, and that `FI_API_KEY`, `FI_SECRET_KEY` and `FI_PROJECT_NAME` are set. The exporter URL must be the full traces URL, `{FI_BASE_URL}/tracer/v1/traces`.
-- **Export errors about the body.** Keep `otlptracehttp.WithCompression(otlptracehttp.NoCompression)`; the collector rejects gzip.
+- **Export errors about the body.** Check that the content type matches the decoded OTLP body and that gzip bodies declare `Content-Encoding: gzip` (or `x-gzip`). For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it).
- **Spans in the wrong project or none at all.** `project_name` must be a resource attribute on the tracer provider, not a span attribute.
- **No token counts on a streamed span.** Set `StreamOptions.IncludeUsage`; without it the API sends no usage chunk.
- **Every retry is a separate trace.** Start a parent span around the call (step 5).
diff --git a/src/pages/docs/integrations/traceai/gradio.mdx b/src/pages/docs/integrations/traceai/gradio.mdx
index fb4774db5..1679d6732 100644
--- a/src/pages/docs/integrations/traceai/gradio.mdx
+++ b/src/pages/docs/integrations/traceai/gradio.mdx
@@ -116,7 +116,7 @@ No Future AGI key, secret or model key is exported. The recipe never prints the
- **Turns are not grouped.** `predict` ran without a `gr.Request`, so no session id was set.
- **Generator chat functions.** Generator callbacks are outside this synchronous recipe. Gradio can run generator steps in separate context copies; do not assume a span opened around a generator remains its parent.
- **Reload mode (`gradio app.py`).** Gradio re-runs the file in the same process; the first instrumentation stays in force. Changes to the tracing setup need a restart. You will see `Attempting to instrument while already instrumented` once at startup and once per save.
-- **Nothing exported after kill.** `register()` flushes on normal exit, SIGINT and SIGTERM. SIGKILL skips the flush.
+- **Nothing exported after kill.** `register()` installs an `atexit` hook for normal interpreter exit, not handlers for `SIGINT` or `SIGTERM`. Default `SIGTERM` termination does not run that hook; `SIGINT` normally raises `KeyboardInterrupt`. Keep the provider returned by `init_tracing()` and call `provider.shutdown()` from your application's or framework's graceful-shutdown hook after active spans end, preserving its signal handlers and exit policy. Flush at task boundaries with `provider.force_flush()` too: cleanup must finish within the termination grace period, and no hook can run after `SIGKILL`.
## Limitations
diff --git a/src/pages/docs/knowledge-base/guides/manage-with-the-sdk.mdx b/src/pages/docs/knowledge-base/guides/manage-with-the-sdk.mdx
index 550f15afa..beac79c59 100644
--- a/src/pages/docs/knowledge-base/guides/manage-with-the-sdk.mdx
+++ b/src/pages/docs/knowledge-base/guides/manage-with-the-sdk.mdx
@@ -19,7 +19,7 @@ pip install futureagi
## Set your credentials
-The client reads your API key and secret from the environment. Get `FI_API_KEY` and `FI_SECRET_KEY` from Settings > API Keys (see [API Keys](/docs/admin-settings/api-keys)), then set them before you run your script:
+The client reads your API key and secret from the environment. Get `FI_API_KEY` and `FI_SECRET_KEY` from **Keys** under **Build** in the main sidebar (see [API Keys](/docs/admin-settings/api-keys)), then set them before you run your script:
```bash
export FI_API_KEY="your-api-key"
diff --git a/src/pages/docs/observe/features/quickstart.mdx b/src/pages/docs/observe/features/quickstart.mdx
index 46a2e4754..ba084ec2e 100644
--- a/src/pages/docs/observe/features/quickstart.mdx
+++ b/src/pages/docs/observe/features/quickstart.mdx
@@ -61,7 +61,7 @@ This is how you connect your application to Future AGI so LLM calls are captured
trace_provider = register(
project_type=ProjectType.OBSERVE,
project_name="FUTURE_AGI",
- transport=Transport.GRPC,
+ transport=Transport.HTTP,
)
```
```typescript
diff --git a/src/pages/docs/observe/quickstart.mdx b/src/pages/docs/observe/quickstart.mdx
index 02ab1e095..7e9a6f980 100644
--- a/src/pages/docs/observe/quickstart.mdx
+++ b/src/pages/docs/observe/quickstart.mdx
@@ -59,7 +59,7 @@ Pin the packages to the version you test against, so a later release cannot chan
trace_provider = register(
project_type=ProjectType.OBSERVE,
project_name="self-improving-agent",
- transport=Transport.GRPC,
+ transport=Transport.HTTP,
)
# Auto-instrument OpenAI: every call is now traced
@@ -77,7 +77,7 @@ Pin the packages to the version you test against, so a later release cannot chan
import { register, ProjectType } from "@traceai/fi-core";
import { OpenAIInstrumentation } from "@traceai/openai";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
- import OpenAI from "openai";
+ import * as OpenAI from "openai";
// Connect to Future AGI and create (or reuse) an Observe project
const traceProvider = register({
@@ -85,14 +85,16 @@ Pin the packages to the version you test against, so a later release cannot chan
projectName: "self-improving-agent",
});
- // Auto-instrument OpenAI: every call is now traced
+ const openAIInstrumentation = new OpenAIInstrumentation({});
registerInstrumentations({
- instrumentations: [new OpenAIInstrumentation({})],
+ instrumentations: [openAIInstrumentation],
tracerProvider: traceProvider,
});
+ // Native ESM imports need an explicit patch before creating the client
+ openAIInstrumentation.manuallyInstrument(OpenAI);
// Use OpenAI exactly as you normally would
- const client = new OpenAI();
+ const client = new OpenAI.OpenAI();
const completion = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Suggest one habit I could build to keep improving every day." }],
diff --git a/src/pages/docs/observe/reference/export-formats.mdx b/src/pages/docs/observe/reference/export-formats.mdx
index d7bd319d8..30f1570f2 100644
--- a/src/pages/docs/observe/reference/export-formats.mdx
+++ b/src/pages/docs/observe/reference/export-formats.mdx
@@ -1,6 +1,6 @@
---
title: "Export and endpoints"
-description: "Reference for getting trace data out of Observe — exporting the current view, the available formats, and the OTLP endpoints traces are sent to (cloud and self-hosted)."
+description: "Reference for Observe's current CSV export limitation and the OTLP endpoints traces are sent to (cloud and self-hosted)."
slug: "export-formats"
page_type: "reference"
products: ["Observe"]
@@ -13,12 +13,12 @@ last_tested: "2026-05-25"
schema_type: "TechArticle"
seo:
title: "Trace export and endpoints in Observe"
- description: "Reference for getting trace data out of Observe — exporting the current view, the available formats, and the OTLP endpoints traces are sent to (cloud and self-hosted)."
+ description: "Reference for Observe's current CSV export limitation and the OTLP endpoints traces are sent to (cloud and self-hosted)."
primary_keyword: "export traces endpoints futureagi"
direct_answer: true
geo:
answer_target: "How do I export traces from Observe, and what endpoints does traceAI send to?"
- llm_summary: "Observe exports the current trace-explorer view (filters and time range) to CSV from the download icon. Inbound, the traceAI SDK sends spans over OTLP to FutureAGI — set FI_BASE_URL (HTTP) or FI_GRPC_URL (gRPC), defaulting to FutureAGI cloud, or point them at your own collector when self-hosted."
+ llm_summary: "Exact CSV export from the trace explorer is unavailable; the download control is disabled. Inbound, the traceAI SDK sends spans over OTLP to FutureAGI — set FI_BASE_URL (HTTP) or FI_GRPC_URL (gRPC), defaulting to FutureAGI cloud, or point them at your own collector when self-hosted."
canonical: "/docs/observe/reference/export-formats"
related:
- "/docs/observe/guides/explore-dashboard"
@@ -32,19 +32,9 @@ There are two directions for trace data: **out of** Observe (exporting what you'
## Export from the trace explorer
-The download icon in the [trace explorer](/docs/observe/guides/explore-dashboard) header exports the **current view** — the traces that match your active filters and time range.
+Exact CSV export from the [trace explorer](/docs/observe/guides/explore-dashboard) is **unavailable**. The download control in the LLM Tracing and Sessions headers is disabled and displays “Exact CSV export is temporarily unavailable.”
-**CSV is the only export format.** There is no JSON or Parquet download from the trace explorer.
-
-| Format | Use for |
-|---|---|
-| CSV | Spreadsheet analysis, sharing, importing elsewhere. |
-
-The CSV holds one row per span in the current view, with the columns the explorer shows: trace ID, span name, status, model, provider, latency, token counts, cost, timestamp, and any eval scores present on the span.
-
-
- The export reflects the current view, so a very large view may be truncated. Narrow the filters or time range to export a complete slice.
-
+You cannot download the current view from this control. Changing filters or the time range does not enable it.
## Ingestion endpoints
@@ -103,7 +93,7 @@ OTEL_BSP_MAX_EXPORT_BATCH_SIZE=256
- Filter, then export the current view.
+ Explore traces in the dashboard.
Configure the OTLP endpoint and transport.
diff --git a/src/pages/docs/prompt/reference/sdk-api.mdx b/src/pages/docs/prompt/reference/sdk-api.mdx
index 38433f673..12955951e 100644
--- a/src/pages/docs/prompt/reference/sdk-api.mdx
+++ b/src/pages/docs/prompt/reference/sdk-api.mdx
@@ -32,7 +32,7 @@ export FI_API_KEY="your-api-key"
export FI_SECRET_KEY="your-secret-key"
```
-Find both under **Settings → API Keys** in the platform.
+Create a pair from **Keys** under **Build** in the main sidebar ([API Keys](/docs/admin-settings/api-keys)).
## Construct a template
diff --git a/src/pages/docs/protect/concepts/understanding-protect.mdx b/src/pages/docs/protect/concepts/understanding-protect.mdx
index 528194459..c7045dd76 100644
--- a/src/pages/docs/protect/concepts/understanding-protect.mdx
+++ b/src/pages/docs/protect/concepts/understanding-protect.mdx
@@ -5,27 +5,27 @@ description: "Protect screens your traffic for safety issues through guardrails
## A guardrail is a named check on your traffic
-A **guardrail** wraps a check and adds three settings you configure in [Agent Command Center](/docs/command-center/features/guardrails), under **Gateway** > **Guardrails**:
+A **guardrail** wraps a check. In [Agent Command Center](/docs/command-center/features/guardrails), under **Gateway** > **Guardrails**, its rule has:
-- An **action** for when it triggers: Block, Warn, Mask, or Log
-- A **stage** for when it runs: **pre** runs before the model sees the request, **post** runs before the response reaches the caller (on a streamed response, after it has been sent; see [Streaming Guardrails](/docs/command-center/features/guardrails#streaming-guardrails)), and **both** runs on each side
-- A **confidence threshold** for how sure the check has to be before it fires
+- An **action** for when it triggers: Block, Warn, or Log, with Mask available for checks that can rewrite the request
+- A fixed **stage**, determined by the check type: request (**pre**) or response (**post**)
+- A **match threshold**, where the check exposes one, for the score that counts as a match
-A stage of **both** isn't a third mode. It asks for the same check to run twice, once on the way in and once on the way out. As a setting, stage belongs to the guardrail rather than to the check it wraps, which is why a single **gateway**, the Agent Command Center surface your guardrails live on, can carry a mix: some watching only what the caller sent, others only what the model sent back. Today the gateway doesn't apply the setting: it runs each check at its built-in stage, so Data Leakage Prevention and Hallucination Detection check the response, and the others check the request.
+Organization checks do not have a dashboard stage selector. **Hallucination Detection** and **Data Leakage Prevention** read the model's answer; the other catalog checks, including **PII Detection**, read the request before the model is called. The Rules drawer describes this stage but cannot move a check to post or make it run on both sides. On a streamed response, post-stage checks run after the response text has been sent; see [Streaming Guardrails](/docs/command-center/features/guardrails#streaming-guardrails).
The threshold runs from 0.0 to 1.0: raising it makes the check fire only on higher-confidence matches, lowering it makes it fire more readily. Threshold and action are independent settings. The threshold decides how often a check fires; the action decides what the caller experiences when it does. Loosening a threshold on a check set to Log changes only whether the call carries the `x-agentcc-guardrail-triggered` header. For a non-streamed response, Block from either stage returns `403`; Warn and Log let the call return `200`. On a stream, only a pre-stage check can change the already-sent HTTP status. For the status each action produces, see [Guardrail checks](/docs/protect/reference/guardrail-checks#response-statuses).
-Every guardrail wraps a check, and the Rules tab groups checks under two headers that mark a single split: where the check runs. **Rule-Based Checks** are [first-party](/docs/protect/reference/guardrail-checks): they run inside Protect without an external provider. **AI-Powered Checks** call a provider you configure under **Provider Settings** in that check's own settings.
+Where a check executes is separate from which side it reads. [Built-in checks](/docs/protect/reference/guardrail-checks) run inside the gateway; provider-backed checks call a service. See the reference for which checks the console offers and which saved integrations are read-only.
A["Action: Block, Warn, Mask, or Log"]
- G --> S["Stage: pre, post, or both"]
- G --> T["Confidence threshold"]
+ accDescr: An organization guardrail has an action, a stage fixed by its check type, and a match threshold where applicable. Built-in checks run in the gateway; provider-backed checks call a service.
+ G["Guardrail"] --> A["Action: Block, Warn, or Log; Mask where supported"]
+ G --> S["Fixed stage: request or response"]
+ G --> T["Match threshold where applicable"]
G --> C["Check"]
- C --> RBC["Rule-Based: runs inside Protect"]
- C --> AIP["AI-Powered: external provider"]`} />
+ C --> RBC["Built-in: runs in the gateway"]
+ C --> AIP["Provider-backed: calls a service"]`} />
## protect() screens one input in your code
@@ -52,13 +52,13 @@ This record belongs to the dashboard-guardrail surface, while `protect()` return
Treating a guardrail and `protect()` as one surface means assuming a check you configured in one place is protecting you in the other, when it isn't.
-Two guardrails with the same check can still behave completely differently once their action, stage, or threshold diverge, because a guardrail is this fixed shape, not a single on/off switch.
+Rules using the same check can behave differently when their action or supported settings differ. Their request or response stage remains fixed by the check type.
## Keep exploring
- Enable a check, set its stage, threshold, and action
+ Add a check, set its action, and confirm where it runs
Browse the full list of first-party and provider-backed checks
diff --git a/src/pages/docs/protect/guides/run-protect-from-the-sdk.mdx b/src/pages/docs/protect/guides/run-protect-from-the-sdk.mdx
index 41cfced0e..43db63a69 100644
--- a/src/pages/docs/protect/guides/run-protect-from-the-sdk.mdx
+++ b/src/pages/docs/protect/guides/run-protect-from-the-sdk.mdx
@@ -36,7 +36,9 @@ rules = [
For how these checks fit into the guardrail model, see [Understanding Protect](/docs/protect/concepts/understanding-protect).
-The SDK accepts these `metric` values: `toxicity`, `bias`, `bias_detection`, `sexist`, `prompt_injection`, `data_privacy_compliance`, and `pii`. A `metric` name outside this list raises an error, so confirm the name before you ship it.
+Use one of these four canonical `metric` values: `toxicity`, `bias_detection`, `prompt_injection`, or `data_privacy_compliance`. The names `bias`, `sexist`, and `pii` are not accepted and raise an error.
+
+Two deprecated aliases remain supported: `content_moderation` maps to `toxicity`, and `security` maps to `prompt_injection`. Use the canonical names in new code.
## Run the check
diff --git a/src/pages/docs/protect/guides/test-a-guardrail.mdx b/src/pages/docs/protect/guides/test-a-guardrail.mdx
index 2e1d23dda..b3fb1368a 100644
--- a/src/pages/docs/protect/guides/test-a-guardrail.mdx
+++ b/src/pages/docs/protect/guides/test-a-guardrail.mdx
@@ -5,6 +5,12 @@ description: "Run a prompt through your active guardrails from the Test tab and
The Test tab runs a single prompt through your active guardrail configuration and shows you the verdict immediately, so you can check how a guardrail behaves before any traffic from your app reaches it.
+## Before you start
+
+You must be an organization **Owner** or **Admin** to run a test. Members and Viewers can open the tab, but **Run test** stays disabled with **Owner or Admin only**; entering a prompt does not grant permission.
+
+A gateway must be selected. If it reports **Guardrails are off on this gateway**, testing is disabled: ask the gateway operator to enable its guardrail pipeline. Saving or enabling an individual rule does not turn that pipeline on.
+
You need a guardrail already turned on for a run to mean anything; with nothing configured, every prompt just passes through. If you haven't set one up yet, [turn on a guardrail](/docs/protect/guides/turn-on-a-guardrail) first.
@@ -25,10 +31,10 @@ Below the prompt box, "Model (optional)" takes a model name, with placeholder te
## Run the test
-Press **Run Test**. It stays disabled until there's a prompt in the box, whether you typed one or picked a chip, and while a test is in flight it reads "Running...".
+Press **Run test**. It is enabled only when you have the Owner/Admin role, a selected gateway, a non-blank prompt, and no test already in flight, and the gateway does not report its guardrail pipeline as off. While a test is in flight, the button reads **Running**.
-*The **PII test** chip fills the box for you, so Run Test is live without typing anything*
+*The **PII test** chip supplies the prompt; the role, gateway, pipeline, and in-flight-test conditions still apply.*
## Read the result
@@ -42,6 +48,8 @@ Below the chip, **Guardrail Headers** and **Response Body** show the guardrail h
## If something doesn't look right
+**Run test is disabled.** Check your organization role, selected gateway, prompt, whether another test is running, and any pipeline-off warning. A Member or Viewer needs an Owner or Admin to run the test.
+
**The test didn't run.** If the test call itself can't complete, an error alert replaces the result chip.
**The verdict was wrong.** A chip that doesn't match what you expected, for example a prompt you thought would block coming back **OK**, points back to the guardrail's configuration rather than the panel. See [Guardrail fires on the wrong requests](/docs/protect/troubleshooting/guardrail-fires-on-the-wrong-requests) to fix it.
diff --git a/src/pages/docs/protect/guides/turn-on-a-guardrail.mdx b/src/pages/docs/protect/guides/turn-on-a-guardrail.mdx
index 4df383aad..fdfa87391 100644
--- a/src/pages/docs/protect/guides/turn-on-a-guardrail.mdx
+++ b/src/pages/docs/protect/guides/turn-on-a-guardrail.mdx
@@ -1,59 +1,42 @@
---
title: "Turn on a guardrail"
-description: "Enable a check in Protect's guardrail catalog and set how it behaves."
+description: "Add a check in Protect's guardrail catalog and set how it behaves."
---
-Every check in Protect's guardrail catalog starts out switched off. This guide flips one on end to end, using **PII Detection** configured for a support agent as the running example: enable it, configure it, and push it live. The same steps apply to any other check in the catalog.
+This guide adds **PII Detection** for a support agent: choose the check, set its action, and save it to the gateway.
-This guide assumes Agent Command Center, where Protect's guardrails live under Gateway, is already set up with at least one provider connected. If it isn't yet, start with the [Agent Command Center Quickstart](/docs/command-center/quickstart).
-
-
-*PII Detection sits below every AI-Powered check, so expect to scroll a fair way down the Rules catalog before you reach it*
+You need an organization **Owner** or **Admin** role and Agent Command Center set up with a connected gateway and at least one provider. If it isn't set up yet, start with the [Agent Command Center Quickstart](/docs/command-center/quickstart).
## Find the check in Rules
-Go to **Gateway** > **Guardrails** > **Rules**. The Rules tab lists the full guardrail catalog as a list of check cards, split between **AI-Powered Checks** and **Rule-Based Checks**. **PII Detection** sits under **Rule-Based Checks**; find its card there. Each card carries a **Switch** and a pencil icon button.
-
-## Turn it on
+Go to **Gateway** > **Guardrails** > **Rules**. If no checks are configured, the page shows **No guardrails**. Click **Add check**, then choose **PII Detection** under **Built into the gateway**. You can also use **Search checks** to find it.
-Click the **Switch** on the **PII Detection** card. This only stages the change: it's what raises the unsaved-changes banner covered in [Push the change to the gateway](#push-the-change-to-the-gateway) below, and nothing reaches the gateway until you save there. How the check behaves is set separately, from the pencil.
+If PII Detection is already configured, open its saved row in Rules instead of adding it again.
## Set how it behaves
-Click the pencil icon button on the card to open **Configure: PII Detection**, then set:
+In the **PII Detection** drawer:
-1. **Enabled**: leave it on (it mirrors the card's **Switch**, so it already matches what you just set)
-2. **Action**: set to **Block**
-3. **Confidence Threshold**: a slider running 0.0 to 1.0, with the current value shown in the label; leave it at its default of 0.80
+1. Leave the switch **On**.
+2. Set **Action** to **Block**.
-For provider-backed checks, the dialog also has a Provider Settings section (see the note below). It always has Cancel and Save buttons. For what action and threshold actually do to a request, see [Understanding Protect](/docs/protect/concepts/understanding-protect).
+The drawer describes where the check runs: PII Detection reads the request before the model is called. It has no provider connection to configure. With its default settings, it has no **Match threshold** slider; see [Guardrail fires on the wrong requests](/docs/protect/troubleshooting/guardrail-fires-on-the-wrong-requests) for tuning limits.
-Click **Save** to close the dialog. The card changes to show it's now carrying your settings: its badge flips from **Inherited** to **Custom**, a **Reset** button appears beside the pencil, and a summary line reads **Action: block** and **Threshold: 0.80**.
-
-
-If the check calls out to an external provider, **Provider Settings** is where that check's own provider connection lives (not the model provider connected during Agent Command Center setup). **PII Detection**'s dialog has no Provider Settings section, so skip it. (See [Guardrail checks](/docs/protect/reference/guardrail-checks) for which checks are provider-backed.) For a check that does use a provider, a credential you've already saved shows up masked; click into the field and it clears, ready for you to enter a new one.
-
+For what the actions do, see [Understanding Protect](/docs/protect/concepts/understanding-protect). Other checks can have their own **Settings** fields; complete any required fields before saving.
## Push the change to the gateway
-Nothing you set in the check's dialog reaches the gateway until you save here. Back on the Rules tab, an info banner now reads "You have unsaved changes. Save to push guardrail config to the gateway.", with **Reset** and **Save & Activate** buttons. This **Reset** discards the unsaved changes shown in the banner, not the check's saved configuration. It's a separate control from the **Reset** button that appears on a check's card once the check has been customized: that one clears the customization, switching the check back off and its action and threshold back to the defaults (**Block**, **0.8**). Click **Save & Activate**; while the save is in flight the button reads **Saving...**, and a toast confirms with "Config saved and activated".
-
-## Confirm it in Overview
+Click **Add check** in the drawer, or **Save** when editing an existing rule. This sends the change immediately; there is no separate **Save & Activate** step. On success, the drawer closes and the toast reads **PII Detection saved**.
-Switch to the **Overview** tab. The row shows up as **pii-detector**, enabled, and its Stage column reads **Before LLM** (before any guardrail is enabled, this tab reads "No guardrails configured" instead).
+Read any warnings returned by the save. **Saved, but the gateway did not apply it** means the configuration was stored, but the gateway did not confirm the change. The drawer stays open with the warning; don't treat the saved row alone as proof that the gateway is running it. See [Guardrail changes not taking effect](/docs/protect/troubleshooting/guardrail-changes-not-taking-effect).
-## Confirm the stage
+## Confirm the saved rule
-Click the row's **Edit** icon button to open **Edit Guardrail: pii-detector**. **Stage** already shows **Pre**, matching the **Before LLM** you just saw in Overview; this dialog is where you'd change it if you needed a different stage. The dialog also shows **Action** and **Threshold** (with the helper text "Numeric threshold for the guardrail (optional)"), both carried over from the pencil dialog: **Block** and **0.8**, plus **Cancel** and **Save** / **Saving...** buttons. Click **Save**. A toast confirms: `Guardrail "pii-detector" updated`.
+Back in **Rules**, find the PII Detection row and confirm it is on. Open the row to review its action and request stage. The same drawer is used for later edits; there is no separate Overview editor.
-**PII Detection** is now enabled, blocking at a 0.8 confidence threshold, running at the **pre** stage.
+
+If the page says **Guardrails are off on this gateway**, rules can still be saved, but no check runs. The gateway operator must turn on the guardrail pipeline in the gateway configuration before those rules are enforced.
+
## Dive deeper
diff --git a/src/pages/docs/protect/reference/guardrail-checks.mdx b/src/pages/docs/protect/reference/guardrail-checks.mdx
index 9b619c8f2..072b76aed 100644
--- a/src/pages/docs/protect/reference/guardrail-checks.mdx
+++ b/src/pages/docs/protect/reference/guardrail-checks.mdx
@@ -24,49 +24,57 @@ These ten checks run inside Protect without an external provider. Three of them
## Provider-backed checks
-Each of these 18 checks has a **Provider Settings** section in its Rules dialog (the per-check settings dialog under **Gateway > Guardrails**, covered in [Configuration fields](#configuration-fields)). Provider Settings is where that check's own provider configuration lives, not always a credential connection to an outside service.
-
-| Check |
-|---|
-| `futureagi-eval` |
-| `llama-guard` |
-| `azure-content-safety` |
-| `presidio-pii` |
-| `lakera-guard` |
-| `bedrock-guardrails` |
-| `hiddenlayer-guard` |
-| `aporia-guard` |
-| `pangea-guard` |
-| `dynamoai-guard` |
-| `enkrypt-guard` |
-| `ibm-ai-detector` |
-| `grayswan-guard` |
-| `lasso-guard` |
-| `crowdstrike-aidr` |
-| `zscaler-guard` |
-| `tool-permissions` |
-| `mcp-security` |
+The table includes provider-backed and tool-policy checks retained in saved configurations; it is **not** a list of new dashboard integrations. **Add check** offers built-in gateway checks. Future AGI Eval and Lakera Guard appear in Rules only when their required account configuration is already saved.
+
+**Stored-only** checks have read-only saved rows: you cannot add them, enable them, or edit their settings in this console. Existing configuration is preserved and sent to the gateway, so an already enabled check may still run. An Owner or Admin can turn one off or remove it after confirmation, but cannot turn it back on here.
+
+| Check | Console availability |
+|---|---|
+| `futureagi-eval` | Existing configured account only; not in Add check |
+| `llama-guard` | Stored-only |
+| `azure-content-safety` | Stored-only |
+| `presidio-pii` | Stored-only |
+| `lakera-guard` | Existing configured account only; not in Add check |
+| `bedrock-guardrails` | Stored-only |
+| `hiddenlayer-guard` | Stored-only |
+| `aporia-guard` | Stored-only |
+| `pangea-guard` | Stored-only |
+| `dynamoai-guard` | Stored-only |
+| `enkrypt-guard` | Stored-only |
+| `ibm-ai-detector` | Stored-only |
+| `grayswan-guard` | Stored-only |
+| `lasso-guard` | Stored-only |
+| `crowdstrike-aidr` | Stored-only |
+| `zscaler-guard` | Stored-only |
+| `tool-permissions` | Stored-only; not offered in the catalog |
+| `mcp-security` | Not offered in the catalog |
## PII entities
-A `pii-detector` check (shown as `pii-detection` in the settings tab) can look for these 14 entity ids:
+The gateway's `pii-detection` check (also stored under the rule alias `pii-detector`) accepts the following **case-sensitive** identifiers in `config.entities`. For example, `["email", "ssn", "credit_card"]` selects those three detectors. Omitting `entities` uses all supported detectors.
-| Entity id | Label |
+| Entity id | Pattern detected |
|---|---|
-| `SSN` | Social Security Number |
-| `CREDIT_CARD` | Credit Card Number |
-| `EMAIL` | Email Address |
-| `PHONE` | Phone Number |
-| `ADDRESS` | Physical Address |
-| `NAME` | Person Name |
-| `DOB` | Date of Birth |
-| `PASSPORT` | Passport Number |
-| `DRIVER_LICENSE` | Driver's License |
-| `IP_ADDRESS` | IP Address |
-| `BANK_ACCOUNT` | Bank Account Number |
-| `MEDICAL_RECORD` | Medical Record Number |
-| `AWS_KEY` | AWS Access Key |
-| `API_KEY` | API Key / Secret |
+| `email` | Email address |
+| `phone` | US phone number |
+| `ssn` | US Social Security number |
+| `credit_card` | Credit card number |
+| `ipv4` | IPv4 address |
+| `ipv6` | Full eight-group IPv6 address |
+| `dob` | Date of birth pattern |
+| `passport` | Passport number following a passport keyword |
+| `drivers_license` | Driver's license number following a license keyword |
+| `iban` | International Bank Account Number |
+| `zip_code` | US ZIP+4 code |
+| `aws_key` | AWS access key identifier |
+| `api_key` | API key / secret pattern |
+| `url_credentials` | URL containing credentials |
+| `mac_address` | Colon-separated MAC address |
+| `ein` | US Employer Identification Number |
+| `mrn` | Medical record number with an MRN prefix |
+| `bitcoin` | Bitcoin address pattern |
+
+Uppercase legacy reference-data labels such as `EMAIL` and `SSN` are not gateway configuration identifiers and are not translated. `ADDRESS` and `NAME` have no detector in this check. Use the exact identifiers above rather than lowercasing the legacy list: for example, IP addresses use `ipv4` and `ipv6`, and medical record numbers use `mrn`.
## Topic categories
@@ -85,36 +93,36 @@ A `topic-restriction` check groups its topics under 8 categories, most with thei
## Configuration fields
-Most fields below apply to every check in both the [first-party](#first-party-checks) and [provider-backed](#provider-backed-checks) tables above. A check that carries provider fields additionally has a Provider Settings section holding them, which is why `keyword-blocklist` shows its Blocked Keywords there despite being first-party. Confidence Threshold appears on every check except `futureagi-eval` and `presidio-pii`.
-
-A check is configured from one of two dialogs: the per-check dialog covered under [Rules dialog](#rules-dialog), and the guardrail-level dialog covered under [Overview dialog](#overview-dialog). The two expose different fields: the Rules dialog offers Mask as an action option, plus a Confidence Threshold; the Overview dialog sets Stage.
+Open an editable check's row in **Rules** to change its drawer. Check-specific fields appear under **Settings**, such as **Blocked keywords** for Keyword Blocklist. The [availability restrictions](#provider-backed-checks) apply before these fields: stored-only checks do not expose an editable provider-settings form.
### Rules dialog
-| Field | Values | Default |
+The Rules editor is a drawer. Click **Add check** for a new rule or **Save** for an existing rule; either sends the change without a second activation step.
+
+| Field | Values | New-rule default |
|---|---|---|
-| Enabled | Toggle | On |
-| Action | Block, Warn, Mask, Log | Block |
-| Confidence Threshold | Slider from 0.0 to 1.0, marked at 0.0 / 0.5 / 1.0 | 0.8 |
-| Provider Settings (checks that have provider fields) | The check's provider configuration | |
+| On / Off | Toggle | On |
+| Action | Block, Warn, Log; Mask where the check supports request rewriting | Block |
+| Match threshold | Slider from 0.0 to 1.0, where applicable | 0.8 |
+| Settings | Fields specific to the check, when it has them | Check-specific |
+
+Fixed-score checks, including default PII Detection, have no Match threshold slider. Future AGI Eval also hides it. See [threshold tuning](/docs/protect/troubleshooting/guardrail-fires-on-the-wrong-requests).
### Overview dialog
-| Field | Values | Default |
-|---|---|---|
-| Action | Block, Warn, Log | Block |
-| Stage | pre, post, both | pre |
-| Threshold | Numeric (optional) | |
+There is no separate Overview editor. Legacy Overview links open Rules. The drawer describes the check's fixed request or response stage; it does not offer a pre/post/both selector.
### Pipeline settings
-These apply to every check on the gateway rather than to an individual check.
+The **Pipeline** section in Rules separates the organization's editable timeout from gateway/operator-owned behavior. Organization checks run one at a time and stop at the first block; an organization `pipeline_mode` value does not enable parallel execution.
+
+| Field | Organization control | What it does |
+|---|---|---|
+| Mode | Read-only; sequential | Shows how the gateway runs organization checks, not an organization-selectable mode |
+| Fail Open | Read-only; set by the gateway operator | For a check that does not rewrite the request, a timeout or recovered panic skips the check when on; when off, it produces a failed result evaluated against the check's threshold and action |
+| Check timeout (ms) | Editable by Owner/Admin, 100–60000 ms; the UI starts at 5000 ms when unset | Limits each organization check's runtime; click **Save timeout** to save it. Without an organization timeout, execution uses the gateway's default |
-| Field | Values | Default | What it does |
-|---|---|---|---|
-| Mode | Parallel, Sequential | Parallel | Whether the gateway's checks run at the same time or one after another |
-| Fail Open | Toggle | On | What happens when a check doesn't return a verdict before Timeout runs out. On lets the request through unchecked; off applies the check's configured action instead |
-| Timeout | Milliseconds | 5000 ms | How long a check is given to return a verdict before Fail Open decides what happens next |
+A request-rewriting check that times out or panics blocks the request rather than sending an unfinished rewrite. Remote-service errors returned as failed check results follow the check's threshold and action. Setting organization `guardrails.fail_open` in an update-config request does not change the gateway's failure handling; ask the gateway operator about that behavior.
## Response statuses
@@ -123,7 +131,7 @@ These are the statuses the [Agent Command Center](/docs/command-center) gateway
| Action | Status | What the caller gets |
|---|---|---|
| Block, pre-stage check | `403` | An error body with `"type": "guardrail_error"` and `"code": "content_blocked"`, returned before the model is ever called |
-| Block, post-stage check | `403` (non-streamed) | The model response is withheld. On a stream the HTTP status is already sent; see Streaming Guardrails above. Organization checks `data-leakage-prevention` and `hallucination-detection` run post-stage regardless of the Stage control |
+| Block, post-stage check | `403` (non-streamed) | The model response is withheld. On a stream the HTTP status is already sent; see Streaming Guardrails above. Organization checks `data-leakage-prevention` and `hallucination-detection` have a fixed post-stage |
| Warn | `200` | The normal response, plus the header `x-agentcc-guardrail-triggered: true` |
| Log | `200` | The normal response, plus the header `x-agentcc-guardrail-triggered: true` |
diff --git a/src/pages/docs/protect/troubleshooting/guardrail-changes-not-taking-effect.mdx b/src/pages/docs/protect/troubleshooting/guardrail-changes-not-taking-effect.mdx
index 87902ac7c..3a14b134e 100644
--- a/src/pages/docs/protect/troubleshooting/guardrail-changes-not-taking-effect.mdx
+++ b/src/pages/docs/protect/troubleshooting/guardrail-changes-not-taking-effect.mdx
@@ -1,29 +1,23 @@
---
title: "Guardrail changes not taking effect"
-description: "Diagnose a guardrail edit that isn't taking effect: an unsaved Rules tab banner, a failed enable toggle, or a Stage that doesn't cover where you're testing."
+description: "Check whether a Rules drawer edit was saved and applied, whether the gateway pipeline is on, and which side of the exchange the check reads."
---
-You changed a guardrail's action, threshold, or stage, but requests still come back exactly like they did before. Guardrails have two edit paths: the check cards on the **Rules** tab go through **Save & Activate**, while the **Overview** tab's Edit dialog saves on its own. Three causes account for almost every case, plus one run to confirm the fix landed.
+You changed a guardrail's action or settings, but requests still come back exactly like they did before. **Rules** has one editor: open the saved rule's row to edit its drawer. **Save** sends an edit immediately; a new rule uses **Add check**. There is no separate **Save & Activate** step or Overview editor.
-## Check for the unsaved-changes banner
+## Check the save result
-Go to **Gateway** > **Guardrails** > **Rules**. If an info banner sits above the check cards reading "You have unsaved changes. Save to push guardrail config to the gateway.", with **Reset** and **Save & Activate** buttons, your edit never went live.
+Go to **Gateway** > **Guardrails** > **Rules** and open the rule. Confirm that you saved the drawer, then read the result. **Saved, but the gateway did not apply it** means the configuration was stored but the gateway did not confirm the change; the drawer stays open with the warning. Follow the reason in that warning rather than assuming the saved rule is live. A request error is shown separately and is not a successful save.
-Click **Save & Activate**. If you've saved and behavior still hasn't changed, move on to the next check.
+If the page says **Guardrails are off on this gateway**, no check runs even though you can save rules. Ask the gateway operator to enable the pipeline in the gateway configuration. If the save applied and the pipeline is on, move on to the next check.
## Check the guardrail's Stage
-If the banner is clear and the change definitely saved, go to **Gateway** > **Guardrails** > **Overview**, click the guardrail row's pencil icon button, and look at **Stage**. It's one of **pre**, **post**, or **both**.
-
-- Staged to **pre** but expecting it to catch something in the response? It never will: pre only looks at what goes in
-- Staged to **post** but expecting it to block the request itself? It never will either: post only looks at what comes back
-- Staged to **both**? It sees both sides, since **both** is the same check running once on the way in and once on the way out; see [Understanding Protect](/docs/protect/concepts/understanding-protect)
-
-Match **Stage** to the side of the exchange you actually need checked.
+Open the rule's row in **Rules** and read the stage description in its drawer. It tells you whether the check reads the request or the model's answer; it is not a selector. **PII Detection** reads requests only. **Hallucination Detection** and **Data Leakage Prevention** read responses. You cannot move an organization check to post or both by editing its rule. Choose and test a check that reads the side you need covered. See [Understanding Protect](/docs/protect/concepts/understanding-protect).
## If the toggle failed
-If you flipped the **Enabled** switch on the guardrail's row in the **Overview** tab and saw the "Failed to toggle guardrail" error toast instead of a success toast reading "`` enabled", the enabled state itself never changed. Retry the toggle and don't move on until you see the success toast instead of the error toast.
+Use the rule's switch in **Rules**. A successful change reports that the check was **turned on** or **turned off**. If you see an error, check the saved state before retrying. A **Saved, but the gateway did not apply it** warning is different: the saved state changed, but enforcement has not been confirmed. Only an organization Owner or Admin can change the rule.
## Confirm with a Test tab run
@@ -31,13 +25,13 @@ The fastest way to know whether any of these fixes worked, rather than waiting o
## If the test still doesn't fire
-If the banner was clear, the toggle succeeded, the Stage already matched what you needed, and the Test tab run still shows the guardrail not firing, the cause isn't a leftover unsaved change or a stage mismatch. Contact support@futureagi.com with the guardrail's name, its Stage and Action, and the prompt you tested with.
+If the save applied, the pipeline is on, the rule is on, and you're testing the side it reads, inspect the rule's settings next. Contact support@futureagi.com with the guardrail's name, its stage and action, and the prompt you tested with if the behavior is still unexplained.
## Dive deeper
- Enable a check and set its action, threshold, and stage
+ Add a check, set its action, and confirm the save
Fire a prompt at your guardrails and read the result chip
diff --git a/src/pages/docs/protect/troubleshooting/guardrail-fires-on-the-wrong-requests.mdx b/src/pages/docs/protect/troubleshooting/guardrail-fires-on-the-wrong-requests.mdx
index 885c96d7b..def3d5c87 100644
--- a/src/pages/docs/protect/troubleshooting/guardrail-fires-on-the-wrong-requests.mdx
+++ b/src/pages/docs/protect/troubleshooting/guardrail-fires-on-the-wrong-requests.mdx
@@ -1,11 +1,11 @@
---
title: "Guardrail fires on the wrong requests"
-description: "Switch Action to Warn or Log while you tune, move Confidence Threshold, re-test the change, then record the verdict as feedback."
+description: "Tune a check's supported settings or match threshold, re-test the change, and record feedback. Default PII matches have no threshold slider."
---
A guardrail check can go wrong in two directions: it blocks or warns on requests that were fine, or it lets through content it should have caught. Either way, the fix is the same set of levers, worked in order on the one check that's causing it.
-Go to **Gateway** > **Guardrails**. **Overview**, **Rules**, **Analytics**, **Feedback**, **Test**, and **Logs** are all tabs on that section; **Analytics** is where you diagnose which check is misfiring, and **Rules** is where you open that check's card to edit it.
+Go to **Gateway** > **Guardrails**. **Rules**, **Analytics**, **Feedback**, **Test**, and **Logs** are the tabs in that section; **Analytics** is where you diagnose which check is misfiring, and **Rules** is where you open that check's row to edit it.
## Find the check that's firing
@@ -16,43 +16,58 @@ Checks show up under **Rule** in the **Top Triggered Rules** table alongside **T
## Switch Action to Warn or Log while you tune
-Switch **Action** from Block to Warn or Log before you touch the Confidence Threshold below, so a check you're still tuning doesn't stop real traffic. Go to **Gateway** > **Guardrails** > **Rules**, find the check's card, and click its pencil icon button to open the dialog (see [Turn on a guardrail](/docs/protect/guides/turn-on-a-guardrail) for the fuller walkthrough of setting these up). In the dialog's **Action** select:
+Switch **Action** from Block to Warn or Log before tuning, so a check you're still tuning doesn't stop real traffic. As an organization Owner or Admin, go to **Gateway** > **Guardrails** > **Rules** and open the check's row to edit its drawer (see [Turn on a guardrail](/docs/protect/guides/turn-on-a-guardrail) for the setup walkthrough). In the drawer's **Action** select:
- **Block** stops the request, and counts toward that check's Block total in Top Triggered Rules
- **Warn** lets the request through, and counts toward that check's Warn total in Top Triggered Rules
- **Log** lets the request through too, and still counts toward Triggers, but the check's Block and Warn totals stay flat, for a quieter pass while you compare several changes
-**Mask** is also on the select, but it's a different use case outside this tuning flow. Set **Action** back to Block once you're satisfied with where the threshold lands.
+**Mask** is offered only for checks that can rewrite the request, and serves a different use case. Set **Action** back to Block once you're satisfied with the check's behavior.
-## Move its Confidence Threshold
+## Tune the settings the check supports
-In the same dialog, find **Confidence Threshold**, a slider marked at 0.0, 0.5, and 1.0. Its untouched value is 0.8. Future AGI Eval and Presidio PII checks don't render this slider at all, so this lever isn't available for them. For what action and threshold actually do to a request, see [Understanding Protect](/docs/protect/concepts/understanding-protect).
+### Checks with a Match threshold
-Raise the threshold and the check catches less, so move it up if the check is blocking or warning on requests that were fine. Lower it and the check catches more, so move it down if it's letting through content it should have caught. As a first move, try adjusting it by about 0.05 to 0.1, then re-test before adjusting further.
+If the drawer shows **Match threshold**, it runs from 0.0 to 1.0 in steps of 0.05; a new rule starts at 0.8. A match scoring at or below the threshold is ignored. For checks whose match scores vary, raising the threshold can reduce matches and lowering it can include lower-scoring matches. Try a change of 0.05 to 0.1 and re-test before adjusting further; a threshold change will not affect examples whose scores remain on the same side of it.
-Click **Save** in the dialog to stage the change, then back on the Rules tab, click **Save & Activate** on the "You have unsaved changes. Save to push guardrail config to the gateway." banner; the change doesn't reach the gateway until you do. If you re-test and nothing's changed, see [Guardrail changes not taking effect](/docs/protect/troubleshooting/guardrail-changes-not-taking-effect).
+### Fixed-score checks
+
+Default **PII Detection** has no Match threshold slider. Its `max_entities` defaults to 1, and its score is the detected entity count divided by that maximum, capped at 1. Any detected entity therefore scores 1. Raising a threshold from 0.8 to 0.9 would not change which requests match.
+
+The same limitation applies to default **Secret Detection** and **Keyword Blocklist**. Their sliders appear only if a saved `max_secrets` or `max_matches` exceeds 1; PII's slider similarly depends on a saved `max_entities` above 1. **Topic Restriction** and **Language Detection** always score 1 on a match and have no threshold slider. A threshold of 1 ignores these fixed-score matches rather than selectively reducing false positives.
+
+For fixed-score checks, edit the rule-specific **Settings** that the drawer actually offers:
+
+- **Keyword Blocklist**: adjust **Blocked keywords**.
+- **Topic Restriction**: adjust `forbidden_topics` or `allowed_topics` and their keywords.
+- **Language Detection**: adjust **Allowed scripts**.
+- **PII Detection** and **Secret Detection**: the drawer does not expose detector-specific tuning fields. Change the action or choose a different check rather than looking for a sensitivity slider.
+
+**Future AGI Eval** also hides the threshold slider, and stored-only integrations such as Presidio PII are not editable in this console. See [Guardrail checks](/docs/protect/reference/guardrail-checks) for availability and [Understanding Protect](/docs/protect/concepts/understanding-protect) for actions and stages.
+
+Click **Save** in the drawer to send the change immediately. There is no second activation step. If you see **Saved, but the gateway did not apply it**, or re-testing shows no change, see [Guardrail changes not taking effect](/docs/protect/troubleshooting/guardrail-changes-not-taking-effect) before assuming the edit is enforced.
## Re-test after each change
-After each change, switch to **Gateway** > **Guardrails** > **Test** (see [Test a guardrail](/docs/protect/guides/test-a-guardrail) for what it shows) and click the example chip that matches the case you're chasing, whether that's PII, an injection attempt, secrets, or toxic content, then run it. The result chip in the Result card reads BLOCKED, which the gateway returns as `403`, or OK with the status code the call came back on; that's the verdict to check against what you meant to happen. Re-running that chip after every threshold or Action change is how you confirm the change did what you meant, instead of stacking up several changes and losing track of which one mattered.
+After each change, switch to **Gateway** > **Guardrails** > **Test** (see [Test a guardrail](/docs/protect/guides/test-a-guardrail) for what it shows) and click the example chip that matches the case you're chasing, whether that's PII, an injection attempt, secrets, or toxic content, then run it. The result chip in the Result card reads BLOCKED, which the gateway returns as `403`, or OK with the status code the call came back on; that's the verdict to check against what you meant to happen. Re-running that chip after every settings, threshold, or Action change is how you confirm the change did what you meant, instead of stacking up several changes and losing track of which one mattered.
## Record the verdict as feedback
-Feedback is a record, not a control. Submitting a False Positive or False Negative doesn't retune the check. Marking one here and skipping the Action and Confidence Threshold changes above leaves the check exactly as it was.
+Feedback is a record, not a control. Submitting a False Positive or False Negative doesn't retune the check. Marking one here without changing the rule's action or supported settings leaves the check exactly as it was.
From **Gateway** > **Guardrails** > **Logs**, open the request the check got wrong in its detail drawer. The drawer's **Guardrails** tab only appears when at least one check fired on that request; a request that nothing caught has no feedback controls at all, so it can't be marked False Negative from its own drawer. On the drawer's **Guardrails** tab, one set of feedback controls appears for each check that fired on that request; find the set for the check you're tuning and mark it **False Positive** if it fired on a request that was fine, or **False Negative** if it let through something it should have caught, then press **Submit Feedback**. That verdict rolls up into **Gateway** > **Guardrails** > **Feedback** alongside every other correction submitted for that check.
## If it still fires wrong
-If the check still fires wrong after switching Action, moving the Confidence Threshold, and re-testing, the fix may be more than this check can offer on its own. Check whether a different check in the **Top Triggered Rules** table above is the actual source, or contact support@futureagi.com with the check's name, the Action and threshold you tried, and an example request it still gets wrong.
+If the check still fires wrong after tuning its supported settings and re-testing, the fix may be more than this check can offer on its own. Check whether a different check in the **Top Triggered Rules** table above is the actual source, or contact support@futureagi.com with the check's name, the action and settings you tried, and an example request it still gets wrong.
## Dive deeper
- Set a check's Action and Confidence Threshold for the first time
+ Set a check's action and supported settings for the first time
Run example or custom prompts through a check from the Test tab
diff --git a/src/pages/docs/quickstart/generate-synthetic-data.mdx b/src/pages/docs/quickstart/generate-synthetic-data.mdx
index ef0b8bd9b..c607f17cf 100644
--- a/src/pages/docs/quickstart/generate-synthetic-data.mdx
+++ b/src/pages/docs/quickstart/generate-synthetic-data.mdx
@@ -23,6 +23,7 @@ description: "Generate synthetic datasets with Future AGI. Define schemas, colum
- **Name** (required): a clear, descriptive title for the dataset.
- **Description** (required): what the dataset is for and how it will be used.
+ - **Enter No. of rows** (required): enter at least **10**, for example `10`. You must fill this in before proceeding to the schema.
- **Use Case**: the intended application, e.g. *"Simulated customer support logs for LLM fine-tuning"*.
- **Pattern** (optional): structural or stylistic rules, e.g. *"Follow a conversational pattern"* or *"Keep tone formal"*.
diff --git a/src/pages/docs/quickstart/running-evals-in-simulation.mdx b/src/pages/docs/quickstart/running-evals-in-simulation.mdx
index 8c542ffa9..1f6789740 100644
--- a/src/pages/docs/quickstart/running-evals-in-simulation.mdx
+++ b/src/pages/docs/quickstart/running-evals-in-simulation.mdx
@@ -9,7 +9,20 @@ description: "Run evaluations in Future AGI simulations to test AI agents agains
---
-**Prerequisites:** Before starting, make sure you have set up your [Agent Definition](/docs/simulation/concepts/agent-definitions), [Scenarios](/docs/simulation/concepts/scenarios), and [Personas](/docs/simulation/concepts/personas).
+## Run with an Environment
+
+1. Open **Agents → Connected agents**, then select **Connect agents** from the Agents creation menu to open Environment's build tab. Follow [Connect your no code agent](/docs/get-started/connect-no-code-agents) to complete the connection and build an environment.
+2. Open the environment from **Simulate → Environment** and review its **Scenarios** and **Evaluations** tabs.
+3. Once the environment is ready and any setup blockers are resolved, click **Run simulation**.
+4. Inspect the completed run under **Runs**.
+
+
+The standalone **Run Simulation** and **Agent Definition** workflow below is legacy and is not available to Environments-only organizations. For those organizations, use the Environment workflow above instead.
+
+
+## Legacy Run Simulation workflow
+
+**Prerequisites for the legacy workflow:** Your organization must still have access to the legacy simulation pages. Before starting, set up your [Agent Definition](/docs/simulation/concepts/agent-definitions), [Scenarios](/docs/simulation/concepts/scenarios), and [Personas](/docs/simulation/concepts/personas). The old Agent Definition list link now redirects to **Connected agents**.
diff --git a/src/pages/docs/quickstart/setup-observability.mdx b/src/pages/docs/quickstart/setup-observability.mdx
index 7b6087b90..4a22757c2 100644
--- a/src/pages/docs/quickstart/setup-observability.mdx
+++ b/src/pages/docs/quickstart/setup-observability.mdx
@@ -111,14 +111,12 @@ Observe supports auto-instrumentation for OpenAI, Anthropic, LangChain, LlamaInd
```typescript JS/TS
import { OpenAIInstrumentation } from "@traceai/openai";
- import { registerInstrumentations } from "@opentelemetry/instrumentation";
- import { OpenAI } from "openai";
+ import OpenAI from "openai";
- // Enable auto-instrumentation
- registerInstrumentations({
- instrumentations: [new OpenAIInstrumentation({})],
- tracerProvider: traceProvider,
- });
+ // Explicitly instrument the imported SDK for ESM/TypeScript
+ const instrumentation = new OpenAIInstrumentation({});
+ instrumentation.setTracerProvider(traceProvider);
+ instrumentation.manuallyInstrument(OpenAI);
// Use OpenAI as normal
const client = new OpenAI();
diff --git a/src/pages/docs/roles-and-permissions.mdx b/src/pages/docs/roles-and-permissions.mdx
index e9309d9e5..e8940bd84 100644
--- a/src/pages/docs/roles-and-permissions.mdx
+++ b/src/pages/docs/roles-and-permissions.mdx
@@ -52,14 +52,14 @@ A user's effective workspace access is the **higher** of their organization role
## Inviting Users
-Org Admins and Workspace Admins can invite new users to the organization. You can manage all members from the **Organization > Members** page.
+Org Admins and Workspace Admins can invite new users to the organization. You can manage all members from the **Settings > Members** page.

- Go to **Organization > Members** in the sidebar.
+ Go to **Settings > Members** in the sidebar.
Click the invite button and enter the user's email address.
@@ -90,7 +90,7 @@ Org Admins and Workspace Admins can invite new users to the organization. You ca
### Changing a user's role
-Org Admins and Owners can change any member's organization role or workspace role from the **Users** page.
+Org Admins and Owners can change any member's organization role or workspace role from the **Settings > Members** page.

diff --git a/src/pages/docs/sdk/annotation-queues/queues.mdx b/src/pages/docs/sdk/annotation-queues/queues.mdx
index 71b5a0e1e..c9d57de15 100644
--- a/src/pages/docs/sdk/annotation-queues/queues.mdx
+++ b/src/pages/docs/sdk/annotation-queues/queues.mdx
@@ -6,13 +6,16 @@ description: "Create, list, get, update, and delete annotation queues."
## `create`
-Creates a new annotation queue.
+Creates a new, unscoped annotation queue in draft status. To use a project's, dataset's, or agent definition's scoped default queue, call [`get_or_create_default()`](#get_or_create_default) instead. Pass a non-empty `label_ids` list from `create_label()` or `list_labels()`, then call `activate(queue.id)` when the queue is ready for annotation.
+
+If `label_ids` is omitted, the deprecated compatibility path requires exactly one of `project`, `dataset`, or `agent_definition`. It calls `get_or_create_default()` for that scope and reuses its labels; it raises `DefaultQueueLabelError` if there is no single scope or the default queue has no usable labels. The lookup can create or restore the default queue even when the label check fails. It does not scope the new queue. Pass explicit label IDs to avoid this lookup.
```python
def create(
self,
name: str,
*,
+ label_ids: Sequence[Union[str, UUID]] = OMITTED_LABEL_IDS,
description: Optional[str] = None,
instructions: Optional[str] = None,
assignment_strategy: Optional[str] = None,
@@ -28,21 +31,42 @@ def create(
- **Arguments:**
- `name` (str): Name of the annotation queue.
+ - `label_ids` (Sequence[Union[str, UUID]]): Non-empty list of annotation label IDs. Explicit `None` or an empty list is invalid; omission uses only the compatibility path described above.
- `description` (Optional[str]): Description of the queue's purpose.
- `instructions` (Optional[str]): Guidelines for annotators.
- `assignment_strategy` (Optional[str]): How items are assigned — `"manual"`, `"round_robin"`, or `"load_balanced"`.
- `annotations_required` (Optional[int]): Number of annotations needed per item.
- `reservation_timeout_minutes` (Optional[int]): Time limit (in minutes) for an annotator to complete an item.
- `requires_review` (Optional[bool]): Whether completed annotations require reviewer approval.
- - `project` (Optional[str]): Project ID to scope the queue to.
- - `dataset` (Optional[str]): Dataset ID to associate with the queue.
- - `agent_definition` (Optional[str]): Agent definition ID to associate with the queue.
+ - `project` (Optional[str]): Compatibility-only project ID for default-label lookup when `label_ids` is omitted. Never sent as an association on create.
+ - `dataset` (Optional[str]): Compatibility-only dataset ID for the same lookup. Does not associate the new queue with the dataset.
+ - `agent_definition` (Optional[str]): Compatibility-only agent definition ID for the same lookup. Does not associate the new queue with the agent definition.
+ - With explicit `label_ids`, these three compatibility arguments are dropped with a warning.
- `timeout` (Optional[int]): Request timeout in seconds.
- **Returns:**
- `QueueDetail` instance
---
+## `get_or_create_default`
+
+Gets the scoped default queue. Pass exactly one of `project_id`, `dataset_id`, or `agent_definition_id`; passing none or more than one raises `ValueError`.
+
+```python
+def get_or_create_default(
+ self,
+ *,
+ project_id: Optional[Union[str, UUID]] = None,
+ dataset_id: Optional[Union[str, UUID]] = None,
+ agent_definition_id: Optional[Union[str, UUID]] = None,
+ timeout: Optional[int] = None,
+) -> DefaultQueue
+```
+
+The server fetches, restores, or creates that entity's default queue and returns it with its labels. This is distinct from `create()`, which never scopes a new queue.
+
+---
+
## `list_queues`
Lists annotation queues with optional filters.
diff --git a/src/pages/docs/sdk/evals/cloud-evals.mdx b/src/pages/docs/sdk/evals/cloud-evals.mdx
index 28601424a..ff2a9a67a 100644
--- a/src/pages/docs/sdk/evals/cloud-evals.mdx
+++ b/src/pages/docs/sdk/evals/cloud-evals.mdx
@@ -42,8 +42,8 @@ print(f"Total templates: {len(templates)}")
t = templates[0]
print(t["name"]) # "toxicity"
print(t["description"]) # what it checks
-print(t["evalTags"]) # categories like ["SAFETY", "TEXT"]
-print(t["config"]["requiredKeys"]) # what inputs you need to pass
+print(t["eval_tags"]) # server-returned tags, such as "Red Teaming"
+print(t["config"]["required_keys"]) # what inputs you need to pass
```
### Filtering by tag
@@ -54,31 +54,22 @@ from fi.evals import Evaluator
evaluator = Evaluator()
templates = evaluator.list_evaluations()
-# Get all safety templates
-safety = [t for t in templates if "SAFETY" in t.get("evalTags", [])]
-print(f"Safety templates: {len(safety)}")
-for t in safety:
+# Filter by an exact tag returned by the server
+selected_tag = "Red Teaming"
+matching = [t for t in templates if selected_tag in (t.get("eval_tags") or [])]
+print(f"{selected_tag} templates: {len(matching)}")
+for t in matching:
print(f" {t['name']}: {t['description'][:80]}")
-
-# Get all RAG templates
-rag = [t for t in templates if "RAG" in t.get("evalTags", [])]
-print(f"RAG templates: {len(rag)}")
```
### Available tags
-| Tag | What it covers |
-|-----|---------------|
-| `SAFETY` | Toxicity, bias, PII, content moderation, prompt injection |
-| `RAG` | Context adherence, chunk attribution, faithfulness, retrieval metrics |
-| `HALLUCINATION` | Hallucination detection, factual accuracy, groundedness |
-| `CONVERSATION` | Coherence, resolution, customer agent behaviors |
-| `CHAT` | General chat quality metrics |
-| `AUDIO` | Transcription accuracy, audio quality, TTS/ASR |
-| `IMAGE` | Caption hallucination, image instruction adherence |
-| `TEXT` | General text quality (completeness, tone, helpfulness) |
-| `FUNCTION` | Deterministic checks (contains, regex, JSON, similarity) |
-| `LLMS` | LLM-specific checks (bias, completeness, attribution) |
+Tags come from the server, not a fixed uppercase enum. Inspect the returned values before choosing a filter:
+
+```python
+available_tags = sorted({tag for t in templates for tag in (t.get("eval_tags") or [])})
+print(available_tags)
+```
### Checking required inputs
@@ -92,8 +83,8 @@ templates = evaluator.list_evaluations()
# Find a specific template
toxicity = next(t for t in templates if t["name"] == "toxicity")
-print(toxicity["config"]["requiredKeys"]) # what you need to pass
-print(toxicity["config"].get("configParamsDesc", {})) # parameter descriptions
+print(toxicity["config"]["required_keys"]) # what you need to pass
+print(toxicity["config"].get("config_params_desc", {})) # parameter descriptions
```
## Turing Models
@@ -123,10 +114,10 @@ result = evaluate("toxicity", output="...", model="turing_large")
```python
from fi.evals import evaluate
-# Single template
-result = evaluate("tone", output="Dear Sir, I hope this finds you well.", model="turing_flash")
-print(result.score) # 1.0
-print(result.passed) # True
+# Single pass/fail template
+result = evaluate("toxicity", output="Dear Sir, I hope this finds you well.", model="turing_flash")
+print(result.score) # e.g. 1.0 for a Passed verdict
+print(result.passed) # True when the normalized score is at least 0.5
print(result.reason) # detailed explanation from Turing
# Multiple inputs
@@ -138,6 +129,23 @@ result = evaluate(
)
```
+### Categorical results
+
+`tone` returns categorical labels, such as `neutral`, rather than a numeric pass/fail verdict. The unified `evaluate()` wrapper does not preserve label lists or choice-score dictionaries as a score: `score` and `passed` are `None` for those outputs. Use the raw `Evaluator` result to read them:
+
+```python
+from fi.evals import Evaluator
+
+batch = Evaluator().evaluate(
+ eval_templates="tone",
+ inputs={"output": "Dear Sir, I hope this finds you well."},
+ model_name="turing_flash",
+)
+for result in batch.eval_results:
+ print(result.output) # categorical output, for example ["neutral"]
+ print(result.reason)
+```
+
### With the Evaluator class
The `Evaluator` class provides additional features for cloud evals: pipeline execution, async results, and batch processing.
@@ -152,8 +160,8 @@ result = evaluator.evaluate_pipeline(
project_name="my-project",
version="v1",
eval_data=[
- {"template": "toxicity", "output": "Hello world", "model_name": "turing_flash"},
- {"template": "tone", "output": "Dear Sir...", "model_name": "turing_flash"},
+ {"eval_template": "toxicity", "model_name": "turing_flash", "inputs": {"output": ["Hello world"]}},
+ {"eval_template": "tone", "model_name": "turing_flash", "inputs": {"output": ["Dear Sir..."]}},
],
)
diff --git a/src/pages/docs/sdk/evals/evaluate.mdx b/src/pages/docs/sdk/evals/evaluate.mdx
index b097a7c7f..ebb7db7fe 100644
--- a/src/pages/docs/sdk/evals/evaluate.mdx
+++ b/src/pages/docs/sdk/evals/evaluate.mdx
@@ -113,7 +113,7 @@ def evaluate(
- Custom evaluation criteria for LLM-as-Judge mode. Use `{field_name}` placeholders to reference input fields. Requires `engine="llm"` and a `model`.
+ Custom evaluation criteria for LLM-as-Judge mode. Refer to the separately supplied input fields by name; `{field_name}` placeholders inside `prompt` are not interpolated. Format the string in your own Python code if you need values embedded in the criteria. Requires `engine="llm"` and a `model`.
@@ -188,11 +188,11 @@ def evaluate(
- `"completed"` or `"error"`.
+ `"completed"` or `"failed"`. This describes execution, not the evaluation's pass/fail verdict.
- Error message if `status` is `"error"`.
+ Error message if `status` is `"failed"`.
@@ -213,13 +213,20 @@ for r in results:
# Access by name
toxicity = results.get("toxicity")
-# Check overall pass rate
+# Execution completion rate, not a quality pass rate
print(results.success_rate) # 0.0 to 1.0
+# Quality pass rate across all requested evaluations
+passed_count = sum(r.status == "completed" and r.passed is True for r in results)
+quality_pass_rate = passed_count / len(results) if len(results) else 0.0
+print(quality_pass_rate)
+
# Count
print(len(results)) # 2
```
+`success_rate` counts results with `status="completed"`, even when their verdict is `passed=False`. The quality calculation above counts only completed results with `passed=True`; failed executions and unscored results stay in the denominator and do not count as passes. Both rates are `0.0` for an empty batch.
+
## Engine Routing
If you don't set `engine` explicitly, the function picks one:
@@ -229,7 +236,7 @@ If you don't set `engine` explicitly, the function picks one:
3. **Any other model string** → LLM-as-Judge engine
4. **`augment=True`** → local first, then LLM refinement
-You can check which engine ran via `result.metadata["engine"]`.
+Normal local, Turing, and LLM results do not include an `engine` metadata key; use the routing rules above or set `engine` explicitly. The key is currently added only on augmentation paths. Read it defensively with `result.metadata.get("engine")`, which returns `None` when absent.
## Common Patterns
@@ -238,7 +245,7 @@ You can check which engine ran via `result.metadata["engine"]`.
```python
result = evaluate("toxicity", output="test", model="turing_flash")
-if result.status == "error":
+if result.status != "completed":
print(f"Eval failed: {result.error}")
else:
print(f"Score: {result.score}")
diff --git a/src/pages/docs/sdk/evals/llm-judge.mdx b/src/pages/docs/sdk/evals/llm-judge.mdx
index f2c551433..1d72321e5 100644
--- a/src/pages/docs/sdk/evals/llm-judge.mdx
+++ b/src/pages/docs/sdk/evals/llm-judge.mdx
@@ -37,7 +37,7 @@ print(result.reason) # JSON with score and explanation
When you pass `engine="llm"` (or a non-Turing model string), the SDK:
1. Takes your `prompt` as the grading criteria
-2. Substitutes any `{field_name}` placeholders with your input values
+2. Includes your input fields separately alongside the criteria; it does not substitute `{field_name}` placeholders inside `prompt`
3. Sends the criteria + inputs to the LLM
4. Parses the response into an `EvalResult` with score, passed, and reason
@@ -59,11 +59,11 @@ result = evaluate(
### Criteria with input references
-Use `{field_name}` placeholders to reference any input field in your criteria.
+Refer to input fields by name in your criteria. The SDK sends their values separately; it does not interpolate placeholders in `prompt`. If you need values embedded in the criteria text, format the string in your own Python code before calling `evaluate()`.
```python
result = evaluate(
- prompt="Does the response answer the question '{query}'? Score 0 if it ignores the question, 1 if it fully answers it.",
+ prompt="Does the output field answer the question in the query field? Score 0 if it ignores the question, 1 if it fully answers it.",
output="The capital of France is Paris.",
query="What is the capital of France?",
engine="llm",
diff --git a/src/pages/docs/sdk/evals/local.mdx b/src/pages/docs/sdk/evals/local.mdx
index 3f1d904ca..681097f40 100644
--- a/src/pages/docs/sdk/evals/local.mdx
+++ b/src/pages/docs/sdk/evals/local.mdx
@@ -314,23 +314,33 @@ metric_cls = registry.get("bleu_score")
### Registering custom metrics
+Specialize `BaseMetric` with an input model, implement `metric_name` and `compute_one`, and return the score under `output`. Run registered custom metrics with the unified `evaluate()` local engine; `LocalEvaluator` also checks its fixed local-capable metric set.
+
```python
+from fi.evals import evaluate
from fi.evals.local import get_registry
from fi.evals.metrics.base_metric import BaseMetric
+from fi.evals.types import TextMetricInput
+
+class MyCustomMetric(BaseMetric[TextMetricInput]):
+ @property
+ def metric_name(self) -> str:
+ return "my_custom"
-class MyCustomMetric(BaseMetric):
- def compute(self, inputs):
- response = inputs.get("response", "")
+ def compute_one(self, inputs: TextMetricInput) -> dict:
+ response = inputs.response
score = 1.0 if len(response) > 50 else 0.0
- return {"score": score, "reason": f"Length: {len(response)}"}
+ return {"output": score, "reason": f"Length: {len(response)}"}
registry = get_registry()
registry.register("my_custom", MyCustomMetric)
-# Now use it with LocalEvaluator
-from fi.evals.local import LocalEvaluator
-evaluator = LocalEvaluator()
-result = evaluator.evaluate("my_custom", [{"response": "A sufficiently long response for testing purposes here"}])
+result = evaluate(
+ "my_custom",
+ output="A sufficiently long response for testing purposes here",
+ engine="local",
+)
+print(result.score) # 1.0
```
### Lazy registration
diff --git a/src/pages/docs/sdk/evals/metrics/json.mdx b/src/pages/docs/sdk/evals/metrics/json.mdx
index baf1e36ec..f0f4d9ac3 100644
--- a/src/pages/docs/sdk/evals/metrics/json.mdx
+++ b/src/pages/docs/sdk/evals/metrics/json.mdx
@@ -54,13 +54,13 @@ result = evaluate("contains_json", output='Here is the result: {"name": "Alice"}
### json_schema
-Output matches a provided JSON schema. Pass the schema via config.
+Output matches a provided JSON schema. Pass `schema` as an input keyword argument, not inside `config`.
```python
result = evaluate(
"json_schema",
output='{"name": "Alice", "age": 30}',
- config={"schema": {"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}, "required": ["name", "age"]}},
+ schema={"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}, "required": ["name", "age"]},
)
# score → 1.0
```
diff --git a/src/pages/docs/sdk/evals/metrics/string.mdx b/src/pages/docs/sdk/evals/metrics/string.mdx
index e75a67a64..c86241c37 100644
--- a/src/pages/docs/sdk/evals/metrics/string.mdx
+++ b/src/pages/docs/sdk/evals/metrics/string.mdx
@@ -35,7 +35,7 @@ Binary scores: 1 (pass) or 0 (fail). No config required unless noted.
| `contains_valid_link` | Output contains a reachable URL (makes HTTP request) |
| `is_email` | Entire output is a valid email address |
| `one_line` | Output has no newlines |
-| `equals` | Exact match with `expected_output` |
+| `equals` | String equality with `expected_output` (case-insensitive by default) |
| `starts_with` | Output starts with a given string |
| `ends_with` | Output ends with a given string |
| `regex` | Output matches a regular expression |
@@ -141,13 +141,16 @@ result = evaluate("one_line", output="The capital of France is Paris.")
### equals
-Exact string match against `expected_output`. Case-sensitive.
+String equality against `expected_output`. Case-insensitive by default; set `config={"case_sensitive": True}` to require matching capitalization.
```python
result = evaluate("equals", output="Paris", expected_output="Paris")
# score → 1.0
result = evaluate("equals", output="paris", expected_output="Paris")
+# score → 1.0 (case-insensitive default)
+
+result = evaluate("equals", output="paris", expected_output="Paris", config={"case_sensitive": True})
# score → 0.0
```
@@ -285,7 +288,7 @@ result = evaluate("embedding_similarity", output="...", expected_output="...", c
### semantic_list_contains
-Checks whether the output contains phrases semantically similar to keywords. Uses embeddings.
+Checks whether the output is semantically similar to reference phrases. Pass the phrases as the `expected_response` input; keep options such as `similarity_threshold` in `config`. Uses embeddings.
Requires `pip install ai-evaluation[embeddings]`
@@ -293,9 +296,10 @@ Checks whether the output contains phrases semantically similar to keywords. Use
result = evaluate(
"semantic_list_contains",
output="Greetings! How can I assist you?",
- config={"keywords": ["hello", "help"], "similarity_threshold": 0.7},
+ expected_response=["hello", "help"],
+ config={"similarity_threshold": 0.7},
)
-# score → 1.0 ("Greetings" is semantically close to "hello")
+# score → 1.0 if at least one reference meets the threshold; otherwise 0.0
# similarity_threshold: float, default 0.7 — lower = more permissive
```
diff --git a/src/pages/docs/sdk/index.mdx b/src/pages/docs/sdk/index.mdx
index b1d562705..bd895fc4e 100644
--- a/src/pages/docs/sdk/index.mdx
+++ b/src/pages/docs/sdk/index.mdx
@@ -7,7 +7,7 @@ description: "Evaluate LLM outputs, trace AI calls, optimize prompts, and test v
- **Python:** evals, tracing, datasets, prompts, optimization, simulation
- **TypeScript:** evals, tracing, datasets, prompts
- **Java / C#:** tracing
-- `pip install ai-evaluation` or `npm install @future-agi/ai-evaluation` to get started
+- Python local evals: Python 3.11+ and `pip install "futureagi[evals]"`; TypeScript: `npm install @future-agi/ai-evaluation`
Future AGI is a set of packages that evaluate LLM outputs, trace calls across your stack, optimize prompts, and load-test voice agents. Install what you need, skip what you don't.
@@ -28,10 +28,10 @@ Future AGI is a set of packages that evaluate LLM outputs, trace calls across yo
```bash
- pip install ai-evaluation
+ pip install "futureagi[evals]"
```
- Requires Python 3.10+. This also installs `futureagi` ([datasets](/docs/sdk/datasets), [prompts](/docs/sdk/datasets), [knowledge bases](/docs/sdk/knowledgebase)) automatically.
+ This quickstart uses a local metric, so it requires Python 3.11+ and the `evals` extra. The base `futureagi` package supports Python 3.10+ for cloud evaluations, [datasets](/docs/sdk/datasets), prompts, and [knowledge bases](/docs/sdk/knowledgebase); local metrics and LLM judges require Python 3.11+ with `futureagi[evals]`.
```bash
export FI_API_KEY="your-api-key"
@@ -123,14 +123,14 @@ Future AGI is a set of packages that evaluate LLM outputs, trace calls across yo
- **`ModuleNotFoundError: No module named 'fi'`** — The package is called `ai-evaluation`, not `future-agi` or `futureagi-sdk`:
+ **`ModuleNotFoundError: No module named 'fi'`** — Install the current `futureagi` package with the evaluation extra for this quickstart:
```bash
- pip install ai-evaluation
+ pip install "futureagi[evals]"
```
**`AuthenticationError`** — Both `FI_API_KEY` and `FI_SECRET_KEY` must be set. The API key alone is not enough.
- **`Python version error`** — `ai-evaluation` requires Python 3.10+. Check with `python --version`.
+ **`Python version error`** — Check with `python --version`. The base SDK requires Python 3.10+, but local metrics and LLM judges require Python 3.11+ and `futureagi[evals]`.
@@ -138,21 +138,20 @@ Future AGI is a set of packages that evaluate LLM outputs, trace calls across yo
### Python
-Six packages, each installable independently:
+Choose the base SDK or its evaluation extra, then add other packages as needed:
| Package | Install | What it does | Python |
|---------|---------|--------------|--------|
-| **futureagi** | `pip install futureagi` | Datasets, prompt versioning, knowledge bases | 3.9+ |
-| **ai-evaluation** | `pip install ai-evaluation` | 76+ local metrics + 100+ cloud templates, guardrails, streaming eval | 3.10+ |
+| **futureagi** | `pip install futureagi` | Datasets, prompt versioning, knowledge bases, cloud evaluations | 3.10+ |
+| **futureagi[evals]** | `pip install "futureagi[evals]"` | Base SDK plus local metrics and LLM judges | 3.11+ |
| **fi-instrumentation-otel** | `pip install fi-instrumentation-otel` | OpenTelemetry tracing for AI apps | 3.9+ |
| **traceai-\*** | `pip install traceAI-openai` | Auto-instrumentation for 45+ frameworks | 3.9+ |
| **agent-opt** | `pip install agent-opt` | Prompt optimization (6 algorithms) | 3.10+ |
| **agent-simulate** | `pip install agent-simulate` | Simulate voice AI agents at scale | 3.10+ |
```
-futureagi ← standalone base layer
- └── ai-evaluation ← installs futureagi automatically
- └── agent-opt ← installs ai-evaluation automatically
+futureagi ← base SDK (Python 3.10+)
+ └── futureagi[evals] ← local metrics and LLM judges (Python 3.11+)
fi-instrumentation-otel ← standalone tracing layer
├── traceai-* ← each installs fi-instrumentation-otel
@@ -160,7 +159,7 @@ fi-instrumentation-otel ← standalone tracing layer
```
- You don't need to install dependencies manually. `pip install ai-evaluation` gives you `futureagi` too. `pip install traceAI-openai` gives you `fi-instrumentation-otel` too.
+ `pip install "futureagi[evals]"` installs the base SDK and its evaluation dependencies together. `pip install traceAI-openai` gives you `fi-instrumentation-otel` too.
### TypeScript
diff --git a/src/pages/docs/sdk/protect.mdx b/src/pages/docs/sdk/protect.mdx
index 17e6641ad..15f8c5efe 100644
--- a/src/pages/docs/sdk/protect.mdx
+++ b/src/pages/docs/sdk/protect.mdx
@@ -30,22 +30,28 @@ result = protect.protect(
],
)
-print(result["status"]) # "failed"
-print(result["failed_rule"]) # "toxicity"
-print(result["messages"]) # action message
+print(result["status"]) # e.g. "failed"
+print(result["failed_rule"]) # e.g. ["toxicity"]
+print(result["messages"]) # action message on failure
```
## Protect Class
+Set both credentials in the environment before constructing the client. Replace the placeholders with your own credentials:
+
+```bash
+export FI_API_KEY="YOUR_API_KEY"
+export FI_SECRET_KEY="YOUR_SECRET_KEY"
+```
+
```python
from fi.evals import Protect
-protect = Protect(
- fi_api_key="...", # or FI_API_KEY env var
- fi_secret_key="...", # or FI_SECRET_KEY env var
-)
+protect = Protect()
```
+Both `FI_API_KEY` and `FI_SECRET_KEY` must be non-empty. This SDK checks the environment pair before considering constructor credentials, so passing `fi_api_key` and `fi_secret_key` is not an alternative when either environment variable is missing; construction raises `InvalidAuthError`.
+
## protect() Method
```python
@@ -96,14 +102,16 @@ rules = [
### Return Value
+The default path (`use_flash=False`) returns a dictionary. For example, a completed check that flags toxicity can return:
+
```python
{
- "status": "passed" | "failed",
+ "status": "failed",
"completed_rules": ["toxicity", "bias_detection"],
"uncompleted_rules": [],
- "failed_rule": None | "prompt_injection",
- "messages": "Input rejected" | "original input text",
- "reasons": ["..."],
+ "failed_rule": ["toxicity"],
+ "messages": "Input rejected — fails safety checks",
+ "reasons": ["All checks passed"], # fallback when no failure reason is collected
"time_taken": 0.45,
}
```
@@ -113,11 +121,17 @@ rules = [
| `status` | str | `"passed"` or `"failed"` |
| `completed_rules` | list | Rules that ran to completion |
| `uncompleted_rules` | list | Rules that didn't finish (timeout, error) |
-| `failed_rule` | str or None | First rule that failed |
-| `messages` | str | Action message if failed, original input if passed |
-| `reasons` | list | Reasoning for each rule (if `reason=True`) |
+| `failed_rule` | list[str] | Names of triggered rules reported by the evaluated batch; can contain multiple names or be empty |
+| `messages` | str | Action message on a rule failure; `"All checks passed"` on a normal pass; an adapter error message when input adaptation fails |
+| `reasons` | list | Collected failure explanations when available; otherwise a fallback message. Use `status` for the verdict |
| `time_taken` | float | Execution time in seconds |
+Use membership, not scalar equality, when checking a specific rule: `"toxicity" in result["failed_rule"]`. An empty list does not by itself prove success: input-adapter failures also return `failed_rule=[]` with `status="failed"`.
+
+### Protect Flash results
+
+With `use_flash=True`, the Flash branch returns a different `failed_rule` type: `"ProtectFlash"` for a harmful result, or `None` for a passing result. Its `completed_rules` is `["ProtectFlash"]`, not the individual metric names. If Flash returns no evaluation results, `status` is `"error"`, `failed_rule` is `None`, and `uncompleted_rules` is `["ProtectFlash"]`. Do not apply the Flash scalar contract to the default rule-based path.
+
## Common Patterns
### Check before sending to LLM
diff --git a/src/pages/docs/sdk/testcase.mdx b/src/pages/docs/sdk/testcase.mdx
index 9e7dcf091..b1597c3c3 100644
--- a/src/pages/docs/sdk/testcase.mdx
+++ b/src/pages/docs/sdk/testcase.mdx
@@ -1,165 +1,47 @@
---
-title: "Test Case: Future AGI Python SDK Test Reference"
-description: "Reference for the TestCase class in the Future AGI Python SDK. Create, manage, and load test cases for batch evaluation and scoring workflows."
+title: "Evaluation inputs"
+description: "Pass evaluation inputs to evaluate() and use the current metric input models in the Future AGI Python SDK."
---
-# Test Case Classes
+## Pass inputs to `evaluate()`
-The Test Case classes define the structure for test cases used in evaluations, including support for text, conversational, LLM, multimodal (image/audio), and more.
+The current `futureagi` package does not provide `fi.testcase` or the `TestCase`, `LLMTestCase`, `ConversationalTestCase`, and `MLLMTestCase` classes. Pass the inputs required by your chosen evaluation as keyword arguments to `fi.evals.evaluate()` instead.
----
-
-## `TestCase` Class
-
-Represents a general test case for evaluation.
-
-```python
-class TestCase(BaseModel):
- text: Optional[str] = None
- document: Optional[str] = None
- input: Optional[str] = None
- output: Optional[str] = None
- prompt: Optional[str] = None
- criteria: Optional[str] = None
- actual_json: Optional[dict] = None
- expected_json: Optional[dict] = None
- expected_text: Optional[str] = None
- query: Optional[str] = None
- response: Optional[str] = None
- context: Union[List[str], str] = None
-```
-
-**Attributes:**
-
-- `text` (Optional[str]): Text for the test case.
-- `document` (Optional[str]): Document content.
-- `input` (Optional[str]): Input string.
-- `output` (Optional[str]): Output string.
-- `prompt` (Optional[str]): Prompt used.
-- `criteria` (Optional[str]): Evaluation criteria.
-- `actual_json` (Optional[dict]): Actual JSON object.
-- `expected_json` (Optional[dict]): Expected JSON object.
-- `expected_text` (Optional[str]): Expected text output.
-- `query` (Optional[str]): Query string.
-- `response` (Optional[str]): Response string.
-- `context` (Union[List[str], str]): Context for the test case.
-
----
-
-## `ConversationalTestCase` Class
-
-Represents a conversational test case, consisting of a list of LLM test cases (messages).
-
-```python
-class ConversationalTestCase(BaseModel):
- messages: List[LLMTestCase]
-```
-
-**Attributes:**
-
-- `messages` (List[LLMTestCase]): List of LLM test case messages.
-
----
-
-## `LLMTestCase` Class
-
-Represents a test case for LLM (Language Model) evaluation.
-
-```python
-class LLMTestCase(BaseModel):
- query: str
- response: str
- context: Optional[Union[str, List[str]]] = None
- expected_response: Optional[str] = None
-```
-
-**Attributes:**
+For this local metric example, use Python 3.11+ and install the evaluation extra:
-- `query` (str): The input query.
-- `response` (str): The model's response.
-- `context` (Optional[Union[str, List[str]]]): Context for the test case.
-- `expected_response` (Optional[str]): The expected response.
-
----
-
-## `MLLMImage` Class
-
-Represents an image input for multimodal LLM test cases.
-
-```python
-class MLLMImage(BaseModel):
- url: str
- local: Optional[bool] = None
+```bash
+pip install "futureagi[evals]"
```
-**Attributes:**
-
-- `url` (str): URL or local path to the image.
-- `local` (Optional[bool]): Whether the image is local.
-
----
-
-## `MLLMAudio` Class
-
-Represents an audio input for multimodal LLM test cases.
-
```python
-class MLLMAudio(BaseModel):
- url: str
- local: Optional[bool] = None
- is_plain_text: bool = False
-```
+from fi.evals import evaluate
-**Attributes:**
-
-- `url` (str): URL or local path to the audio file.
-- `local` (Optional[bool]): Whether the audio is local.
-- `is_plain_text` (bool): Whether the input is plain text (not audio).
-
----
-
-## `MLLMTestCase` Class
-
-Represents a multimodal LLM test case, supporting image and audio inputs.
-
-```python
-class MLLMTestCase(TestCase):
- image_url: Optional[Union[str, MLLMImage]] = None
- input_image_url: Optional[Union[str, MLLMImage]] = None
- output_image_url: Optional[Union[str, MLLMImage]] = None
- input_audio: Optional[Union[str, MLLMAudio]] = None
- call_type: Optional[str] = None
+result = evaluate(
+ "contains",
+ output="The capital of France is Paris.",
+ keyword="Paris",
+)
+print(result.score)
```
-**Attributes:**
+`evaluate()` accepts a metric or template name and evaluation-specific inputs such as `output`, `input`, `context`, and `keyword`. Input names depend on the evaluation; there is no single test-case class that supplies the fields for every metric.
-- `image_url` (Optional[Union[str, MLLMImage]]): Image input.
-- `input_image_url` (Optional[Union[str, MLLMImage]]): Input image.
-- `output_image_url` (Optional[Union[str, MLLMImage]]): Output image.
-- `input_audio` (Optional[Union[str, MLLMAudio]]): Input audio.
-- `call_type` (Optional[str]): Type of call (if applicable).
+## Metric input models
----
+For APIs that accept metric input models, the current models live in `fi.evals.types`:
-## Example Usage
+| Model | Fields |
+|---|---|
+| `TextMetricInput` | Required `response: str`; optional `expected_response: str` or list of strings. |
+| `JsonMetricInput` | Required `response: str`, dictionary, or list; optional `expected_response` of the same types and optional `schema` as a string or dictionary. |
```python
-from fi.testcase import TestCase, LLMTestCase, ConversationalTestCase, MLLMTestCase
-
-# Simple test case
-tc = TestCase(input="What is the capital of France?", output="Paris")
-
-# LLM test case
-llm_tc = LLMTestCase(query="Who wrote 1984?", response="George Orwell")
-
-# Conversational test case
-conv_tc = ConversationalTestCase(messages=[llm_tc])
+from fi.evals.types import TextMetricInput
-# Multimodal test case
-mllm_tc = MLLMTestCase(
- input="Describe this image.",
- image_url="path/to/image.jpg"
+text_input = TextMetricInput(
+ response="The capital of France is Paris.",
+ expected_response="Paris",
)
```
----
\ No newline at end of file
+These models do not replace the keyword arguments to `evaluate()` automatically. Use the input names expected by the metric or template you selected.
diff --git a/src/pages/docs/sdk/tracing/environment-variables.mdx b/src/pages/docs/sdk/tracing/environment-variables.mdx
index 867a13a43..161c93347 100644
--- a/src/pages/docs/sdk/tracing/environment-variables.mdx
+++ b/src/pages/docs/sdk/tracing/environment-variables.mdx
@@ -3,29 +3,61 @@ title: "Environment variables"
description: "Environment variables that configure traceAI tracing."
---
+Environment-variable support differs by language. Set variables before initializing tracing; do not assume a variable supported by one SDK is read by another. Native OpenTelemetry integrations use the controls documented in their integration guide.
-traceAI SDKs share the following environment-variable names. Support varies by SDK and integration; native OpenTelemetry integrations use the controls documented in their integration guide.
-
-| Variable | Purpose | Default |
-|----------|---------|---------|
-| `FI_API_KEY` | Authentication | required |
-| `FI_SECRET_KEY` | Authentication | required |
-| `FI_BASE_URL` | HTTP collector endpoint | `https://api.futureagi.com` |
-| `FI_GRPC_URL` | gRPC collector endpoint | `https://grpc.futureagi.com` |
-| `FI_PROJECT_NAME` | Default project name | None |
-| `FI_PROJECT_VERSION_NAME` | Default version | None |
-| `FI_HIDE_INPUTS` | Redact inputs | False |
-| `FI_HIDE_OUTPUTS` | Redact outputs | False |
-| `FI_HIDE_INPUT_MESSAGES` | Redact input messages | False |
-| `FI_HIDE_OUTPUT_MESSAGES` | Redact output messages | False |
-| `FI_HIDE_INPUT_IMAGES` | Redact input images | False |
-| `FI_HIDE_INPUT_TEXT` | Redact input text | False |
-| `FI_HIDE_OUTPUT_TEXT` | Redact output text | False |
-| `FI_HIDE_EMBEDDING_VECTORS` | Redact embedding vectors | False |
-| `FI_HIDE_LLM_INVOCATION_PARAMETERS` | Redact model parameters | False |
-| `FI_BASE64_IMAGE_MAX_LENGTH` | Max base64 image chars | 32000 |
-| `FI_PII_REDACTION` | Auto-mask PII (Python) | False |
-| `FI_HIDE_ERROR_DETAILS` | Hide error messages and stack traces on supported traceAI spans. Use Python `fi-instrumentation-otel` 1.2.3 or later or TypeScript `@traceai/fi-core` 1.1.1 or later. See [Errors and privacy](/docs/sdk/tracing/trace-config#errors-and-privacy) | When unset, derived from the resolved input/output content flags in code or the environment; see below |
-
-For Python and TypeScript, `FI_HIDE_ERROR_DETAILS` defaults to true when any of `hide_inputs`, `hide_outputs`, `hide_input_messages`, `hide_output_messages`, `hide_input_text`, or `hide_output_text` is true (camelCase in TypeScript). These flags can come from code or their `FI_` environment variables. Otherwise, it defaults to false. Image-only, embedding-vector, invocation-parameter, and PII masking do not enable it. An explicit error-details setting in code takes precedence over `FI_HIDE_ERROR_DETAILS`; an explicit environment value takes precedence over the derived default.
+## Connection variables
+| Variable | Purpose |
+|----------|---------|
+| `FI_API_KEY` | Authentication |
+| `FI_SECRET_KEY` | Authentication |
+| `FI_BASE_URL` | HTTP collector base URL |
+| `FI_GRPC_URL` | gRPC collector endpoint; not read by Java's `TraceConfig` |
+| `FI_PROJECT_NAME` | Default project name |
+| `FI_PROJECT_VERSION_NAME` | Default version; not read by Java's `TraceConfig` |
+
+Java reads `FI_BASE_URL`, `FI_API_KEY`, `FI_SECRET_KEY`, and `FI_PROJECT_NAME`. Set `FI_BASE_URL` explicitly (for cloud, `https://api.futureagi.com`): Java's `TraceConfig` has no built-in base URL default. See [register()](/docs/sdk/tracing/register) for programmatic connection options.
+
+## Privacy variables by language
+
+“Yes” means the SDK reads the environment variable. It does not mean the setting is enabled by default. Input/output hiding flags default to `false`.
+
+| Variable | Python | TypeScript | Java | C# |
+|----------|--------|------------|------|----|
+| `FI_HIDE_INPUTS` | Yes | Yes | Yes | Yes |
+| `FI_HIDE_OUTPUTS` | Yes | Yes | Yes | Yes |
+| `FI_HIDE_INPUT_MESSAGES` | Yes | Yes | No; use builder | Yes |
+| `FI_HIDE_OUTPUT_MESSAGES` | Yes | Yes | No; use builder | Yes |
+| `FI_HIDE_INPUT_IMAGES` | Yes | Yes | No | Yes |
+| `FI_HIDE_INPUT_TEXT` | Yes | Yes | No | Yes |
+| `FI_HIDE_OUTPUT_TEXT` | Yes | Yes | No | Yes |
+| `FI_HIDE_EMBEDDING_VECTORS` | Yes | Yes | No | Yes |
+| `FI_HIDE_LLM_INVOCATION_PARAMETERS` | Yes | Yes | No | Yes |
+| `FI_HIDE_ERROR_DETAILS` | Yes | Yes | Yes | Yes |
+| `FI_BASE64_IMAGE_MAX_LENGTH` | Yes; default `0` | Yes; default `0` | No | Yes; default `32000` |
+| `FI_PII_REDACTION` | Yes; default `false` | Yes; default `false` | No | No |
+
+In Java, `FI_HIDE_ERROR_DETAILS` defaults to `false`. In Python, TypeScript, and C#, an unset error-detail flag is derived from the input/output content-hiding flags. Explicit programmatic values take precedence over environment defaults. For C#, environment defaults are read by `TraceConfig.Builder()`; constructing `new TraceConfig()` does not read them.
+
+`FI_HIDE_ERROR_DETAILS` hides error messages and stack traces on supported traceAI spans. Use Python `fi-instrumentation-otel` 1.2.3 or later or TypeScript `@traceai/fi-core` 1.1.1 or later. See [Errors and privacy](/docs/sdk/tracing/trace-config#errors-and-privacy) for the supported scope.
+
+For Python and TypeScript's `TraceConfig`, error-detail hiding defaults to true when any of `hide_inputs`, `hide_outputs`, `hide_input_messages`, `hide_output_messages`, `hide_input_text`, or `hide_output_text` is true (camelCase in TypeScript). These flags can come from code or their `FI_` environment variables. An explicit error-details setting in code takes precedence over `FI_HIDE_ERROR_DETAILS`; an explicit environment value takes precedence over the derived default. Export-time controls can apply additional hiding.
+
+## Java message hiding
+
+Java does not read `FI_HIDE_INPUT_MESSAGES` or `FI_HIDE_OUTPUT_MESSAGES`. Set the corresponding builder methods instead:
+
+```java
+import ai.traceai.TraceAI;
+import ai.traceai.TraceConfig;
+
+// Connection values come from FI_BASE_URL, FI_API_KEY,
+// FI_SECRET_KEY, and FI_PROJECT_NAME.
+TraceAI.init(TraceConfig.builder()
+ .hideInputMessages(true)
+ .hideOutputMessages(true)
+ .build()
+);
+```
+
+The Java builder also reads `FI_HIDE_INPUTS`, `FI_HIDE_OUTPUTS`, and `FI_HIDE_ERROR_DETAILS` when their corresponding builder methods are not set. It does not read the image, text, embedding, invocation-parameter, image-length, or PII variables in the matrix.
diff --git a/src/pages/docs/sdk/tracing/instrument-with-traceai-helpers.mdx b/src/pages/docs/sdk/tracing/instrument-with-traceai-helpers.mdx
index b6ee07d67..213ab9439 100644
--- a/src/pages/docs/sdk/tracing/instrument-with-traceai-helpers.mdx
+++ b/src/pages/docs/sdk/tracing/instrument-with-traceai-helpers.mdx
@@ -101,8 +101,8 @@ Manual tracing with raw OpenTelemetry means writing a lot of setup code for ever
```javascript JS/TS
tracer.startActiveSpan("my-span-name", { attributes: { "fi.span.kind": "chain" } }, (span) => {
- span.setAttribute("input", "input");
- span.setAttribute("output", "output");
+ span.setAttribute("input.value", "input");
+ span.setAttribute("output.value", "output");
span.setStatus({ code: SpanStatusCode.OK });
span.end();
});
@@ -163,8 +163,8 @@ Manual tracing with raw OpenTelemetry means writing a lot of setup code for ever
```javascript JS/TS
tracer.startActiveSpan("agent-span-with-plain-text-io", { attributes: { "fi.span.kind": "agent" } }, (span) => {
- span.setAttribute("input", "input");
- span.setAttribute("output", "output");
+ span.setAttribute("input.value", "input");
+ span.setAttribute("output.value", "output");
span.setStatus({ code: SpanStatusCode.OK });
span.end();
});
@@ -208,8 +208,8 @@ Manual tracing with raw OpenTelemetry means writing a lot of setup code for ever
```javascript JS/TS
tracer.startActiveSpan("tool-span", { attributes: { "fi.span.kind": "tool" } }, (span) => {
- span.setAttribute("input", "input");
- span.setAttribute("output", "output");
+ span.setAttribute("input.value", "input");
+ span.setAttribute("output.value", "output");
span.setAttribute("tool.name", "tool-name");
span.setAttribute("tool.description", "tool-description");
span.setAttribute("tool.parameters", JSON.stringify({"input": "input"}));
@@ -253,8 +253,8 @@ Manual tracing with raw OpenTelemetry means writing a lot of setup code for ever
```javascript JS/TS
tracer.startActiveSpan("llm-span", { attributes: { "fi.span.kind": "llm" } }, (span) => {
- span.setAttribute("input", "input");
- span.setAttribute("output", "output");
+ span.setAttribute("input.value", "input");
+ span.setAttribute("output.value", "output");
span.setStatus({ code: SpanStatusCode.OK });
span.end();
});
@@ -280,8 +280,8 @@ Manual tracing with raw OpenTelemetry means writing a lot of setup code for ever
```javascript JS/TS
tracer.startActiveSpan("retriever-span", { attributes: { "fi.span.kind": "retriever" } }, (span) => {
- span.setAttribute("input", "input");
- span.setAttribute("output", "output");
+ span.setAttribute("input.value", "input");
+ span.setAttribute("output.value", "output");
span.setStatus({ code: SpanStatusCode.OK });
span.end();
});
diff --git a/src/pages/docs/sdk/tracing/register.mdx b/src/pages/docs/sdk/tracing/register.mdx
index 83908feb4..bbb0e18ca 100644
--- a/src/pages/docs/sdk/tracing/register.mdx
+++ b/src/pages/docs/sdk/tracing/register.mdx
@@ -9,8 +9,8 @@ Creates an OpenTelemetry tracer provider configured to export spans to your Futu
```python
- from fi_instrumentation import register
- from fi_instrumentation.fi_types import ProjectType, Transport
+ from fi_instrumentation import register, Transport
+ from fi_instrumentation.fi_types import ProjectType
trace_provider = register(
project_name="my-project",
@@ -33,7 +33,7 @@ Creates an OpenTelemetry tracer provider configured to export spans to your Futu
| `headers` | dict / None | None | Custom HTTP headers (auto-populated from API keys if not set) |
| `verbose` | bool | True | Print configuration details on startup |
| `transport` | Transport | `HTTP` | `HTTP` or `GRPC` |
- | `semantic_convention` | SemanticConvention | `FI` | Attribute naming convention |
+ | `semantic_convention` | SemanticConvention | `FI` | Resource metadata describing the convention emitted by the application or instrumentor; does not convert span attributes |
**Returns:** `TracerProvider` - pass this to `.instrument(tracer_provider=...)` on any instrumentor.
@@ -64,8 +64,11 @@ Creates an OpenTelemetry tracer provider configured to export spans to your Futu
| `verbose` | boolean | false | Verbose logging |
| `endpoint` | string | `FI_BASE_URL` | Custom endpoint |
| `transport` | Transport | `HTTP` | `HTTP` or `GRPC` |
+ | `sampler` | Sampler | undefined (OpenTelemetry environment/default) | Optional OpenTelemetry head sampler. See [Sampling](#sampling) |
**Returns:** `FITracerProvider`
+
+ TypeScript accepts `sampler?: Sampler` and forwards it to the OpenTelemetry provider. Alternatively, leave it unset to use the standard `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` environment variables (see [Sampling](#sampling)). Call `await tracerProvider.forceFlush()` before a short-lived process ends.
```java
@@ -165,18 +168,42 @@ Creates an OpenTelemetry tracer provider configured to export spans to your Futu
| `EXPERIMENT` | Development and testing. Supports eval tags and version names. |
| `OBSERVE` | Production monitoring. No eval tags, no version names. |
-## SemanticConvention (Python/TypeScript)
+## SemanticConvention (Python)
+
+Python's `semantic_convention` option records a `semantic_convention` attribute on the provider's resource. It identifies the convention your application or instrumentor actually emits; it does **not** rewrite, rename, or add span attributes. TypeScript `register()` does not expose this option.
+
+Import `SemanticConvention` from `fi_instrumentation` and select the value that describes your emitted spans:
+
+| Value | Resource value | Convention described |
+|-------|----------------|----------------------|
+| `OTEL_GENAI` | `otel_genai` | OpenTelemetry GenAI |
+| `FI` | `fi` | Future AGI native convention (default) |
+| `OPENINFERENCE` | `openinference` | OpenInference |
+| `OPENLLMETRY` | `openllmetry` | OpenLLMetry |
+
+
+ Passing `semantic_convention=SemanticConvention.OTEL_GENAI` does not make an instrumentor emit GenAI attributes. Configure the application or instrumentor itself to emit the attributes required by your downstream tools.
+
+
+## Sampling
+
+Pass an OpenTelemetry sampler to TypeScript `register()`. For example, record new root traces and respect an existing parent's sampling decision:
+
+```typescript
+import { register, ProjectType } from "@traceai/fi-core";
+import { AlwaysOnSampler, ParentBasedSampler } from "@opentelemetry/sdk-trace-base";
-Controls how span attributes are named. We recommend `OTEL_GENAI` for standard OpenTelemetry GenAI conventions.
+const tracerProvider = register({
+ projectName: "my-project",
+ projectType: ProjectType.OBSERVE,
+ sampler: new ParentBasedSampler({ root: new AlwaysOnSampler() }),
+});
+```
-| Value | Attribute prefix | Use for |
-|-------|-----------------|---------|
-| `OTEL_GENAI` | `gen_ai.*` | Recommended - OpenTelemetry GenAI standard |
-| `FI` | `fi.*` | Legacy Future AGI format (default) |
-| `OPENINFERENCE` | `openinference.*` | Arize Phoenix compatibility |
-| `OPENLLMETRY` | `traceloop.*` | Traceloop / OpenLLMetry compatibility |
+To configure sampling through the environment instead, omit `sampler`. Set these variables before creating the provider to sample approximately 10% of new root traces while respecting an existing parent's sampling decision:
-
- Pass `semantic_convention=SemanticConvention.OTEL_GENAI` for the best interoperability with other OpenTelemetry tools.
-
+```bash
+export OTEL_TRACES_SAMPLER=parentbased_traceidratio
+export OTEL_TRACES_SAMPLER_ARG=0.1
+```
diff --git a/src/pages/docs/sdk/tracing/set-session-user-id.mdx b/src/pages/docs/sdk/tracing/set-session-user-id.mdx
index 72735aa00..2680ac838 100644
--- a/src/pages/docs/sdk/tracing/set-session-user-id.mdx
+++ b/src/pages/docs/sdk/tracing/set-session-user-id.mdx
@@ -5,7 +5,7 @@ description: "Add SessionID and UserID as span attributes to group and filter tr
## About
-Traces are isolated by default. Without a session or user identifier, there is no way to connect multiple traces that belong to the same conversation or the same end user. Setting `session.id` and `user.id` on spans links them together so traces can be grouped by conversation and filtered by user in the dashboard. Both values are added to the OpenTelemetry context and automatically picked up by traceAI auto-instrumentors as span attributes.
+Traces are isolated by default. Without a session or user identifier, there is no way to connect multiple traces that belong to the same conversation or the same end user. Setting `session.id` and `user.id` on spans links them together so traces can be grouped by conversation and filtered by user in the dashboard. Python context helpers pass these values to traceAI instrumentors. In JS/TS, use FI context helpers with `FITracer.startActiveSpan()` for explicitly wrapped spans, as shown below. OpenTelemetry baggage alone does not attach these attributes to traceAI spans; do not assume an auto-instrumented OpenAI child span inherits them.
---
@@ -30,19 +30,30 @@ Traces are isolated by default. Without a session or user identifier, there is n
```
```javascript JS/TS
- npm install @opentelemetry/api # or yarn add @opentelemetry/api
- # Assuming your traceAI or equivalent auto-instrumentation package is already installed.
+ npm install @traceai/fi-core @opentelemetry/api
```
- Choose your approach:`using_session`, `using_user`, or `using_attributes`.
+ In Python, choose `using_session`, `using_user`, or `using_attributes`. For the JS/TS examples, first create an `FITracer` (with `FI_API_KEY` and `FI_SECRET_KEY` in your environment):
+
+ ```javascript JS/TS
+ import { FITracer, register, ProjectType } from "@traceai/fi-core";
+
+ const provider = register({
+ projectName: "session-user-example",
+ projectType: ProjectType.OBSERVE,
+ });
+ const tracer = new FITracer({ tracer: provider.getTracer("session-user-example") });
+ ```
+
+ Each JS/TS example below attaches identifiers to the span it creates, not automatically to every instrumented child. Call `await provider.forceFlush()` after your spans end in a short-lived script.
- Add a session ID to the current OpenTelemetry context. Any LLM call within the block will include `session.id` as a span attribute. The session ID must be a non-empty string.
+ In Python, add a session ID to the current OpenTelemetry context with `using_session`. In JS/TS, pass an FI session context to `FITracer.startActiveSpan()` to attach `session.id` to that span. Use a non-empty string.
@@ -58,20 +69,16 @@ Traces are isolated by default. Without a session or user identifier, there is n
```
```javascript JS/TS
- import { context, propagation } from "@opentelemetry/api";
-
- const sessionId = "my-js-session-id"; // Example session ID
-
- const activeContext = context.active();
- const baggageWithSession = propagation.createBaggage({
- "session.id": { value: sessionId }
- });
- const newContext = propagation.setBaggage(activeContext, baggageWithSession);
-
- context.with(newContext, () => {
- // Calls within this block by auto-instrumented libraries (like traceAI)
- // should generate spans with the attribute: "session.id" = "my-js-session-id"
- // e.g., myInstrumentedFunction();
+ import { context } from "@opentelemetry/api";
+ import { setSession } from "@traceai/fi-core";
+
+ const sessionContext = setSession(context.active(), { sessionId: "my-js-session-id" });
+ tracer.startActiveSpan("session-work", {}, sessionContext, (span) => {
+ try {
+ // This span carries session.id. Run your work here.
+ } finally {
+ span.end();
+ }
});
```
@@ -90,7 +97,7 @@ Traces are isolated by default. Without a session or user identifier, there is n
- Add a user ID to the current OpenTelemetry context. Any LLM call within the block will include `user.id` as a span attribute. The user ID must be a non-empty string.
+ In Python, add a user ID to the current OpenTelemetry context with `using_user`. In JS/TS, pass an FI user context to `FITracer.startActiveSpan()` to attach `user.id` to that span. Use a non-empty string.
@@ -106,20 +113,16 @@ Traces are isolated by default. Without a session or user identifier, there is n
```
```javascript JS/TS
- import { context, propagation } from "@opentelemetry/api";
-
- const userId = "my-js-user-id"; // Example user ID
-
- const activeContext = context.active();
- const baggageWithUser = propagation.createBaggage({
- "user.id": { value: userId }
- });
- const newContext = propagation.setBaggage(activeContext, baggageWithUser);
-
- context.with(newContext, () => {
- // Calls within this block by auto-instrumented libraries (like traceAI)
- // should generate spans with the attribute: "user.id" = "my-js-user-id"
- // e.g., myInstrumentedFunction();
+ import { context } from "@opentelemetry/api";
+ import { setUser } from "@traceai/fi-core";
+
+ const userContext = setUser(context.active(), { userId: "my-js-user-id" });
+ tracer.startActiveSpan("user-work", {}, userContext, (span) => {
+ try {
+ // This span carries user.id. Run your work here.
+ } finally {
+ span.end();
+ }
});
```
@@ -138,7 +141,7 @@ Traces are isolated by default. Without a session or user identifier, there is n
- Use `using_attributes` to set session ID, user ID, or both in a single call alongside an LLM client.
+ In Python, use `using_attributes` to set session ID, user ID, or both alongside an LLM client. The JS/TS alternatives use FI context helpers for an explicitly wrapped span.
@@ -162,27 +165,16 @@ Traces are isolated by default. Without a session or user identifier, there is n
```
```javascript JS/TS
- import { context, propagation } from "@opentelemetry/api";
- // Assume 'openai' client or equivalent is initialized and used here.
- // import OpenAI from 'openai';
- // const client = new OpenAI();
-
- const sessionId = "my-js-session-id";
-
- const activeContext = context.active();
- const baggageWithSession = propagation.createBaggage({
- "session.id": { value: sessionId }
- });
- const newContext = propagation.setBaggage(activeContext, baggageWithSession);
-
- context.with(newContext, () => {
- // Example LLM call that would pick up the session.id from context
- // response = client.chat.completions.create(
- // model="gpt-3.5-turbo",
- // messages=[{"role": "user", "content": "Write a haiku in JavaScript context."}],
- // max_tokens=20,
- // );
- console.log('In context with session.id set via Baggage');
+ import { context } from "@opentelemetry/api";
+ import { setSession } from "@traceai/fi-core";
+
+ const sessionContext = setSession(context.active(), { sessionId: "my-js-session-id" });
+ tracer.startActiveSpan("session-request", {}, sessionContext, (span) => {
+ try {
+ // Run your work here. session.id is attached to this span.
+ } finally {
+ span.end();
+ }
});
```
@@ -203,21 +195,16 @@ Traces are isolated by default. Without a session or user identifier, there is n
```
```javascript JS/TS
- import { context, propagation } from "@opentelemetry/api";
- // Assume 'client' (e.g., OpenAI client) is initialized and used here.
-
- const userId = "my-js-user-id";
-
- const activeContext = context.active();
- const baggageWithUser = propagation.createBaggage({
- "user.id": { value: userId }
- });
- const newContext = propagation.setBaggage(activeContext, baggageWithUser);
-
- context.with(newContext, () => {
- // Example LLM call that would pick up the user.id from context
- // response = client.chat.completions.create(...);
- console.log('In context with user.id set via Baggage');
+ import { context } from "@opentelemetry/api";
+ import { setUser } from "@traceai/fi-core";
+
+ const userContext = setUser(context.active(), { userId: "my-js-user-id" });
+ tracer.startActiveSpan("user-request", {}, userContext, (span) => {
+ try {
+ // Run your work here. user.id is attached to this span.
+ } finally {
+ span.end();
+ }
});
```
@@ -241,23 +228,19 @@ Traces are isolated by default. Without a session or user identifier, there is n
```
```javascript JS/TS
- import { context, propagation } from "@opentelemetry/api";
- // Assume 'client' (e.g., OpenAI client) is initialized and used here.
-
- const sessionId = "my-js-session-id";
- const userId = "my-js-user-id";
-
- const activeContext = context.active();
- const baggageWithBoth = propagation.createBaggage({
- "session.id": { value: sessionId },
- "user.id": { value: userId }
- });
- const newContext = propagation.setBaggage(activeContext, baggageWithBoth);
-
- context.with(newContext, () => {
- // Example LLM call that would pick up both session.id and user.id from context
- // response = client.chat.completions.create(...);
- console.log('In context with session.id and user.id set via Baggage');
+ import { context } from "@opentelemetry/api";
+ import { setSession, setUser } from "@traceai/fi-core";
+
+ const requestContext = setUser(
+ setSession(context.active(), { sessionId: "my-js-session-id" }),
+ { userId: "my-js-user-id" },
+ );
+ tracer.startActiveSpan("session-user-request", {}, requestContext, (span) => {
+ try {
+ // This span carries both identifiers. Run your work here.
+ } finally {
+ span.end();
+ }
});
```
@@ -302,7 +285,7 @@ Traces are isolated by default. Without a session or user identifier, there is n
- **`using_session`**:Context manager that adds `session.id` to the OpenTelemetry context. All spans from traceAI auto-instrumentors within the block will carry this attribute. Input must be a non-empty string.
- **`using_user`**:Context manager that adds `user.id` to the OpenTelemetry context. All spans within the block will carry this attribute. Input must be a non-empty string.
- **`using_attributes`**:General-purpose context manager that accepts both `session_id` and `user_id` (and other attributes). Useful when setting multiple context attributes in one call.
-- **Baggage (JS/TS)**:The JS/TS equivalent of Python context managers. Use `propagation.createBaggage()` and `context.with()` to propagate session and user IDs to child spans.
+- **FI context helpers (JS/TS)**: `setSession` and `setUser` return a context that `FITracer.startActiveSpan()` reads to attach identifiers to the span it creates. They are not OpenTelemetry baggage, and do not guarantee that auto-instrumented OpenAI child spans receive the same attributes.
---
diff --git a/src/pages/docs/self-hosting.mdx b/src/pages/docs/self-hosting.mdx
index bebabea78..58849f849 100644
--- a/src/pages/docs/self-hosting.mdx
+++ b/src/pages/docs/self-hosting.mdx
@@ -104,7 +104,7 @@ The Helm chart runs the Distributed topology on Kubernetes, with one Deployment
| Model downloads from the Hugging Face Hub by `serving`, the first time each model loads | When `serving` runs: always on Distributed, with the `ml` profile on Standalone, with `serving.enabled` on Helm | Pre-seed the model cache, then set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`; on Helm, `global.airgap` |
| Fonts, icons, a code editor, document previews, videos and logos, loaded by users' browsers from public hosts | Yes: fonts and icons on every page, the rest on the screens that use them | Not switchable. Without `cdn.jsdelivr.net` the code editor does not load, so code evals cannot be written |
-Deployment telemetry registers the install once, with the email addresses of its owner, admin, staff and superuser accounts, then sends usage counts every 6 hours, never traces, prompts or other content. With it off, one minimal registration without email addresses still goes out. [Telemetry](/docs/self-hosting/configuration/telemetry) covers exactly what is sent and how to turn it off or block it.
+Deployment telemetry registers the install without a `users` field or email addresses by default, then sends usage counts every 6 hours, never traces, prompts or other content. Sharing owner, admin, staff and superuser email addresses requires `FUTURE_AGI_TELEMETRY_SHARE_ADMIN_EMAILS=true` (Helm: `config.telemetryShareAdminEmails=true`); leave it false to omit them. With it off, no registration, heartbeats or events are sent, including retries and buffered replay. [Telemetry](/docs/self-hosting/configuration/telemetry) covers exactly what is sent and how to turn it off or block it.
## Where to go next
diff --git a/src/pages/docs/self-hosting/configuration/environment.mdx b/src/pages/docs/self-hosting/configuration/environment.mdx
index 1c9f9fec9..6cc778c8c 100644
--- a/src/pages/docs/self-hosting/configuration/environment.mdx
+++ b/src/pages/docs/self-hosting/configuration/environment.mdx
@@ -113,7 +113,7 @@ Two more keys control how the gateway reaches providers:
| Key | Default | What it does |
|---|---|---|
| `AGENTCC_ALLOW_PRIVATE_PROVIDER_URLS` | `false` | `true` lets providers added in the UI or the org API use base URLs on private networks (LAN addresses, Docker service names, `host.docker.internal`), such as an Ollama or vLLM server. One `.env` line reaches both the API and the gateway. Loopback, link-local and cloud metadata addresses stay refused. Providers listed in the gateway's `config.yaml` need no setting. |
-| `AGENTCC_SYNC_INTERVAL` | `60s` | How often the gateway re-reads keys and organization settings from the app, as a Go duration (`60s`, `5m`). See [Keys and org settings sync](/docs/self-hosting/configuration/system#keys-and-org-settings-sync). |
+| `AGENTCC_SYNC_INTERVAL` | `60s` | How often the gateway re-reads keys and organization settings from the app, as a positive Go duration up to `60s` (for example `30s` or `60s`). With a control-plane URL configured, values above `60s` or at/below zero are clamped to `60s`; zero does not disable polling. See [Keys and org settings sync](/docs/self-hosting/configuration/system#keys-and-org-settings-sync). |
With `AGENTCC_ALLOW_PRIVATE_PROVIDER_URLS=true`, anyone who can add a provider can point the gateway and the API at the databases and every other service on those networks, including a ClickHouse without a password on installs made before Standalone became the default. Turn it on only where everyone who can add a provider is trusted: see [Private and local provider URLs](/docs/command-center/features/self-hosted-models#private-and-local-provider-urls).
@@ -175,7 +175,7 @@ An install made before Standalone became the default may still have `OSS_RETURN_
## Telemetry
-Deployment telemetry is on by default. Set `FUTURE_AGI_TELEMETRY_DISABLED=true` in `.env` and run `docker compose up -d` to turn it off, or install with `./bin/install --no-telemetry`, which writes that line before anything starts. [Telemetry and outbound connections](/docs/self-hosting/configuration/telemetry) covers what is sent, what the opt-out still sends, and every other outbound connection.
+Deployment telemetry is on by default. Set `FUTURE_AGI_TELEMETRY_DISABLED=true` in `.env` and run `docker compose up -d` to turn it off, or install with `./bin/install --no-telemetry`, which writes that line before anything starts. [Telemetry and outbound connections](/docs/self-hosting/configuration/telemetry) covers what is sent, how opting out stops registration, heartbeats and event sends (including retries and buffered replay), and every other outbound connection.
## Image versions
diff --git a/src/pages/docs/self-hosting/configuration/reference.mdx b/src/pages/docs/self-hosting/configuration/reference.mdx
index bcc8733d9..3c39b94d9 100644
--- a/src/pages/docs/self-hosting/configuration/reference.mdx
+++ b/src/pages/docs/self-hosting/configuration/reference.mdx
@@ -306,13 +306,13 @@ Distributed's `backend` read them too.
Deployment telemetry is **on** by default, so Future AGI knows how many
installs run which version. It never sends traces, prompts, completions,
-datasets or any other content. What it sends and when, what the opt-out still
-sends, and how to see what your install sent:
+datasets or any other content. What it sends and when, how to turn it off,
+and how to see what your install sent:
[Telemetry and outbound connections](/docs/self-hosting/configuration/telemetry).
| Key | Default | Setups | What it does |
|---|---|---|---|
-| `FUTURE_AGI_TELEMETRY_DISABLED` | `false` | S D H | `true` opts out (`1`, `yes` and `on` also work): no email addresses and no heartbeats are sent, and buffered heartbeats are deleted; one minimal registration remains ([what it contains](/docs/self-hosting/configuration/telemetry#turn-it-off)). `./bin/install --no-telemetry` (`.\bin\install.ps1 -NoTelemetry`) writes it before anything starts. Helm: `config.telemetry=false`. |
+| `FUTURE_AGI_TELEMETRY_DISABLED` | `false` | S D H | `true` opts out (`1`, `yes` and `on` also work): no registration, heartbeats or events are sent, including retries and buffered replay; the disabled telemetry cycle clears both outboxes. See [Turn it off](/docs/self-hosting/configuration/telemetry#turn-it-off). `./bin/install --no-telemetry` (`.\bin\install.ps1 -NoTelemetry`) writes it before anything starts. Helm: `config.telemetry=false`. |
| `FUTURE_AGI_TELEMETRY_URL` | `https://api.futureagi.com` | S D H | Where registrations and heartbeats go. Change it only to test against another receiver. |
| `FUTURE_AGI_TELEMETRY_INTERVAL_HOURS` | `6` | S D H | Heartbeat interval: 1, 2, 3, 4, 6, 8, 12 or 24. Other values fall back to 6. |
| `FUTURE_AGI_TELEMETRY_JITTER_SECONDS` | `1800` | S D H | Maximum random delay added to each scheduled run, so installs do not all send at once. |
@@ -471,7 +471,7 @@ database, override `PG_HOST`, `PGBOUNCER_HOST` or `CH_HOST` in a
| `EXTRA_CSRF_ORIGINS` | empty | S D H | Extra trusted CSRF origins, comma-separated. |
| `AGENTCC_CONFIG_PATH` | `agentcc-gateway/config.example.yaml` | S D | Gateway provider config, relative to the repository root, mounted into the gateway. Copy the example to enable Anthropic, Gemini, Bedrock, Vertex and others; the copy is git-ignored. A path that does not exist stops the stack from starting. |
| `AGENTCC_ALLOW_PRIVATE_PROVIDER_URLS` | `false`; Helm: `agentccGateway.allowPrivateProviderURLs` | S D H | `true` lets org providers use base URLs on private networks (RFC 1918 addresses, Docker service names): a local Ollama or vLLM, see [INSTALLATION.md](https://github.com/future-agi/future-agi/blob/main/INSTALLATION.md#local-models-ollama-vllm). Read by the gateway and the API; a gateway or API from before this setting ignores it and refuses every private address. Loopback, link-local and cloud metadata addresses stay refused. Anyone who can add a provider can then reach every service on that network, ClickHouse included (its `default` user has no password while `CH_PASSWORD` is empty, as on installs made before the installer generated one), so turn it on only when they are all trusted. |
-| `AGENTCC_SYNC_INTERVAL` | `60s` (Helm: `60s`, from `agentccGateway.config.control_plane.sync_interval`) | S D H | A Go duration (`60s`, `5m`): the gateway also re-reads keys and org settings from the app this often, so a gateway that started before the app (Distributed on a slow host) or several gateway replicas catch up. A gateway from before this setting ignores it and syncs only on start: its `control plane sync enabled` log line shows an `interval` of `0s`. Helm sets it in the gateway's config file, which every gateway reads. |
+| `AGENTCC_SYNC_INTERVAL` | `60s` (Helm: `60s`, from `agentccGateway.config.control_plane.sync_interval`) | S D H | A positive Go duration up to `60s` (`30s`, `60s`): the gateway also re-reads keys and org settings from the app this often, so a gateway that started before the app (Distributed on a slow host) or several gateway replicas catch up. With a control-plane URL configured, values above `60s` or at/below zero are clamped to `60s`; zero does not disable polling. A gateway from before this setting ignores it and syncs only on start: its `control plane sync enabled` log line shows an `interval` of `0s`. Helm sets it in the gateway's config file, which every gateway reads. |
| `APP_VERSION` | unset | S D H | The running version, reported by the license check, Sentry and the model server. Helm sets it to the backend image tag. |
| `SIM_COLLECTOR_OTLP_ENDPOINT` | `fi-collector:4317` | D H | OTLP/gRPC `host:port` where simulations and voice calls send their spans. Helm sets it to the release's collector. |
| `OTEL_ENABLED` | `false` | S D H | Export the platform's own OpenTelemetry traces (monitoring Future AGI itself, not your application's traces). |
diff --git a/src/pages/docs/self-hosting/configuration/telemetry.mdx b/src/pages/docs/self-hosting/configuration/telemetry.mdx
index 0c37ca2d9..4cd7a154c 100644
--- a/src/pages/docs/self-hosting/configuration/telemetry.mdx
+++ b/src/pages/docs/self-hosting/configuration/telemetry.mdx
@@ -5,10 +5,10 @@ description: "What a self-hosted Future AGI install sends to Future AGI and to o
## In this page
-A self-hosted install makes one kind of call home on its own: deployment telemetry to Future AGI, on by default. An install with an Enterprise licence makes a second: licence activation and a licence heartbeat, which carries the same instance id and usage counts. This page covers what it sends and when, what it never sends, how to turn it off, how to see what your install sent, and every other outbound connection the platform and the browser UI make. It applies to Standalone, Distributed and Helm. The exact payloads are under [Registration](#registration) and [Heartbeat](#heartbeat).
+A self-hosted install makes one kind of call home on its own: deployment telemetry to Future AGI, on by default. An install with an Enterprise licence makes a second: licence activation and a licence heartbeat, which carries the same instance id and usage counts. This page covers what it sends and when, what it never sends, how to turn it off, how to see what your install sent, and every other outbound connection the platform and the browser UI make. It applies to Standalone, Distributed and Helm. The payloads are under [Registration](#registration), [Heartbeat](#heartbeat) and [Lifecycle and sign-in events](#lifecycle-and-sign-in-events).
-After the first account is created, the install registers once with `api.futureagi.com`, including the email addresses of its owner, admin, staff and superuser accounts, then sends usage counts every 6 hours. It never sends traces, prompts, outputs, datasets or keys. Turn it off with `./bin/install --no-telemetry` or `FUTURE_AGI_TELEMETRY_DISABLED=true`; one minimal registration without email addresses still goes out, so block `api.futureagi.com` to send nothing. The only other connections made by default are litellm's price-list download, the model server's downloads from the Hugging Face Hub, and the assets users' browsers load from public hosts. An Enterprise licence adds its own heartbeat. Every other outside service stays off until you give it a key.
+After the first account is created, the install registers with `api.futureagi.com`, without a `users` field or email addresses by default, then sends usage counts every 6 hours and best-effort background event batches for lifecycle and sign-in activity. Sharing owner, admin, staff and superuser email addresses requires `FUTURE_AGI_TELEMETRY_SHARE_ADMIN_EMAILS=true` (Helm: `config.telemetryShareAdminEmails=true`). It never sends traces, prompts, outputs, datasets or keys. Turn it off with `./bin/install --no-telemetry` or `FUTURE_AGI_TELEMETRY_DISABLED=true`; registration, heartbeats and event sends stop, including retries and buffered replay. The only other connections made by default are litellm's price-list download, the model server's downloads from the Hugging Face Hub, and the assets users' browsers load from public hosts. An Enterprise licence adds its own heartbeat. Every other outside service stays off until you give it a key.
## At a glance
@@ -39,15 +39,17 @@ Model providers, SSO providers and integrations you add in the product are calle
Future AGI uses deployment telemetry to count the installs that are running, which versions they run, and roughly how much they are used. It never includes anything you send through the product.
-**Registration**, once per install. The app tries to register right after the first account is created, whether through the sign-up page or `python manage.py create_user`, which `./bin/install` runs. It runs on a background thread, so the sign-up page returns without waiting for it. `create_user` waits for it before exiting, which takes a few seconds, longer if the telemetry URL does not answer: three tries, each bounded by `FUTURE_AGI_TELEMETRY_TIMEOUT_SECONDS`, which `./bin/install` sets to 2 for that command unless you set it. If that attempt fails, the scheduled job below retries it. Registration is sent again only if you turn telemetry off or back on, or if the install loses the signing secret it received the first time.
+**Registration**, once per install. The app tries to register right after the first account is created, whether through the sign-up page or `python manage.py create_user`, which `./bin/install` runs. It runs on a background thread, so the sign-up page returns without waiting for it. `create_user` waits for it before exiting, which takes a few seconds, longer if the telemetry URL does not answer: three tries, each bounded by `FUTURE_AGI_TELEMETRY_TIMEOUT_SECONDS`, which `./bin/install` sets to 2 for that command unless you set it. If that attempt fails, the scheduled job below retries it. Turning telemetry off does not send a registration or discard an existing full registration. Losing the signing secret requires registration again before signed telemetry can resume.
**Heartbeats**, every `FUTURE_AGI_TELEMETRY_INTERVAL_HOURS` (6 by default), started by a recurring Temporal schedule with up to `FUTURE_AGI_TELEMETRY_JITTER_SECONDS` (30 minutes by default) of random delay. Each heartbeat covers the previous fixed UTC window, for example 06:00 to 12:00, and holds counts only, across the whole install.
-Nothing is sent at startup or on login.
+**Lifecycle and sign-in events**, separately from the aggregate heartbeats. Registration queues `deployment_booted` and `telemetry_registered`; account creation can queue `user_created`, and successful password sign-in can queue `user_logged_in`. These events are stored locally and trigger a best-effort background flush. Sending requires enabled self-hosted telemetry and a registration signing secret: it is not a synchronous upload on every login, and it does not wait for the six-hour heartbeat.
### Registration
-A registration payload, and what each field means:
+By default, `FUTURE_AGI_TELEMETRY_SHARE_ADMIN_EMAILS=false`: registration contains `schema_version`, `instance_id`, `version`, `deployment_type`, `timestamp` and `telemetry_disabled`, but no `users` field. Email addresses and domains are not read for this payload. On Helm, use `config.telemetryShareAdminEmails` for the same opt-in.
+
+The example below is **opt-in only**, with `FUTURE_AGI_TELEMETRY_SHARE_ADMIN_EMAILS=true` (Helm: `config.telemetryShareAdminEmails=true`). Changing this setting on an already registered, telemetry-enabled install triggers re-registration with or without `users`.
{/* generated:begin telemetry-registration: from future-agi futureagi/tfc/deployment_telemetry/wire_reference.toml; edit there, not here */}
@@ -75,8 +77,8 @@ A registration payload, and what each field means:
| `version` | The first of `FUTURE_AGI_VERSION`, `SERVICE_VERSION` and `GIT_SHA` that names a release (`unknown` and `latest` are skipped), else `unknown`. `FUTURE_AGI_VERSION` is the image tag in `.env`; `SERVICE_VERSION` and `GIT_SHA` come from the image. |
| `deployment_type` | `FUTURE_AGI_DEPLOYMENT_TYPE` if set (`docker` in both compose files); otherwise `kubernetes`, `docker` or `bare_metal`, detected. |
| `timestamp` | When the payload was built (UTC). |
-| `telemetry_disabled` | `false` here. `true` on the opt-out registration. |
-| `users` | One entry for each active account that is an organization owner or admin, or a Django staff or superuser account (for example one made with `manage.py createsuperuser`), at the time of registration, most recent login first, at most 500. On a new install that is normally only the first account. |
+| `telemetry_disabled` | `false`. Disabled installs do not send a registration. |
+| `users` | Omitted unless email sharing is explicitly enabled. When opted in, one entry for each active account that is an organization owner or admin, or a Django staff or superuser account (for example one made with `manage.py createsuperuser`), at the time of registration, most recent login first, at most 500. On a new install that is normally only the first account. |
| `users[].email` | The account's email address, lowercased. |
| `users[].domain` | The part of `email` after the `@`. |
@@ -142,10 +144,24 @@ unavailable), so a failure is never reported as zero.
{/* generated:end telemetry-heartbeat */}
+### Lifecycle and sign-in events
+
+Event batches go to `/telemetry/events/`. The batch contains `schema_version`, `instance_id` and an `events` array of up to 50 events. Unlike aggregate heartbeats, each event describes activity by an actor:
+
+| Event fields | Meaning |
+|---|---|
+| `event_id`, `event_name`, `occurred_at` | A random event UUID, the event name and its UTC timestamp. |
+| `instance_id`, `version`, `deployment_type` | The install and release that produced the event. |
+| `actor_type`, `actor_id`, `source` | The actor category, an install-scoped pseudonymous identifier for identified actors, and a source such as `web` or `system`. The source user or API-key identifier is hashed before enqueueing; it is not an email address or credential. |
+| `organization_id`, `workspace_id` | Organization and workspace identifiers when supplied, otherwise `null`. These are separate from the pseudonymous actor identifier. The password-login event includes the organization identifier. |
+| `operation_id`, `outcome`, `duration_ms`, `properties` | Operation metadata when supplied. Properties are restricted to `method`, `route`, `status_code`, `auth_method`, `feature`, `deployment_type` and `version`. |
+
+`FUTURE_AGI_TELEMETRY_DISABLED=true` suppresses both event enqueueing and sending, including buffered replay, as well as registration and aggregate heartbeats.
+
### How it is sent
-- Each payload is an HTTPS `POST` of JSON to `FUTURE_AGI_TELEMETRY_URL` (`https://api.futureagi.com` by default) plus `/telemetry/register/` or `/telemetry/heartbeat/`, with a `FUTURE_AGI_TELEMETRY_TIMEOUT_SECONDS` timeout (5 by default), tried up to 3 times. A payload over 512 KiB is not sent.
-- The registration response carries a per-install secret. Heartbeats are signed with it (HMAC-SHA256, in the `X-FAGI-Telemetry-Signature` header), so the receiver can reject forged ones.
+- Each payload is an HTTPS `POST` of JSON to `FUTURE_AGI_TELEMETRY_URL` (`https://api.futureagi.com` by default) plus `/telemetry/register/`, `/telemetry/heartbeat/` or `/telemetry/events/`, with a `FUTURE_AGI_TELEMETRY_TIMEOUT_SECONDS` timeout (5 by default), tried up to 3 times. A payload over 512 KiB is not sent.
+- The registration response carries a per-install secret. Heartbeats and event batches are signed with it (HMAC-SHA256, in the `X-FAGI-Telemetry-Signature` header), so the receiver can reject forged ones.
- A heartbeat that cannot be delivered is kept as a JSON file in `FUTURE_AGI_TELEMETRY_BUFFER_DIR`, a directory only the app's user can read, and sent on a later run, oldest first. Files older than 30 days are deleted.
- Telemetry failures are logged as warnings and never stop the app, a sign-up or a login.
@@ -157,18 +173,18 @@ Prompts, model outputs, trace or span contents and attributes, dataset rows, eva
## Turn it off
-- **At install:** `./bin/install --no-telemetry` (Windows: `.\bin\install.ps1 -NoTelemetry`) writes `FUTURE_AGI_TELEMETRY_DISABLED=true` to `.env` before anything starts, so even the first account's registration is the minimal one. The installer prints what telemetry sends before it asks for the first account's email.
+- **At install:** `./bin/install --no-telemetry` (Windows: `.\bin\install.ps1 -NoTelemetry`) writes `FUTURE_AGI_TELEMETRY_DISABLED=true` to `.env` before anything starts, so even the first account's registration is suppressed. The installer prints what telemetry sends before it asks for the first account's email.
- **Standalone or Distributed, after install:** uncomment or add `FUTURE_AGI_TELEMETRY_DISABLED=true` in `.env` (`1`, `yes` and `on` also work), then recreate the containers. Both compose files pass the variable to every backend process.
```bash
docker compose up -d
```
-- **Helm:** `config.telemetry=false` in your values sets the variable for the backend, the workers and the bootstrap job. `global.airgap=true` turns telemetry off too, along with the licence heartbeat (unless `license.heartbeat` is `true`), the price-list download and model serving's downloads. Offline, the one opt-out registration below then fails the same harmless way and is retried. See [Configure](/docs/self-hosting/helm#configure).
+- **Helm:** `config.telemetry=false` in your values sets the variable for the backend, the workers and the bootstrap job. `global.airgap=true` turns telemetry off too, along with the licence heartbeat (unless `license.heartbeat` is `true`), the price-list download and model serving's downloads. See [Configure](/docs/self-hosting/helm#configure).
-With telemetry off, no email addresses and no heartbeats are sent, and buffered heartbeats are deleted. One opt-out registration is still sent, so Future AGI can count installs that opted out. It contains only `schema_version`, `instance_id`, `version`, `deployment_type`, `timestamp` and `telemetry_disabled: true`, and is sent again only if you turn telemetry back on and off. Until one gets through, it is attempted at every telemetry run.
+With telemetry off, no registration, heartbeats or events are sent. This includes subsequent retries and replay of buffered payloads. The disabled telemetry cycle clears the heartbeat and event outboxes; it does not send an opt-out registration. No firewall block is needed to suppress deployment telemetry.
-To make no connection to Future AGI at all, also block outbound traffic to `api.futureagi.com` (or the host in your `FUTURE_AGI_TELEMETRY_URL`) at your network. The failed attempts are logged as warnings and change nothing else.
+Enterprise licence activation and licence heartbeats are separate from deployment telemetry; see [Other outbound connections](#other-outbound-connections).
## Settings
@@ -178,8 +194,9 @@ Every setting of deployment telemetry, with its default in each setup (S Standal
| Key | Default | Setups | What it does |
|---|---|---|---|
-| `FUTURE_AGI_TELEMETRY_DISABLED` | `false` | S D H | `true` opts out (`1`, `yes` and `on` also work): no email addresses and no heartbeats are sent, and buffered heartbeats are deleted; one minimal registration remains ([what it contains](#turn-it-off)). `./bin/install --no-telemetry` (`.\bin\install.ps1 -NoTelemetry`) writes it before anything starts. Helm: `config.telemetry=false`. |
-| `FUTURE_AGI_TELEMETRY_URL` | `https://api.futureagi.com` | S D H | Where registrations and heartbeats go. Change it only to test against another receiver. |
+| `FUTURE_AGI_TELEMETRY_DISABLED` | `false` | S D H | `true` opts out (`1`, `yes` and `on` also work): no registration, heartbeats or events are sent, including retries and buffered replay; the disabled telemetry cycle clears both outboxes. See [Turn it off](#turn-it-off). `./bin/install --no-telemetry` (`.\bin\install.ps1 -NoTelemetry`) writes it before anything starts. Helm: `config.telemetry=false`. |
+| `FUTURE_AGI_TELEMETRY_SHARE_ADMIN_EMAILS` | `false` (app default and Compose fallback) | S D H | Opt in to owner, admin, staff and superuser email addresses and domains in registration. When false, the `users` field is omitted. Helm: `config.telemetryShareAdminEmails`. Changing it while telemetry is enabled triggers re-registration. |
+| `FUTURE_AGI_TELEMETRY_URL` | `https://api.futureagi.com` | S D H | Where registrations, heartbeats and event batches go. Change it only to test against another receiver. |
| `FUTURE_AGI_TELEMETRY_INTERVAL_HOURS` | `6` | S D H | Heartbeat interval: 1, 2, 3, 4, 6, 8, 12 or 24. Other values fall back to 6. |
| `FUTURE_AGI_TELEMETRY_JITTER_SECONDS` | `1800` | S D H | Maximum random delay added to each scheduled run, so installs do not all send at once. |
| `FUTURE_AGI_TELEMETRY_TIMEOUT_SECONDS` | `5` | S D H | Timeout of each request. `./bin/install` runs its `create_user` with `2` unless this is set, so an unreachable telemetry URL delays the first account only briefly. |
@@ -271,7 +288,7 @@ Analytics, error reports and ad pixels send nothing unless the UI was built with
To run an install with no outbound access, or to keep it from trying to reach outside hosts:
-- Set `FUTURE_AGI_TELEMETRY_DISABLED=true`, and block `api.futureagi.com` so the one minimal registration fails harmlessly.
+- Set `FUTURE_AGI_TELEMETRY_DISABLED=true` to suppress deployment registration, heartbeats and events, including retries and buffered replay.
- Set `LITELLM_LOCAL_MODEL_COST_MAP=True` so no process downloads the price list.
- Leave the third-party keys above empty.
- With an Enterprise licence, decide on the licence heartbeat: `FUTURE_AGI_ENTERPRISE_HEARTBEAT_DISABLED=true` stops it.
diff --git a/src/pages/docs/self-hosting/helm.mdx b/src/pages/docs/self-hosting/helm.mdx
index 5453bce6a..e7d817cb8 100644
--- a/src/pages/docs/self-hosting/helm.mdx
+++ b/src/pages/docs/self-hosting/helm.mdx
@@ -278,7 +278,7 @@ Set everything in `my-values.yaml` and apply it with `helm upgrade`, passing the
- **Code sandbox.** `codeExecutor.enabled=true` runs the nsjail sandbox for custom code evals. It needs privileged pods, which `baseline` and `restricted` Pod Security namespaces and some managed clusters refuse. With the sandbox off, custom code evals are refused. `codeExecutor.localFallback=true` runs them inside the worker pods instead, next to the platform's secrets: set it only when everyone who can create evals is trusted.
- **Scaling and placement.** `worker.queues` gives a busy queue its own Deployment, `autoscaling` on each component adds a HorizontalPodAutoscaler (it needs metrics-server), PodDisruptionBudgets are on by default, and `topologySpread.preset` (`soft` by default, `hard` or `none`) spreads replicas over zones and nodes. See [Sizing presets](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#sizing-presets).
- **Network and pod security.** `networkPolicy.enabled=true` limits the datastores, the gateway, serving and the sandbox to this release's pods, and blocks the sandbox from private, link-local and metadata addresses. With the code sandbox off, every pod meets the Pod Security `restricted` level. See [Security](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#security).
-- **Telemetry and air-gap.** Deployment telemetry is on by default: a one-time registration with the owner, admin, staff and superuser emails, then usage counts every 6 hours, never traces, prompts or other content. `config.telemetry=false` stops the emails and heartbeats, but one minimal registration without emails is still attempted: block `api.futureagi.com` to send nothing. `global.airgap=true` turns telemetry off the same way, along with the license heartbeat (unless `license.heartbeat` is `true`), the price-list download and model serving's downloads. See [Telemetry](/docs/self-hosting/configuration/telemetry).
+- **Telemetry and air-gap.** Deployment telemetry is on by default: registration, then usage counts every 6 hours, never traces, prompts or other content. With `config.telemetryShareAdminEmails=false`, registration omits the `users` field and email addresses; set it to `true` only to opt in to sharing owner, admin, staff and superuser emails. `config.telemetry=false` stops registration, heartbeats and event sends, including retries and buffered replay. `global.airgap=true` turns telemetry off the same way, along with the license heartbeat (unless `license.heartbeat` is `true`), the price-list download and model serving's downloads. See [Telemetry](/docs/self-hosting/configuration/telemetry).
- **Email.** `config.email.mailgunSenderDomain` with `secrets.mailgunApiKey`, or `config.email.existingSecret` holding `MAILGUN_API_KEY`. Until both are set, email stays off and invites return a link to share yourself.
- **Enterprise.** `edition=ee` with `license.existingSecret` (a Secret holding `EE_LICENSE_KEY`) turns on the Enterprise features in the same public image. `examples/enterprise.yaml` adds SSO (`auth.google`, `auth.github`, `auth.microsoft`), a corporate proxy (`global.proxy`; the LLM gateway does not use it yet), a CA bundle (`global.caBundle`) and a private registry mirror (`global.imageRegistry`). See [Enterprise](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#enterprise).
- **Private provider URLs.** `agentccGateway.allowPrivateProviderURLs=true` lets organization providers use base URLs on private networks, such as an Ollama or vLLM server in the cluster or on your network. It sets `AGENTCC_ALLOW_PRIVATE_PROVIDER_URLS` on the backend and the gateway. Loopback, link-local and cloud metadata addresses stay refused. Set it only when everyone who can add a provider is trusted: it puts every in-cluster service in reach of a provider URL. See [Private and local provider URLs](/docs/command-center/features/self-hosted-models#private-and-local-provider-urls).
diff --git a/src/pages/docs/self-hosting/production/checklist.mdx b/src/pages/docs/self-hosting/production/checklist.mdx
index 17d4a8d76..aecb59ff6 100644
--- a/src/pages/docs/self-hosting/production/checklist.mdx
+++ b/src/pages/docs/self-hosting/production/checklist.mdx
@@ -13,7 +13,7 @@ Apply a `.env` change with `docker compose up -d`, which recreates every contain
## Decide on telemetry
-Decide before the first account exists. A self-hosted install sends deployment telemetry to Future AGI by default. Right after the first account is created, which `./bin/install` does, it sends a registration with the email addresses of the owner, admin, staff and superuser accounts, then usage counts.
+Decide before the first account exists. A self-hosted install sends deployment telemetry to Future AGI by default. Right after the first account is created, which `./bin/install` does, it sends a registration without a `users` field or email addresses by default, then usage counts. Keep `FUTURE_AGI_TELEMETRY_SHARE_ADMIN_EMAILS=false` (Helm: `config.telemetryShareAdminEmails=false`) to omit them; set it to `true` only to opt in to sharing owner, admin, staff and superuser emails.
- To opt out from the start, install with `./bin/install --no-telemetry`, or with the production overlay, set `FUTURE_AGI_TELEMETRY_DISABLED=true` in `deploy/.env.production` before you start the stack.
- Setting `FUTURE_AGI_TELEMETRY_DISABLED=true` later stops the heartbeats, but the registration already sent cannot be taken back.