From 6ba7b8a09db9f8b243536bfdffb855648b22843b Mon Sep 17 00:00:00 2001 From: nik13 <8347993+nik13@users.noreply.github.com> Date: Mon, 9 Nov 2026 10:49:28 -0800 Subject: [PATCH] docs: correct first-run, SDK, gateway and API reference instructions Fixes instructions that did not match how the product currently works. Each change was checked against the current product and SDK source. - First trace: create a key pair with Add API Key and copy both values from the one-time dialog. The keys list cannot reveal saved secrets. JS/TS examples now instrument OpenAI before the client is created. - Evaluations: install the current SDK and eval packages. Built-in TypeScript examples import the published package. Code-eval, cloud-eval payload and CI examples are corrected. - Gateway and Protect: gateway keys are created under AI Gateway > API keys. Key-creation requests, guardrail setup, provider formats, rate limits, budget alerts and request headers are corrected. - Tracing SDK: the gRPC/HTTP transport extra, register() options, the TypeScript sampler, environment variables by language, and shutdown/flush guidance. - Datasets, prompts, experiments and annotations: required fields, current menus and wizard steps, queue labels, and SDK examples. - Self-hosting and API reference: telemetry behaviour, the gateway sync interval, dataset-eval column IDs, run-prompt option values, and voice capabilities. - Integrations and cookbooks: LangChain, Vercel and OpenTelemetry setup, and the RAG cookbook evaluation calls. There are no product code changes; documentation only. --- src/pages/docs/admin-settings/api-keys.mdx | 12 +- src/pages/docs/admin-settings/index.mdx | 7 +- .../agent-playground/guides/create-agent.mdx | 8 +- .../docs/annotations/reference/sdk-api.mdx | 26 ++- .../api/dataset-evals/add-dataset-eval.mdx | 6 +- .../api/dataset-evals/edit-and-run-eval.mdx | 6 +- src/pages/docs/api/datasets/add-as-new.mdx | 6 +- .../datasets/run-prompt/get-model-voices.mdx | 4 +- .../retrieve-run-prompt-options.mdx | 18 +- .../docs/billing/guides/control-spend.mdx | 4 +- .../command-center/admin/organizations.mdx | 23 +-- src/pages/docs/command-center/api/chat.mdx | 2 +- src/pages/docs/command-center/api/headers.mdx | 5 +- .../command-center/concepts/virtual-keys.mdx | 59 ++---- .../command-center/features/cost-tracking.mdx | 53 ++---- .../command-center/features/guardrails.mdx | 54 +++--- .../command-center/features/providers.mdx | 2 + .../command-center/features/rate-limiting.mdx | 9 +- src/pages/docs/cookbook/rag-langchain.mdx | 84 +++++---- src/pages/docs/dataset/guides/add-rows.mdx | 17 +- .../docs/dataset/guides/create-a-dataset.mdx | 56 +++--- .../docs/dataset/guides/run-an-experiment.mdx | 23 ++- .../builtin/caption-hallucination.mdx | 19 +- .../builtin/detect-hallucination.mdx | 19 +- src/pages/docs/evaluation/builtin/index.mdx | 10 +- src/pages/docs/evaluation/builtin/pii.mdx | 18 +- .../builtin/translation-accuracy.mdx | 19 +- src/pages/docs/evaluation/guides/cicd.mdx | 80 ++++++--- .../docs/evaluation/guides/custom-evals.mdx | 8 +- .../docs/evaluation/guides/guardrails.mdx | 2 +- .../docs/evaluation/reference/sdk-api.mdx | 12 +- .../falcon-ai/guides/chat-with-falcon-ai.mdx | 2 +- src/pages/docs/falcon-ai/index.mdx | 2 +- src/pages/docs/faq.mdx | 4 +- .../get-started/connect-no-code-agents.mdx | 50 +++--- .../docs/get-started/migrate/langfuse.mdx | 10 +- .../route-your-first-llm-request.mdx | 10 +- .../get-started/send-your-first-trace.mdx | 18 +- .../integrations/coding-agents/gemini-cli.mdx | 2 +- .../integrations/frameworks/tanstack-ai.mdx | 8 +- .../traceai/bedrock-agentcore.mdx | 2 +- .../docs/integrations/traceai/go/a2a.mdx | 4 +- .../docs/integrations/traceai/go/eino.mdx | 4 +- .../docs/integrations/traceai/go/gemini.mdx | 4 +- .../docs/integrations/traceai/go/openai.mdx | 4 +- .../docs/integrations/traceai/gradio.mdx | 2 +- .../guides/manage-with-the-sdk.mdx | 2 +- .../docs/observe/features/quickstart.mdx | 2 +- src/pages/docs/observe/quickstart.mdx | 12 +- .../docs/observe/reference/export-formats.mdx | 22 +-- src/pages/docs/prompt/reference/sdk-api.mdx | 2 +- .../concepts/understanding-protect.mdx | 28 +-- .../guides/run-protect-from-the-sdk.mdx | 4 +- .../docs/protect/guides/test-a-guardrail.mdx | 12 +- .../protect/guides/turn-on-a-guardrail.mdx | 51 ++---- .../protect/reference/guardrail-checks.mdx | 124 +++++++------ .../guardrail-changes-not-taking-effect.mdx | 24 +-- .../guardrail-fires-on-the-wrong-requests.mdx | 39 ++-- .../quickstart/generate-synthetic-data.mdx | 1 + .../running-evals-in-simulation.mdx | 15 +- .../docs/quickstart/setup-observability.mdx | 12 +- src/pages/docs/roles-and-permissions.mdx | 6 +- .../docs/sdk/annotation-queues/queues.mdx | 32 +++- src/pages/docs/sdk/evals/cloud-evals.mdx | 68 +++---- src/pages/docs/sdk/evals/evaluate.mdx | 19 +- src/pages/docs/sdk/evals/llm-judge.mdx | 6 +- src/pages/docs/sdk/evals/local.mdx | 26 ++- src/pages/docs/sdk/evals/metrics/json.mdx | 4 +- src/pages/docs/sdk/evals/metrics/string.mdx | 14 +- src/pages/docs/sdk/index.mdx | 25 ++- src/pages/docs/sdk/protect.mdx | 42 +++-- src/pages/docs/sdk/testcase.mdx | 170 +++--------------- .../sdk/tracing/environment-variables.mdx | 80 ++++++--- .../instrument-with-traceai-helpers.mdx | 20 +-- src/pages/docs/sdk/tracing/register.mdx | 55 ++++-- .../docs/sdk/tracing/set-session-user-id.mdx | 161 ++++++++--------- src/pages/docs/self-hosting.mdx | 2 +- .../configuration/environment.mdx | 4 +- .../self-hosting/configuration/reference.mdx | 8 +- .../self-hosting/configuration/telemetry.mdx | 49 +++-- src/pages/docs/self-hosting/helm.mdx | 2 +- .../self-hosting/production/checklist.mdx | 2 +- 82 files changed, 967 insertions(+), 975 deletions(-) diff --git a/src/pages/docs/admin-settings/api-keys.mdx b/src/pages/docs/admin-settings/api-keys.mdx index ca5912ad3..87f568a37 100644 --- a/src/pages/docs/admin-settings/api-keys.mdx +++ b/src/pages/docs/admin-settings/api-keys.mdx @@ -7,19 +7,19 @@ description: "Create, copy, and rotate FI_API_KEY and FI_SECRET_KEY credentials API keys authenticate your application with Future AGI. Each key pair consists of an API Key (`FI_API_KEY`) and a Secret Key (`FI_SECRET_KEY`). You need these to use the Python SDK, TypeScript SDK, or REST API. -Access: **Owner** only. +Access: active organization members (**Owner**, **Admin**, **Member**, and **Viewer**) can create their own personal keys. Sign in interactively with a password or OAuth; API-key authentication and SAML-scoped sessions cannot issue personal keys. Managing other members' keys is restricted to **Owners** and **Admins**. ## How to - - Navigate to **Settings > API Keys** at [https://app.futureagi.com/dashboard/keys](https://app.futureagi.com/dashboard/keys). + + Open **Keys** under **Build** in the main sidebar, or go directly to [https://app.futureagi.com/dashboard/keys](https://app.futureagi.com/dashboard/keys). Click **Add API Key**. Enter a name for the key. - Copy both the API Key and Secret Key. The Secret Key is only shown once at creation time. + In **Save your personal key**, copy and securely save both the API Key and Secret Key before closing. Both values are shown only once, in this creation result; you cannot recover them from the key table later. Set them as environment variables. @@ -41,8 +41,8 @@ Access: **Owner** only. | Action | How | |--------|-----| -| View keys | API Key is visible in the table. Secret Key is masked. | -| Copy a key | Click the copy icon next to the key. | +| View keys | Both API Key and Secret Key are masked in existing rows. The table cannot reveal either value. | +| Copy a key pair | Copy both values from the one-time creation result, before closing it. Existing rows do not offer copying of usable credentials. | | Delete a key | Click the delete icon. This is permanent and cannot be undone. | | Rotate keys | Delete the old key and create a new one. Update your application with the new credentials. | diff --git a/src/pages/docs/admin-settings/index.mdx b/src/pages/docs/admin-settings/index.mdx index 97dbe2984..2c77ba52f 100644 --- a/src/pages/docs/admin-settings/index.mdx +++ b/src/pages/docs/admin-settings/index.mdx @@ -5,7 +5,7 @@ description: "Manage your Future AGI account: profile, organization, team member ## About -The Settings page is where you manage everything about your Future AGI account: your profile and security, organization configuration, team members, workspaces, API keys, AI provider connections, external integrations, and usage tracking. +The Settings page is where you manage your Future AGI account: your profile and security, organization configuration, team members, workspaces, AI provider connections, external integrations, and usage tracking. Personal API keys are under **Keys** in the main sidebar's **Build** section. Access to different settings depends on your role. See [Roles & Permissions](/docs/roles-and-permissions) for details. @@ -44,9 +44,12 @@ Access to different settings depends on your role. See [Roles & Permissions](/do ## Access by Role +Personal key creation requires active organization membership and an interactive password or OAuth sign-in, not API-key authentication or a SAML-scoped session. See [API Keys](/docs/admin-settings/api-keys). + | Section | Owner | Admin | Member | Viewer | |---|---|---|---|---| -| API Keys | Yes | No | No | No | +| Create own personal API keys | Yes | Yes | Yes | Yes | +| Manage other members' API keys | Yes | Yes | No | No | | Profile & Security | Yes | Yes | Yes | Yes | | Organization Settings | Yes | Yes | No | No | | User Management | Yes | Yes | No | No | diff --git a/src/pages/docs/agent-playground/guides/create-agent.mdx b/src/pages/docs/agent-playground/guides/create-agent.mdx index 401828ba9..0ff653b6f 100644 --- a/src/pages/docs/agent-playground/guides/create-agent.mdx +++ b/src/pages/docs/agent-playground/guides/create-agent.mdx @@ -3,7 +3,7 @@ title: "Create an agent" description: "Create a new agent from scratch or a template" --- -Every agent in Agent Playground starts in the same place: a list you open from the sidebar, and a button that drops you onto a blank canvas. This guide gets a new agent onto that canvas and back out again if you ever need to delete it, nothing more. +Every agent in Agent Playground starts in the same place: **Agents** in the sidebar, then the creation menu's **Playground** option. This guide gets a new agent onto that canvas and back out again if you ever need to delete it, nothing more. The example built up across this section is an agent called **Invoice Triage**. Create yours under that name so later guides line up with what's on your screen. @@ -11,15 +11,15 @@ The example built up across this section is an agent called **Invoice Triage**. ## Open the list and create an agent -Click **Agents** in the sidebar to see every agent in the workspace. Clicking anywhere on a row opens that agent, so there's no separate open button to look for. Once there are more agents than fit on one page, use the Search box above the table to find one by name, and the pagination controls beneath it to page through the rest. If the workspace has no agents yet, the list is replaced by an empty state instead: a **Create your first agent** heading, the line "Break down complex tasks into sequential steps that build upon each other." underneath, and a **Start creating** button in place of **Create Agent**. It opens the same empty canvas. +Click **Agents** in the sidebar. An initial visit opens **Connected agents**, which lists agents connected from outside the Playground. Select the **Agent Playground** tab to find your saved graphs. Click a row to open one; use Search and pagination to find it in a longer list. -Click **Create Agent** in the top right to start a new one. You land straight on an empty builder canvas for a brand-new agent. That agent already exists as an empty draft the moment the canvas opens. +To start a new graph, open **Create agents** in the top right, then choose **Playground**. Clicking the creation button alone only opens the menu. Choosing **Playground** creates the draft and opens its empty builder canvas. The Agent Playground list with five agents in the table, showing node count, creator, collaborators, and created and updated timestamps, with Create Agent in the top right *An agent you never renamed keeps its generated timestamp name, which is what the **No. of nodes** column is for: it's the only thing in the row that tells you how much is in there* -If **Create Agent** or, later, **Delete** looks greyed out, hover it: a tooltip explains why, either `You don't have permission to create agents.` or `You don't have permission to delete agents.` +The **Playground** menu option is disabled when you cannot create Playground agents. If **Delete** later looks greyed out, hover it for the permission tooltip. ## Name your agent diff --git a/src/pages/docs/annotations/reference/sdk-api.mdx b/src/pages/docs/annotations/reference/sdk-api.mdx index 748df193f..2a42405db 100644 --- a/src/pages/docs/annotations/reference/sdk-api.mdx +++ b/src/pages/docs/annotations/reference/sdk-api.mdx @@ -33,19 +33,39 @@ client = AnnotationQueue( ) ``` -You can also set `FI_API_KEY` and `FI_SECRET_KEY` as environment variables and drop both arguments; the client picks them up automatically. Find both under **Settings → API Keys** in the platform. +You can also set `FI_API_KEY` and `FI_SECRET_KEY` as environment variables and drop both arguments; the client picks them up automatically. Create a pair from **Keys** under **Build** in the main sidebar ([API Keys](/docs/admin-settings/api-keys)). ## An end-to-end example -Creating `Support quality review`, pushing two traces into it, checking progress, then pulling the completed results back out: +Create a label and a queue, add two traces, activate the queue, then submit ratings and export the results. Replace the trace IDs with traces in your organization. The ratings below are examples; review each trace before submitting a real rating. ```python -queue = client.create(name="Support quality review", instructions="Rate response quality 1-5") +label = client.create_label( + name="Support quality", + type="star", + settings={"no_of_stars": 5}, +) +queue = client.create( + name="Support quality review", + label_ids=[label.id], + instructions="Rate response quality 1-5", + auto_assign=True, + annotations_required=1, + requires_review=False, +) client.add_items(queue.id, items=[ {"source_type": "trace", "source_id": "trace_abc123"}, {"source_type": "trace", "source_id": "trace_def456"}, ]) +client.activate(queue.id) + +for item in client.list_items(queue.id): + client.submit_annotations( + queue.id, + item.id, + annotations=[{"label_id": label.id, "value": 4}], + ) progress = client.get_progress(queue.id) print(f"{progress.completed} of {progress.total} done") diff --git a/src/pages/docs/api/dataset-evals/add-dataset-eval.mdx b/src/pages/docs/api/dataset-evals/add-dataset-eval.mdx index 5eaa08cbb..8bb7e2051 100644 --- a/src/pages/docs/api/dataset-evals/add-dataset-eval.mdx +++ b/src/pages/docs/api/dataset-evals/add-dataset-eval.mdx @@ -17,8 +17,8 @@ description: "Add an evaluation to a dataset by selecting a template and configu config: {}, params: {}, mapping: { - input: "question_column", - output: "response_column" + input: "11111111-1111-4111-8111-111111111111", + output: "22222222-2222-4222-8222-222222222222" }, reason_column: true }, @@ -67,7 +67,7 @@ description: "Add an evaluation to a dataset by selecting a template and configu - Mapping of eval template variable keys to dataset column names. + Map each template input key (such as `input` or `output`) to a dataset column UUID, not a column name. Get column IDs from `GET /model-hub/dataset/columns/{dataset_id}/` and replace the example UUIDs with the IDs for your input and output columns. diff --git a/src/pages/docs/api/dataset-evals/edit-and-run-eval.mdx b/src/pages/docs/api/dataset-evals/edit-and-run-eval.mdx index a887f7fed..687d84fd9 100644 --- a/src/pages/docs/api/dataset-evals/edit-and-run-eval.mdx +++ b/src/pages/docs/api/dataset-evals/edit-and-run-eval.mdx @@ -16,8 +16,8 @@ description: "Update an evaluation's config and key mapping, then optionally re- config: {}, params: {}, mapping: { - input: "question_column", - output: "response_column" + input: "11111111-1111-4111-8111-111111111111", + output: "22222222-2222-4222-8222-222222222222" }, reason_column: true }, @@ -66,7 +66,7 @@ description: "Update an evaluation's config and key mapping, then optionally re- - Mapping of eval template variable keys to dataset column names. + Map each template input key (such as `input` or `output`) to a dataset column UUID, not a column name. Get column IDs from `GET /model-hub/dataset/columns/{dataset_id}/` and replace the example UUIDs with the IDs for your input and output columns. diff --git a/src/pages/docs/api/datasets/add-as-new.mdx b/src/pages/docs/api/datasets/add-as-new.mdx index 94e2d6ecc..2fc4033cd 100644 --- a/src/pages/docs/api/datasets/add-as-new.mdx +++ b/src/pages/docs/api/datasets/add-as-new.mdx @@ -11,8 +11,8 @@ description: "Create a new dataset from selected columns of an existing dataset dataset_id: "a1b2c3d4-e5f6-7890-abcd-ef1234567890", name: "My New Dataset", columns: { - "col-uuid-1": "input_text", - "col-uuid-2": "expected_output" + "550e8400-e29b-41d4-a716-446655440011": "input_text", + "550e8400-e29b-41d4-a716-446655440012": "expected_output" } }} responseExample={{ @@ -38,7 +38,7 @@ description: "Create a new dataset from selected columns of an existing dataset Name for the new dataset. - Mapping of source column UUIDs to new column names. + A nonempty mapping from existing column UUIDs in the source dataset or experiment to new column names. Include at least one entry and use distinct new names. Keys must be bare UUIDs, without a `col-` prefix; replace the example UUIDs with your source column IDs. Omitting `columns` or sending `{}` returns `400 Bad Request`. diff --git a/src/pages/docs/api/datasets/run-prompt/get-model-voices.mdx b/src/pages/docs/api/datasets/run-prompt/get-model-voices.mdx index 901ef06ad..b1c559aae 100644 --- a/src/pages/docs/api/datasets/run-prompt/get-model-voices.mdx +++ b/src/pages/docs/api/datasets/run-prompt/get-model-voices.mdx @@ -15,7 +15,7 @@ description: "Retrieve the available voice options for a specific model's audio data: { model_name: "gpt-4o-audio-preview", provider: "openai", - custom_voice_supported: true, + custom_voice_supported: false, supported_voices: [ { id: "alloy", name: "Alloy", type: "system" }, { id: "echo", name: "Echo", type: "system" }, @@ -54,7 +54,7 @@ description: "Retrieve the available voice options for a specific model's audio Model provider (e.g., `openai`, `elevenlabs`). - Whether the model supports custom voices. + `true` only when `provider` is `elevenlabs` or `cartesia`; `false` for `openai`. This capability flag is independent of any stored custom voice records included in `supported_voices`. Available voice objects. diff --git a/src/pages/docs/api/datasets/run-prompt/retrieve-run-prompt-options.mdx b/src/pages/docs/api/datasets/run-prompt/retrieve-run-prompt-options.mdx index 0355d081a..46b572fbc 100644 --- a/src/pages/docs/api/datasets/run-prompt/retrieve-run-prompt-options.mdx +++ b/src/pages/docs/api/datasets/run-prompt/retrieve-run-prompt-options.mdx @@ -34,14 +34,16 @@ description: "Retrieve available models, tools, output formats, and tool choices } ], output_formats: [ - { value: "string", label: "String" }, - { value: "audio", label: "Audio" }, - { value: "json", label: "JSON" } + { value: "array", label: "array" }, + { value: "string", label: "string" }, + { value: "number", label: "number" }, + { value: "object", label: "object" }, + { value: "audio", label: "audio" }, + { value: "image", label: "image" } ], tool_choices: [ - { value: "auto", label: "Auto" }, - { value: "none", label: "None" }, - { value: "required", label: "Required" } + { value: "auto", label: "auto" }, + { value: "required", label: "required" } ] } }} @@ -87,9 +89,9 @@ description: "Retrieve available models, tools, output formats, and tool choices Tool description. - Supported output format options with `value` and `label`. + Supported output format options with `value` and `label`. Values are `array`, `string`, `number`, `object`, `audio` and `image`. Use `object`, not `json`, for structured object output. - Supported tool choice options with `value` and `label`. + Supported tool choice options with `value` and `label`. Only `auto` and `required` are returned. When adding or editing a run prompt column, `config.tool_choice` also accepts JSON `null` to clear the tool preference; the string `none` is not accepted or returned. diff --git a/src/pages/docs/billing/guides/control-spend.mdx b/src/pages/docs/billing/guides/control-spend.mdx index 74adfceff..abc29913e 100644 --- a/src/pages/docs/billing/guides/control-spend.mdx +++ b/src/pages/docs/billing/guides/control-spend.mdx @@ -19,7 +19,7 @@ Go to **Settings > Usage Summary**. The page shows one card per meter for the cu *A fresh period: every meter still inside its free allowance, so the period total is just the platform fee* -Each card carries what you have used so far, the free allowance that usage counts against, the share of the allowance you have spent, the projected total by the end of the period, and the estimated cost with the tier bands that produced it. Above the cards sit your plan name, the platform fee, the start and end of the billing period, and the total estimated cost across every meter. Two filters narrow the view: one picks the period, the other scopes everything to a single workspace. +Each card carries what you have used so far, the free allowance that usage counts against, the share of the allowance you have spent, the projected total by the end of the period, and the estimated cost with the tier bands that produced it. Above the cards sit your plan name, the platform fee, the start and end of the billing period, and the total estimated cost across every meter. The period filter selects the time range for this organization-wide overview. To compare workspaces, select a meter and read its **Usage by workspace** breakdown. Read the projection as a pace, not a promise. It extrapolates from how fast the meter has moved so far, so it shifts as the month goes on and is least reliable in the first few days, when a single busy afternoon can drag the whole line upward. @@ -31,7 +31,7 @@ This is the view that answers whether a jump was one bad day or a new baseline. ## Split usage by workspace -Go to **Settings > Usage Summary**, open a meter's trend, then break it down by workspace for a period. This is distinct from the workspace filter on the overview page: the filter scopes the whole page to one workspace, while this view compares workspaces against each other for a single meter, telling you which team is driving the number rather than just how big it got. +Go to **Settings > Usage Summary**, choose a period and select a meter. Its **Usage by workspace** table compares workspaces for that meter and period, telling you which team is driving the number rather than just how big it got. The overview cards and totals remain organization-wide; there is no global workspace filter. Reach for this before you set a budget. A ceiling on total spend catches everything but tells you nothing, while knowing which workspace is driving a meter tells you which meter is worth putting a ceiling on. diff --git a/src/pages/docs/command-center/admin/organizations.mdx b/src/pages/docs/command-center/admin/organizations.mdx index 27e5ad2f7..65718d099 100644 --- a/src/pages/docs/command-center/admin/organizations.mdx +++ b/src/pages/docs/command-center/admin/organizations.mdx @@ -103,8 +103,8 @@ Organizations can have multiple members with different roles. |---|---| | **Owner** | Full access. Can delete the org, manage billing, and change all settings. | | **Admin** | Can manage providers, keys, routing, budgets, and members (except owner). | -| **Member** | Can view config and create API keys. Cannot change org settings. | -| **Viewer** | Read-only access to dashboard, logs, and analytics. | +| **Member** | Cannot write gateway configuration or provider credentials (`403`), but active members can create and revoke their own personal gateway keys through the signed-in dashboard. | +| **Viewer** | Read-only access to gateway configuration, logs, and analytics, plus personal-key self-service: active Viewers can create and revoke their own gateway keys through the signed-in dashboard. | ### Managing members @@ -116,24 +116,9 @@ Members are managed through the Future AGI dashboard at **Settings > Members**. Each organization has its own pool of API keys (virtual keys). Keys inherit org-level settings and can have additional per-key restrictions. -```python -# List keys for an org -keys = client.keys.list(org_id="your-org-id") -for key in keys: - print(f"{key.name}: {key.key_prefix}...") +To create a personal key, sign in and select your organization, then open **Gateway > API keys > Create key**. Enter a **Name**, optionally choose **Allowed models** and **Allowed providers**, and click **Create**. Copy the key from the one-time reveal and store it securely. Active Members and Viewers can also use this flow and revoke their own keys in the dashboard. Creating a key with `X-Api-Key` / `X-Secret-Key` returns `403`, including for Owners and Admins. -# Create a new key -new_key = client.keys.create( - org_id="your-org-id", - name="backend-service", - rate_limit_rpm=100, - allowed_models=["gpt-4o", "gpt-4o-mini"], -) -print(f"Key: {new_key.key}") # full key shown only at creation - -# Revoke a key -client.keys.delete(key_id=new_key.id) -``` +The [control-plane list and revoke examples](/docs/command-center/concepts/virtual-keys#creating-and-managing-keys) use a Future AGI user key and follow its owner's organization and workspace permissions. Owners and Admins can access organization keys; other members can access only their own. Use the dashboard for personal-key self-service without workspace write access. See [Virtual keys & access control](/docs/command-center/concepts/virtual-keys) for detailed key configuration (RBAC, IP ACL, model restrictions). diff --git a/src/pages/docs/command-center/api/chat.mdx b/src/pages/docs/command-center/api/chat.mdx index a26b0c96c..b37f44221 100644 --- a/src/pages/docs/command-center/api/chat.mdx +++ b/src/pages/docs/command-center/api/chat.mdx @@ -129,7 +129,7 @@ All standard OpenAI chat completion parameters are supported: | `audio` | object | Audio output config: `{voice: "alloy", format: "wav"}`. | -Agent Command Center passes through unknown fields to the provider. Provider-specific parameters (like Anthropic's `thinking` or any vendor extension) work without Agent Command Center needing to know about them. +Unknown-field passthrough depends on the provider adapter. The OpenAI-compatible adapter preserves unknown top-level fields, but the provider must still support them. The Anthropic adapter translates `/v1/chat/completions` into a typed Messages request and does not forward top-level `thinking` or arbitrary unknown fields, for either streaming or non-streaming calls. The native Anthropic `/v1/messages` route is separate; do not assume its capabilities apply to translated chat requests. --- diff --git a/src/pages/docs/command-center/api/headers.mdx b/src/pages/docs/command-center/api/headers.mdx index 50f4e2c81..eccc46ada 100644 --- a/src/pages/docs/command-center/api/headers.mdx +++ b/src/pages/docs/command-center/api/headers.mdx @@ -21,7 +21,7 @@ The Agent Command Center SDK handles these automatically. If you're using the Op | `x-agentcc-session-id` | string | Group related requests into a logical session for analytics. | | `x-agentcc-session-name` | string | Human-readable label for the session (used alongside `session-id`). | | `x-agentcc-session-path` | string | Hierarchical path within a session, e.g. `/search/rerank`. | -| `x-agentcc-request-id` | string | Client-generated request ID for idempotency and log correlation. | +| `x-agentcc-request-id` | string | Caller correlation ID, stored as `client_request_id` metadata and echoed as `x-agentcc-client-request-id`. The gateway generates its own response `x-agentcc-request-id`. Reusing the caller ID does not deduplicate requests or replay a prior response; a retry can make another billable inference call. | | `x-agentcc-user-id` | string | User identifier for per-user tracking, budgets, and analytics. | ### Metadata and properties @@ -74,7 +74,7 @@ A value that does not parse is ignored rather than rejected. | Header | Example | Description | |---|---|---| -| `x-agentcc-request-id` | `req-a1b2c3` | Unique identifier for this request. Use this when filing support tickets or searching logs. | +| `x-agentcc-request-id` | `req-a1b2c3` | Fresh gateway-generated identifier for this request, not the caller's supplied ID. Use this when filing support tickets or searching logs. | | `x-agentcc-trace-id` | `trace-x7y8z9` | Trace ID for distributed tracing. Matches the request header if one was sent. | | `x-agentcc-provider` | `openai` | Which provider served this request. | | `x-agentcc-model-used` | `gpt-4o-2024-08-06` | Actual model returned by the provider. May differ from the requested model if routing redirected the request. | @@ -85,6 +85,7 @@ A value that does not parse is ignored rather than rejected. | Header | Present when | Value | |---|---|---| +| `x-agentcc-client-request-id` | A valid caller `x-agentcc-request-id` was supplied | Echo of the caller's correlation ID; does not provide idempotency. | | `x-agentcc-cost` | Model has pricing data | Estimated cost in USD (e.g. `0.00234`). Returns `0` on exact cache hits. | | `x-agentcc-cache` | Caching is enabled | `hit`, `hit_exact`, `hit_semantic`, `miss`, or `skip` | | `x-agentcc-guardrail-triggered` | A guardrail fired | `true` | diff --git a/src/pages/docs/command-center/concepts/virtual-keys.mdx b/src/pages/docs/command-center/concepts/virtual-keys.mdx index 5a1811737..57e46905a 100644 --- a/src/pages/docs/command-center/concepts/virtual-keys.mdx +++ b/src/pages/docs/command-center/concepts/virtual-keys.mdx @@ -44,62 +44,41 @@ Each virtual key can have the following restrictions: ## Creating and managing keys - +Creating a hosted personal gateway key requires an interactive sign-in. Active organization members, including Members and Viewers, can create and revoke their own keys in the dashboard. Editing key access settings and syncing keys remain Owner/Admin operations. A gateway key is for inference, not these management calls. + + -Go to **Settings > API Keys** in the Future AGI dashboard to create, view, and revoke keys. +1. Sign in to the Future AGI dashboard and select your organization. +2. Open **Gateway > API keys** and click **Create key**. +3. Enter a **Name**, optionally choose **Allowed models** and **Allowed providers**, then click **Create**. +4. Copy the key from the one-time reveal and store it securely; it will not be shown again. - - -All key operations require the admin token in the `Authorization` header. - -**Create a key:** - -```bash -curl -X POST https://gateway.futureagi.com/-/keys \ - -H "Authorization: Bearer your-admin-token" \ - -H "Content-Type: application/json" \ - -d '{ - "name": "production-backend", - "owner": "alice@example.com", - "models": ["gpt-4o", "claude-sonnet-4-6"], - "providers": ["openai", "anthropic"], - "rate_limit_rpm": 100, - "rate_limit_tpm": 50000, - "allowed_ips": ["10.0.0.0/8"], - "metadata": {"team": "ml", "env": "production"}, - "expires_at": "2026-12-31T23:59:59Z" - }' -``` + -The response includes the raw key value. This is the only time it's shown - store it securely. +Create keys through the signed-in dashboard flow above. `POST /agentcc/api-keys/` rejects `X-Api-Key` / `X-Secret-Key` authentication with `403`, even for Owners and Admins. -**List keys:** +For the list and revoke calls below, use a Future AGI **user key** in `X-Api-Key` and `X-Secret-Key`. Access follows the user's organization membership and workspace permissions: Owners and Admins can manage organization keys; other members can access only their own keys. Use the dashboard for personal-key self-service without workspace write access. -```bash -curl https://gateway.futureagi.com/-/keys \ - -H "Authorization: Bearer your-admin-token" -``` +Responses use a `{"status": true, "result": ...}` envelope. -**Revoke a key:** +**List accessible keys:** ```bash -curl -X DELETE "https://gateway.futureagi.com/-/keys/key_123" \ - -H "Authorization: Bearer your-admin-token" +curl https://api.futureagi.com/agentcc/api-keys/ \ + -H "X-Api-Key: $FI_API_KEY" \ + -H "X-Secret-Key: $FI_SECRET_KEY" ``` -Revocations are broadcast to all gateway replicas via Redis pub/sub immediately. - -**Add credits (managed keys):** +**Revoke an accessible key:** ```bash -curl -X POST "https://gateway.futureagi.com/-/keys/key_123/credits" \ - -H "Authorization: Bearer your-admin-token" \ - -H "Content-Type: application/json" \ - -d '{"amount": 50.00}' +curl -X POST https://api.futureagi.com/agentcc/api-keys/KEY_ID/revoke/ \ + -H "X-Api-Key: $FI_API_KEY" \ + -H "X-Secret-Key: $FI_SECRET_KEY" ``` diff --git a/src/pages/docs/command-center/features/cost-tracking.mdx b/src/pages/docs/command-center/features/cost-tracking.mdx index 1f0825e86..aa48b9b1f 100644 --- a/src/pages/docs/command-center/features/cost-tracking.mdx +++ b/src/pages/docs/command-center/features/cost-tracking.mdx @@ -190,51 +190,24 @@ comparison = client.analytics.model_comparison( Get notified when spending crosses a threshold. Alerts are configured per organization. - +Sign in as an organization **Owner or Admin** and open **Gateway > Alerts** (under **Observability**), not Gateway settings. - +1. Select **Channels**, then **Add channel**. Enter a **Channel name** and select **Type**: **Webhook**, **Slack**, or **Log Only**. For Webhook or Slack, enter the destination's **URL / Endpoint**; Log Only does not require one. Click **Add channel**. +2. Select **Rules**, then **Create rule**. Enter a **Rule name**, choose **Total Cost ($)** as the **Metric**, and set the **Condition** and **Threshold**. +3. Set the **Window**, **Cooldown**, and **Severity**, then select the destination under **Notify channels**. A rule with no selected channel notifies nobody. +4. Click **Create rule**. Check for any gateway-sync warning and the rule's status: **Active** means the gateway reports that it enforces organization alerting. A saved rule is not proof that the gateway runs it. -Go to **Agent Command Center > Settings > Alerts** in the Future AGI dashboard. Create a new alert by selecting the event type, setting recipients, and configuring severity. +These rules and channels use the organization's `alerting` config. They are separate from the former standalone email-alert settings flow; **Gateway > Alerts** does not offer an email or SMTP channel. Rules compare metrics over a window rather than selecting budget events or email recipients. - - - - -```python -alert = client.alerts.create( - name="Budget warning at 80%", - condition="cost > 80", - recipients=["team@example.com"], - severity="high", -) -``` - - - - - -```typescript -const alert = await client.alerts.create({ - name: "Budget warning at 80%", - condition: "cost > 80", - recipients: ["team@example.com"], - severity: "high", -}); -``` - - - - - -### Alert types +### Alert metrics -| Event | Trigger | +| Metric | Field | |---|---| -| `budget_exceeded` | Spend crosses the budget limit | -| `budget_threshold` | Spend crosses a percentage threshold (e.g. 80%) | -| `error_spike` | Error rate exceeds configured threshold | -| `latency_spike` | P95 latency exceeds configured threshold | -| `guardrail_triggered` | A guardrail blocks or flags a request | +| Total Cost ($) | `cost_total` | +| Error Count | `error_count` | +| Request Count | `request_count` | +| Avg Latency (ms) | `latency_avg` | +| Total Tokens | `tokens_total` | Configure a cooldown period to prevent alert flooding when thresholds are repeatedly crossed. diff --git a/src/pages/docs/command-center/features/guardrails.mdx b/src/pages/docs/command-center/features/guardrails.mdx index cd13d9368..ff2674df4 100644 --- a/src/pages/docs/command-center/features/guardrails.mdx +++ b/src/pages/docs/command-center/features/guardrails.mdx @@ -25,7 +25,7 @@ Guardrails that check the request (Pre stage) run before the [response cache](/d ## Built-in Guardrail Types -Agent Command Center includes 16 guardrail types covering common safety scenarios. +Agent Command Center includes 16 guardrail types covering common safety scenarios. This feature list is broader than the dashboard's **Add check** catalog: **Tool Permissions** is stored-only/read-only, **MCP Security** is not offered in the catalog, and **Future AGI Evaluation** appears only when its required account configuration is already saved. See [console availability](/docs/protect/reference/guardrail-checks#provider-backed-checks) before choosing an integration. | Guardrail Type | Name in the org config | Stage | What it detects | |---|---|---|---| @@ -52,20 +52,21 @@ The Stage column is where an organization's check runs: Hallucination Detection ## External Integrations -Agent Command Center integrates with leading guardrail and security providers. +These integrations are not all newly configurable in the dashboard: -| Provider | Capabilities | +| Provider | Console availability | |---|---| -| Lakera Guard | PII, prompt injection, content moderation | -| Presidio | PII detection and redaction | -| Llama Guard | Content moderation | -| AWS Bedrock Guardrails | Multi-modal content safety | -| Azure Content Safety | Content moderation and PII detection | -| Pangea | Data security and compliance | -| Aporia | AI monitoring and anomaly detection | -| Enkrypt AI | Encryption and data protection | - -Additional integrations available: HiddenLayer, DynamoAI, IBM AI, Zscaler, Crowdstrike, Lasso, Grayswan. +| Lakera Guard | Shown only for an already configured account; not in Add check | +| Presidio | Stored-only/read-only | +| Llama Guard | Stored-only/read-only | +| AWS Bedrock Guardrails | Stored-only/read-only | +| Azure Content Safety | Stored-only/read-only | +| Pangea | Stored-only/read-only | +| Aporia | Stored-only/read-only | +| Enkrypt AI | Stored-only/read-only | +| HiddenLayer, DynamoAI, IBM AI, Zscaler, Crowdstrike, Lasso, Grayswan | Stored-only/read-only | + +Stored-only checks cannot be added, enabled, or edited in this console. Their saved configuration is retained and pushed to the gateway, so existing enabled checks may still run. An organization Owner or Admin can turn one off or remove it after confirmation, but cannot turn it back on here. For the complete check-name table, see [Guardrail checks](/docs/protect/reference/guardrail-checks#provider-backed-checks). --- @@ -86,14 +87,11 @@ Start with `log` or `warn` to understand traffic patterns before switching to `b ### Fail-open vs fail-closed -What happens when a check itself fails (an error or a timeout) is the gateway's `guardrails.fail_open` setting: - -- **Fail-open** (`true`, the default in a self-hosted gateway's config file): the request proceeds. -- **Fail-closed** (`false`): the failure counts as a trigger with a score of 1.0, so a check with the action `block` refuses the request. +Failure handling is owned by the gateway operator. **Mode and Fail Open** are read-only on the organization surface; saving `guardrails.fail_open` through the control-plane update-config endpoint does not change this behavior. Organization checks execute sequentially and stop at the first block. -A check that rewrites the request (a PII remediation or the action `mask`) and does not finish refuses the request either way, since its rewrite was not applied. +For a check that does not rewrite the request, a timeout or recovered panic skips that check when the gateway's fail-open setting is on. When it is off, the check returns a failed result that is evaluated against its threshold and action; that does not always mean the request is blocked. A request-rewriting check that times out or panics blocks the request. Remote-service errors returned as failed check results follow the check's threshold and action. -On **Gateway > Guardrails > Rules**, the Mode and Fail Open controls are read-only and show how the gateway runs your checks. The organization's `timeout_ms` sets how long each of its checks may run. +The organization-editable pipeline control is **Check timeout (ms)** in Rules. See [Pipeline settings](/docs/protect/reference/guardrail-checks#pipeline-settings) for the limits and save behavior. --- @@ -122,12 +120,12 @@ Set your organization's checks in the dashboard or in its org config. A self-hos - 1. Go to **Gateway > Guardrails** in the Future AGI dashboard and open the **Rules** tab - 2. Turn on a check (for example **PII Detection**) with its switch. A check you turn on starts with the action Block and a threshold of 0.80 - 3. Click the check's edit icon to set its **Action** (Block, Warn, or Log, and Mask for PII Detection), its **Confidence Threshold**, and any settings the check has, and click **Save** - 4. Click **Save & Activate** + 1. Sign in as an organization **Owner or Admin**, go to **Gateway > Guardrails**, and open **Rules**. + 2. Click **Add check** and choose an offered check, such as **PII Detection**, or open an existing editable row. + 3. Set its **Action** and any check-specific **Settings**. A **Match threshold** is available only where applicable; default PII Detection has no slider. The drawer describes the check's fixed request or response stage. + 4. Click **Add check** for a new rule or **Save** for an edit, then check the save result and any gateway-sync warning. There is no separate **Save & Activate** step or Overview editor. - The **Overview** tab lists your checks and the stage each one runs at. + Stored-only rows cannot be enabled or edited. See [Turn on a guardrail](/docs/protect/guides/turn-on-a-guardrail) for the full flow. @@ -587,10 +585,10 @@ Block requests that contain specific words or phrases with the **Blocklist** che - 1. Go to **Gateway > Guardrails** and open the **Rules** tab - 2. Turn on **Keyword Blocklist** and click its edit icon - 3. Enter the **Blocked Keywords**, one per line (commas also work), and choose the **Action** - 4. Click **Save**, then **Save & Activate** + 1. Go to **Gateway > Guardrails** and open **Rules** as an organization **Owner or Admin**. + 2. Click **Add check** and choose **Keyword Blocklist**, or open its existing row. + 3. Enter **Blocked keywords** under **Settings** and choose the **Action**. + 4. Click **Add check** for a new rule or **Save** for an edit, then check the save result and any gateway-sync warning. diff --git a/src/pages/docs/command-center/features/providers.mdx b/src/pages/docs/command-center/features/providers.mdx index a3b5b1bcc..638b518e7 100644 --- a/src/pages/docs/command-center/features/providers.mdx +++ b/src/pages/docs/command-center/features/providers.mdx @@ -53,6 +53,8 @@ On a self-hosted Future AGI install, a provider you add in the dashboard can use ## Adding a provider +When creating or updating a provider credential through `/agentcc/provider-credentials/`, `api_format` accepts `openai`, `anthropic`, `gemini` (or `google`), `bedrock`, `cohere`, `azure`, `deepgram` and `cartesia`. Some versions also accept `systemone` and `typesafe`. On creation, the default is `openai` regardless of `provider_name`, so set `anthropic` for Anthropic. An unsupported value returns `400` with an `Unsupported api_format` validation error; the change is not saved. It does not create a provider that fails later during inference. + diff --git a/src/pages/docs/command-center/features/rate-limiting.mdx b/src/pages/docs/command-center/features/rate-limiting.mdx index 5930201a5..9b7fc7cc3 100644 --- a/src/pages/docs/command-center/features/rate-limiting.mdx +++ b/src/pages/docs/command-center/features/rate-limiting.mdx @@ -36,9 +36,14 @@ The most restrictive limit applies. If the global limit is 1000 RPM and a key's -Go to **Agent Command Center > Rate Limits** in the Future AGI dashboard to set global and per-org limits. +Sign in as an organization **Owner or Admin**, then: -Per-key limits are set when creating or editing a key in **Settings > API Keys**. +1. Open **Gateway > Gateway settings > Org config**. +2. Select the **Rate limiting** section row to open the **Rate Limiting** editor tab. If the organization has no custom config yet, click **Create Custom Config**, then select **Rate Limiting**. +3. Turn on **Enable per-org rate limiting** and set **Requests per Minute (RPM)** and **Tokens per Minute (TPM)** for the organization. +4. Click **Save & Activate**. Check the save result and any gateway-sync warning before relying on the new limits. + +These controls set organization-wide limits, not gateway-global or per-key limits. Hosted keys do not accept `rate_limit_rpm` or `rate_limit_tpm`; the API refuses those fields. Gateway-global limits are operator-owned configuration. diff --git a/src/pages/docs/cookbook/rag-langchain.mdx b/src/pages/docs/cookbook/rag-langchain.mdx index 83ed3fa26..50fd89daf 100644 --- a/src/pages/docs/cookbook/rag-langchain.mdx +++ b/src/pages/docs/cookbook/rag-langchain.mdx @@ -94,7 +94,11 @@ Indexed 187 chunks from 3 pages -Register a trace provider with `traceai-langchain` so every LLM call, retrieval, and chain step is captured, and set up an `Evaluator` to score answers after the fact. +Register a trace provider with `traceai-langchain` so every LLM call, retrieval, and chain step is captured, and set up an `Evaluator` to score answers after the fact. This recipe uses explicit `Evaluator` calls rather than span evaluation tags; the scores are returned to the script, not attached to the traced spans. + + +This example registers an `EXPERIMENT` project. Its former **Prototype** dashboard viewer is unavailable. Use the scored results in the script to inspect this comparison; the separate **Tracing** page is not a replacement viewer for this experiment setup. + ```python from getpass import getpass @@ -102,14 +106,7 @@ import os from fi.evals import Evaluator from fi_instrumentation import register -from fi_instrumentation.fi_types import ( - ProjectType, - EvalName, - EvalSpanKind, - EvalTag, - EvalTagType, - ModelChoices, -) +from fi_instrumentation.fi_types import ProjectType from traceai_langchain import LangChainInstrumentor os.environ["FI_API_KEY"] = os.environ.get("FI_API_KEY") or getpass("Enter your FI API key: ") @@ -117,27 +114,10 @@ os.environ["FI_SECRET_KEY"] = os.environ.get("FI_SECRET_KEY") or getpass("Enter evaluator = Evaluator(fi_base_url="https://api.futureagi.com") -# Tag every LLM span for context adherence and groundedness as it's traced -eval_tags = [ - EvalTag( - type=EvalTagType.OBSERVATION_SPAN, - value=EvalSpanKind.LLM, - eval_name=EvalName.CONTEXT_ADHERENCE, - model=ModelChoices.TURING_FLASH, - ), - EvalTag( - type=EvalTagType.OBSERVATION_SPAN, - value=EvalSpanKind.LLM, - eval_name=EvalName.GROUNDEDNESS, - model=ModelChoices.TURING_FLASH, - ), -] - trace_provider = register( project_type=ProjectType.EXPERIMENT, project_name="RAG-Cookbook", project_version_name="v1", - eval_tags=eval_tags, ) LangChainInstrumentor().instrument(tracer_provider=trace_provider) @@ -146,8 +126,30 @@ LangChainInstrumentor().instrument(tracer_provider=trace_provider) Confirm the wiring works end to end before building the full comparison: retrieve for one question, answer it, and score the answer with `fi.evals`. ```python +from math import isfinite from fi.evals.templates import ContextRelevance + +def numeric_score(batch): + if not batch.eval_results or batch.eval_results[0] is None: + raise ValueError("Evaluation returned no result") + result = batch.eval_results[0] + if result.output is None: + raise ValueError(result.reason or "Evaluation returned no output") + if result.output_type == "boolean": + if isinstance(result.output, bool): + return float(result.output) + if isinstance(result.output, str) and result.output.lower() in {"true", "false"}: + return float(result.output.lower() == "true") + raise ValueError("Expected a boolean evaluation output") + if result.output_type != "score" or isinstance(result.output, bool): + raise ValueError("Expected a score or boolean evaluation output") + score = float(result.output) + if not isfinite(score) or not 0 <= score <= 1: + raise ValueError("Expected a finite score between 0 and 1") + return score + + sample_docs = recursive_retriever.invoke("What is a transformer?") sample_context = "\n\n".join(doc.page_content for doc in sample_docs) sample_answer = llm.invoke( @@ -156,13 +158,13 @@ sample_answer = llm.invoke( check = evaluator.evaluate( eval_templates=[ContextRelevance(config={"check_internet": False})], - inputs=[{"input": "What is a transformer?", "context": sample_context}], + inputs={"input": "What is a transformer?", "context": sample_context}, model_name="turing_flash", ) -print(check.eval_results[0].metrics[0].value) +print(numeric_score(check)) ``` -You should see a single score between 0 and 1, for example `0.9`. That confirms the chain, the tracer, and the evaluator are all wired correctly before the three-strategy comparison in the next steps. +You should see a single score between 0 and 1, for example `0.9`. The helper reads `EvalResult.output` according to `output_type`: numeric scores stay on the 0–1 scale, and boolean results become 1 for true or 0 for false. It stops on missing, failed, or unexpected outputs instead of averaging them. A successful score confirms the evaluation request worked, not that traces are visible in the dashboard. @@ -205,14 +207,14 @@ def score_pipeline(rows, model="turing_flash"): scored = [] for row in rows: inputs = {"input": row["query"], "context": row["context"], "output": row["answer"]} - relevance = evaluator.evaluate(eval_templates=[relevance_template], inputs=[inputs], model_name=model) - utilization = evaluator.evaluate(eval_templates=[utilization_template], inputs=[inputs], model_name=model) - grounded = evaluator.evaluate(eval_templates=[groundedness_template], inputs=[inputs], model_name=model) + relevance = evaluator.evaluate(eval_templates=[relevance_template], inputs=inputs, model_name=model) + utilization = evaluator.evaluate(eval_templates=[utilization_template], inputs=inputs, model_name=model) + grounded = evaluator.evaluate(eval_templates=[groundedness_template], inputs=inputs, model_name=model) scored.append({ **row, - "context_relevance": relevance.eval_results[0].metrics[0].value, - "chunk_utilization": utilization.eval_results[0].metrics[0].value, - "groundedness": grounded.eval_results[0].metrics[0].value, + "context_relevance": numeric_score(relevance), + "chunk_utilization": numeric_score(utilization), + "groundedness": numeric_score(grounded), }) return scored @@ -351,15 +353,11 @@ You should see a bar chart with three groups (illustrative values, from the runs *Context relevance stays nearly flat across all three strategies; chunk utilization and groundedness are what actually move* - - -Every `llm.invoke()` and `retriever.invoke()` call from the three runs was captured by `LangChainInstrumentor`. Open the Future AGI dashboard to inspect them. - -Open the **Prototype** tab and find the `RAG-Cookbook` project → open any trace to see the retrieval span, the LLM call, and the `context_adherence` and `groundedness` eval scores attached to it. This span-attached `groundedness` score is computed independently of the printed averages from Step 3-5, so the two don't need to match. + -Future AGI dashboard showing traced spans for the RAG-Cookbook experiment with retrieval and LLM steps +Review `recursive_scored`, `semantic_scored`, and `subq_scored` to compare individual answers rather than only the averages. Each row contains the question (`query`), retrieved `context`, generated `answer`, and the three scores returned by the explicit `Evaluator` calls. -*Each row is one trace: expand it to see the exact chunks retrieved and the eval score attached to the LLM span* +The old **Prototype** trace-inspection step is unavailable. This recipe does not attach evaluation scores to spans or require that viewer to complete the comparison. @@ -375,6 +373,6 @@ Open the **Prototype** tab and find the `RAG-Cookbook` project → open any trac | `score_pipeline` scores look identical across `recursive_avg`, `semantic_avg`, and `subq_avg` | `chroma_recursive` or `chroma_semantic` already had vectors from a previous run, so the new documents were appended instead of replacing them | Delete the `chroma_recursive/` and `chroma_semantic/` directories before each fresh comparison run | | `openai.RateLimitError` mid-run on `run_pipeline` | Three questions run back to back against the OpenAI API with no delay | Add `time.sleep(1)` between questions, or lower `test_questions` to 1-2 while iterating | | `evaluator.evaluate(...)` raises an authentication error | `FI_API_KEY` or `FI_SECRET_KEY` missing or unexported | Re-run the `export` block, then re-run the script | -| No traces appear under the `RAG-Cookbook` project | `LangChainInstrumentor().instrument()` ran after the chain was already built | Call `instrument()` right after `register()`, before defining or invoking any chain | +| Cannot find the `RAG-Cookbook` experiment in a Prototype tab | The Prototype viewer is unavailable | Inspect `recursive_scored`, `semantic_scored`, and `subq_scored` in the script; the Tracing page is not an alias for the experiment viewer | Next: read how the platform scores retrieval and groundedness in [Evaluate a RAG pipeline](/docs/cookbook/evaluate-rag). diff --git a/src/pages/docs/dataset/guides/add-rows.mdx b/src/pages/docs/dataset/guides/add-rows.mdx index 31f422f7b..2e9094ed6 100644 --- a/src/pages/docs/dataset/guides/add-rows.mdx +++ b/src/pages/docs/dataset/guides/add-rows.mdx @@ -1,6 +1,6 @@ --- title: "Add rows" -description: "Five ways to add new examples to a dataset you've already created." +description: "Add new examples to an existing dataset with the SDK, blank rows, file uploads, or dataset copies." --- Adding rows brings more examples into a [dataset](/docs/dataset/concepts/understanding-datasets) that already exists. If you don't have a dataset yet, start with [Create a dataset](/docs/dataset/guides/create-a-dataset). Columns stay put unless imported data brings a name the dataset doesn't already have. Whichever path you pick, new rows always land after whatever's already in the dataset, appended in the order you add them. @@ -126,7 +126,7 @@ Use this when the rows you need already exist somewhere else. - In the Add Row drawer, select **Add from existing model dataset or experiment**, then pick the source: another dataset, or an experiment snapshot. + In the Add Row drawer, select **Copy from another dataset**, then pick the source: another dataset, or an experiment snapshot. Map each source column to a column in this dataset. Only mapped columns copy over; anything left unmapped in the source is skipped. @@ -135,16 +135,7 @@ Use this when the rows you need already exist somewhere else. ## Import rows from Hugging Face -Use this to pull in a public dataset instead of typing examples by hand. - - - - In the Add Row drawer, select **Import from Hugging Face**, then search for the dataset and pick the subset next to the split. Set how many rows to import. - - - Start the import. Each Hugging Face feature is matched to a column by name; a name that doesn't exist yet becomes a new column, backfilled with empty cells on the rows that already existed. - - +Hugging Face row imports are disabled. See the [Hugging Face availability notice](/docs/dataset/guides/create-a-dataset#import-from-hugging-face). Use **Upload a file** or **Copy from another dataset** instead. ## Add rows from a file @@ -152,7 +143,7 @@ Use this for a CSV, Excel, JSON or JSONL export you already have. - In the Add Row drawer, select **Upload a file (JSONl/ JSON/ CSV)**, then upload it. + In the Add Row drawer, select **Upload a file**, then upload it. Column names in the file are matched to existing columns by name. A name that isn't already a column gets created, backfilled with empty cells on the rows that already existed. diff --git a/src/pages/docs/dataset/guides/create-a-dataset.mdx b/src/pages/docs/dataset/guides/create-a-dataset.mdx index 8abce2db5..e323d3a57 100644 --- a/src/pages/docs/dataset/guides/create-a-dataset.mdx +++ b/src/pages/docs/dataset/guides/create-a-dataset.mdx @@ -1,20 +1,21 @@ --- title: "Create a dataset" -description: "Six ways to create a dataset and get data into it." +description: "Create a dataset with the SDK or choose an available import method in the dashboard." --- -A dataset starts as just a name in your organization; everything else comes from how you fill it. Future AGI gives you six ways to do that, each landing you on the same [dataset](/docs/dataset/concepts/understanding-datasets) table of rows and columns you can keep editing afterward. +A dataset starts as just a name in your organization; everything else comes from how you fill it. Create it in code or in the dashboard, then keep editing its [rows and columns](/docs/dataset/concepts/understanding-datasets). -Every method starts the same way: from the dataset list (**Dataset** in the left nav), click **Add Dataset**. That opens a panel with six tiles: +For dashboard creation, open the dataset list and click **New dataset**. The current launch chooser offers these entries; synthetic generation also depends on your plan: | Method | Reach for it when | | --- | --- | -| Add data using SDK | You're scripting the setup or pushing rows from your own pipeline | -| Upload a file (JSON, CSV) | You already have test cases in a CSV, Excel, or JSON export | -| Create Synthetic Data | You don't have real data yet, but know the shape of the examples you need | -| Add datasets Manually | You're hand-building a small set and want an empty grid to fill in | -| Import from HuggingFace | The data you need already exists as a Hugging Face dataset | -| Add from existing model dataset or experiment | You want to branch off a dataset or experiment you already have | +| Upload a file | You already have test cases in a CSV, Excel, or JSON export | +| Start with a blank grid | You're hand-building a small set and want an empty grid to fill in | +| Copy from another dataset | You want to branch off a dataset or experiment you already have | +| Hugging Face | Disabled | +| Generate synthetic rows | You don't have real data yet, but know the shape of the examples you need | + +The SDK is a separate programmatic workflow, not a tile in this chooser. Whichever method you pick, the dataset's name must be unique inside your organization. A name that's already taken is rejected. For the full set of dataset and column limits, see [Limits & Data Types](/docs/dataset/reference/limits-and-data-types). @@ -22,9 +23,7 @@ Every method starts the same way: from the dataset list (**Dataset** in the left For scripting dataset creation, or when you'd rather write rows in code than click through the grid. -In the Add dataset panel, pick **Add data using SDK** and name the dataset. Future AGI creates an empty dataset and drops you on its Data tab, which shows the dataset's name, ID, API key, and secret key alongside a ready-to-run code snippet. Copy the snippet below and run it against your dataset to add [columns](/docs/dataset/concepts/static-and-dynamic-columns) and rows. - -An empty dataset starts with at most 10 rows; add the rest with the SDK. +Create the dataset directly with the SDK, then add [columns](/docs/dataset/concepts/static-and-dynamic-columns) and rows. The Python and TypeScript examples below each create a new dataset named `support-agent-eval`; choose an unused name before running one. The cURL example only adds columns to an existing dataset, so replace `` with that dataset's ID. @@ -33,13 +32,17 @@ An empty dataset starts with at most 10 rows; add the rest with the SDK. import os from fi.datasets import Dataset -from fi.datasets.types import Cell, Column, DataTypeChoices, Row, SourceChoices +from fi.datasets.types import Cell, Column, DatasetConfig, DataTypeChoices, Row, SourceChoices +from fi.utils.types import ModelTypes os.environ["FI_API_KEY"] = "" os.environ["FI_SECRET_KEY"] = "" -# Get the dataset you just created -dataset = Dataset.get_dataset_config("support-agent-eval") +# Create a new dataset before adding columns and rows +dataset = Dataset(dataset_config=DatasetConfig( + name="support-agent-eval", + model_type=ModelTypes.GENERATIVE_LLM, +)).create() # Define columns columns = [ @@ -67,14 +70,17 @@ dataset = dataset.add_rows(rows=rows) ``` ```typescript Typescript -import { Dataset, DataTypeChoices, createRow, createCell } from "@future-agi/sdk"; +import { Dataset, DataTypeChoices, ModelTypes, createRow, createCell } from "@future-agi/sdk"; process.env["FI_API_KEY"] = ""; process.env["FI_SECRET_KEY"] = ""; async function main() { - // Get the dataset you just created - const dataset = await Dataset.open("support-agent-eval"); + // Create a new dataset before adding columns and rows + const dataset = await Dataset.createDataset({ + name: "support-agent-eval", + model_type: ModelTypes.GENERATIVE_LLM, + }); // Define columns const columns = [ @@ -131,31 +137,31 @@ See the [Datasets SDK reference](/docs/sdk/datasets) for the full `Dataset` API. For bringing in test cases you already have as a file, instead of typing them in. -In the Add dataset panel, pick **Upload a file (JSON, CSV)** and name the dataset. Drop or browse to your file: accepted formats are `.csv`, `.xls`, `.xlsx`, `.json`, and `.jsonl`, up to 25 MB. The dataset appears on your list right away. Future AGI processes the file in the background with a visible progress state until it's done. +In the Add dataset panel, pick **Upload a file** and name the dataset. Drop or browse to your file: accepted formats are `.csv`, `.xls`, `.xlsx`, `.json`, and `.jsonl`, up to 25 MB. The dataset appears on your list right away. Future AGI processes the file in the background with a visible progress state until it's done. ## Create synthetic data For when you don't have real data yet, but know the shape of the examples you need. -In the Add dataset panel, pick **Create Synthetic Data** and name the dataset. From there, Future AGI walks you through describing the schema and generates rows for you. See [Synthetic Data](/docs/dataset/concepts/synthetic-data) if you want to regenerate later. +In the Add dataset panel, pick **Generate synthetic rows** and name the dataset. From there, Future AGI walks you through describing the schema and generates rows for you. See [Synthetic Data](/docs/dataset/concepts/synthetic-data) if you want to regenerate later. ## Add a dataset manually For hand-building a small dataset from scratch when you already know its shape. -In the Add dataset panel, pick **Add datasets Manually**, name the dataset, and choose how many rows and how many columns to start with, up to 100 of each. Future AGI creates the dataset with that many empty rows and columns, ready for you to fill in. +In the Add dataset panel, pick **Start with a blank grid**, name the dataset, and choose how many rows and how many columns to start with, up to 100 of each. Future AGI creates the dataset with that many empty rows and columns, ready for you to fill in. ## Import from Hugging Face -For pulling in a Hugging Face dataset instead of typing test cases by hand. - -In the Add dataset panel, pick **Import from HuggingFace** and paste the Hugging Face dataset ID. Click **Load Dataset**, then pick the **Subset** and **Split** you want. Name the new dataset to finish. Only the first 100 rows of the source are ingested. + +Hugging Face imports are disabled because imported cells can remain blank. The chooser may list **Hugging Face**, but it cannot start an import. Use **Upload a file** or **Copy from another dataset** instead. This restriction also applies when adding rows to an existing dataset. + ## Add from an existing dataset or experiment For branching off a dataset or experiment you already have. -In the Add dataset panel, pick **Add from existing model dataset or experiment** and choose the dataset or experiment you want to copy from. Choose whether to bring over **Import Data** or **Import data and prompt configuration**, then select which columns to include. Name the new dataset to finish. +In the Add dataset panel, pick **Copy from another dataset** and choose the dataset or experiment you want to copy from. Choose whether to bring over **Import Data** or **Import data and prompt configuration**, then select which columns to include. Name the new dataset to finish. ## Dive deeper diff --git a/src/pages/docs/dataset/guides/run-an-experiment.mdx b/src/pages/docs/dataset/guides/run-an-experiment.mdx index 4b87234da..2be839071 100644 --- a/src/pages/docs/dataset/guides/run-an-experiment.mdx +++ b/src/pages/docs/dataset/guides/run-an-experiment.mdx @@ -23,21 +23,30 @@ One experiment lays your dataset's rows against every configuration you add. A * ## Build the experiment -Click **Experiment** on the dataset to open the creation flow. It's a three-step form. +Click **New experiment** on the dataset to open the creation flow. The current default wizard has four steps: **Configuration → Data → Evaluations → Review & run**. - - Name the experiment and choose its type: **LLM**, **TTS**, **STT**, or **Image**. The type decides the output format and which models you can attach. - - Add the prompts or agents you want to compare and attach a model to each. Every prompt/agent-model pair becomes its own configuration column. LLM experiments can mix prompts and agents in the same run and attach tools to a prompt, useful for deciding whether an agent earns its extra complexity over a plain prompt; TTS, STT, and Image experiments take prompts only. + Name the experiment. New experiments in the experiment wizard are **LLM-only**; **TTS**, **STT**, and **Image** are not choices for a fresh experiment. + + Add the prompts or agents you want to compare and select a saved version of each. Attach a model to each prompt. Every prompt-model pair becomes its own configuration column; an agent uses its saved version. LLM experiments can mix prompts and agents in the same run. + + + Check the dataset data the experiment will use. If the **Dataset version** selector is available in your build, choose the version to run against. Map every required prompt or agent input variable to a dataset column and check the sample values. Unmapped inputs block **Continue** and the final run action. Choose a reference column here if your evaluations need one. - Optionally pick a column to compare outputs against as a baseline, then add the evals that will score every configuration. + Add at least one evaluation to score the configurations, then continue to review. + + + Check the runs, data, and evaluations in the summary. Resolve any issues reported by the earlier steps, then click **Run experiment** to start the configured prompt and agent runs. -Click **Run Experiment**, and every row runs against every configuration, with each output scored by your evals as it comes in. +The configurations run against the dataset data, with each output scored by your evals as it comes in. + + +An existing TTS or STT experiment retains its selected type. Builds with experiment enhancements explicitly disabled also expose the classic type choices, including Image. Those are separate paths from the default new-experiment flow above. + An eval you add after the run sits queued for a few seconds before it starts scoring. diff --git a/src/pages/docs/evaluation/builtin/caption-hallucination.mdx b/src/pages/docs/evaluation/builtin/caption-hallucination.mdx index 4ac4e11b2..3d3a37d1c 100644 --- a/src/pages/docs/evaluation/builtin/caption-hallucination.mdx +++ b/src/pages/docs/evaluation/builtin/caption-hallucination.mdx @@ -47,23 +47,12 @@ print(result.score) print(result.reason) ``` -```typescript TypeScript -import { evaluate } from "@future-agi/ai-evaluation"; - -const result = await evaluate( - "caption_hallucination", - { - image: "https://www.esparklearning.com/app/uploads/2024/04/Albert-Einstein-generated-by-AI-1024x683.webp", - caption: "old man" - }, - { modelName: "turing_flash" } -); - -console.log(result); -``` - + +The released TypeScript SDK (`@future-agi/sdk` 0.2.0) does not export the standalone `evaluate()` API. The legacy TypeScript evaluation snippet is not supported by that package; use the Python example above for this API. + + ## When to use Run Caption Hallucination wherever a caption is generated for an image and needs to stay grounded in what the image actually shows. diff --git a/src/pages/docs/evaluation/builtin/detect-hallucination.mdx b/src/pages/docs/evaluation/builtin/detect-hallucination.mdx index cde5c12ae..2109ba001 100644 --- a/src/pages/docs/evaluation/builtin/detect-hallucination.mdx +++ b/src/pages/docs/evaluation/builtin/detect-hallucination.mdx @@ -51,23 +51,12 @@ print(result.score) print(result.reason) ``` -```typescript TypeScript -import { evaluate } from "@future-agi/ai-evaluation"; - -const result = await evaluate( - "detect_hallucination", - { - context: "Honey never spoils because it has low moisture content and high acidity, creating an environment that resists bacteria and microorganisms. Archaeologists have even found pots of honey in ancient Egyptian tombs that are still perfectly edible.", - output: "Honey doesn't spoil because its low moisture and high acidity prevent the growth of bacteria and other microbes." - }, - { modelName: "turing_flash" } -); - -console.log(result); -``` - + +The released TypeScript SDK (`@future-agi/sdk` 0.2.0) does not export the standalone `evaluate()` API. The legacy TypeScript evaluation snippet is not supported by that package; use the Python example above for this API. + + ## When to use Run Detect Hallucination wherever fabricated facts would be costly, and you want a direct signal on made-up content rather than a broader adherence score. diff --git a/src/pages/docs/evaluation/builtin/index.mdx b/src/pages/docs/evaluation/builtin/index.mdx index a6dfc6dbd..b221b8177 100644 --- a/src/pages/docs/evaluation/builtin/index.mdx +++ b/src/pages/docs/evaluation/builtin/index.mdx @@ -21,7 +21,7 @@ Whether what you retrieved was right, and whether the response stayed inside it | [**Chunk Attribution**](/docs/evaluation/builtin/chunk-attribution) | Tracks if the context chunk is used in generating the response. | `output`, `context` | RAG & Retrieval | LLM as Judge | | [**Chunk Utilization**](/docs/evaluation/builtin/chunk-utilization) | Measures how effectively context chunks are used in responses. | `output`, `context` | RAG & Retrieval | LLM as Judge | | [**Groundedness**](/docs/evaluation/builtin/groundedness) | Ensures response strictly adheres to the provided context without external information. | `output`, `context` | Text, Audio, Chat, RAG & Retrieval, Hallucination | LLM as Judge | -| [**Detect Hallucination**](/docs/evaluation/builtin/detect-hallucination) | Identifies fabricated facts not present in the input or reference. | `input`, `output` | Text, Audio, Image, Chat, RAG & Retrieval, Hallucination | LLM as Judge | +| [**Detect Hallucination**](/docs/evaluation/builtin/detect-hallucination) | Identifies fabricated facts not present in the input or reference. | `output`, `context` (optional: `input`) | Text, Audio, Image, Chat, RAG & Retrieval, Hallucination | LLM as Judge | | [**Eval Ranking**](/docs/evaluation/builtin/eval-ranking) | Provides a ranking score for each context based on specified criteria. | `input`, `context` | RAG & Retrieval, Custom | LLM as Ranker | | [**Recall@K**](/docs/evaluation/builtin/recall-at-k) | Evaluates recall at K for retrieval-based systems. | `output`, `context` | RAG & Retrieval | Statistical Metric | | [**Precision@K**](/docs/evaluation/builtin/precision-at-k) | Evaluates precision at K for retrieval-based systems. | `output`, `context` | RAG & Retrieval | Statistical Metric | @@ -38,10 +38,10 @@ Anything that must never reach your user | Eval | Description | Required Inputs | Use Cases | Evaluation Method | |------|-------------|-----------------|-----------|-------------------| -| [**PII Detection**](/docs/evaluation/builtin/pii) | Detects personally identifiable information (PII) in text. | `output` | Text, Audio, Image, Chat, Safety | LLM as Judge | +| [**PII Detection**](/docs/evaluation/builtin/pii) | Detects personally identifiable information (PII) in text. | `input` | Text, Audio, Image, Chat, Safety | LLM as Judge | | [**Toxicity**](/docs/evaluation/builtin/toxicity) | Evaluates content for toxic or harmful language. | `output` | Text, Audio, Image, Chat, Safety | LLM as Judge | | [**Sexist**](/docs/evaluation/builtin/sexist) | Detects sexist content and gender bias. | `output` | Text, Audio, Image, Chat, Safety | LLM as Judge | -| [**Prompt Injection**](/docs/evaluation/builtin/prompt-injection) | Evaluates text for potential prompt injection attempts. | `input`, `output` | Text, Audio, Image, Chat, Safety | LLM as Judge | +| [**Prompt Injection**](/docs/evaluation/builtin/prompt-injection) | Evaluates text for potential prompt injection attempts. | `input` | Text, Audio, Image, Chat, Safety | LLM as Judge | | [**Data Privacy Compliance**](/docs/evaluation/builtin/data-privacy) | Checks output for GDPR, HIPAA, and other privacy regulation compliance. | `output` | Text, Audio, Image, Chat, Safety | LLM as Judge | | [**Cultural Sensitivity**](/docs/evaluation/builtin/cultural-sensitivity) | Analyzes output for cultural appropriateness and inclusive language. | `output` | Text, Audio, Image, Chat, Safety | LLM as Judge | | [**Bias Detection**](/docs/evaluation/builtin/bias-detection) | Identifies gender, racial, cultural, or ideological bias in output. | `output` | Text, Audio, Image, Chat, Safety | LLM as Judge | @@ -90,7 +90,7 @@ Whether a single response is well-made: tone, brevity, structure, and format che | [**Tone**](/docs/evaluation/builtin/tone) | Analyzes the tone and sentiment of content. | `output` | Text, Audio, Chat, Safety | LLM as Judge | | [**Instruction Adherence**](/docs/evaluation/builtin/instruction-adherence) | Assesses how closely the output follows prompt instructions. | `input`, `output` | Text, Audio, Chat, Hallucination | LLM as Judge | | [**Summary Quality**](/docs/evaluation/builtin/summary-quality) | Evaluates if a summary captures main points and achieves appropriate length. | `input`, `output` | Text, Audio, Image, RAG & Retrieval | LLM as Judge | -| [**Translation Accuracy**](/docs/evaluation/builtin/translation-accuracy) | Evaluates translation quality, accuracy, and cultural appropriateness. | `output`, `expected_response` | Text, Audio, RAG & Retrieval | LLM as Judge | +| [**Translation Accuracy**](/docs/evaluation/builtin/translation-accuracy) | Evaluates translation quality, accuracy, and cultural appropriateness. | `input`, `output` | Text, Audio, RAG & Retrieval | LLM as Judge | | [**No LLM Reference**](/docs/evaluation/builtin/no-llm-reference) | Ensures output does not reference being an LLM or OpenAI model. | `output` | Text, Audio, Chat, Safety | LLM as Judge | | [**No Apologies**](/docs/evaluation/builtin/no-apologies) | Checks if the model unnecessarily apologizes. | `output` | Text, Audio, Chat | LLM as Judge | | [**Is Polite**](/docs/evaluation/builtin/is-polite) | Ensures output maintains a respectful and non-aggressive tone. | `output` | Text, Audio, Chat | LLM as Judge | @@ -155,7 +155,7 @@ Generated images and document extraction | Eval | Description | Required Inputs | Use Cases | Evaluation Method | |------|-------------|-----------------|-----------|-------------------| -| [**Caption Hallucination**](/docs/evaluation/builtin/caption-hallucination) | Detects hallucinated or fabricated details in image captions. | `instruction`, `output` | Image, RAG & Retrieval, Hallucination | LLM as Judge | +| [**Caption Hallucination**](/docs/evaluation/builtin/caption-hallucination) | Detects hallucinated or fabricated details in image captions. | `image`, `caption` | Image, RAG & Retrieval, Hallucination | LLM as Judge | | [**Synthetic Image Evaluator**](/docs/evaluation/builtin/synthetic-image-evaluator) | Evaluates synthetic or AI-generated images against criteria. | `image`, `instruction` | Image | LLM as Judge | | [**OCR Evaluation**](/docs/evaluation/builtin/ocr-evaluation) | Evaluates the accuracy of optical character recognition (OCR) output. | `input_pdf`, `json_content` | Text, PDF / Document | LLM as Judge | | [**FID Score**](/docs/evaluation/builtin/fid-score) | Computes the Fréchet Inception Distance between two sets of images; lower scores indicate more similar image distributions. | `real_images`, `fake_images` | Image | Statistical Metric | diff --git a/src/pages/docs/evaluation/builtin/pii.mdx b/src/pages/docs/evaluation/builtin/pii.mdx index 410726917..93d8507f6 100644 --- a/src/pages/docs/evaluation/builtin/pii.mdx +++ b/src/pages/docs/evaluation/builtin/pii.mdx @@ -45,22 +45,12 @@ print(result.score) print(result.reason) ``` -```typescript TypeScript -import { evaluate } from "@future-agi/ai-evaluation"; - -const result = await evaluate( - "pii", - { - input: "My name is John Doe." - }, - { modelName: "turing_flash" } -); - -console.log(result); -``` - + +The released TypeScript SDK (`@future-agi/sdk` 0.2.0) does not export the standalone `evaluate()` API. The legacy TypeScript evaluation snippet is not supported by that package; use the Python example above for this API. + + ## When to use Run PII Detection wherever generated text could carry personal data that shouldn't leave the system. diff --git a/src/pages/docs/evaluation/builtin/translation-accuracy.mdx b/src/pages/docs/evaluation/builtin/translation-accuracy.mdx index 1e71669cf..c696f9b6e 100644 --- a/src/pages/docs/evaluation/builtin/translation-accuracy.mdx +++ b/src/pages/docs/evaluation/builtin/translation-accuracy.mdx @@ -47,23 +47,12 @@ print(result.score) print(result.reason) ``` -```typescript TypeScript -import { evaluate } from "@future-agi/ai-evaluation"; - -const result = await evaluate( - "translation_accuracy", - { - input: "Hello, how are you?", - output: "¡Hola, cómo estás?" - }, - { modelName: "turing_flash" } -); - -console.log(result); -``` - + +The released TypeScript SDK (`@future-agi/sdk` 0.2.0) does not export the standalone `evaluate()` API. The legacy TypeScript evaluation snippet is not supported by that package; use the Python example above for this API. + + ## When to use Run Translation Accuracy wherever content moves from one language to another and meaning needs to survive the transfer. diff --git a/src/pages/docs/evaluation/guides/cicd.mdx b/src/pages/docs/evaluation/guides/cicd.mdx index 4f806c63a..03a39fd98 100644 --- a/src/pages/docs/evaluation/guides/cicd.mdx +++ b/src/pages/docs/evaluation/guides/cicd.mdx @@ -13,7 +13,7 @@ Future AGI never triggers or runs your pipeline. You call the SDK from your own ## When to use -- **Gate PRs on quality**: Run evals on every PR so regressions in tone, factual consistency, or custom metrics block or flag merges before they land. +- **Review quality on every PR**: Report tone, factual consistency, and custom metrics so reviewers can identify regressions before merging. - **Compare versions in CI**: Submit evaluations with a version tag and compare results across versions in one place. - **Automate quality reporting**: Post eval results as a PR comment so reviewers see model performance without leaving GitHub. - **Repeatable checks**: Use the same eval templates and inputs in CI so every run is directly comparable. @@ -24,7 +24,7 @@ Future AGI never triggers or runs your pipeline. You call the SDK from your own - A Future AGI account with API key and secret key - A CI system that can run Python (GitHub Actions, GitLab CI, Jenkins, or any runner with Python and network access) -- The `ai-evaluation` package (`pip install ai-evaluation>=0.1.7`) +- The `futureagi` package (`pip install futureagi`) ### Required secrets @@ -41,8 +41,10 @@ Set these as environment variables or in your CI's secret store. Do not commit t | Variable | Description | Default | |---|---|---| | `PROJECT_NAME` | Future AGI project name | `Voice Agent` | -| `VERSION` | Current version identifier | `v0.1.0` | -| `COMPARISON_VERSIONS` | Comma-separated versions to compare against | *(empty)* | +| `VERSION` | New version identifier, unique within the project for each submission | Generated by the workflow from the commit SHA, run ID, and run attempt | +| `COMPARISON_VERSIONS` | Comma-separated, previously submitted versions to compare against | *(empty)* | + +A project cannot accept the same `VERSION` twice. The workflow below generates a new identifier for each run and rerun. On another runner or when running the script manually, set `VERSION` to a fresh identifier before each submission. Keep baseline identifiers in `COMPARISON_VERSIONS`; do not reuse a baseline as the new `VERSION`. --- @@ -50,6 +52,8 @@ Set these as environment variables or in your CI's secret store. Do not commit t Set this up once. On every pull request, GitHub Actions installs the SDK, runs your eval suite, tags the results with a version, and posts a comparison table back to the PR. +This is a **quality-reporting workflow**, not an automatic score-threshold merge gate. The script exits nonzero if initialization, submission, or polling fails, but it does not fail on low scores or regressions. A completed run means processing finished, not that quality passed. To enforce a quality gate, add your own threshold or regression check after retrieving results, exit nonzero when it fails, and make the job a required check in your repository. + This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, and the `post_github_comment` function all use GitHub's API. The `evaluate_pipeline` and `get_pipeline_results` calls are identical on any runner, so to use GitLab CI, Jenkins, or another system, keep the eval script and swap `post_github_comment` (and the workflow file) for your platform's equivalent. @@ -62,7 +66,7 @@ This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, an pandas requests tabulate - ai-evaluation>=0.1.7 + futureagi python-dotenv ``` @@ -178,23 +182,30 @@ This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, an print(f"Timeout after {max_wait_time} seconds") return None + def flatten_summary(summary): + """Keep scalar scores and expand nested choice percentages.""" + flattened = {} + for metric, value in summary.items(): + if isinstance(value, dict): + for choice, percentage in value.items(): + flattened[f"{metric}_{choice}"] = percentage + else: + flattened[metric] = value + return flattened + def format_results(evaluation_runs, current_version): """Formats results into a markdown comparison table.""" if not evaluation_runs: return "No evaluation results found." - version_data = {run.get('version'): run.get('results_summary', {}) - for run in evaluation_runs} + version_data = { + run['version']: flatten_summary(run.get('results_summary', {})) + for run in evaluation_runs + } - # Collect all metrics - all_metrics = set() - for run in evaluation_runs: - for key, value in run.get('results_summary', {}).items(): - if isinstance(value, dict): - for sub_key in value.keys(): - all_metrics.add(f"{key}_{sub_key}") - else: - all_metrics.add(key) + # Collect and look up metrics using the same flattened keys. + all_metrics = {metric for results in version_data.values() + for metric in results} comparison_data = [] for metric in sorted(all_metrics): @@ -215,7 +226,7 @@ This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, an def main(): project_name = os.getenv("PROJECT_NAME", "Voice Agent") - version = os.getenv("VERSION", "v0.1.0") + version = os.environ["VERSION"] comparison_versions = os.getenv("COMPARISON_VERSIONS", "") try: @@ -225,7 +236,7 @@ This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, an ) except Exception as e: post_github_comment(f"## Evaluation Failed\n\n**Reason:** Failed to initialize evaluator: {e}") - return + raise SystemExit(1) try: result = evaluator.evaluate_pipeline( @@ -235,16 +246,16 @@ This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, an ) if not result.get('status'): post_github_comment(f"## Evaluation Failed\n\n**Reason:** {result}") - return + raise SystemExit(1) except Exception as e: post_github_comment(f"## Evaluation Failed\n\n**Reason:** Error submitting evaluation: {e}") - return + raise SystemExit(1) all_runs = poll_for_completion(evaluator, project_name, version, comparison_versions) if not all_runs: post_github_comment("## Evaluation Failed\n\n**Reason:** Timed out or failed during processing") - return + raise SystemExit(1) comment_body = format_results(all_runs, version) post_github_comment(comment_body) @@ -252,6 +263,25 @@ This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, an if __name__ == "__main__": main() ``` + + Choice metrics such as `tone` return percentages nested under the template name. The formatter flattens each version's summary before building the table, so `tone.neutral` is displayed as **Tone Neutral**, not `N/A`. Check it without an API call using this fixture: + + ```python + from evaluate_pipeline import flatten_summary, format_results + + summary = {"tone": {"neutral": 100.0}, "groundedness": 0.75} + assert flatten_summary(summary) == { + "tone_neutral": 100.0, + "groundedness": 0.75, + } + report = format_results( + [{"version": "fixture", "results_summary": summary}], + current_version="fixture", + ) + assert "Tone Neutral" in report + assert "N/A" not in report + print(report) + ``` @@ -293,7 +323,7 @@ This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, an FI_API_KEY: ${{ secrets.FI_API_KEY }} FI_SECRET_KEY: ${{ secrets.FI_SECRET_KEY }} PROJECT_NAME: ${{ vars.PROJECT_NAME || 'Voice Agent' }} - VERSION: ${{ vars.VERSION || 'v0.1.0' }} + VERSION: ${{ github.sha }}-${{ github.run_id }}-${{ github.run_attempt }} COMPARISON_VERSIONS: ${{ vars.COMPARISON_VERSIONS || '' }} ``` @@ -303,7 +333,7 @@ This walkthrough is GitHub Actions specific: the workflow file, `PAT_GITHUB`, an - In your repository settings, add the [secrets and variables](#prerequisites) the workflow reads: `FI_API_KEY`, `FI_SECRET_KEY`, and `PAT_GITHUB` as secrets, and `PROJECT_NAME`, `VERSION`, and `COMPARISON_VERSIONS` as repository variables. Never commit them to the repo. + In your repository settings, add the [secrets and variables](#prerequisites) the workflow reads: `FI_API_KEY`, `FI_SECRET_KEY`, and `PAT_GITHUB` as secrets, and `PROJECT_NAME` and `COMPARISON_VERSIONS` as repository variables. The workflow generates `VERSION`; do not set it to a fixed repository variable. Never commit secrets to the repo. @@ -337,7 +367,7 @@ Submits a list of eval configs tagged to a version. Each config has an `eval_tem ```python result = evaluator.evaluate_pipeline( project_name="my-project", - version="v0.1.5", + version=os.environ["VERSION"], # Fresh identifier for this submission eval_data=eval_data, ) ``` @@ -345,7 +375,7 @@ result = evaluator.evaluate_pipeline( | Parameter | Description | |---|---| | `project_name` | Your project identifier | -| `version` | Version tag for this run (e.g. branch name or commit SHA) | +| `version` | New tag unique within the project for every submission, including reruns (for example, commit SHA + workflow run ID + run attempt) | | `eval_data` | List of evaluation configurations (template, model, inputs) | ### get_pipeline_results diff --git a/src/pages/docs/evaluation/guides/custom-evals.mdx b/src/pages/docs/evaluation/guides/custom-evals.mdx index 8456f7b83..3f3cbfb8b 100644 --- a/src/pages/docs/evaluation/guides/custom-evals.mdx +++ b/src/pages/docs/evaluation/guides/custom-evals.mdx @@ -5,7 +5,7 @@ description: "Write your own eval when no built-in template fits" Every AI product has its own definition of a good response. When no [built-in template](/docs/evaluation/builtin) fits, a custom eval lets you encode that definition yourself: write the criteria once, in plain language, and Future AGI scores every response against it. -A custom eval is a [template](/docs/evaluation/concepts/eval-templates) you define. Pick the [type](/docs/evaluation/concepts/eval-types) that fits: an **Agent Evaluator** for multi-step reasoning, an **LLM-as-Judge** for a single pass, or a **Code Eval** for a deterministic rule. Then write the criteria and choose what the result looks like. Once saved, it works exactly like a built-in: use it on a dataset, a trace, a simulation, or from the SDK, by its name. This guide builds a `hateful_speech` Agent Evaluator as the example. +A custom eval is a [template](/docs/evaluation/concepts/eval-templates) you define. Pick the [type](/docs/evaluation/concepts/eval-types) that fits: an **Agent Evaluator** for multi-step reasoning or an **LLM-as-Judge** for a single pass. Custom **Code Eval** creation is unavailable in the builder. Then write the criteria and choose what the result looks like. Once saved, it works exactly like a built-in: use it on a dataset, a trace, a simulation, or from the SDK, by its name. This guide builds a `hateful_speech` Agent Evaluator as the example. ## When to use @@ -23,7 +23,7 @@ On the Evals list, click **Create evals** at the top right. ## Name the eval and pick its type -The builder opens on **Single**, a single eval rather than a [composite](/docs/evaluation/concepts/composite-evals). Give it a unique **Eval Name**, lowercase letters, numbers, hyphens, and underscores only, like `hateful_speech`. Then pick the eval type: **Agents**, **LLM-As-A-Judge**, or **Code**. This example uses **Agents**. +The builder opens on **Single**, a single eval rather than a [composite](/docs/evaluation/concepts/composite-evals). Give it a unique **Eval Name**, lowercase letters, numbers, hyphens, and underscores only, like `hateful_speech`. For this model-based example, pick **Agents**; **LLM-As-A-Judge** is the single-pass alternative. There is no **Code** tab in this builder. Eval details on the Single tab with the name hateful_speech, the Agents eval type, the Mustache variable format, the instructions, and Pass/fail as the output type *Set the name, eval type, instructions, and output type on one screen* @@ -91,12 +91,12 @@ Click **Save Evaluation**. The custom eval appears in the Evals list under your ## Create it from the SDK -Create the template with a POST to the Future AGI API, then run it with the `Evaluator` from the `ai-evaluation` SDK. +Create the template with a POST to the Future AGI API, then run it with the `Evaluator` from the `futureagi` SDK. ### Install the SDK ```bash -pip install ai-evaluation +pip install futureagi ``` ### Create the template diff --git a/src/pages/docs/evaluation/guides/guardrails.mdx b/src/pages/docs/evaluation/guides/guardrails.mdx index 0e5cfc683..93877239b 100644 --- a/src/pages/docs/evaluation/guides/guardrails.mdx +++ b/src/pages/docs/evaluation/guides/guardrails.mdx @@ -14,7 +14,7 @@ If your traffic already flows through Agent Command Center, you can turn on chec Protect is the in-code path: you call the check exactly where your application handles text, so the verdict comes back inline and nothing ships until it passes. -Before running: install the SDK (`pip install ai-evaluation` or `npm install @future-agi/ai-evaluation`) and set `FI_API_KEY` / `FI_SECRET_KEY` in your environment. +Before running: install the SDK (`pip install futureagi` or `npm install @future-agi/ai-evaluation`) and set `FI_API_KEY` / `FI_SECRET_KEY` in your environment. ### Run your first check diff --git a/src/pages/docs/evaluation/reference/sdk-api.mdx b/src/pages/docs/evaluation/reference/sdk-api.mdx index bb5b20c6c..8130197de 100644 --- a/src/pages/docs/evaluation/reference/sdk-api.mdx +++ b/src/pages/docs/evaluation/reference/sdk-api.mdx @@ -5,14 +5,22 @@ description: "Run evaluations from code with the evaluate() function" ## Evals from code -Everything you can do with evals in the platform, you can also do from code. The `ai-evaluation` package gives you a single `evaluate()` function that runs a metric locally, in the cloud, or as an LLM judge, and returns a result you can assert on. +Everything you can do with evals in the platform, you can also do from code. The `futureagi` package gives you a single `evaluate()` function that runs a metric locally, in the cloud, or as an LLM judge, and returns a result you can assert on. This page is the reference for reaching evals programmatically. For the full SDK docs, see the [SDK evals section](/docs/sdk/evals). ## Install and authenticate +For cloud-only evaluations: + +```bash +pip install futureagi +``` + +For the local metrics and LLM-as-Judge examples below, use Python 3.11+ and install the evals extra: + ```bash -pip install ai-evaluation +pip install "futureagi[evals]" ``` ```python diff --git a/src/pages/docs/falcon-ai/guides/chat-with-falcon-ai.mdx b/src/pages/docs/falcon-ai/guides/chat-with-falcon-ai.mdx index e96508f60..fc9230c63 100644 --- a/src/pages/docs/falcon-ai/guides/chat-with-falcon-ai.mdx +++ b/src/pages/docs/falcon-ai/guides/chat-with-falcon-ai.mdx @@ -7,7 +7,7 @@ Falcon AI is the AI copilot built into the Future AGI dashboard, reachable from ## Open Falcon AI -Click **Falcon AI** in the dashboard's navigation sidebar. It opens the full-page view at `/dashboard/falcon-ai`, with your conversations listed down the left and the chat itself in the center. +In the dashboard's navigation sidebar, click **Falcon AI** (labelled **Chat** in newer versions). It opens the full-page Falcon AI view at `/dashboard/falcon-ai`, with your conversations listed down the left and the chat itself in the center. The Falcon AI full page with the Falcon AI nav item selected in the sidebar, three past conversations listed on the left, and an answer in the center *Every conversation you have ever had is one click away in the left rail, which is the full page's whole advantage over the panel* diff --git a/src/pages/docs/falcon-ai/index.mdx b/src/pages/docs/falcon-ai/index.mdx index 67d9466ea..bf35a11cb 100644 --- a/src/pages/docs/falcon-ai/index.mdx +++ b/src/pages/docs/falcon-ai/index.mdx @@ -11,7 +11,7 @@ description: "Where Falcon AI lives in the dashboard and how it differs from the It's available on Future AGI Cloud and Enterprise plans. Falcon AI runs on two surfaces: a full page, which carries the conversation list and lets you rename or delete conversations, and a side panel, which keeps the page underneath in view. -- **Full page:** open the **Falcon AI** entry at the top of the left navigation, or land there automatically right after logging in +- **Full page:** open **Falcon AI** in the left navigation (labelled **Chat** in newer versions). It opens the full-page Falcon AI surface at `/dashboard/falcon-ai` - **Side panel:** from any other page in the dashboard, click the button in its bottom-right corner, or press `⌘K` (Mac) or `Ctrl+K` (Windows/Linux) ## What you can use it for diff --git a/src/pages/docs/faq.mdx b/src/pages/docs/faq.mdx index 75c23abec..121911605 100644 --- a/src/pages/docs/faq.mdx +++ b/src/pages/docs/faq.mdx @@ -177,7 +177,7 @@ PDF, DOCX, DOC, TXT, and RTF. Maximum 5MB per file. See [Understanding Knowledge **Where do I find my API keys?** -Go to Settings > API Keys. See [API Keys](/docs/admin-settings/api-keys). +Sign in interactively and open [Build > Keys](https://app.futureagi.com/dashboard/keys) to create a personal key. Active organization members can create their own keys; organization Owners/Admins can also manage other members' keys. Gateway credentials are separate: open **AI Gateway** in the product switcher, then **API keys**. See [API Keys](/docs/admin-settings/api-keys). **How do I manage team members?** @@ -201,4 +201,4 @@ Some built-in evaluations require an evaluator model. Pass `model_name="turing_f **I can't find my API keys.** -Go to [Settings > API Keys](https://app.futureagi.com/dashboard/keys). You need the Owner role. See [API Keys](/docs/admin-settings/api-keys). +Sign in interactively as an active organization member, then go to [Build > Keys](https://app.futureagi.com/dashboard/keys) and select **Add API Key**. Save both values from the one-time creation dialog; the list cannot reveal them again. You do not need the Owner role to create your own key. Organization Owners/Admins can manage other members' keys. For gateway requests, use **AI Gateway > API keys** instead of the personal API/secret pair. See [API Keys](/docs/admin-settings/api-keys). diff --git a/src/pages/docs/get-started/connect-no-code-agents.mdx b/src/pages/docs/get-started/connect-no-code-agents.mdx index 45b17b67e..cf6c3041b 100644 --- a/src/pages/docs/get-started/connect-no-code-agents.mdx +++ b/src/pages/docs/get-started/connect-no-code-agents.mdx @@ -3,31 +3,39 @@ title: "Connect your no code agent" description: "Sync your no code agent definition in Future AGI to run simulations" --- -An agent definition tells Future AGI which agent you're testing and how to reach it. This page walks you through creating your first **voice** agent definition in **Simulate**, so you can start running simulations against it +Connect an existing agent from **Agents → Connected agents**, then use an **Environment** to configure and run simulations. This guide starts with a hosted **voice** agent on **Vapi** or **Retell**. -Any voice agent can be simulated as long as it's reachable by a **phone number**. **Vapi** and **Retell** are natively supported, so you can sync the agent's name and prompt straight from them +## Prerequisites - -Building a **chat** agent? Chat simulations run through the **SDK**, not this form. See [Chat Simulation Using SDK](/docs/simulation/guides/run-chat-simulation) - +- A Future AGI account +- A Vapi **Assistant ID** or Retell **Agent ID**, and the matching provider **API key** +- The contact information required by your chosen voice connection mode -Every definition is versioned, so once you save it you can run tests against a specific version, compare versions, or roll back +## 1. Open the connection flow -## Prerequisites +Open **Agents → Connected agents**. In the Agents creation menu, select **Connect agents**. If the list is empty, you can also click **Connect an agent**. Both open the build tab of **Simulate → Environment**. -- A Future AGI account -- A **voice agent reachable via a phone number** -- If the agent is on **Vapi** or **Retell**: its **Assistant ID** and provider **API key** (these also enable **Sync**) +## 2. Connect your hosted agent -## 1. Open Agent Definition +Choose **Hosted platform**, select **Voice** as the agent type, and choose **Vapi** or **Retell**. Enter the provider's agent ID and API key, then complete the contact information and other required fields shown in the form. -In the dashboard, open **Simulate** from the sidebar, select **Agent Definition**, and click **Start agent testing**: +Click **Run preflight** and resolve any failed checks. Once the form and preflight allow it, click **Build environment**. -![Welcome to Agent Simulation screen with the Start Agent Testing button in Future AGI Simulate](/images/docs/get-started/connect-no-code-agents/agent-simulation-welcome.png) +## 3. Run your first simulation -This opens the **Create new agent definition** wizard, which has three steps: **Basic Info**, **Configuration**, and **Behaviour** +When the environment is ready, review its **Scenarios** and **Evaluations** tabs. Resolve any setup blockers, then click **Run simulation**. Open **Runs** to inspect the results. -## 2. Add the basic information + +The older **Agent Definition** wizard and standalone **Run Simulation** flow below are legacy features. They are unavailable to Environments-only organizations. The old Agent Definition list now redirects to **Connected agents**; it is not a sidebar entry. Use the Environment flow above for new connections. + + +## Legacy Agent Definition wizard + +The remaining instructions and screenshots apply only if your organization still has access to the legacy **Create new agent definition** page. That wizard has three steps: **Basic Info**, **Configuration**, and **Behaviour**. + +![Legacy Agent Simulation welcome screen](/images/docs/get-started/connect-no-code-agents/agent-simulation-welcome.png) + +### Add the basic information On the **Basic Info** step, fill in the basics: @@ -39,7 +47,7 @@ On the **Basic Info** step, fill in the basics: ![Basic Info step of the wizard with agent type, agent name, and language fields](/images/docs/get-started/connect-no-code-agents/basic-information.png) -## 3. Configure and connect your agent +### Configure and connect your agent On the **Configuration** step, choose the provider and set how Future AGI reaches your agent @@ -56,7 +64,7 @@ Then set **Contact Information**, since voice agents are reached by phone: enter For **Outbound** agents, both the **Assistant ID** and **API Key** are required. Otherwise saving fails with a validation error -## 4. Define behavior and save +### Define behavior and save On the **Behaviour** step, set how your agent thinks and responds. Add the **system prompt** and, if needed, attach a **Knowledge Base** so evals can check the agent's answers against your real content: FAQs, SOPs, or product docs @@ -64,13 +72,11 @@ Review the **Summary** panel, enter a **commit message** (e.g. "Initial support ![Behaviour step with the system prompt, knowledge base, commit message, and the configuration summary panel](/images/docs/get-started/connect-no-code-agents/behaviour-and-summary.png) -## Verify - -Your agent now appears in the **Agent Definition** list. Open it to see version **v1** and its configuration +### Verify the legacy definition -Congratulations! You've created your first agent definition!🎉 +After saving, inspect the definition's version and configuration on its detail page. The old **Agent Definition** list link now opens **Connected agents**. -## Troubleshooting +### Troubleshoot the legacy wizard Save not going through? Try checking these: diff --git a/src/pages/docs/get-started/migrate/langfuse.mdx b/src/pages/docs/get-started/migrate/langfuse.mdx index 59929e66f..f677723d0 100644 --- a/src/pages/docs/get-started/migrate/langfuse.mdx +++ b/src/pages/docs/get-started/migrate/langfuse.mdx @@ -12,8 +12,8 @@ You can turn the import off at any time, too. Anything already imported stays in ## Prerequisites - A **Langfuse** account with at least one project that already has traces -- Your Langfuse **Public Key** (`pk-lf-...`) and **Secret Key** (`sk-lf-...`), from **Langfuse → Settings → API Keys** -- Atleast **Member** level access in your Future AGI workspace. +- Your Langfuse **Public Key** (`pk-lf-...`) and **Secret Key**, from **Langfuse → Settings → API Keys** +- An organization **Owner** or **Admin** role, or an **Admin** role in the current Future AGI workspace. Ordinary members have read access to existing connections but must ask an admin to create or change one. For a detailed breakdown of Role based access, see [Roles & Permissions](/docs/roles-and-permissions). @@ -34,7 +34,7 @@ On the **Credentials** step, paste the values from Langfuse: |---|---| | **Host URL** | `https://cloud.langfuse.com`, or your self-hosted Langfuse URL | | **Public Key** | Your Langfuse public key (`pk-lf-...`) | -| **Secret Key** | Your Langfuse secret key (`sk-lf-...`) | +| **Secret Key** | Your Langfuse secret key | Self-hosted Langfuse with a private certificate? Expand **Advanced Settings** and paste your **CA certificate (PEM)**. Then click **Validate & Continue**, and Future AGI verifies the keys before moving on @@ -85,11 +85,11 @@ Congratulations! Your Langfuse traces are now flowing into Future AGI 🎉 | Token counts and cost | Span attributes | | Scores | Evaluation scores, trace-level and span-level, numeric and categorical | -The sync is idempotent, so running it again never creates duplicates. New scores and metadata edits on existing Langfuse traces are picked up on the next cycle +Sync upserts the traces it fetches, updating matching imported records. After the initial backfill, each cycle queries trace timestamps from five minutes before the last successful sync cursor through the start of the current cycle. This lookback can pick up recent traces again, but it is not a historical-change feed: new scores or metadata edits on older, already-complete traces outside that window are not automatically discovered. A separate repair pass revisits imported traces with no spans; it does not refresh all historical traces. ### **Manage the connection** -From the connection's detail page you can: +From the connection's detail page, organization Owners/Admins and Admins of the current workspace can manage the connection. Ordinary members retain read and **Sync Now** access: - **Sync Now**: pull the latest traces immediately, with a 60-second cooldown between manual syncs - **Edit**: update the display name, API keys, host URL, or sync interval. Changing keys triggers re-validation diff --git a/src/pages/docs/get-started/route-your-first-llm-request.mdx b/src/pages/docs/get-started/route-your-first-llm-request.mdx index e375ff371..c080f0376 100644 --- a/src/pages/docs/get-started/route-your-first-llm-request.mdx +++ b/src/pages/docs/get-started/route-your-first-llm-request.mdx @@ -16,7 +16,7 @@ You can switch providers later without touching your code, too ## 1. Create your API key -Create an Agent Command Center API key in **Dashboard → Settings → API Keys** and copy the value that starts with `sk-agentcc-`: +In the dashboard's product switcher, open **AI Gateway**, then select **API keys** under **Providers, Tools & Keys**. Create a gateway API key and copy its value. This is separate from the personal API Key and Secret Key pair at **Build → Keys**; that pair cannot authenticate gateway requests.
Add API key modal in Agent Command Center @@ -61,7 +61,7 @@ npm install openai Set your Agent Command Center key in your terminal: ```bash -export AGENTCC_API_KEY="sk-agentcc-your-api-key-here" +export AGENTCC_API_KEY="your-gateway-api-key" ``` ## 5. Point the SDK at the gateway @@ -76,7 +76,7 @@ from openai import OpenAI client = OpenAI( base_url="https://gateway.futureagi.com/v1", # 1. point at the gateway - api_key=os.environ["AGENTCC_API_KEY"], # 2. use your sk-agentcc- key + api_key=os.environ["AGENTCC_API_KEY"], # 2. use your gateway key ) response = client.chat.completions.create( @@ -91,7 +91,7 @@ import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://gateway.futureagi.com/v1", // 1. point at the gateway - apiKey: process.env.AGENTCC_API_KEY, // 2. use your sk-agentcc- key + apiKey: process.env.AGENTCC_API_KEY, // 2. use your gateway key }); const response = await client.chat.completions.create({ @@ -133,7 +133,7 @@ Congratulations! You've successfully routed your first request!🎉 Not seeing a response? Try checking these: -- **Authentication error**: confirm your key starts with `sk-agentcc-` (Settings → API Keys) +- **Authentication error**: confirm you are using the gateway key created at **AI Gateway → API keys**, not the personal credentials from **Build → Keys**. Open **AI Gateway** from the product switcher to manage gateway keys - **Model or provider error**: make sure that provider is configured in Agent Command Center → Providers - **Wrong base URL**: it must be `https://gateway.futureagi.com/v1` diff --git a/src/pages/docs/get-started/send-your-first-trace.mdx b/src/pages/docs/get-started/send-your-first-trace.mdx index a30cb80f3..f0d7b891e 100644 --- a/src/pages/docs/get-started/send-your-first-trace.mdx +++ b/src/pages/docs/get-started/send-your-first-trace.mdx @@ -12,12 +12,12 @@ You can add [custom spans](/docs/cookbook/quickstart/manual-tracing), too. ## Prerequisites - A Future AGI account and your **`FI_API_KEY`** and **`FI_SECRET_KEY`** (Dashboard → Build → Keys) -- Python 3.10+ +- Python 3.10–3.13 for the Python example (`traceAI-openai` requires Python below 3.14) - An OpenAI API key -Find both keys at **Dashboard → Build → Keys**. Copy the **API Key** (`FI_API_KEY`) and **Secret Key** (`FI_SECRET_KEY`): +Open **Dashboard → Build → Keys → Add API Key** and create a personal key. Copy both the **API Key** (`FI_API_KEY`) and **Secret Key** (`FI_SECRET_KEY`) from the one-time creation dialog before closing it. Existing entries in the Keys list cannot reveal these values again. -![Future AGI Keys page showing where to copy the API Key (FI_API_KEY) and Secret Key (FI_SECRET_KEY)](/images/docs/get-started/send-your-first-trace/keys-page.png) +![Future AGI Keys page](/images/docs/get-started/send-your-first-trace/keys-page.png) ## 1. Install traceAI @@ -74,7 +74,6 @@ print(response.choices[0].message.content) ```typescript JS/TS import { register, ProjectType } from "@traceai/fi-core"; import { OpenAIInstrumentation } from "@traceai/openai"; -import { registerInstrumentations } from "@opentelemetry/instrumentation"; import OpenAI from "openai"; // Connect to Future AGI and create (or reuse) a project @@ -83,11 +82,10 @@ const tracerProvider = register({ projectName: "my-llm-app", }); -// Auto-instrument OpenAI: every call is now traced -registerInstrumentations({ - instrumentations: [new OpenAIInstrumentation({})], - tracerProvider, -}); +// Explicitly instrument the imported SDK for ESM/TypeScript +const instrumentation = new OpenAIInstrumentation({}); +instrumentation.setTracerProvider(tracerProvider); +instrumentation.manuallyInstrument(OpenAI); // Use OpenAI exactly as you normally would const client = new OpenAI(); @@ -142,7 +140,7 @@ Congratulations! You've successfully sent your first trace!🎉 Not seeing your traces? Try checking these: -- **Authentication error**: re-check `FI_API_KEY` and `FI_SECRET_KEY` against Dashboard → Build → Keys +- **Authentication error**: check that `FI_API_KEY` and `FI_SECRET_KEY` are the pair saved from the creation dialog. If you did not save both values, create a new personal key at **Dashboard → Build → Keys → Add API Key** and copy both before closing - **No trace in Tracing**: let the script finish. Traces flush as the program exits, so don't stop it early - **Wrong place**: confirm you're on the **Tracing** tab of platform and in the `my-llm-app` project diff --git a/src/pages/docs/integrations/coding-agents/gemini-cli.mdx b/src/pages/docs/integrations/coding-agents/gemini-cli.mdx index c9dcd2e9a..503057fc8 100644 --- a/src/pages/docs/integrations/coding-agents/gemini-cli.mdx +++ b/src/pages/docs/integrations/coding-agents/gemini-cli.mdx @@ -139,7 +139,7 @@ Model messages and tool arguments/results then appear as input and output in Fut **Port 4318 is already in use.** Set `GEMINI_OTLP_LISTEN=127.0.0.1:24318` before starting the collector and change Gemini CLI's `otlpEndpoint` to `http://127.0.0.1:24318`. -**The collector reports an export error.** For `401` or `403`, check your Future AGI keys. For `400`, keep `compression: none` in the exporter: the Future AGI endpoint requires uncompressed OTLP requests for this setup. +**The collector reports an export error.** For `401` or `403`, check your Future AGI keys. For `400`, check the response and that the content type matches the decoded OTLP body. HTTP accepts uncompressed bodies or a single `gzip`/`x-gzip` layer with a matching `Content-Encoding` header, so `compression: none` is optional. For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it). **Model spans have no token counts.** Set `telemetry.traces` to `true` and restart Gemini CLI. diff --git a/src/pages/docs/integrations/frameworks/tanstack-ai.mdx b/src/pages/docs/integrations/frameworks/tanstack-ai.mdx index ddc25ae5b..084c91d76 100644 --- a/src/pages/docs/integrations/frameworks/tanstack-ai.mdx +++ b/src/pages/docs/integrations/frameworks/tanstack-ai.mdx @@ -63,9 +63,9 @@ export const GEN_AI_SPAN_KIND = "gen_ai.span.kind"; * sets the project_name and project_type resource attributes the collector * requires. A batch without project_name is dropped. * - * `batch: true` is not passed: with @traceai/fi-core 1.0.0 on - * @opentelemetry/sdk-trace(-base) 2.11.0 it does not attach the batch - * processor, so spans go through register()'s default SimpleSpanProcessor. + * This recipe leaves `batch` unset and uses the default SimpleSpanProcessor. + * @traceai/fi-core 1.2.3 supports `batch: true`; if you upgrade and enable + * batching, await forceFlush() before returning from a short-lived invocation. * * `setGlobalTracerProvider: false`: the recipe passes this provider's tracer * to otelMiddleware, so it never needs the global provider, and leaving the @@ -520,7 +520,7 @@ TanStack starts its chat span under whatever span is active. If your app has its - **Tokens are counted once.** TanStack also writes the sum of every model call's usage onto the chat span. Future AGI sums `gen_ai.usage.*` over every span of a trace for the trace view, sessions, dashboards and cost, so the recipe moves the chat span's copy to `tanstack.ai.root_usage.*` whenever the model-call spans carry usage. If no model call reported usage, the chat span keeps it. - **Sessions.** With `threadIdAsSession: true` and a `threadId` on `chat()`, every span carries `session.id` and Future AGI groups the conversation. Without a `threadId`, TanStack generates `thread--` per call, so the recipe sets no session. - **Exporter down.** With nothing listening at `FI_BASE_URL`, `chat()` and the route still answer; `flushTraces()` logs `[futureagi] span export failed` and never throws. A collector that accepts but never answers costs the request at most the 2 s flush bound. -- **Batching.** `register({ batch: true })` has no effect with `@traceai/fi-core` 1.0.0 on OpenTelemetry SDK 2.x, so the recipe uses the default per-span export. With a very slow collector the OTLP exporter's limit of 30 exports in flight can drop spans (about 8 concurrent chats of 4 spans). +- **Batching.** This recipe leaves `batch` unset, using the default per-span export (`batch: false`). In `@traceai/fi-core` 1.2.3, `register({ batch: true })` creates an OpenTelemetry `BatchSpanProcessor`; no separate provider is needed. If you upgrade and enable batching, await `tracerProvider.forceFlush()` after active spans end and before returning or exiting, so queued or in-flight spans are not cut off. **Historical compatibility:** the recipe's pinned 1.0.0 package did not attach a batch processor on OpenTelemetry SDK 2.x. With that per-span export and a very slow collector, the OTLP exporter's limit of 30 exports in flight can drop spans (about 8 concurrent chats of 4 spans). - **Global provider untouched.** See [Apps that already run OpenTelemetry](#apps-that-already-run-opentelemetry). ## Limitations diff --git a/src/pages/docs/integrations/traceai/bedrock-agentcore.mdx b/src/pages/docs/integrations/traceai/bedrock-agentcore.mdx index ee1baab14..1f111ee8d 100644 --- a/src/pages/docs/integrations/traceai/bedrock-agentcore.mdx +++ b/src/pages/docs/integrations/traceai/bedrock-agentcore.mdx @@ -150,7 +150,7 @@ export OTEL_SERVICE_NAME=order-agent export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,gen_ai_span_attributes_only,gen_ai_unredacted_attributes= ``` -The trace-specific endpoint is the **full URL**, not an origin. Remove any conflicting `OTEL_EXPORTER_OTLP_TRACES_HEADERS` override. Use uncompressed HTTP/protobuf; do not enable gzip. Plain OTLP exporters do not read `FI_*` authentication variables. +The trace-specific endpoint is the **full URL**, not an origin. Remove any conflicting `OTEL_EXPORTER_OTLP_TRACES_HEADERS` override. Use HTTP/protobuf, either uncompressed or with a single `gzip` or `x-gzip` layer and a matching `Content-Encoding` header. Both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it). Plain OTLP exporters do not read `FI_*` authentication variables. Strands' `gen_ai_unredacted_attributes=` setting redacts messages and tool arguments/results, **not exception messages or stack traces**. Those can contain sensitive data. The `FI_HIDE_*` settings do not apply to Strands' native spans. If you need error-detail hiding, use the LangChain/traceAI path above or a separately configured trusted collector that enforces your policy before export. diff --git a/src/pages/docs/integrations/traceai/go/a2a.mdx b/src/pages/docs/integrations/traceai/go/a2a.mdx index 7baf9262a..df1d90e30 100644 --- a/src/pages/docs/integrations/traceai/go/a2a.mdx +++ b/src/pages/docs/integrations/traceai/go/a2a.mdx @@ -41,7 +41,7 @@ For self-hosting, set `FI_BASE_URL` to your Future AGI API base URL. The default ## 3. Configure OpenTelemetry -The package uses your tracer provider. Configure the standard OTLP/HTTP exporter with both authentication headers, the `project_name` resource attribute and no compression. +The package uses your tracer provider. Configure the standard OTLP/HTTP exporter with both authentication headers and the `project_name` resource attribute. This example sends uncompressed bodies; HTTP also accepts a single `gzip` or `x-gzip` layer. OTLP/gRPC export remains uncompressed-only. ```go import ( @@ -223,6 +223,6 @@ Task IDs, states, agent URLs, card metadata, session IDs and user IDs remain vis ## Troubleshooting - **No traces:** check both Future AGI keys, `FI_PROJECT_NAME`, and provider shutdown. The exporter URL must end in `/tracer/v1/traces`, and `project_name` must be a resource attribute, not a span attribute. -- **Export body errors:** use `otlptracehttp.NoCompression`; the collector does not accept gzip. +- **Export body errors:** check that the content type matches the decoded OTLP body and that gzip bodies declare `Content-Encoding: gzip` (or `x-gzip`). For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it). - **Missing stream spans:** call the returned traceAI client rather than `.Client`, use `WrapHandler` on the server, and set a deadline for blocked reads. - **Separate client and server traces:** install the server interceptor and preserve W3C trace-context headers or gRPC metadata across proxies. An uninstrumented remote agent contributes no child spans. diff --git a/src/pages/docs/integrations/traceai/go/eino.mdx b/src/pages/docs/integrations/traceai/go/eino.mdx index b54f1427b..c28742668 100644 --- a/src/pages/docs/integrations/traceai/go/eino.mdx +++ b/src/pages/docs/integrations/traceai/go/eino.mdx @@ -63,7 +63,7 @@ export FI_PROJECT_NAME="my-service" ## 3. Create the tracer provider -The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with the three things Future AGI needs: both auth headers, the `project_name` resource attribute, and an uncompressed body (the collector rejects gzip). +The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with both auth headers and the `project_name` resource attribute. This example sends uncompressed bodies; HTTP also accepts a single `gzip` or `x-gzip` layer. OTLP/gRPC export remains uncompressed-only. ```go import ( @@ -315,7 +315,7 @@ The embedding callback mapping omits embedding vectors. The module adds no resou ## Troubleshooting - **No traces.** Check that the process calls `h.Shutdown` and `tp.Shutdown` before it exits, that the handler was registered before the first run, and that `FI_API_KEY`, `FI_SECRET_KEY` and `FI_PROJECT_NAME` are set. The exporter URL must be the full traces URL, `{FI_BASE_URL}/tracer/v1/traces`. -- **Export errors about the body.** Keep `otlptracehttp.WithCompression(otlptracehttp.NoCompression)`; the collector rejects gzip. +- **Export errors about the body.** Check that the content type matches the decoded OTLP body and that gzip bodies declare `Content-Encoding: gzip` (or `x-gzip`). For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it). - **Spans in the wrong project or none at all.** `project_name` must be a resource attribute on the tracer provider, not a span attribute. - **Two spans for every node.** The handler is registered twice: `AppendGlobalHandlers` ran twice, or the handler is global and also passed with `compose.WithCallbacks` / `adk.WithCallbacks`. - **No model name or token counts on a model span.** The chat model fires no callbacks itself, so the graph reports only its messages. The eino-ext models report both; for your own model, fire `callbacks.OnStart` / `OnEnd` with `*model.CallbackInput` / `*model.CallbackOutput`. diff --git a/src/pages/docs/integrations/traceai/go/gemini.mdx b/src/pages/docs/integrations/traceai/go/gemini.mdx index 7e305434f..546626b0d 100644 --- a/src/pages/docs/integrations/traceai/go/gemini.mdx +++ b/src/pages/docs/integrations/traceai/go/gemini.mdx @@ -48,7 +48,7 @@ export GEMINI_API_KEY="your-gemini-api-key" ## 3. Create the tracer provider -The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with the three things Future AGI needs: both auth headers, the `project_name` resource attribute, and an uncompressed body (the collector rejects gzip). +The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with both auth headers and the `project_name` resource attribute. This example sends uncompressed bodies; HTTP also accepts a single `gzip` or `x-gzip` layer. OTLP/gRPC export remains uncompressed-only. ```go import ( @@ -267,7 +267,7 @@ With `FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true` the model, token counts, functio ## Troubleshooting - **No traces.** Check that the process calls `shutdown(tp)` (or `tp.Shutdown`) before it exits, and that `FI_API_KEY`, `FI_SECRET_KEY` and `FI_PROJECT_NAME` are set. The exporter URL must be the full traces URL, `{FI_BASE_URL}/tracer/v1/traces`. -- **Export errors about the body.** Keep `otlptracehttp.WithCompression(otlptracehttp.NoCompression)`; the collector rejects gzip. +- **Export errors about the body.** Check that the content type matches the decoded OTLP body and that gzip bodies declare `Content-Encoding: gzip` (or `x-gzip`). For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it). - **Spans in the wrong project or none at all.** `project_name` must be a resource attribute on the tracer provider, not a span attribute. - **No span for a call.** Check that `HTTPClient` on the `genai.ClientConfig` is the traced client. Only generation and embedding calls are spanned. - **ERROR spans with `RESOURCE_EXHAUSTED` (429) or `UNAVAILABLE` (503).** Those are Gemini quota or capacity errors, recorded as they happened. On the free tier, space your requests or use another model. diff --git a/src/pages/docs/integrations/traceai/go/openai.mdx b/src/pages/docs/integrations/traceai/go/openai.mdx index d74dc28a0..fcd306bea 100644 --- a/src/pages/docs/integrations/traceai/go/openai.mdx +++ b/src/pages/docs/integrations/traceai/go/openai.mdx @@ -36,7 +36,7 @@ export OPENAI_API_KEY="your-openai-api-key" ## 3. Create the tracer provider -The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with the three things Future AGI needs: both auth headers, the `project_name` resource attribute, and an uncompressed body (the collector rejects gzip). +The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with both auth headers and the `project_name` resource attribute. This example sends uncompressed bodies; HTTP also accepts a single `gzip` or `x-gzip` layer. OTLP/gRPC export remains uncompressed-only. ```go import ( @@ -238,7 +238,7 @@ A boolean switch counts only when it is `true` (any case). With both broad switc ## Troubleshooting - **No traces.** Check that the process calls `shutdown(tp)` (or `tp.Shutdown`) before it exits, and that `FI_API_KEY`, `FI_SECRET_KEY` and `FI_PROJECT_NAME` are set. The exporter URL must be the full traces URL, `{FI_BASE_URL}/tracer/v1/traces`. -- **Export errors about the body.** Keep `otlptracehttp.WithCompression(otlptracehttp.NoCompression)`; the collector rejects gzip. +- **Export errors about the body.** Check that the content type matches the decoded OTLP body and that gzip bodies declare `Content-Encoding: gzip` (or `x-gzip`). For `413`, reduce the batch size: both wire and decompressed bodies must fit the HTTP size cap (16 MiB by default; self-hosted configuration and the collector's memory budget can change it). - **Spans in the wrong project or none at all.** `project_name` must be a resource attribute on the tracer provider, not a span attribute. - **No token counts on a streamed span.** Set `StreamOptions.IncludeUsage`; without it the API sends no usage chunk. - **Every retry is a separate trace.** Start a parent span around the call (step 5). diff --git a/src/pages/docs/integrations/traceai/gradio.mdx b/src/pages/docs/integrations/traceai/gradio.mdx index fb4774db5..1679d6732 100644 --- a/src/pages/docs/integrations/traceai/gradio.mdx +++ b/src/pages/docs/integrations/traceai/gradio.mdx @@ -116,7 +116,7 @@ No Future AGI key, secret or model key is exported. The recipe never prints the - **Turns are not grouped.** `predict` ran without a `gr.Request`, so no session id was set. - **Generator chat functions.** Generator callbacks are outside this synchronous recipe. Gradio can run generator steps in separate context copies; do not assume a span opened around a generator remains its parent. - **Reload mode (`gradio app.py`).** Gradio re-runs the file in the same process; the first instrumentation stays in force. Changes to the tracing setup need a restart. You will see `Attempting to instrument while already instrumented` once at startup and once per save. -- **Nothing exported after kill.** `register()` flushes on normal exit, SIGINT and SIGTERM. SIGKILL skips the flush. +- **Nothing exported after kill.** `register()` installs an `atexit` hook for normal interpreter exit, not handlers for `SIGINT` or `SIGTERM`. Default `SIGTERM` termination does not run that hook; `SIGINT` normally raises `KeyboardInterrupt`. Keep the provider returned by `init_tracing()` and call `provider.shutdown()` from your application's or framework's graceful-shutdown hook after active spans end, preserving its signal handlers and exit policy. Flush at task boundaries with `provider.force_flush()` too: cleanup must finish within the termination grace period, and no hook can run after `SIGKILL`. ## Limitations diff --git a/src/pages/docs/knowledge-base/guides/manage-with-the-sdk.mdx b/src/pages/docs/knowledge-base/guides/manage-with-the-sdk.mdx index 550f15afa..beac79c59 100644 --- a/src/pages/docs/knowledge-base/guides/manage-with-the-sdk.mdx +++ b/src/pages/docs/knowledge-base/guides/manage-with-the-sdk.mdx @@ -19,7 +19,7 @@ pip install futureagi ## Set your credentials -The client reads your API key and secret from the environment. Get `FI_API_KEY` and `FI_SECRET_KEY` from Settings > API Keys (see [API Keys](/docs/admin-settings/api-keys)), then set them before you run your script: +The client reads your API key and secret from the environment. Get `FI_API_KEY` and `FI_SECRET_KEY` from **Keys** under **Build** in the main sidebar (see [API Keys](/docs/admin-settings/api-keys)), then set them before you run your script: ```bash export FI_API_KEY="your-api-key" diff --git a/src/pages/docs/observe/features/quickstart.mdx b/src/pages/docs/observe/features/quickstart.mdx index 46a2e4754..ba084ec2e 100644 --- a/src/pages/docs/observe/features/quickstart.mdx +++ b/src/pages/docs/observe/features/quickstart.mdx @@ -61,7 +61,7 @@ This is how you connect your application to Future AGI so LLM calls are captured trace_provider = register( project_type=ProjectType.OBSERVE, project_name="FUTURE_AGI", - transport=Transport.GRPC, + transport=Transport.HTTP, ) ``` ```typescript diff --git a/src/pages/docs/observe/quickstart.mdx b/src/pages/docs/observe/quickstart.mdx index 02ab1e095..7e9a6f980 100644 --- a/src/pages/docs/observe/quickstart.mdx +++ b/src/pages/docs/observe/quickstart.mdx @@ -59,7 +59,7 @@ Pin the packages to the version you test against, so a later release cannot chan trace_provider = register( project_type=ProjectType.OBSERVE, project_name="self-improving-agent", - transport=Transport.GRPC, + transport=Transport.HTTP, ) # Auto-instrument OpenAI: every call is now traced @@ -77,7 +77,7 @@ Pin the packages to the version you test against, so a later release cannot chan import { register, ProjectType } from "@traceai/fi-core"; import { OpenAIInstrumentation } from "@traceai/openai"; import { registerInstrumentations } from "@opentelemetry/instrumentation"; - import OpenAI from "openai"; + import * as OpenAI from "openai"; // Connect to Future AGI and create (or reuse) an Observe project const traceProvider = register({ @@ -85,14 +85,16 @@ Pin the packages to the version you test against, so a later release cannot chan projectName: "self-improving-agent", }); - // Auto-instrument OpenAI: every call is now traced + const openAIInstrumentation = new OpenAIInstrumentation({}); registerInstrumentations({ - instrumentations: [new OpenAIInstrumentation({})], + instrumentations: [openAIInstrumentation], tracerProvider: traceProvider, }); + // Native ESM imports need an explicit patch before creating the client + openAIInstrumentation.manuallyInstrument(OpenAI); // Use OpenAI exactly as you normally would - const client = new OpenAI(); + const client = new OpenAI.OpenAI(); const completion = await client.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: "Suggest one habit I could build to keep improving every day." }], diff --git a/src/pages/docs/observe/reference/export-formats.mdx b/src/pages/docs/observe/reference/export-formats.mdx index d7bd319d8..30f1570f2 100644 --- a/src/pages/docs/observe/reference/export-formats.mdx +++ b/src/pages/docs/observe/reference/export-formats.mdx @@ -1,6 +1,6 @@ --- title: "Export and endpoints" -description: "Reference for getting trace data out of Observe — exporting the current view, the available formats, and the OTLP endpoints traces are sent to (cloud and self-hosted)." +description: "Reference for Observe's current CSV export limitation and the OTLP endpoints traces are sent to (cloud and self-hosted)." slug: "export-formats" page_type: "reference" products: ["Observe"] @@ -13,12 +13,12 @@ last_tested: "2026-05-25" schema_type: "TechArticle" seo: title: "Trace export and endpoints in Observe" - description: "Reference for getting trace data out of Observe — exporting the current view, the available formats, and the OTLP endpoints traces are sent to (cloud and self-hosted)." + description: "Reference for Observe's current CSV export limitation and the OTLP endpoints traces are sent to (cloud and self-hosted)." primary_keyword: "export traces endpoints futureagi" direct_answer: true geo: answer_target: "How do I export traces from Observe, and what endpoints does traceAI send to?" - llm_summary: "Observe exports the current trace-explorer view (filters and time range) to CSV from the download icon. Inbound, the traceAI SDK sends spans over OTLP to FutureAGI — set FI_BASE_URL (HTTP) or FI_GRPC_URL (gRPC), defaulting to FutureAGI cloud, or point them at your own collector when self-hosted." + llm_summary: "Exact CSV export from the trace explorer is unavailable; the download control is disabled. Inbound, the traceAI SDK sends spans over OTLP to FutureAGI — set FI_BASE_URL (HTTP) or FI_GRPC_URL (gRPC), defaulting to FutureAGI cloud, or point them at your own collector when self-hosted." canonical: "/docs/observe/reference/export-formats" related: - "/docs/observe/guides/explore-dashboard" @@ -32,19 +32,9 @@ There are two directions for trace data: **out of** Observe (exporting what you' ## Export from the trace explorer -The download icon in the [trace explorer](/docs/observe/guides/explore-dashboard) header exports the **current view** — the traces that match your active filters and time range. +Exact CSV export from the [trace explorer](/docs/observe/guides/explore-dashboard) is **unavailable**. The download control in the LLM Tracing and Sessions headers is disabled and displays “Exact CSV export is temporarily unavailable.” -**CSV is the only export format.** There is no JSON or Parquet download from the trace explorer. - -| Format | Use for | -|---|---| -| CSV | Spreadsheet analysis, sharing, importing elsewhere. | - -The CSV holds one row per span in the current view, with the columns the explorer shows: trace ID, span name, status, model, provider, latency, token counts, cost, timestamp, and any eval scores present on the span. - - - The export reflects the current view, so a very large view may be truncated. Narrow the filters or time range to export a complete slice. - +You cannot download the current view from this control. Changing filters or the time range does not enable it. ## Ingestion endpoints @@ -103,7 +93,7 @@ OTEL_BSP_MAX_EXPORT_BATCH_SIZE=256 - Filter, then export the current view. + Explore traces in the dashboard. Configure the OTLP endpoint and transport. diff --git a/src/pages/docs/prompt/reference/sdk-api.mdx b/src/pages/docs/prompt/reference/sdk-api.mdx index 38433f673..12955951e 100644 --- a/src/pages/docs/prompt/reference/sdk-api.mdx +++ b/src/pages/docs/prompt/reference/sdk-api.mdx @@ -32,7 +32,7 @@ export FI_API_KEY="your-api-key" export FI_SECRET_KEY="your-secret-key" ``` -Find both under **Settings → API Keys** in the platform. +Create a pair from **Keys** under **Build** in the main sidebar ([API Keys](/docs/admin-settings/api-keys)). ## Construct a template diff --git a/src/pages/docs/protect/concepts/understanding-protect.mdx b/src/pages/docs/protect/concepts/understanding-protect.mdx index 528194459..c7045dd76 100644 --- a/src/pages/docs/protect/concepts/understanding-protect.mdx +++ b/src/pages/docs/protect/concepts/understanding-protect.mdx @@ -5,27 +5,27 @@ description: "Protect screens your traffic for safety issues through guardrails ## A guardrail is a named check on your traffic -A **guardrail** wraps a check and adds three settings you configure in [Agent Command Center](/docs/command-center/features/guardrails), under **Gateway** > **Guardrails**: +A **guardrail** wraps a check. In [Agent Command Center](/docs/command-center/features/guardrails), under **Gateway** > **Guardrails**, its rule has: -- An **action** for when it triggers: Block, Warn, Mask, or Log -- A **stage** for when it runs: **pre** runs before the model sees the request, **post** runs before the response reaches the caller (on a streamed response, after it has been sent; see [Streaming Guardrails](/docs/command-center/features/guardrails#streaming-guardrails)), and **both** runs on each side -- A **confidence threshold** for how sure the check has to be before it fires +- An **action** for when it triggers: Block, Warn, or Log, with Mask available for checks that can rewrite the request +- A fixed **stage**, determined by the check type: request (**pre**) or response (**post**) +- A **match threshold**, where the check exposes one, for the score that counts as a match -A stage of **both** isn't a third mode. It asks for the same check to run twice, once on the way in and once on the way out. As a setting, stage belongs to the guardrail rather than to the check it wraps, which is why a single **gateway**, the Agent Command Center surface your guardrails live on, can carry a mix: some watching only what the caller sent, others only what the model sent back. Today the gateway doesn't apply the setting: it runs each check at its built-in stage, so Data Leakage Prevention and Hallucination Detection check the response, and the others check the request. +Organization checks do not have a dashboard stage selector. **Hallucination Detection** and **Data Leakage Prevention** read the model's answer; the other catalog checks, including **PII Detection**, read the request before the model is called. The Rules drawer describes this stage but cannot move a check to post or make it run on both sides. On a streamed response, post-stage checks run after the response text has been sent; see [Streaming Guardrails](/docs/command-center/features/guardrails#streaming-guardrails). The threshold runs from 0.0 to 1.0: raising it makes the check fire only on higher-confidence matches, lowering it makes it fire more readily. Threshold and action are independent settings. The threshold decides how often a check fires; the action decides what the caller experiences when it does. Loosening a threshold on a check set to Log changes only whether the call carries the `x-agentcc-guardrail-triggered` header. For a non-streamed response, Block from either stage returns `403`; Warn and Log let the call return `200`. On a stream, only a pre-stage check can change the already-sent HTTP status. For the status each action produces, see [Guardrail checks](/docs/protect/reference/guardrail-checks#response-statuses). -Every guardrail wraps a check, and the Rules tab groups checks under two headers that mark a single split: where the check runs. **Rule-Based Checks** are [first-party](/docs/protect/reference/guardrail-checks): they run inside Protect without an external provider. **AI-Powered Checks** call a provider you configure under **Provider Settings** in that check's own settings. +Where a check executes is separate from which side it reads. [Built-in checks](/docs/protect/reference/guardrail-checks) run inside the gateway; provider-backed checks call a service. See the reference for which checks the console offers and which saved integrations are read-only. A["Action: Block, Warn, Mask, or Log"] - G --> S["Stage: pre, post, or both"] - G --> T["Confidence threshold"] + accDescr: An organization guardrail has an action, a stage fixed by its check type, and a match threshold where applicable. Built-in checks run in the gateway; provider-backed checks call a service. + G["Guardrail"] --> A["Action: Block, Warn, or Log; Mask where supported"] + G --> S["Fixed stage: request or response"] + G --> T["Match threshold where applicable"] G --> C["Check"] - C --> RBC["Rule-Based: runs inside Protect"] - C --> AIP["AI-Powered: external provider"]`} /> + C --> RBC["Built-in: runs in the gateway"] + C --> AIP["Provider-backed: calls a service"]`} /> ## protect() screens one input in your code @@ -52,13 +52,13 @@ This record belongs to the dashboard-guardrail surface, while `protect()` return Treating a guardrail and `protect()` as one surface means assuming a check you configured in one place is protecting you in the other, when it isn't. -Two guardrails with the same check can still behave completely differently once their action, stage, or threshold diverge, because a guardrail is this fixed shape, not a single on/off switch. +Rules using the same check can behave differently when their action or supported settings differ. Their request or response stage remains fixed by the check type. ## Keep exploring - Enable a check, set its stage, threshold, and action + Add a check, set its action, and confirm where it runs Browse the full list of first-party and provider-backed checks diff --git a/src/pages/docs/protect/guides/run-protect-from-the-sdk.mdx b/src/pages/docs/protect/guides/run-protect-from-the-sdk.mdx index 41cfced0e..43db63a69 100644 --- a/src/pages/docs/protect/guides/run-protect-from-the-sdk.mdx +++ b/src/pages/docs/protect/guides/run-protect-from-the-sdk.mdx @@ -36,7 +36,9 @@ rules = [ For how these checks fit into the guardrail model, see [Understanding Protect](/docs/protect/concepts/understanding-protect). -The SDK accepts these `metric` values: `toxicity`, `bias`, `bias_detection`, `sexist`, `prompt_injection`, `data_privacy_compliance`, and `pii`. A `metric` name outside this list raises an error, so confirm the name before you ship it. +Use one of these four canonical `metric` values: `toxicity`, `bias_detection`, `prompt_injection`, or `data_privacy_compliance`. The names `bias`, `sexist`, and `pii` are not accepted and raise an error. + +Two deprecated aliases remain supported: `content_moderation` maps to `toxicity`, and `security` maps to `prompt_injection`. Use the canonical names in new code. ## Run the check diff --git a/src/pages/docs/protect/guides/test-a-guardrail.mdx b/src/pages/docs/protect/guides/test-a-guardrail.mdx index 2e1d23dda..b3fb1368a 100644 --- a/src/pages/docs/protect/guides/test-a-guardrail.mdx +++ b/src/pages/docs/protect/guides/test-a-guardrail.mdx @@ -5,6 +5,12 @@ description: "Run a prompt through your active guardrails from the Test tab and The Test tab runs a single prompt through your active guardrail configuration and shows you the verdict immediately, so you can check how a guardrail behaves before any traffic from your app reaches it. +## Before you start + +You must be an organization **Owner** or **Admin** to run a test. Members and Viewers can open the tab, but **Run test** stays disabled with **Owner or Admin only**; entering a prompt does not grant permission. + +A gateway must be selected. If it reports **Guardrails are off on this gateway**, testing is disabled: ask the gateway operator to enable its guardrail pipeline. Saving or enabling an individual rule does not turn that pipeline on. + You need a guardrail already turned on for a run to mean anything; with nothing configured, every prompt just passes through. If you haven't set one up yet, [turn on a guardrail](/docs/protect/guides/turn-on-a-guardrail) first. @@ -25,10 +31,10 @@ Below the prompt box, "Model (optional)" takes a model name, with placeholder te ## Run the test -Press **Run Test**. It stays disabled until there's a prompt in the box, whether you typed one or picked a chip, and while a test is in flight it reads "Running...". +Press **Run test**. It is enabled only when you have the Owner/Admin role, a selected gateway, a non-blank prompt, and no test already in flight, and the gateway does not report its guardrail pipeline as off. While a test is in flight, the button reads **Running**. The Test Guardrails panel with the PII test chip selected, its prompt filled into the box, the optional Model field, and the Run Test button -*The **PII test** chip fills the box for you, so Run Test is live without typing anything* +*The **PII test** chip supplies the prompt; the role, gateway, pipeline, and in-flight-test conditions still apply.* ## Read the result @@ -42,6 +48,8 @@ Below the chip, **Guardrail Headers** and **Response Body** show the guardrail h ## If something doesn't look right +**Run test is disabled.** Check your organization role, selected gateway, prompt, whether another test is running, and any pipeline-off warning. A Member or Viewer needs an Owner or Admin to run the test. + **The test didn't run.** If the test call itself can't complete, an error alert replaces the result chip. **The verdict was wrong.** A chip that doesn't match what you expected, for example a prompt you thought would block coming back **OK**, points back to the guardrail's configuration rather than the panel. See [Guardrail fires on the wrong requests](/docs/protect/troubleshooting/guardrail-fires-on-the-wrong-requests) to fix it. diff --git a/src/pages/docs/protect/guides/turn-on-a-guardrail.mdx b/src/pages/docs/protect/guides/turn-on-a-guardrail.mdx index 4df383aad..fdfa87391 100644 --- a/src/pages/docs/protect/guides/turn-on-a-guardrail.mdx +++ b/src/pages/docs/protect/guides/turn-on-a-guardrail.mdx @@ -1,59 +1,42 @@ --- title: "Turn on a guardrail" -description: "Enable a check in Protect's guardrail catalog and set how it behaves." +description: "Add a check in Protect's guardrail catalog and set how it behaves." --- -Every check in Protect's guardrail catalog starts out switched off. This guide flips one on end to end, using **PII Detection** configured for a support agent as the running example: enable it, configure it, and push it live. The same steps apply to any other check in the catalog. +This guide adds **PII Detection** for a support agent: choose the check, set its action, and save it to the gateway. -This guide assumes Agent Command Center, where Protect's guardrails live under Gateway, is already set up with at least one provider connected. If it isn't yet, start with the [Agent Command Center Quickstart](/docs/command-center/quickstart). - -